Improving maternal and child health in Pakistan: a programme evaluation using a difference in difference analysis

Abstract

Introduction

Pakistan is a country with high maternal and infant mortality. Several large foreign funded projects were targeted at improving maternal, neonatal and child health. The Norway-Pakistan Partnership Initiative (NPPI) was one of these projects. This study aims to evaluate whether NPPI was successful in improving access and use of skilled maternal healthcare.

Methods

We used data from three rounds (2009–2010, 2011–2012 and 2013–2014) of the Pakistan Social and Living Standards Measurement Survey (PSLM). A difference-in-difference regression framework was used to estimate the effectiveness of NPPI and its different programme components with respect to maternal healthcare seeking behaviour of pregnant women. Various parts of the PSLM were combined to examine the healthcare seeking behaviour response of pregnant women to exposure to NPPI.

Results

Trends in maternal care seeking behaviour of pregnant women were similar in districts exposed to NPPI and control districts. Consequently, only a weak and insignificant impact of NPPI on maternal care seeking behaviour was found. However, women in districts which used vouchers or which implemented contracting were more likely to seek skilled assistance with their delivery.

Conclusion

We conclude that the objective to improve access to and use of skilled care was not achieved by NPPI. The small effects identified for vouchers and contracts on skilled birth attendance hold some promise for further experimentation.

Key questions.

What is already known?

• The Norway Pakistan Partnership Initiative (NPPI) has made a substantial investment in maternal and child health (MCH) in rural districts of the province of Sindh, Pakistan.

• Previous research suggests that NPPI improved the coverage of skilled and institutional-based care for pregnant women.

What are the new findings?

• A research design with better controls rejects earlier claims of the contributions of NPPI towards improved maternal care-seeking behaviour of pregnant women.

• NPPI subcomponents such as contracting and voucher schemes has, however, led to an increase in the uptake of skilled assistance during births.

What do the new findings imply?

• Our findings do not support earlier claims about the effectiveness of NPPI.

• Despite substantial investment, the low uptake of MCH services suggests a revisit of the intervention model underlying MCH projects such as NPPI.

Introduction

Pakistan is a classic case of growth without development.¹ Historically, the growth rate of Gross Domestic Product (GDP) in Pakistan has been around an average of 5% per annum.² Yet, its ranking in social development—especially in health outcomes—is lagging behind countries at a similar economic level.³ Pakistan missed infant and maternal mortality targets set in the Millennium Development Goals (MDG) 4 and 5.⁴ Recently, it was ranked first among the countries with the highest newborn mortality rate.⁵

One of the often-alleged causes of the poor health outcomes is the low priority given to the health sector by successive governments when allocating public resources. According to the most recent National Health Accounts 2015–2016, public spending on health was less than 1% of GDP.⁶ Geographical disparities in resource allocation further aggravate the provision of essential health services. Nearly 80% of public spending on health was allocated to secondary and tertiary care hospitals situated in large metropolitan areas, while the remaining 20% was left for primary healthcare facilities and programmes.

While resource scarcity in the health sector still holds since the onset of the new millennium, the Government of Pakistan (GoP) and its development partners have prioritised maternal and child health (MCH) on their health agenda. As a result, the MCH subsector received a substantial increase in funding. In recent decades, three major foreign-funded projects devoted roughly PKR 24 billion (US$225 million, in 2015 prices) toward improving MCH: (1) the Women’s Health Project (2000–2006)⁷; (2) the Pakistan Initiatives for Mothers and Newborns (PAIMAN, 2004–2010)⁸ and (3) the Norway-Pakistan Partnership Initiative (NPPI, 2009–2014).⁹ During 2000–2014, these projects collectively intervened in the MCH delivery system of 44 districts (out of 113 in total) that cover around 40% of the population of the country, mainly in rural areas.

In this study, we evaluate the effects of the most recent of these projects, NPPI, on maternal care-seeking behaviour. Although there have been some earlier evaluations of the NPPI programme, they were all subject to various limitations on which we elaborate in section 1.2.^{9 10}

The NPPI project setup

Rural and urban disparities in wealth and health were among the worst in Sindh, the second largest but very poor province of Pakistan.¹¹ According to the Multiple Indicator Cluster Survey 2003–2004, the maternal mortality ratio (MMR) in rural areas of Sindh was 410 (240 in urban areas) per 100 000 live births. The infant mortality rate (IMR) in rural areas was 80 (50 in urban areas) per 1000 live births.¹¹ The high MMR (276 deaths per 100 000 live births) and IMR (78 per 1000 live births) in Pakistan in general¹² and in rural areas in Sindh in particular, are mainly due to the delay in, or absence of, access to skilled care for pregnant women. In rural areas of the province, 23% of pregnant women (68% in urban areas) were assisted by a skilled birth attendant, and 29% of pregnant women (72% in urban areas) had sought skilled antenatal care (ANC).¹² Most of these health disparities are associated with inequalities in the distribution of wealth.¹³ Second, many studies identified supply-side challenges that limited the choices for pregnant women to seek care from the district-based healthcare delivery system.^14–17

The NPPI was considered an important investment programme for several reasons.¹⁰ First, it was one of the first foreign-funded projects that was implemented exclusively in one province, Sindh. Second, NPPI was the first project that was implemented under the OneUN Programme involving three UN agencies (UNICEF, UNFPA and WHO) in collaboration with the government of Sindh. Third, it was a resource-intensive (approximately US$23 million) large-scale project covering ten rural districts in Sindh. Finally, it innovated by adopting supply-side (contracting out) and demand-side (vouchers) financing tools to stimulate the use of MCH services by pregnant women.^{9 10} The main objective of NPPI was to reduce maternal, neonatal, and child mortality in the ten selected districts (see the map of Sindh province in online supplemental appendix A). According to the project plans, the selection of treatment districts was based on their vulnerability in maternal, new-born, and under-5 mortality as well as below average functioning health systems.^{9 10}

To achieve its objectives, NPPI envisaged more skilled and institutional-based care for pregnant women. A two-pronged strategy simultaneously targeted healthcare providers and communities. First, resources were devoted to enhancing the capacity of healthcare providers in routine and emergency management of obstetric and newborn care. This included trainings of medical professionals on clinical skills, as well as provision of equipment and medicines to health facilities. Second, community health workers were engaged in spreading knowledge and improving practices of appropriate maternity care. Furthermore, NPPI trained and deployed nearly 200 community midwives to facilitate community based skilled birth attendance. Finally, NPPI also provided technical assistance to include more MCH in the graduate and postgraduate curriculum of medical and public health education, as well as to improve quality of health management information systems and maternal mortality registration.^{9 10}

In addition, 2 of the 10 treatment districts were exposed to a voucher scheme and two other districts to a contracting out scheme, while the remaining six districts were exposed only to the common interventions in treatment districts.⁹

UNFPA supervised the voucher scheme. It contracted a non-governmental organisation (NGO), Greenstar Social Marketing, to implement the scheme that aimed to minimise financial barriers to access institutional-based MCH services to women of poor families (Population Council, 2014). Vouchers could be redeemed at 48 public and private health facilities. Eligible women were provided with voucher booklets (monetary value of PKR13 776 (approximately US$83)) for ANC visits, institutional births (IB) including normal vaginal and caesarean section delivery, postnatal care (PNC) visits, key vaccinations, family planning service and transportation costs.⁹

The contracting scheme was supervised by UNICEF. The management of 22 public health facilities was contracted out to the NGO Integrated Health Services (IHS).¹⁰ IHS was tasked to improve the quality and volume of MCH services through a set of interventions that included operational management, deployment of staff, provision of supplies and equipment, and social mobilisation of the target communities of the contracted facilities.

The NPPI project reviews

The effectiveness of NPPI has been reported in a project completion report⁹ and an independent study conducted by the Population Council.¹⁰ The final report of NPPI submitted to the Norwegian government concluded that most of the physical and financial targets were achieved.⁹ On the supply side, the project provided multiple inputs to 95%–100% of the facilities to enable them to manage the maternal and child healthcare needs of the target population.⁹ Against the target of 28 facilities, by the end of the project, 22 health facilities were providing 24/7 MCH services in two NPPI districts. On the demand side, 39 923 voucher booklets were distributed, of which 41% were redeemed.⁹

The project completion report assessed the impact as a before and after comparison of maternal, infant, and neonatal mortality in Sindh from the Pakistan Demographic and Health Surveys 2006–2007 and 2012–2013. While the IMR had dropped from 81 to 74 per 1000 live births, the neonatal mortality rate had remained stable (from 53 to 54 per 1000 live births) in Sindh.⁹ On the services delivery side, ANC coverage by a skilled provider increased from 70.4% to 78.2%, and deliveries attended by a skilled person increased from 44.4% to 60.5% between 2006–2007 and 2012–2013, respectively.⁹ A major limitation of this evaluation is that it did not compare health outcomes in treated districts to those in control districts not covered by NPPI.

In 2014, the Population Council compared four treated NPPI districts with two control districts. The inclusion criteria of the treatment districts were multiple project interventions, the presence of all three UN agencies, and a secure environment. The selection of control districts was based on similar development indicators and the absence of other large-scale health interventions.¹⁰ Using data from two rounds of the Pakistan Social and Living Standards Measurement Survey (PSLM), the authors calculated the coverage of essential MCH services comparing treated (T) and control (C) districts at baseline (2008–2009) and at the end-line (2012–2013) of the project period.¹⁰ The results show that during this period, coverage of skilled birth attendance increased by 12.3%-points (C) and 14.6%-points (T) while IB increased by 7.6%-points (C) and 16.4%-points (T). ANC and PNC visits increased by 0.8%-points (C) and 3.9%-points (T), and 1%-point (C) and 17.1%-points (T), respectively.¹⁰

Compared with the evaluation report of the project team, the Population Council’s analysis was an improvement but still suffered from several shortcomings: (1) it associated all births reported to the year of survey, while these births referred to a 3 year recall period; (2) the study did not isolate the effectiveness of crucial programme subcomponents like the contracting and voucher scheme and (3) generalisability of the results is unclear given the possible bias in the selection of 4 (out of 10) treatment districts and 2 (out 17) control districts.

We aim to improve on these estimates by using a more robust estimation strategy. To make better use of the timing of events, first, we obtained the birth dates from different parts of the PSLM survey and second, we extracted the exact timing of the roll-out of NPPI and its components from project reports. This allows us to (1) use an additional round of the PSLM (2013–2014) and (2) to obtain separate estimates of the effectiveness of the contracting and voucher schemes, that is, two essential programme subcomponents.

Methods

Data

Our data were drawn from three rounds (2009–2010, 2011–2012 and 2013–2014) of the PSLM. Starting in 2001, the PSLM is an ongoing cross-sectional survey designed to track progress on the MDGs and poverty reduction strategies of the GoP.¹⁸ The maternal health section of the survey contains questions related to the health-seeking behaviour of women who have delivered a live birth in the 3 years preceding the survey. More specifically, these questions pertain to the use of ANC, Tetanus Toxoid vaccinations, place of delivery, and PNC. Combining data from the three rounds of PSLM provided a raw sample of 95 910 women who had delivered a live birth in the past 3 years. To link the maternal care-seeking behaviour of these women to the NPPI project we needed more specific information on the timing of (1) when healthcare was used and (2) when and where NPPI was implemented.

To determine this more precisely than the broad 3-year window, we exploited the information on timing (year and month) of the most recent live births as reported in the immunisation section of the PSLM. As a result, we created a new time series with the time variable equal to the year of birth as opposed to the year of data collection in the survey. This reduced the sample size to 77 885 pregnancies, as many women who reported maternal healthcare use in the past 3 years did not report to have any children below the age of 3.

To define the actual implementation period of NPPI, we went beyond the documented project period because the findings of the project midterm report indicated that the actual start of activities was often later than the anticipated starting dates. We define this as the effective project period which reflects the period of core physical activities. In general, the effective project period was shorter than the documented project period. Moreover, in 2014 USAID started the Maternal and Child Health Integrated Programme (MChip) which was implemented in 15 districts of Sindh, including 5 NPPI districts.¹⁹ To avoid contamination, we dropped the year 2014 from our analysis leaving the effective project period to 2012–2013. To check for pretreatment parallel trends, we define the pre-treatment period as 36 months preceding the effective project period, that is, 2009–2011.

We further restricted the sample to the Sindh province and excluded three urban districts, namely Karachi, Hyderabad and Sukkur from the control units as well as districts that were exposed to the interventions of the PAIMAN project (2004–2010). The final sample used in this paper for the NPPI evaluation consists of 11 686 women (age 15–60 years), who delivered a live birth in the 18 rural districts of Sindh. We estimate the target population of NPPI intervention districts for the project period 2010–2014.

Only maternal healthcare seeking for births could be used to evaluate NPPI effectiveness as the date of birth of the most recent child born could only be associated with obstetric deliveries, as the timing for ANC, Tetanus Toxoid vaccination or PNC is not captured in PSLM surveys.

We examine four types of choices related to maternal health-seeking. Thereby, we assume that household choices regarding the place of delivery are sequential. In the first instance, the household chooses whether (or not) to seek care for the delivery outside their home. If they prefer institutional birth, the next choice is between a public or a private facility. These choices are influenced by multiple factors, including—but not limited to—socioeconomic and cultural aspects, access to health services, quality of care and health of the pregnant woman. Figure 1 provides a graphical description of the demand for maternal health services and the sample sizes for each type of demand in the PSLM.

Figure 1.

Maternal care-seeking behaviour of women who delivered a live birth in 2009–2013 (PSLM data). N denotes total sample size; T denotes sample in treated districts. Sample size excludes mothers who reported maternal health seeking but whose most recent birth was more than three years ago. PAIMAN treatment districts are excluded from the sample. The sample distribution may not sum-up to 100% as the lowest compartment excludes home based deliveries by an unskilled attendant. PAIMAN, Pakistan Initiatives for Mothers and Newborns; PSLM, Pakistan Social and Living Standards Measurement.

When estimating the effects of NPPI, we control for demographic and socioeconomic characteristics of mothers and households. Mother characteristics include age and number of years of schooling. At the household level, we include the formal education and gender of the household head. To account for the policy focus on rural areas and the percentage of the population below the poverty line, we also include rural residence and socioeconomic status (SES) of the household. To account for household SES, we constructed an asset index by principal component analysis using data on household assets and dwelling conditions as recorded in the surveys. Earlier evidence suggests that this type of wealth index provides a very good proxy for household economic status in the absence of expenditure data.²⁰

Analysis

We use a difference-in-difference (DID) approach to evaluate the effectiveness of NPPI by comparing outcomes in districts that were exposed to the NPPI project (all treated districts) and those without such exposure (control districts). We estimate linear models for each binary outcome variable²¹ for the following binary outcomes: births with skilled assistance, institutional births, home-based skilled births and births at private facilities (figure 1). We test for the overall effect of NPPI using a DID model with time and district fixed effects.²¹ The model is as follows:

$$y_{i,d,t}={\beta NPPI}_{i,d,t}+\Omega X_{i,d,t}+{\tau }_{i,d,t}+D_{i,d,t}+{\epsilon }_{i,d,t}$$ (1)

where $y_{i,d,t}$ is an indicator of whether the respondent (mother) i used healthcare for her most recent birth in district $d$ at time t. ${NPPI}_{i,d,t}$ reflects whether a district $d$ in which mother i delivered a live birth was treated by NPPI at time t and the parameter $\beta$ estimates the treatment effect of NPPI. $X_{idt}$ is a vector of individual and district control variables at time t, while $\Omega$ is a vector of coefficient estimates for these control variables. ${\tau }_{i,d,t}$ and $D_{i,d,t}$ indicate year and district level fixed effects to account for unobserved but time-invariant district heterogeneity and for the time trend in the use of services that is common to all districts. ${\epsilon }_{i,d,t}$ is a normally distributed error term. We adjust the SEs for clustering at the level of the primary sampling unit and robust to heteroscedasticity.²²

We test for possible heterogeneity in the effects of the three separate component interventions of NPPI, that is, contracting and voucher scheme, each in two treatment districts, and the remaining treatment districts of NPPI. In this case, we compare districts that received each intervention at a particular time interval with those that did not receive any intervention (controls).

$${\displaystyle y_{i,d,t}=\alpha VO{U}_{i,d,t}+\delta CO{N}_{i,d,t}+\gamma C_{i,d,t}+\mathrm{\Omega }{X}_{i,d,t}+{\tau }_{i,d,t}+D_{i,d,t}+{\epsilon }_{i,d,t}}$$ (2)

$\alpha$, δ and γ capture the separate treatment effects in districts with vouchers (VOU), contracting (CON) and other districts of NPPI.

The identifying assumption is that the utilisation of services in treatment districts would have followed the same time trend as in control districts if no project activities had been undertaken.²¹ This is the so-called parallel trends assumption (PTA). To test whether our assumption of parallel trends holds pretreatment, we perform two checks. First, we visually inspect the graphs of proportions and 95% CIs (obtained by normal approximation) of the selected outcomes for the treated and control districts in the period of analysis. Second, we test for differential pretreatment trends by estimating a regression model using pretreatment data only and testing whether the year fixed effects differ between the control and (later to be treated) districts. Here, we use an F-test for the joint significance of the interaction of the three lagged time dummies and a dummy indicating the treated districts. This model includes district and time fixed effects and all covariates as explained above.

Patient and public involvement

This study used secondary survey data routinely collected by the GoP. No patient or public involved in this study.

Results

First, we examine whether pretreatment trends can be considered parallel. The graphical presentation of outcome pre-trends in NPPI and control districts suggests a fairly parallel picture in most of the four outcomes (see online supplemental appendix B). Second, the p values of the F-test indicate insignificant differences in the pretreatment outcome trends between the control and treated districts (see online supplemental appendix C). We, therefore, assume that the PTA holds for most outcomes across the full sample and the sample restricted to specific reforms.

The estimation results of the generalised DID analysis are provided in table 1. While the estimated effects of most covariates show the expected patterns—schooling and wealth generally raise the probability of seeking care—the overall NPPI programme does not appear to have had any significant effect on any of the outcomes. This is a very sobering finding but it is possible that the overall effect conceals the effects of the two programme subcomponents. In table 2, we test for differential effects of NPPI in the voucher and contracting districts. We find that in these districts the proportion of births with some skilled assistance—at home as well as in a facility—did increase significantly while (surprisingly and significantly) it fell in the remaining districts of NPPI which did not use vouchers or contracting. This would suggest that the other NPPI components (like improving supplies at health facilities and community awareness campaigns) were counterproductive in the NPPI districts without vouchers or contracting interventions. None of the other maternal care-seeking behaviours were significantly affected by any of the NPPI components.

Table 1.

Probability of seeking care of women who delivered a live birth during 2009–2013

	Skilled births	Institutional births	Home-based skilled births	Births at private facilities
	Coefficient (95% CIs)	Coefficient (95% CIs)	Coefficient (95% CIs)	Coefficient (95% CIs)
NPPI districts	0 (−0.05 to 0.04)	−0.01 (−0.05 to 0.03)	0.01 (−0.05 to 0.08)	0.05 (−0.01 to 0.11)
a. Mother characteristics
15–18 years
19–40 years	−0.02 (−0.05 to 0.01)	−0.02 (−0.05 to 0.01)	−0.02 (−0.06 to 0.01)	−0.05** (−0.09 to −0.01)
41–60 years	−0.05** (−0.08 to −0.02)	−0.06** (−0.09 to −0.03)	−0.03 (−0.07 to 0.01)	−0.08** (−0.13 to −0.03)
No schooling
Up to primary schooling	0.06*** (0.03 to 0.09)	0.09*** (0.06 to 0.12)	0.01 (−0.04 to 0.05)	0.02 (−0.02 to 0.06)
>Primary and ≤middle	0.08*** (0.05 to 0.12)	0.14*** (0.1 to 0.18)	0 (−0.06 to 0.06)	0.02 (−0.03 to 0.06)
>Middle and ≤high School	0.09*** (0.05 to 0.14)	0.16*** (0.1 to 0.21)	0.05 (−0.07 to 0.17)	0 (−0.07 to 0.07)
Graduate and above	0.1*** (0.05 to 0.14)	0.19*** (0.13 to 0.25)	0 (−0.19 to 0.18)	0.09** (0.03 to 0.15)
b. Family characteristics
Rural	−0.1*** (−0.13 to −0.07)	−0.12*** (−0.16 to −0.09)	−0.04 (−0.09 to 0.01)	0.05* (0.02 to 0.09)
SES Q1
SES Q2	0.11*** (0.08 to 0.14)	0.06*** (0.03 to 0.09)	0.09*** (0.06 to 0.12)	0.02 (−0.03 to 0.06)
SES Q3	0.15*** (0.12 to 0.19)	0.11*** (0.07 to 0.14)	0.13*** (0.08 to 0.17)	0.02 (−0.03 to 0.07)
SES Q4	0.21*** (0.17 to 0.25)	0.21*** (0.16 to 0.25)	0.17*** (0.11 to 0.23)	0.11*** (0.06 to 0.17)
SES Q5	0.26*** (0.21 to 0.3)	0.29*** (0.23 to 0.34)	0.28*** (0.18 to 0.38)	0.2*** (0.14 to 0.26)
Female head	−0.03 (−0.11 to 0.06)	0.01 (−0.08 to 0.1)	−0.05 (−0.16 to 0.06)	0.07 (−0.04 to 0.18)
Head no schooling
Upto primary schooling	0.03** (0 to 0.05)	0.04** (0.01 to 0.06)	0 (−0.03 to 0.03)	0.01 (−0.03 to 0.05)
>Primary and ≤ middle	0.04** (0.02 to 0.07)	0.05*** (0.03 to 0.08)	0.01 (−0.02 to 0.05)	0.02 (−0.02 to 0.05)
>Middle and ≤ high school	0.07*** (0.04 to 0.11)	0.1*** (0.06 to 0.13)	0.01 (−0.03 to 0.06)	0.09*** (0.04 to 0.13)
Graduate and above	0.08*** (0.04 to 0.12)	0.11*** (0.07 to 0.15)	0.02 (−0.04 to 0.08)	0.09*** (0.05 to 0.14)
N	11 032	11 032	6656	4376

Table shows effect of NPPI on the probability of each outcome estimated using multiple linear regression on treatment effect, all covariates (as explained in table 1, year and district fixed effects. CIs obtained using SEs adjusted for clustering at level of primary sampling units. District fixed effect estimates are not included in the table but can be provided on request.

*, **,*** indicates level of significance at 1%, 5% and 10%.

NPPI, Norway-Pakistan Partnership Initiative; SES, socioeconomic status.

Table 2.

Probability of seeking care of women who delivered a live birth during 2009–2013

	Skilled births	Institutional births	Home-based skilled births	Births at private facilities
	Coefficient (95% CIs)	Coefficient (95% CIs)	Coefficient (95% CIs)	Coefficient (95% CIs)
Voucher districts	0.06* (−0.01 to 0.12)	0.03 (−0.03 to 0.09)	0.05 (−0.04 to 0.13)	0.04 (−0.04 to 0.12)
Contracting districts	0.14*** (0.07 to 0.21)	0 (−0.07 to 0.07)	0.17*** (0.08 to 0.26)	0.08 (−0.02 to 0.19)
Other NPPI districts	−0.08** (−0.13 to −0.02)	−0.03 (−0.08 to 0.02)	−0.05 (−0.12 to 0.02)	0.04 (−0.03 to 0.11)
a. Mother characteristics
15–18 years
19–40 years	−0.02 (−0.05 to 0.01)	−0.02 (−0.05 to 0.01)	−0.02 (−0.06 to 0.01)	−0.05** (−0.09 to −0.01)
41–60 years	−0.05** (−0.08 to −0.02)	−0.06** (−0.09 to −0.03)	−0.03 (−0.06 to 0.01)	−0.08** (−0.13 to −0.03)
No schooling
Upto primary schooling	0.06*** (0.03 to 0.09)	0.09*** (0.06 to 0.12)	0 (−0.04 to 0.05)	0.02 (−0.02 to 0.06)
>Primary and ≤middle	0.08*** (0.05 to 0.11)	0.14*** (0.1 to 0.18)	0 (−0.06 to 0.06)	0.02 (−0.03 to 0.06)
>Middle and ≤high School	0.09*** (0.05 to 0.14)	0.16*** (0.1 to 0.21)	0.05 (−0.06 to 0.17)	0 (−0.07 to 0.07)
Graduate and above	0.09*** (0.05 to 0.14)	0.19*** (0.13 to 0.25)	0 (−0.18 to 0.17)	0.09** (0.03 to 0.15)
b. Family characteristics
Rural	−0.09*** (−0.13 to −0.06)	−0.12*** (−0.15 to −0.09)	−0.04 (−0.09 to 0.01)	0.06** (0.02 to 0.09)
SES Q1
SES Q2	0.11*** (0.08 to 0.14)	0.06*** (0.03 to 0.09)	0.09*** (0.06 to 0.12)	0.02 (−0.03 to 0.06)
SES Q3	0.15*** (0.12 to 0.19)	0.11*** (0.07 to 0.14)	0.13*** (0.08 to 0.17)	0.02 (−0.03 to 0.07)
SES Q4	0.21*** (0.18 to 0.25)	0.21*** (0.17 to 0.25)	0.17*** (0.11 to 0.23)	0.11*** (0.06 to 0.17)
SES Q5	0.26*** (0.22 to 0.3)	0.29*** (0.24 to 0.34)	0.28*** (0.18 to 0.38)	0.2*** (0.14 to 0.26)
Female head	−0.02 (−0.11 to 0.06)	0.01 (−0.08 to 0.1)	−0.04 (−0.15 to 0.07)	0.07 (−0.04 to 0.18)
No schooling
Upto primary schooling	0.03** (0 to 0.05)	0.04*** (0.01 to 0.06)	0 (−0.03 to 0.03)	0.01 (−0.03 to 0.05)
>Primary and ≤middle	0.04** (0.01 to 0.07)	0.05*** (0.03 to 0.08)	0.01 (−0.02 to 0.04)	0.02 (−0.02 to 0.05)
>Middle and ≤high School	0.07*** (0.04 to 0.1)	0.1*** (0.06 to 0.13)	0.01 (−0.04 to 0.06)	0.09*** (0.04 to 0.13)
Graduate and above	0.08*** (0.04 to 0.12)	0.11*** (0.07 to 0.15)	0.02 (−0.04 to 0.08)	0.09*** (0.05 to 0.14)
N	11 032	11 032	6656	4376

Table provides heterogeneous effects of each of the reforms carried out in NPPI. Other notes are same as in table 1.

NPPI, Norway-Pakistan Partnership Initiative; SES, socioeconomic status.

Rural inhabitants have a lower probability of skilled care or institutional care use, and a higher probability of seeking care from private facilities. Schooling of mothers is significantly positively associated with most of the outcomes except home-based skilled births and this is similar for schooling of the head of household (see tables 1 and 2).

Discussion and conclusion

Using a large and representative sample of women from the province of Sindh, we subjected the implementation of NPPI to a rigorous test of effectiveness. Our findings are as follows. First, our estimates confirm that also in this poor and mostly rural population of Sindh, women with low schooling and low SES are less likely to receive maternal care, be it skilled assistance or institutional delivery. This confirms the need to promote such behaviour to improve birth outcomes. Second, in general, we only find a weak and insignificant impact of the NPPI programme on the maternal care-seeking behaviour of pregnant women. For most of the outcomes that we could consider, the broader objective to improve access to and use of skilled care in the province of Sindh was not achieved by NPPI. This is worrisome given the large-scale attempt to improve such behaviour in general, using both demand and supply side interventions. Third, and fortunately, there are two exceptions: women in districts which used vouchers, or which implemented contracting were more likely to seek skilled assistance with their delivery, both at home and at the health facility. This is somewhat a surprising result as the mandate of the NGOs involved in contracting and vouchers was restricted to renovating the health facilities and distribution of vouchers to be redeemed at health facilities, respectively. Such efforts of IHS or Greenstar Marketing in their respective districts were not intended to improve births at home irrespective of the type of assistance.

The finding of increased skilled birth assistance was not obtained for the remaining NPPI treatment districts; on the contrary, we even find it decreased such assistance. It suggests that the provision of supplies and equipment, staff trainings and the general public awareness campaigns—which were implemented across all NPPI districts—were not successful in improving maternal care-seeking behaviour, and possibly even counter-productive.

Compared with earlier work, our study offers several improvements. First, exploiting the timing of births, we construct a 5-year panel of over 11 000 live births in treated and control districts from a representative sample of women drawn from the PSLM surveys done both before and after the introduction of NPPI intervention. Second, access to official documents of NPPI helped us to define treated and control districts, the effective project period and the pretreatment period. Third, the births panel allowed us to adopt a generalised DID design with two-way fixed effects to control for time-invariant unobserved heterogeneity across districts and the common time trends that allow for a better-controlled comparison than the before-after DID design on which the Population Council’s study was based on. Fourth, demographic and socioeconomic characteristics of the household and the women allowed us to control for time-variant characteristics at the individual level.

However, our study also faced certain limitations. By extracting the dates of birth, we were able to more accurately isolate the timing of effects, but in doing so, we lost almost 20% of the original sample, thereby reducing the power of the test. Another limitation of our analysis was that we had to exclude other maternal outcomes like seeking ANC and PNC or TT vaccination because their timing was not recorded in the surveys. We could also not examine neonatal morality since PSLM reported only the live births. Finally, we had to drop 1635 births that occurred in 2014, a year that overlapped with the MChip project.¹⁹

The evidence on the effectiveness of contracting in the Pakistan health sector is somewhat mixed.^{23 24} A few studies demonstrated an improvement in health seeking of pregnant women when this was encouraged with demand side financing. For example, earlier local small scale observational studies find that the distribution of MCH vouchers to the poorest of pregnant women increased institutional births in district Jhang (adjusted OR (AOR) 1.41; 10 treated and 10 control union councils (UC, smallest administrative unit)) and in Dera Ghazi Khan (AOR 4.04; 7 UCs pretreatment and post-treatment) in the province of Punjab, respectively.^{25 26} More recently, a small scale study by Habib et al experimented with an MCH intervention package in the flood-affected areas of Dadu District. Besides the interventions that are similar to interventions in the remaining districts of NPPI, an emergency fund was established to cover transport expenses of pregnant women in low-income families requiring urgent transportation to health facilities. The results show significant gains in wide-ranging MCH outcomes including service delivery (double difference of 33.1% and 30.5% in skilled birth attendance and institutional deliveries, respectively).²⁷

The voucher and contracting schemes seem to have had some success in improving the management of MCH interventions, by showing effects that were not observed in the other NPPI districts. Earlier research has claimed that the social mobilisation strategy and provision of supplies and equipment could deliver, if coupled with innovative strategies to ensure availability of medical staff at the health facilities especially in rural areas. For example, Chaudhri et al managed the staff shortage in district-based health facilities in the Chakwal district by bringing in medical doctors from a tertiary care hospital in Rawalpindi on a rotational basis. The results of their before and after quasi-experimental design indicated that the objectives were broadly achieved, that is, fivefold increase in out-patient visits to the target health facilities, twofold increase in admissions and in institutional births.²⁸ Another study, Jhokio et al, implemented the safe motherhood intervention package in three talukas (subdistricts) of Larkana district that included an intervention similar to NPPI community midwives’ scheme, that is, training of Traditional Birth Attendants and supply of delivery kits to conduct normal deliveries at home.²⁹ The shortage of doctors at health facilities in the treated taluka hospitals was managed by bringing-in obstetricians from Larkana City on a rotational basis. After 6 months of the randomised intervention, a significant decline was observed in perinatal mortality (AOR 0.7) and maternal mortality (AOR 0.74) in the three treatment talukas compared with the controls (four talukas).²⁹ These findings suggest that the routine practice of human resource management in public sector is not sufficient to improve the availability of medical professionals in rural areas. At least for contracting scheme, we could confirm that IHS better managed their human resources with tools such as staff hire-and fire and market based financial incentives for medical professionals.

In general, our findings do not confirm earlier claims of overall success of NPPI.^{9 10} A similar discrepancy can be observed between the claims made in the final evaluation report of PAIMAN project⁷ to those of an independent evaluation by Rashid.³⁰ For example, using data from baseline (2005) and end line (2010) surveys, the PAIMAN final evaluation report claimed that skilled births increased by 11%-points in project districts.⁷ However, using data of PSLM rounds 2004–2005 and 2012–2013, Rashid (2015) finds that women in PAIMAN districts were less likely (OR 0.76) to make use of skilled assistance during their births than women living in control districts.³⁰ Recently, an evaluation of contracting of primary healthcare facilities in Pakistan that had used data of six rounds of PSLM also rejected²⁴ earlier claims of effectiveness based on data collected from community and facility surveys conducted in treated and control districts.²³ Such discrepancies call for improving the quality of programme evaluations to enhance the evidence base of health policies.

Since the dawn of the 21st century, the GoP and its development partners have focused primarily on MCH, often relying on sporadic, short-term projects that have demonstrated little effect. NPPI is one such example, that implied substantial resources on a carefully designed package of interventions. We can only speculate about potential reasons for the lack of effect of NPPI. Some of these reasons are embedded in the short-term project cycle approach of NPPI. These have been mentioned in the progress and evaluation reports such as the delay in the release of funds and the slow progress on a few interventions as well as challenges in operational management between the government and the implementing partners that led the Norwegian government to an extensive revamp of activities including restricting the plans of extension beyond 2014.⁹

In the recent past some of the South Asian countries followed a slightly different approach that enabled them to successfully improve MCH.^31–33 For example, Bangladesh and Nepal adopted an integrated and long-term investment model to achieve MDG targets related to maternal and under-five mortalities. In Bangladesh, this strategy involved the integration of 128 discrete projects and the enhancement of budgetary allocations sustained between 1998 and 2013.³² In Nepal, the government established mechanisms to align donor resources with national priorities and adopted a long-term plan 2002–2017 for the integrated delivery of services with community health volunteers and to address financial barriers to access health services.³¹ Besides, their investments in the health sector in Bangladesh, Nepal and Sri Lanka were coupled with parallel investments in improved education and gender equality which may have had spill-over effects on the health sector.³³

Concluding, our findings do not support earlier reports about programme effectiveness of NPPI. Despite substantial investment, in general, the low uptake of MCH services calls for revisiting the intervention model of MCH projects including NPPI. The small effects identified for vouchers and contracts on skilled birth attendance, however, still hold some promise.

Data availability statement

Data are available in a public, open access repository. We used data sets of Pakistan Social and Living Standard Surveys conducted by Pakistan Bureau of Statistics. The data sets of these surveys are available on request.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

Ethical exemption was obtained from Ethical Review Committee of the Aga Khan University Number 4550-CHS-ERC-16 dated 3 February 2017.