Use of Service Provision Assessments and Service Availability and Readiness Assessments for monitoring quality of maternal and newborn health services in low-income and middle-income countries

Abstract

Improving the quality of maternal and newborn health (MNH) services is key to reducing adverse MNH outcomes in low-income and middle-income countries (LMICs). The Service Provision Assessment (SPA) and Service Availability and Readiness Assessment (SARA) are the most widely employed, standardised tools that generate health service delivery data in LMICs. We ascertained the use of SPA/SARA surveys for assessing the quality of MNH services using a two-step approach: a SPA/SARA questionnaire mapping exercise in line with WHO’s Quality of Care (QoC) Framework for pregnant women and newborns and the WHO quality standards for care around the time of childbirth; and a scoping literature review, searching for articles that report SPA/SARA data. SPA/SARA surveys are well suited to assess the WHO Framework’s cross-cutting dimensions (physical and human resources); SPA also captures elements in the provision and experience of care domains for antenatal care and family planning. Only 4 of 31 proposed WHO quality indicators around the time of childbirth can be fully generated using SPA and SARA surveys, while 19 and 23 quality indicators can be partially obtained from SARA and SPA surveys, respectively; most of these are input indicators. Use of SPA/SARA data is growing, but there is considerable variation in methods employed to measure MNH QoC. With SPA/SARA data available in 30 countries, MNH QoC assessments could benefit from guidance for creating standard metrics. Adding questions in SPA/SARA surveys to assess the WHO QoC Framework’s provision and experience of care dimensions would fill significant data gaps in LMICs.

Summary box.

What is already known?

• There is limited experience with use of low-income and middle-income country (LMIC) data to assess the quality of maternal and newborn health in line with the WHO Quality of Care Framework for Maternal and Newborn Health.

What are the new findings?

• Use of Service Provision Assessment (SPA) and Service Availability and Readiness Assessment (SARA) data is growing, but there is considerable variation in indicators and methods employed to measure quality of care in maternal and newborn health services.

• SPA and SARA surveys are well suited to assess the WHO Framework’s cross-cutting dimensions; the SPA also captures elements in the provision of care and experience of care domains, but only for antenatal care and family planning services.

• Only 4 of 31 proposed WHO quality indicators around the time of childbirth can be fully generated using SPA and SARA surveys, while 19 and 23 quality indicators can be partially obtained from SARA and SPA surveys, respectively; most of these are input indicators.

What do the new findings imply?

• Adding questions in SPA and SARA surveys to assess the WHO Quality of Care Framework’s provision and experience of care dimensions across all maternal and newborn health services would fill significant data gaps in LMICs.

Introduction

Considerable progress has been made during the Millennium Development Goals era to improve maternal and newborn health in low-income and middle-income countries (LMICs).^{1 2} However, in 2015, the maternal mortality ratio (MMR) and the neonatal mortality rate (NMR) were 14 and 5 times higher, respectively, in LMICs than in high-income countries.^{1 2} The Sustainable Development Goals include ambitious global targets for reducing the MMR from 216 to <70 deaths per 100 000 live births and the NMR from 36 to ≤12 deaths per 1000 live births between 2015 and 2030.³ Therefore, addressing current levels of maternal and neonatal mortality in LMICs is a global priority.

Efforts to monitor the implementation of evidence-based, cost-effective interventions to reduce maternal and neonatal mortality have focused on measuring coverage of relevant health services and life-saving, evidence-based interventions.^4–8 Yet, in a majority of LMICs, high maternal and neonatal mortality levels persist despite considerable improvements in coverage of such interventions. This disconnect underscores the critical role that quality of care (QoC) plays in improving health outcomes. Currently, the global health community acknowledges that improving QoC is key to reducing adverse maternal and newborn health outcomes in LMICs.^{9 10}

Recognising the need to better measure QoC, in 2013, the World Health Organization (WHO) convened an expert meeting to establish consensus around assessing QoC in maternal and newborn health. The expert panel developed the WHO QoC Framework for Maternal and Newborn Health (hereinafter referred to as Framework; figure 1), which uses the Donabedian structure–process–outcome model and identifies eight domains key to the provision of quality maternal and newborn care—three domains are related to the experience of care dimension (effective communication, respect and dignity, emotional support), three are related to the provision of care dimension (evidence-based practices for routine care and management of complications, actionable information systems, functional referral systems) and two are cross-cutting between provision and experience of care (competent and motivated human resources, essential physical resources).¹¹ In relation to the Framework’s domains, but only for care around the time of childbirth (ie, thus excluding antenatal care, postpartum care beyond the immediate postpartum period, and family planning), WHO also formulated standards for assessing and monitoring quality of care. Specifically, for each Framework domain, WHO proposed a quality standard (ie, description of requirements to achieve high-quality care around the time of childbirth; 8 total), several quality statements (ie, priorities for measurably improving QoC around the time of childbirth; 31 total) and corresponding indicator measures (input, output/process and outcome indicators; 352 total) for assessing and monitoring the QoC aspects detailed in the quality statements.¹¹ The goal of this work by WHO was the adoption and use of the Framework and quality standards within national strategies for the delivery of high-quality maternal and newborn health services.

Figure 1.

WHO quality of care framework for maternal and newborn health. Reproduced with permission under the terms of the Creative Commons Attribution License from the BJOG: An International Journal of Obstetrics and Gynaecology.

Globally, WHO guidelines and standards are held in high regard and closely followed by governments, policy-makers and clinicians. Yet, use of the Framework and the reporting of proposed quality metrics pose challenges for LMICs given that sources of QoC data are limited and their quality varies widely.¹² The Service Provision Assessment (SPA) and the Service Availability and Readiness Assessment (SARA) are the most widely employed, standardised health facility assessment tools that generate nationally representative data on health service delivery and quality in LMICs. This article aims to ascertain the utility of SPA and SARA surveys for assessing the quality of maternal and newborn health services in LMICs.

Methods

Data sources: SPAs and SARAs

The SPA was developed and is implemented by the Demographic and Health Surveys (DHS) Program with funding from the US Agency for International Development (USAID).¹³ The SARA was developed and is run and funded by WHO in collaboration with key partners.¹⁴ In 2012, SPA and SARA survey questionnaires were revised to address changes in global needs for measuring health service delivery and harmonised so that both surveys collect data on the core SARA indicators established by WHO and USAID.¹⁵ The number of surveys conducted has increased annually (online supplementary e-figure 1) since their launch in 1997 (SPA) and 2009 (SARA). SPA or SARA data are now available in 30 countries, with 13 of them having multiple surveys over time (53 total SPA and SARA surveys).^{13 14}

While the two programmes have similar goals and survey characteristics, some key differences exist (table 1). Both SPA and SARA surveys are implemented as either a census or a nationally/subnationally representative sample of health facilities. The SARA methodology recommends a facility census every five years, with a nationally representative sample of facilities surveyed annually to provide health sector planning guidance. In practice, for both surveys, implementation depends on available resources. While conducted less frequently, SPA surveys provide subnational estimates, whereas SARA surveys often provide only national estimates. For both surveys, health facility samples are drawn using stratified equal probability systematic sampling from a master facility list of all formal sector public and private facilities in the country stratified by region, facility type and/or managing authority. Sampling frames are limited to health facilities included in the master facility list and, depending on country, samples include all, part or no private sector facility. For both surveys, some strata (eg, hospitals) are oversampled, and sampling weights are derived to account for the complex survey design.

Table 1.

Characteristics of Service Provision Assessment (SPA) and Service Availability and Readiness Assessment (SARA) programmes

Characteristics	SPA	SARA
Survey design and sampling	Census or sample of health facilities at national level; sampling for some surveys allows sub-national estimates	Same as SPA
	Facility sample drawn from a Master Facility List of all formal sector public and private facilities in the country, generally stratified by facility type and managing authority	Same as SPA
	Sample of health service providers selected from those present in the facility on the day of the assessment	n/a
	For observation of ANC, FP and sick children care, patients are systematically selected based on the number of clients present at service site on the day of the visit	n/a
Survey questionnaires	Facility inventory for general and specific service availability and readiness	Same as SPA
	Health worker interview	n/a
	Direct observation of care	n/a
	Patient (ANC, FP, sick child caregiver) exit interviews	n/a
Interviewer training* (weeks)	~4	~1–2
Field team composition	Team leader, interviewers, driver	Same as SPA
Technical assistance	Centralised support provided by DHS Program	Decentralised support provided on request by partners including WHO HQ and regional offices, academic institutions, NGOs
Data access	Free public access at www.measuredhs.com	Metadata archive hosted by WHO with country contact information for microdata at http://apps.who.int/healthinfo/systems/datacatalog/index.php/catalog
Survey cost	$$–$$$	$–$$

*Includes field practice.

ANC, antenatal care; DHS, Demographic and Health Survey; FP, family planning; NGO, non-governmental organisation; n/a, not applicable.

SPA and SARA questionnaires both contain a facility inventory module, which collects information on general and specific service availability and readiness (eg, types of services offered, basic amenities, standard precautions for infection prevention, trained staff, practice guidelines, equipment, medicines and commodities, diagnostics) observed by the interviewer on the day of assessment. The SPA survey includes additional modules—a health worker interview, direct care observations and patient exit interviews. The health worker interview collects information on health workers’ professional qualifications, services provided, training and attitudes about the work environment. Through direct observation of antenatal care (ANC), family planning (FP) and sick child visits, the SPA assesses providers’ adherence to accepted guidelines of care delivery. The patient exit interviews collect data from ANC patients, FP patients and caretakers of sick children whose consultations were observed on services received, recollection of instructions and information given by providers, and satisfaction with services.

Both surveys employ similar methodologies for adaptation of questionnaires at the country level, interviewer training, questionnaire pre-testing and field team composition. For the SPA, technical assistance is centralised and provided by the DHS Program; for the SARA, technical support is decentralised and provided by request. Partners providing support for SARA surveys include WHO headquarters, WHO regional offices, academic institutions and non-governmental organisations. Thus, there is more variability in the implementation of SARA surveys than of SPA surveys. All SPA data are made publicly available on the DHS Program website with standardised structure and variable recoding, which allow for comparison across countries and over time. SARA data are not currently available in a single repository; a metadata archive is hosted by WHO with country contact information for microdata.

Analytic approach

We employed a two-step approach to ascertain the use of SPA and SARA surveys in assessing the quality of maternal and newborn health services: a questionnaire mapping exercise and a scoping review.

Following review of SPA and SARA survey questionnaires, individual questions were first mapped to the eight domains in the Framework for five service areas: ANC, FP, prevention of mother-to-child transmission of HIV (PMTCT), delivery and newborn care. The goal was to ascertain the use of SPA and SARA for measuring QoC broadly across the continuum of care for mothers and newborns. Subsequently, for each Framework domain with available SPA/SARA data, we mapped the SPA and SARA questions to the proposed quality standards and statements proposed only for care around the time of childbirth (ie, labour, childbirth and the immediate postpartum period). For each quality statement, we assessed whether corresponding WHO-proposed quality indicators could be derived either fully or partially from the SPA and SARA data. Indicators that could be derived only partially were those that included qualifiers or time restrictions such as ‘routinely’, ‘sufficiently’ and ‘every n months’, which are not available in SPA and SARA data. Of note, there is overlap between quality indicators corresponding to the evidence-based practices and the essential physical resources domains in the WHO QoC Framework. To avoid double counting of indicators that could be captured with SPA/SARA data, these indicators are presented and counted only for the essential physical resources domain since availability of essential resources is implied by the practice of evidence-based interventions.

Next, to understand the scope of contemporary SPA and SARA data applications to assess maternal and newborn QoC in LMICs, we conducted a scoping review of the literature. We used the survey names ‘Service Provision Assessment’ and ‘Service Availability and Readiness Assessment’ to search PubMed, Scopus and Embase databases for relevant articles published before 21 November 2017 (figure 2). Of 184 total references identified, 32 studies were included in the scoping review (26 used SPA, three used SARA, two used a modified SARA, and one used both SPA and SARA survey data). For each of these studies, data were extracted on the following characteristics: country, survey type (SPA or SARA), study objective, types of services assessed, use of any quality of care framework to guide the work, equivalents of Framework dimensions measured, and methods for generating quality metrics.

Figure 2.

Flow chart for identifying contemporary applications of Service Provision Assessment (SPA) and Service Availability and Readiness Assessment (SARA) programs.

Results

Measures of maternal and newborn quality of care in SPA and SARA surveys

The experience of care dimension in the Framework, composed of three QoC domains (effective communication, respect and preservation of dignity, and emotional support), mapped to an equivalent patient experience concept in SPA, but not SARA, and only for ANC and FP services. Available data relate to patients’ knowledge of available services, adequate understanding of the information and explanations received from providers, privacy, respectful treatment, satisfaction with facility infrastructure and services, user fees and trust to confide in the staff (table 2).

Table 2.

Available data elements in Service Provision Assessments (SPA) and Service Availability and Readiness Assessments (SARA) to assess WHO’s Quality of Care (QoC) Framework domains by type of service

WHO QoC Framework		Types of Services					SPA/SARA
Dimensions	Domains	FP	ANC	PMTCT	Delivery	Newborn	Data elements	Concepts
Experience of care	Effective communication	Ο	Ο	—	—	—	Knowledge	Patient experience
		Ο	Ο	—	—	—	Adequate explanations
	Respect and dignity	Ο	Ο	—	—	—	Privacy
		Ο	Ο	—	—	—	Respectful treatment
		Ο	Ο	—	—	—	Satisfaction with facility infrastructure
		Ο	Ο	—	—	—	User fees
	Emotional support	Ο	Ο	—	—	—	Trust to confide in staff
Cross-cutting (experience and provision of care)	Essential physical resources	Ο ◆	Ο ◆	Ο ◆	Ο ◆	Ο ◆	Service availability	Availability and readiness
		Ο ◆	Ο ◆	Ο ◆	Ο ◆	Ο ◆	Infrastructure
		Ο ◆	Ο ◆	Ο ◆	Ο ◆	Ο ◆	Equipment
		Ο ◆	Ο ◆	Ο ◆	Ο ◆	Ο ◆	Standard precautions
		Ο ◆	Ο ◆	Ο ◆	Ο ◆	Ο ◆	Medicines
		n/a	Ο ◆	Ο ◆	Ο ◆	Ο ◆	Diagnostics
	Competent and motivated human resources	Ο ◆	Ο ◆	Ο ◆	Ο ◆	Ο ◆	Staff training
		Ο ◆	Ο ◆	Ο ◆	Ο ◆	Ο ◆	Guidelines displayed
		—	—	—	—	—	Staff motivation
Provision of care	Evidence-based practices for routine care and management of complications	Ο	Ο	—	—	—	Counselling	Provider practices
		Ο	Ο	—	—	—	History taking and examination
		Ο	Ο	—	—	—	Testing and diagnosis
		Ο	Ο	—	—	—	Prophylaxis
		n/a	Ο	—	—	—	Treatment
	Functional referral systems	—	—	—	Ο ◆	—	Referral
	Actionable information systems	Ο	Ο	Ο	Ο	Ο	System in place	Information systems
		—	—	—	—	—	Data used for decision-making

Presence of a circle and/or diamond denotes that SPA and/or SARA, respectively, have ≥1 question that measures ≥1 component of the QoC domain.

O, SPA; ◆, SARA; —, not covered by SPA or SARA; n/a, not applicable to service; ANC, antenatal care; FP, family planning; PMTCT, prevention of mother-to-child transmission of HIV infection.

The two cross-cutting domains in the Framework (competent and motivated human resources, and essential physical resources available) mapped comprehensively to the concepts of service availability and readiness collected mainly with the facility inventory module in both SPA and SARA across all five services of interest. Data exist in SPA/SARA to explore service availability, facility infrastructure, equipment, standard precautions, medicines, diagnostics, staff training and practice guidelines.

The provision of care dimension and its three QoC domains (evidence-based practices for routine care and management of complications, actionable information systems and functional referral systems) mapped to the concepts of provider practices and information systems collected in SPA surveys (mainly for FP and ANC services) and to a very limited extent (referral systems only) in SARA surveys. Relevant data elements included counselling, history taking and examination, testing and diagnosis, prophylaxis, treatment, referral, information system being in place and data from information systems being used for decision-making. The provision of ANC and FP care data are collected in SPA through direct observation of care, the gold standard for assessing QoC.¹⁶ The availability of functional information systems is measured in the SPA via general questions asking about the existence of a system to regularly collect health services information and the frequency of generating reports. Yet, no information is collected on use of these data for decision-making.

Data are available for some degree of assessment for about a fifth of the proposed quality statements (six of 31 with SARA; seven of 31 with SPA) regarding quality of care around the time of childbirth (table 3). More specifically, four quality indicators can be fully generated (ie, as proposed by WHO) using SARA and SPA surveys; 19 and 23 indicators can be only partially obtained from SARA and SPA, respectively (online supplementary e-List). The majority of these quality measures are input indicators assessing the two cross-cutting quality domains in the Framework using elements of infrastructure, equipment, medicines and staff training (ie, availability and readiness concept) in SPA and SARA surveys. Differences noted between use of SPA and SARA relate to data on actionable information systems, available only in SPA.

Table 3.

Available data in Service Provision Assessments (SPA) and Service Availability and Readiness Assessments (SARA) to assess proposed quality standards, statements and measures corresponding to WHO’s Quality of Care (QoC) Framework domains for care around the time of childbirth

WHO QoC Framework		Proposed WHO metrics that can be assessed				SPA/SARA
Dimensions	Domains	Quality standards	Quality statements	Partial quality indicators	Full quality indicators	Data elements	Concepts
Cross-cutting (experience and provision of care)	Essential physical resources	Ο ◆ S8	QS8.1	Four input	One input	Infrastructure	Availability and readiness
		Ο ◆ S8	QS8.3	Six input	—	Equipment
		Ο ◆ S8	QS8.3	Four input+1 process	—	Medicines
		Ο ◆ S8	QS8.3	—	One input	Diagnostics
	Competent and motivated human resources	Ο ◆ S7	QS7.1 QS7.2	One input Two input+1 process	— One process	Staff training
Provision of care	Functional referral systems	Ο ◆ S3	QS3.2	—	One input	Referral	Provider practices
	Actionable information systems	Ο S2	QS2.1 QS2.2	Two input Two input	— —	System in place	Information systems

O, SPA; ◆, SARA; —, not covered by SPA or SARA.

Applications of using SPA and SARA data to assess quality of maternal and newborn care

The first of 32 identified studies that use SPA/SARA data to explore the quality of maternal and newborn care was published in 2006 (online supplementary e-figure 1). Since then, one or two such studies were published annually until 2016; six studies were published in 2016, after the dissemination of the Framework; and 16 studies were published in 2017, after or around the time of release of the related WHO quality standards for care around the time of childbirth.

Across studies, data from 22 countries (18 in Africa) were included in publications representing four of six WHO regions (online supplementary e-table 1). SPA/SARA data from several countries were more frequently analysed (Kenya 15 studies; Tanzania, Namibia and Rwanda 8 studies; Malawi and Uganda 5 studies). Of note, all of these countries are among the 81 countries that currently account for >95% of maternal and >90% of child deaths in the world.¹⁷ Childbirth services were most frequently examined (16 studies), followed by FP (13 studies) and ANC (12 studies); conversely, least examined were newborn care (six studies) and PMTCT (one study). Twenty-three of 32 studies focused on a single service.

The majority of studies (20 of 32 studies) did not reference use of a quality of care framework. Nine studies used Donabedian’s structure–process–outcome framework, and three other studies each employed a different framework (ie, Bruce’s FP framework,¹⁸ the Nesbitt quality index dimensions for maternal and neonatal care,¹⁹ and Hulton’s framework²⁰). Without making reference to the WHO QoC Framework, a large number of studies set to examine one or more of its proposed QoC dimensions. Twenty-six of 32 studies assessed the cross-cutting human/physical resources domain, 17 studies assessed the provision of care dimension, and eight studies assessed the experience of care dimension. Among the eight studies that assessed the experience of care dimension, six were conducted in 2016 and 2017. Eighteen studies assessed one Framework dimension, nine studies assessed two dimensions and five studies assessed all three dimensions.

To create QoC summary measures, most studies used a combination of employing a conceptual framework or a review of the literature, available clinical guidelines, and statistical methods for data reduction. Four common such methods were identified across the 32 studies: individual item frequency (20 studies), index as sum or average number of items in a service domain (20 studies), index using principal components analysis (seven studies) and index using factor analysis (one study).

Discussion

The multitude of QoC definitions^{11 21 22} and frameworks^{20 23–29} available in the literature may have posed a challenge for countries interested in measuring the quality of maternal and newborn care. This diversity has also made comparison and benchmarking difficult both within and across countries. Encouraging countries to use the now well-recognised WHO QoC Framework for maternal and newborn care quality and the related WHO quality standards for care around the time of childbirth is highly important, but data sources are needed to do so. SPA and SARA surveys offer promising platforms for measuring QoC in maternal and newborn health in LMICs as proposed by WHO.

As they stand now, both surveys are well suited to assess the cross-cutting (provision and experience of care) dimension in the Framework, which includes the domains of competent, motivated human resources and essential physical resources. A key data gap for this dimension relates to staff motivation, measured by neither SPA nor SARA. The SPA uses the patient interview modules to assess the experience of care for FP and ANC services, but not for other services. Patient satisfaction is assessed in the SPA by asking if patients had major, minor or no problems with different aspects of the services received. While these questions aim to capture the experience of care, research has shown that their structure and administration during in-person interviews generally results in over-reporting of satisfaction with services due to social desirability bias.³⁰ Other measures may be better suited to capture a valid understanding of patients’ experience of care. The SPA survey also captures the provision of care dimension, yet mainly for FP and ANC services. Only with regard to availability of functional referral systems are both SPA and SARA surveys useful. Yet, both surveys rely on a single set of questions on the availability and functionality of an ambulance for emergency referral use during childbirth. The lack of data on provision of care for delivery and newborn services is particularly relevant as receiving high-quality care at this critical time is key to reducing maternal and newborn mortality. In addition, while essential physical and human resources are necessary for high-quality service provision, they are not sufficient to ensure delivery of high-quality services.³¹ Measuring the provision of care domain is vital to fully understanding how to improve service quality. Thus, despite their being nationally representative and capturing standardised information on health service delivery in LMICs, SPA and SARA surveys do not provide a comprehensive representation of the WHO QoC Framework domains and can only capture a minority of proposed WHO quality indicators for care around the time of childbirth.

SPA and SARA survey data are readily available for analysis across more than 30 countries. The measurement of the quality of maternal and newborn health services would benefit from guidance on using current SPA and SARA questions to generate standard metrics for a wider range of services, practices and experiences. Efforts to design and integrate additional QoC-focused questions in the SPA and SARA questionnaires to fill the gaps identified by our analysis would better equip countries that aim to monitor the quality of health services for mothers and newborns per WHO guidance (ie, Framework, quality standards). In particular, there is critical need to expand assessments of both provision of care and experience of care dimensions across all maternal and newborn services.

Conclusion

Use of SPA and SARA survey data is growing rapidly, a trend that coincides with the global shift towards recognising the importance of QoC for improving maternal and neonatal health outcomes in LMICs. It will be important to continue using harmonised questions in the two surveys as well as to align the surveys as best as possible to address the Framework elements and measure key WHO-proposed quality indicators to facilitate data comparisons between and within countries.