Measurement of district health system strengthening efforts for maternal and newborn health service delivery: synchronized application of evaluation models in the West Nile region, Uganda, 2019–2021

Abstract

Introduction

Uganda has high maternal, neonatal, and under-five mortality rates. Between 2019 and 2021, the United Nations Children’s Fund (UNICEF), in collaboration with the Ministry of Health and other partners, implemented a maternal and newborn health (MNH) quality improvement programme in the West Nile region of Uganda to improve the delivery and utilization of MNH services. This study describes the application of health system evaluation models to monitor changes in health system capacity and the quality of MNH services.

Methods

We designed two novel health system evaluation models—the Progression and ServQual models—which progressively monitor changes in health system capacity and quality of health services. The progression model was designed to measure changes in health system capacity using a set of indicators and benchmarks. For each health system component, the tool generates scores as follows: > 90% for level 4 progression (surpasses basic expectations); 70–90% for level 3 (meets basic expectations); 50- < 70% for level 2 (needs improvement); and < 50% for level 1 progression (needs urgent attention). The quality of MNH services was monitored through annual client satisfaction surveys using the ServQual model. The identified health system capacity and service quality gaps were used to inform corrective actions.

Results

Overall, the average health system capacity scores across the 12 districts increased from 62.9% (level 2) in 2019 to 71.2% (level 3) in 2021. The capacity scores for each health system component increased: governance and leadership increased from 67.7% to 74.1%, health financing increased from 67.4% to 69.3%, access to essential medicines increased from 50.2% to 77.0%, the health information system increased from 71.8% to 74.9%, the health workforce increased from 56.8% to 66.9%, and MNH service delivery slightly increased from 63.5% to 64.5%. Client satisfaction with the quality of MNH services increased from 26.5% in 2019 to 39.8% in 2021.

Conclusion

Synchronized application of the Progression and ServQual models is an important innovative strategy for monitoring changes in district health system capacity and quality of health services. The results of applying these models can be used to guide better targeting for health system capacity improvement.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-025-12276-w.

Background

The West Nile region is located in Northern Uganda and is bordered by the Democratic Republic of the Congo to the South and to the West, by the Republic of South Sudan to the North and by the Albert Nile to the East. The region has experienced some of the poorest maternal and newborn health (MNH) indicators nationally. For example, in 2018, the region registered a very low antenatal care (ANC) attendance in the first trimester of pregnancy of 23.3% according to the National Health Management Information System (HMIS) and high in-facility stillbirth rates of approximately 90 per 1,000 births [1]. The bottlenecks to MNH services in the region included weak governance and leadership structures, an insufficient health workforce, poor infrastructure, the presence of a high number of refugees and barriers such as long distances to health facilities, negative cultural beliefs, poor gender relations and widespread poverty [2, 3].

To address some of the aforementioned challenges, the United Nations Children’s Fund (UNICEF) through the AVSI Foundation, an implementing partner in the region, collaborated with the Ministry of Health and the District Local Governments to support the delivery of an integrated package of high-impact MNH interventions to address key barriers to the availability and utilization of quality MNH services. The aim was to strengthen all levels of the health system to effectively respond to the needs of mothers and newborns and to increase community awareness, demand and utilization of available MNH services. Specifically, interventions aimed at addressing the “three delays” that lead to maternal and newborn death include delays in making the decision to seek care, delays in reaching the health facility and delays in receiving timely and quality care [4]. The interventions are summarized in Table 1.

Table 1.

Key MNH program interventions implemented in the West Nile region, Uganda

Interventions for strengthening capacities of districts to plan, implement and monitor high quality and equity-focused interventions for MNH
	• Training of district health teams and health subdistricts in the use of bottleneck analysis (BNA) for planning for health services • Establishment and training of maternal and perinatal death surveillance and response (MPDSR) committees in the districts • Functionalization of the integrated human resource information system (iHRIS) to support human resource planning • Conducting quarterly performance reviews using the bottle neck analysis and scorecard methodologies • Conducting annual district health system strengthening capacity assessments and client satisfaction surveys using the Progression and ServQual models, respectively
Interventions for enhancing capacity of health facilities to provide a full range of essential MNCH and nutrition services
	• Functionalization of the health unit management committees (HUMCs) at all health facilities • Procurement and distribution of basic MNH medicines, supplies and equipment to health facilities with critical needs • Repair of selected (model) maternity units in health facilities and neonatal intensive care units (NICUs)/special newborn care units in hospitals across the districts • Water and sanitation improvements (establishment of piped water systems, construction of sanitation facilities and bathrooms, at selected health units) • Training and onsite mentorships of health workers in key MNH areas including antenatal care (ANC), postnatal care (PNC), basic emergency obstetric and newborn care (BEmONC), comprehensive emergency obstetric and newborn care (CEmONC), essential newborn care, key family care practices (KFCPs), kangaroo mother care (KMC) and the use of BABIES matrix • Establishing a referral ambulance system for high-risk pregnant women and newborns
Interventions for increasing community awareness, demand and utilization of MNH services
	• Establishment of expert mother groups in the communities across all districts • Training of community health workers (CHWs) on KFCPs and supporting them to deliver on their mandate • Integrated community outreaches for MNH services in hard-to-reach areas • Establishment of a community boda boda (motorcycle taxis) referral system for pregnant mothers and children in hard-to-reach communities • Implementation of FamilyConnect, a digital health solution for increasing uptake of maternal and child health services • Community dialogue meetings on key MNH services including ANC, delivery and PNC

In this paper, we describe the use of the Progression and ServQual models to monitor changes in health system capacity and the quality of health services in the West Nile region of Uganda, respectively

Materials and methods

Setting

There are 12 districts in the West Nile region, namely, Adjumani, Arua, Koboko, Madi-Okollo, Maracha, Moyo, Nebbi, Obongi, Pakwach, Terego, Yumbe and Zombo. The region has a total population of 2,988,300 people living in approximately 500,000 households [5]. The people of the West Nile are predominantly (80%) rural, and subsistence farming is the main livelihood activity in the region [3]. Like in the rest of the country, the health care system in the region uses a numerical ladder to designate the size and function of health centres (HCs): HC level II at the parish level; HC III at subcounty level; HC IV at county level; general hospital (GH) at district level; and regional referral hospital (RRH) at the regional level. The community health workers operate at HC level I with no physical structures. According to the Uganda HMIS, there are a total of 384 health facilities across the 12 districts: 2 RRHs, 13 GHs, 16 HC IVs, 165 HC IIIs, and 188 HC IIs (https://hmis.health.go.ug/dhis).

Evaluation models

The district health systems strengthening progression model

The lack of universally agreed upon metrics limits the ability to effectively monitor changes in health system capacity [6]. To this end, UNICEF, in collaboration with the Ministry of Health and Makerere University School of Public Health, developed a novel evaluation model which progressively measures changes in capacity of the health system building blocks at the district level [7, 8]. The automated Microsoft Excel-based tool was designed to measure changes in capacity for each health system building block using a set of indicators and benchmarks (Supplementary file 1). For each health system building, the tool generates scores with colour codes for each level of progression as follows: > 90% score, dark green colour for level 4 progression (surpasses basic expectations); 70–90% score, light green colour for level 3 (meets basic expectations); 50- < 70% score, yellow colour for level 2 (needs improvement); and < 50% score, red colour for level 1 progression (needs urgent attention) (Fig. 1).

Fig. 1.

The progression model

The ServQual model for measuring client satisfaction

Over the years, Uganda has registered suboptimal quality of care (QoC) of services at all levels of health care service delivery. As such, there has been greater advocacy for improving the QoC in the country. Several methodologies for QoC improvement and assessment were adopted by the Ministry of Health and implemented nationally. In 2018, the Ministry of Health adopted the ServQual model as an essential tool for monitoring and improving service quality [9]. The ServQual model is used to measure client satisfaction with health services and is based on the assumption that service quality is a function of clients’ expectations of a service and their perceptions of the service actually rendered. The level of satisfaction is determined by the difference between clients’ expectations and perceptions of the quality of service actually delivered. The gap between the perceptions and expectations of clients is measured on five dimensions, namely, Reliability, Assurance, Tangibles, Empathy and Responsiveness. Reliability refers to the ability to provide services dependably and accurately; assurance refers to the knowledge and courtesy of service providers and their ability to inspire trust and instil confidence in clients; tangibles describes the appearance of physical facilities, personnel and equipment; empathy refers to the ability to provide caring and individualized attention to clients; and responsiveness refers to the willingness to help clients and provide prompt service (Fig. 2) (Supplementary file 2). The service is considered excellent if the perceptions exceed the expectations; it is regarded as good or adequate if it only equals the expectations; and it is classified as bad, poor or deficient if expectations are not met.

Fig. 2.

The ServQual model

The ServQual model contains 22 pairs (expected vs. perceived service) of Likert scale statements structured around the five dimensions of service quality that link specific service characteristics to clients’ expectations [9]. The scale is composed of a five-point scale ranging from "strongly agree" (5) to "strongly disagree" (1). The "strongly agree" end of the scale is designed to correlate with high expectations and high perceptions. Service quality is operationalized by calculating the difference between expectations and perceptions in relation to the 22 items. It is determined as a function of perception and expectations and is modelled as:

$$SQ={\sum }_{i=1}^k\left(P_{\mathit{ij}},-,E_{\mathit{ij}}\right)$$

where SQ = overall service quality; k = number of attributes; P_ij = Performance perception of stimulus i with respect to attribute j; and E_ij = Service quality expectation for attribute j, which is the relevant norm for stimulus i.

Use of the evaluation models

Both models were applied in all 12 districts in the West Nile region through a partnership between UNICEF and the Makerere University School of Public Health. The support involved training the district health teams in both models and their application, data collation, analysis, report writing and dissemination of findings. The progression model was applied at the district level, and the ServQual model was applied at all 384 health facilities in the region.

Data collection

Assessments of district health system capacity and client satisfaction with the quality of health services were conducted in 2019, 2020 and 2021 using the Progression and ServQual tools, respectively. The progression model data were collected by a team of trained program officers and members of the district health teams, particularly the district biostatisticians. The process involved structured interviews with other district health team members, observations, review of records and retrospective analysis of data for selected key indicators from the district health information software (DHIS2) (https://hmis.health.go.ug/dhis).

The ServQual model data were collected by trained data collectors through face-to-face exit interviews with clients at the health facilities. At each facility, the proportionate number of clients to be interviewed was determined by the probability proportional to the size of the client population at the facility. Systematic sampling was used to select the clients to be interviewed. The facility client registers were used as the sampling frame. The sampling interval was obtained by dividing the total number of clients attending the facility by the required number of clients to be interviewed (N/n). After obtaining a random start from a table of random numbers, the interval was followed until the required number of clients at each facility was obtained. The ServQual tool was designed in the Open Data Toolkit (ODK) form, which was installed on Android-based telephones and enabled offline data collection and submission of data to the central aggregate server.

Data management and statistical analysis

After each progression model assessment, the data from the completed tools were merged for descriptive analysis, which was performed in Ms Excel 2013. The automated Microsoft Excel-based tool generates scores and colour coded as follows: > 90% score, dark green colour code for level 4 progression (surpasses basic expectations); 70–90% score, light green colour code for level 3 (meets basic expectations); 50- < 70% score, yellow colour code for level 2 (needs improvement); and < 50% score, red colour code for level 1 progression (needs urgent attention).

After each ServQual model assessment, the data were downloaded from the central server of the Makerere University School of Public Health and analysed in Microsoft Excel 2013. The mean scores for each of the items (perceptions and expectations) were computed. The gap score for each of the items was computed as the perception score minus the expectation score. The percentage scores were grouped into 4 categories with colour codes of dark green (≥ 75%), light green (50%−70%), yellow (25%- ≤ 50%) and red (< 25%). Data on specific key client satisfaction dimensions were analysed, and differences across the districts were compared.

For the purposes of this paper, the Progression and ServQual model results for 2019 and 2021 were compared.

Quality control

The progression model was applied by trained teams of program officers and district biostatisticians. Experienced and skilled data collectors who spoke the local language of the respective districts were hired and trained in the ServQual data collection. Training for both the Progression and ServQual models involved interpretation of the variables in the tools, application of the tools and electronic data entry and submission. Data collection was closely supervised by a team of central supervisors who oversaw the data collection process and checked for completeness of the collected data.

Results

District health system capacity assessments

Overall, the district health system capacity improved over the three years, with the average capacity scores increasing from 63.5% (level 2 of progression) in 2019 to 73.4% (level 3 of progression) in 2021, representing 9.9% increase. The overall capacity scores for each health system component increased: governance and leadership increased from 67.7% to 74.1%; health financing increased from 67.4% to 69.3%; access to essential medicines increased from 50.2% to 77.0%; health information systems increased from 71.8% to 74.9%; health workforces increased from 56.8% to 66.9%; and MNH service delivery increased from 63.5% to 64.5% (Fig. 3).

Fig. 3.

Overall percentage scores for each health system domain in 2019 and 2021

The district-specific health system capacity scores for 2019 and 2021 are compared in Table 2 below.

Table 2.

Comparison of the district health system capacity scores in 2019 and 2021

Client satisfaction with the quality of health services

Overall, client satisfaction with health services increased from 26.5% in 2019 to 39.8% in 2021. The district-specific client satisfaction levels in 2019 and 2021 were compared. Client satisfaction with health services improved in 9 districts but decreased in 3 districts. In 2021, the highest client satisfaction levels were reported in the Maracha (81.3%), Packwach (80.7%), Zombo (61.6%) and Moyo (61.1%) districts (Table 3).

Table 3.

Comparison of district client satisfaction levels in 2019 and 2021

No	District	Number (%) of clients satisfied with health services		% change
		2019 N = 3,954 (%)	2021 N = 4,684 (%)
1.	Maracha	78 (34.4)	208 (81.3)	46.9
2.	Pakwach	52 (31.5)	151 (80.7)	49.2
3.	Zombo	22 (10.5)	157 (61.6)	51.5
4.	Moyo	12 (4.8)	198 (61.1)	56.3
5.	Madi-Okollo	71 (38.2)	98 (48.0)	9.8
6.	Koboko	72 (41.4)	136 (43.0)	1.6
7.	Yumbe	136 (25.9)	91 (41.0)	15.1
8.	Adjumani	148 (40.7)	254 (40.5)	−0.2
9.	Terego	-----	87 (26.0)	----
10.	Arua	202 (23.4)	35 (24.6)	1.2
11.	Nebbi	84 (28.5)	44 (12.3)	−16.2
12.	Obongi	5 (3.2)	15 (8.5)	5.3
Overall		1,049 (26.5)	1,864 (39.8)	13.3

^---- The district had not yet been created by the time of the assessment

Service quality gaps

In the ideal situation, the difference between the mean scores for client perceptions and expectations should be zero. A reduction in the mean scores for service quality gaps was observed across all five dimensions of service quality between 2019 and 2021. Although the decrease in mean scores was more marked for the Responsiveness and Tangibles dimensions, the two dimensions registered the highest mean gap scores in 2021. The mean scores for the Reliability, Assurance and Empathy dimensions were closer to ideal in 2021 (Fig. 4).

Fig. 4.

Comparison of the mean scores for service quality gaps in 2019 and 2021

Individual statements that contributed to the mean gap scores by dimension

Further analysis of the individual statements making up each dimension was undertaken to determine the contributions of each statement to the total mean gap score for each dimension, thereby identifying the specific areas of service quality that needed improvement. The individual statements with the highest mean gap scores included the following: (i) enough medicines and supplies (−0.55); (ii) responding to clients’ complaints in time (−0.18); (iii) provision of prompt services to clients (−0.15); (iv) willingness of service providers to handle clients’ complaints (−0.14); and (v) a clean environment and setting (−0.12).

Discussion

The results show that the evaluation models were successfully applied to measure changes in the district health system capacity and levels of client satisfaction with health services across all the districts in the West Nile region between 2019 and 2021. These findings may have important implications for national scale-up of the use of the evaluation models to measure changes in health system capacity and quality of health services and to guide better targeting for health system capacity improvement.

Health system capacity assessment

The application of the progression model showed linear improvements in district health system capacity scores and the level of progression. All the districts in the region progressed to level 3 of health system strengthening. This implies that all the districts met the basic expectations for health system capacity. The multifaceted interventions implemented in the region could have contributed to the observed improvements. Across the health system strengthening domains, there was an increase in percentage scores for most domains saved for the health financing domain, which declined between 2019 and 2021. The current financing of the Uganda health sector is very low, at only 6.1% of the total national budget [10]. Across all districts, the per capita health allocation is lower than the 2001 Abuja pledge of 15% [11]. Furthermore, due to the long bureaucratic and delayed procurement processes, most districts are not able to utilize their capital development funds within a given timeframe. These gaps could explain the low performance in the health financing domain. The impact of COVID-19 on the delivery and utilization of essential maternal health services such as antenatal care and health facility deliveries has been well documented [12–14]. The low scores of the service delivery domain could be due to the impact of COVID-19 on the delivery and utilization of maternal health services in the region. In addition, the gaps in the health financing domain could also have contributed to inadequate service delivery.

Client satisfaction with health services

The diverse interventions for addressing a wide range of gaps at the district, health facility and community levels could have contributed to the observed improvement in client satisfaction levels. Although client satisfaction with health services has significantly improved over the years, the realized client satisfaction level is still far below the national target of 79% [15]. The negative mean gaps across all five dimensions of service quality imply that, generally, the clients’ expectations are not met and that service quality is perceived as poor across all five dimensions. The largest mean gap scores were registered in the tangibles and responsiveness dimensions. These are considered most important by clients [16].

Low government expenditures on health continue to contribute to inadequacies in essential medicines and health supplies. In the financial year 2019/20, only 49% of the health facilities in the country had access to essential medicines and health supplies for more than 95% of the time, which is much lower than the national target of 75% of facilities [17]. It is therefore not surprising that the absence of essential drugs and health supplies (tangibles) at health facilities is the leading cause of client dissatisfaction in the region. Although the Ministry of Health has undertaken key steps in strengthening the current supply chain system, including recruiting pharmacists at the district level to support supply chain management at lower-level health facilities and conducting health facility audits to review drug stock cards, prescriptions and facility ordering practices, among others [17], stock outs of essential medicines and health supplies at health facilities are ubiquitous.

The responsiveness of a health care system significantly contributes to clients’ appreciation of the overall quality of care. High service quality gaps related to the inability to provide prompt services and respond to clients’ complaints in time were observed and could have contributed to the low client satisfaction levels. A recently published paper reports that delays from standing in long lines, unprofessional behaviour by medical professionals and lack of medicines at health facilities drive negative perceptions held by clients towards health services [18]. Similar findings were previously reported in a national survey that showed that one in every three clients did not receive quality services [19]. These findings suggest that reinforcing good clinical and client-centred practices through training, mentorship and supportive supervision of health service providers should be considered by all key stakeholders.

Limitations

Four limitations are worth mentioning. First, the progression model heavily relies on retrospective review of records and analysis of health facility data from national health information systems. The information obtained from these secondary data sources is limited by the data already collected and reported. In the event that some of the critical reports are missing and data in the national health information systems is either missing or incomplete, the respective health system domains and subdomains may not be accurately scored. Secondly, the ServQual model mainly focuses on the functional aspect of service quality and not on the technical aspect. The functional dimensions of service quality relate to the manner in which the health care services are provided whereas the technical dimensions relate to accuracy of medical diagnoses and procedures or the conformance to professional specifications. To fully assess the quality of healthcare delivery, it is recommended that both the technical and functional aspects of service quality be considered [20]. Thirdly, in the absence of a counterfactual or a control group, it was not possible to establish whether the observed improvements were due to the MNH programme or not. One would require another study design, such as a quasi-experimental design, to establish a cause-and-effect relationship between the MNH program and the improvements in district health system capacity and quality of health services delivery. Lastly, courtesy bias is common and results from the reluctance of respondents to express negative opinions of a service while they are within the premises of the service provider during the exit interviews, and this could result in over-reporting of satisfaction levels.

Conclusion

Synchronized application of the Progression and ServQual models is an important innovative strategy for monitoring changes in district health system capacity and client satisfaction with health services. The results of applying these models can be used to guide better targeting of health system capacity improvements and improved service delivery to clients. The models may be considered for national scale-up to enhance monitoring of district health system strengthening efforts.

Authors’ contributions

S.M.,G.L., P.P., C.M., F.M., F.G., T.D.H., A.G.A, and A-M.B.-Conceptualized the study. S.M.,G.L., P.P., C.M., and F.M.-Designed the study methods S.M., I.M.L., and J.S.-Performed the analysis. S.M. and A-M.B, -Wrote the original draft of the manuscript. All authors reviewed and edited the manuscript.

Funding

This study was funded by UNICEF Uganda (Contract 43329525).

Data availability

The datasets used and/or analysed during the current study are available from the corresponding author on request.

Declarations

Ethics approval and consent to participate

Ethical clearance for the assessments was obtained from the Makerere University School of Public Health Higher Degrees Research and Ethics Committee (Ref No SPH-2021–162). Permission to conduct the assessments at the district and health facility level was obtained from the respective district health officers. Prior to the assessments, the purpose and procedures of the assessment were explained to the district health team members and to the clients. Written informed consent was obtained from all the participants. Our study adhered to the Helsinki Declaration. The data collectors signed confidentiality agreements to protect the participants and ensure legal follow-up should there be a breach of code of conduct and policy.

Consent for publication

Consent for publication was not obtained since our manuscript does not contain any individual person’s data in any form.

Competing interests

The authors declare no competing interests.