Embedding Community-Based Newborn Care in the Ethiopian health system: lessons from a 4-year programme evaluation

Abstract

Despite remarkable gains, improving neonatal survival globally remains slow paced. Innovative service-delivery packages have been developed for community health workers (CHWs) to maximize system efficiency and increase the reach of services. However, embedding these in health systems needs structural and procedural alignment. The Community-Based Newborn Care (CBNC) programme was a response to high neonatal mortality in Ethiopia. Key aspects include simplified treatment for neonatal illness, integrated outreach services and task-shifting. Using the CHW functionality model by WHO, this study evaluates the health system response to the programme, including quality of care. A before-and-after study was conducted with three survey time points: baseline (November 2013), midline (December 2015) and follow-up (December 2017—4 years after the programme started). Data were collected at a sample of primary healthcare facilities from 101 districts across four regions. Analysis took two perspectives: (1) health system response, through supplies, infrastructure support and supervision, assessed through interviews and observations at health facilities and (2) quality of care, through CHWs’ theoretical capacity to deliver services, as well as their performance, assessed through functional health literacy and direct observation of young infant case management. Results showed gains in services for young infants, with antibiotics and job aids available at over 90% of health centres. However, services at health posts remained inadequate in 2017. In terms of quality of care, only 37% of CHWs correctly diagnosed key conditions in sick young infants at midline. CHWs’ functional health literacy declined by over 70% in basic aspects of case management during the study. Although the frequency of quarterly supportive supervision visits was above 80% during 2013–2017, visits lacked support for managing sick young infants. Infrastructure and resources improved over the course of the CBNC programme implementation. However, embedding and scaling up the programme lacked the systems-thinking and attention to health system building-blocks needed to optimize service delivery.

Key messages.

• Innovations might address high neonatal mortality rates in lower- and middle-income countries: examples include the availability of improved or simplified treatment regimens, integrating outreach services and task-shifting.

• Evidence on implementation, scaling-up and integration of these approaches into the health system would enable their translation into service-delivery packages but is limited.

• A 4-year evaluation of the Community-Based Newborn Care programme in Ethiopia showed gains in services for young infants, with antibiotics and job aids available at primary healthcare facilities. However, community health workers had insufficient supportive supervision, inadequate functional health literacy and provided poor quality of care for key conditions in sick young infants.

• Embedding a community-health-worker-delivered service delivery package within a health system is a complex process, owing to a multitude of structural and procedural alignments. Both ongoing evaluation and course correction are necessary.

Introduction

Globally, despite remarkable gains in child healthcare, there are still challenges to improving neonatal survival in many low- and middle-income countries (LMICs) (United Nation, 2015; You et al., 2015). Progress in neonatal health has been steered by Sustainable Development Goal 3—that all countries should aim to achieve neonatal mortality rates (NMRs) of not more than 12 deaths per 1000 live births by 2030 (United Nation, 2015). Enhancement, extension and strengthening of neonatal health services are imperative for LMICs to meet this target. To address these challenges, several innovative approaches have been developed and recommended in public health, such as increasing the availability of improved or simplified treatment regimens (World Health Organization, 2015a), integrating outreach services and task-shifting (World Health Organization, 2008; Lassi et al., 2014).

Effective implementation at scale is required to translate both WHO guidance and evidence from efficacy trials into population-level gains. Embedding a service delivery package to be delivered by community health workers (CHWs) is a complex process owing to the multitude of structural and procedural alignments within a health system. Not surprisingly, there are very few robust evaluations of these packages when integrated into a routine health service and implemented at scale within a health system.

Countries with high NMRs, such as Ethiopia (NMR 38 per 1000 live births), have taken this challenge very seriously and have made it a public health priority (Ethiopian Public Health Institute, 2019; Tekelab et al., 2019). In response, Community-Based Newborn Care (CBNC) programmes emerged as one of the principal strategies, with variants tested in a number of countries (Khanal et al., 2011; Awasthi et al., 2020; Roy et al., 2020; Wammanda et al., 2020). In Ethiopia, the CBNC programme is a health system initiative to maximize services available in communities, engaging frontline health workers to improve newborn survival. In 2013, Ethiopia was the first country to formally launch this type of programme at scale as a part of the health system. The programme is implemented by strengthening the primary health care system, and all of the components are delivered by CHWs during home visits and static services at health posts. CBNC aims to increase access to essential care for newborns during the most vulnerable period. It comprises a service delivery package of nine components at the community level: early identification of pregnancy, focused antenatal care, promotion of institutional delivery, safe and clean delivery, immediate newborn care, management of newborn asphyxia, prevention and management of hypothermia, management of pre-term and low-birth-weight neonates and management of possible serious bacterial infections (PSBIs). The programme is based on research evidence from low-resource settings in Asia and Africa (Khatri et al., 2016; Gogia et al., 2011). It is important to highlight that these interventions were already embedded within the work of existing CHWs in Ethiopia, through the Health Extension Worker (HEW) programme (Banteyerga, 2011; Karim et al., 2013; 2015)—with the exception of the management of PSBIs, for which HEWs were trained to administer both injectable antibiotics and oral amoxicillin for provisionally diagnosed cases of neonatal sepsis. The CBNC programme is described elsewhere (Berhanu et al., 2021). In brief, it was directly implemented by the Ethiopian Federal Ministry of Health, with support from Save the Children, United Nations Children’s Fund (UNICEF), John Snow Inc. (Last 10 Kilometres initiative) and Pathfinder (the Integrated Family Health Programme). The key activities of the CBNC implementation follow an integrated health system approach, focusing on three key areas: (1) at the initiation of the programme, offering structured training to the HEW cadre on providing neonatal sepsis case management, including injectable antibiotics, case referral to health centres, and regular follow-ups in the home setting (capacity building), (2) following this, ensuring the regular supply of antibiotics, vaccines, and other job essentials to support neonatal health services (system support) and (3) strengthening the referral system within primary care facilities and embedding monitoring and monthly supervision visits by health officers to HEWs (system linkages) (Diaz et al., 2018). Typically, the HEW makes periodic visits to the family during the postnatal period to timely identify a sick newborn. Based on predefined syndromic case management criteria, the HEW immediately starts an injectable antibiotics regimen if the newborn has severe clinical signs. She convinces the family to visit the health centre (next level primary health care referral) for further advice and assessment. She regularly follows up with the family for a week to complete the antibiotics course. In addition, the HEW receives monthly supportive supervision visits from a health officer from the health centre. The purpose of these visits is to check facility records and stockouts and to review the HEW’s knowledge and skills, to make quality neonatal sepsis management services available in the community.

From inception, the CBNC initiative in Ethiopia was supported between 2013 and 2018 by external research to evaluate ongoing implementation and provide evidence for course correction to support scaling-up. This work was grounded in a framework adapted from WHO’s CHW functionality model, which focuses on assessing improvement in CHW-based programmes and services (Crigler et al., 2013) (Figure 1). This framework allowed us to explore the CBNC programme from two perspectives: (1) health system response, in terms of tangible support of infrastructure, supplies and job aids and supportive supervision of linkages within primary health care systems and (2) quality of care, by assessing the theoretical capacity of HEWs to deliver CBNC services, and their actual performance in doing so. This paper aims to present the effect of CBNC on the health system, including human resource capacity, focusing in particular on the management of PSBIs.

Figure 1.

Adapted from WHO’s CHW programme functionality model.

Methodology

The CBNC evaluation plan was co-designed by the authors’ institutes to provide evidence to improve service delivery—especially the management of PSBIs by HEWs (Avan and Berhanu, 2013). Initially, the evaluation was designed with a comparative arm without CBNC. However, the Ministry of Health expedited CBNC implementation in the comparison arm, rendering the intended comparison invalid. Hence, the design was a before-and-after study.

There were three assessment time-points in the CBNC evaluation (Figure 2):

Figure 2.

CBNC timelines: CBNC implementation (below the line) and evaluation (above the line).

1. Baseline survey (October–December 2013): conducted just before the initiation of the programme to identify health system gaps for the delivery of CBNC services.

2. Midline survey (October–December 2015): 20 months after the programme was implemented, the quality of care provided to young infants (0–59 days) was assessed.

3. Follow-up survey (November–December 2017): to determine changes in the health system 4 years after the start of CBNC implementation.

Key features of the three surveys (2013–2017) are as follows:

1. The baseline and follow-up surveys

The CBNC programme was rolled out to 12 zones across the four regions of Ethiopia in two phases: seven zones at the time of CBNC initiation (March 2014) and the remaining five zones in January 2015. In order to get a representative sample of districts from these 12 zones, we selected 101 out of the total of 210 districts by simple random sampling. In the Ethiopian health care system, primary health care units (PHCUs) comprise one health centre (sub-district level) and between three and five health posts (village level). On average two PHCUs per district were randomly selected through computerized number-generation, yielding a total of 209 PHCUs. Within each selected PHCU, in addition to the health centre, one health post was randomly selected. To be eligible for random selection, the health post had to be functional, i.e. have a physical structure for the provision of health services with at least one stationed HEW. The baseline and follow-up surveys were carried out in the same districts and PHCUs. For each survey, the health posts were randomly selected. The total numbers of primary care facilities and health workers in each survey are given in Table 1. A detailed description of the methodology and household-level coverage of CBNC interventions is given elsewhere (Berhanu et al., 2021).

Table 1.

Sample sizes for the CBNC evaluation baseline, quality of care and follow-up surveys

	Baseline survey 2013		Midline survey 2015		Follow-up survey 2017
	Expected	Achieved	Expected	Achieved	Expected	Achieved
Regions	4	4	4	4	4	4
Zones	12	12	12	12	12	12
Districts	101	101	30	30	101	101
Health centres	209	206a	120	117a	206	206
Health posts	209	206a	240	240	206	201a
HEWs	209	206a	240	240	412	335
Observation of sick young infant consultation and re-examination	N/A	N/A	720	893	N/A	NA

^aMissing due to non-accessibility of the health facilities because of poor weather or road infrastructure.

Data were collected through direct observation of facilities and interviews with staff at PHCUs, i.e. health centres and health posts. At the health centre level, the most senior staff member within the relevant department was interviewed. The health centre data collection was primarily about available health services and infrastructure. Therefore, we interviewed the most senior staff member available to gain an overview and understanding of all aspects of the facility. At the health post, the more senior HEWs were interviewed. HEWs routinely work as a team: a more senior HEW usually takes the lead in the management of sick new-borns, while a more junior HEW has a more supportive role. The more senior HEW was preferred for the assessment, as her skills and practices provided a better reflection of the quality of services available at the health post level.

• b The midline (quality of care) survey

The midline survey focused on assessing the quality of CBNC services provided to sick young infants by HEWs. The methodology of the midline survey was distinct from that of the baseline and follow-up surveys. There were two types of participant in the study: from the health system, a senior HEW who had been CBNC-trained and active at the health post for the previous 3 months, and a senior member of health centre staff; and from the community, infants under 2 months, considered sick by their caregiver and being seen for the first time by the HEW at the health post for their current illness episode. Infants (0–2 months) either presented spontaneously by the caregiver at the health post on the day of the survey or were actively identified by community volunteers for consultation on the day of the survey.

A target of at least three eligible infants selected from each health post was expected to be both feasible and to give adequate power to detect differences of at least a 15 percentage point change in the correct classification of young infant health status between intervention and comparison areas. (Note that the changed study design due to roll-out to comparison areas meant that the comparison of areas was no longer relevant. Details of sample-size estimations are given elsewhere) (Berhanu and Avan, 2017). From the sampling frame of 101 districts included in the baseline survey, a subsample of 30 were randomly selected, with the number in each zone selected with probability proportionate to the zonal population. From these 30 districts, a sample of 120 PHCUs were selected, with the number of PHCUs within a district selected with probability proportionate to population size. From the 120 PHCUs, at least two functional health posts (clusters) per health centre were randomly selected. (For a summary of the sample size, see Table 1.)

Survey tools and data collection

The baseline and follow-up surveys employed the same research instruments. These were adapted from Demographic and Health Survey, Service Availability and Readiness Assessment and Integrated Management of Childhood Illness survey instruments (World Health Organization, 2015b; The DHS Program, 2012; World Health Organization, 2014). For the health facility component of the survey, the head of the health centre and the HEW were asked about staff status, supportive supervision and services that had been consistently available in the last 3 months. Direct observations were made to verify physical infrastructure, equipment, job aids and medicines at the time of the study. The HEW survey component included background, training received, knowledge of CBNC, services provided and supervision received in the 3 months preceding the survey.

For the midline survey, additional research instruments followed Ethiopian Health Ministry guidelines for CBNC and for the Integrated Community Case Management (iCCM) of childhood illnesses. We developed new methods for the assessment of young infant health: (1) clinical vignettes for HEWs, specifically addressing CBNC programme-related illnesses including PSBIs and (2) young infant quality-of-care observation and re-examination. Three data-collection stations were established. Enrolment station (health post-entrance): the station where sick young infants were enrolled if they fulfilled the enrolment criteria. Consultations-observation station (main room): the station where HEWs managed the sick young infants. An observer silently observed the consultation and used the observation form to record the HEW’s assessment, classification, treatment and caregiver-counselling. At the end of the consultation, if any information was unclear the observer asked the HEW. Re-examination station (storage or Antenatal care (ANC) room): the station where the re-examiner, a health officer (a cadre providing clinical care at health centre level), assessed the young infant using the re-examination form, following the Ethiopian CBNC/iCCM chart booklet. The re-examination was used in the analysis to derive gold-standard classifications and treatment with which to compare the HEW’s assessment. All data were collected electronically on personal digital assistants (PDAs), except for the examination of the infant which was recorded on paper.

The development of the study protocol and instruments was the result of extensive formative research and iterative field testing to ensure that the tools were comprehensive and accurate and that operational details of data collection were attainable (Avan and Berhanu, 2013). The finalized instruments were translated into local languages: Amharic, Oromifa and Tigrinia. Back-translation into English was carried out to ensure accuracy of the translation.

For the baseline and follow-up surveys, data collectors were selected based on their level of education, previous experiences in their role and ability to speak the required language. Training over 1 week was provided covering both theory and field practice. Twenty data-collection teams each consisted of one supervisor and four data collectors. Detailed field manuals were supplied to each data collector. Baseline data collection was paper based, while the follow-up survey was PDA based. For the midline survey, 12 data-collection teams each comprised four members: a supervisor, a community mobilizer (who encouraged mothers with sick young infants to visit the health post), an observer (who observed the infant’s consultation with an HEW) and a re-examiner (who assessed and classified the infant after they had been seen by an HEW). Observers and re-examiners were government-employed health officers trained in CBNC. The field team followed standard training procedures over 8 days. This included observers and re-examiners having additional training on CBNC guidelines in relation to the assessment, classification and management of sick young infants. The health officer offered counselling to the mother of every assessed child, as well as clinical management. Every infant identified by the HEW as needing emergency treatment was seen immediately by the health officer, who provided necessary advice and management without any time delay caused by travel to the health centre. Parents of any infant needing emergency treatment were also encouraged to take their child to the health centre. At the close of each day, the health officer also provided HEWs with feedback on their performance. Standard quality assurance procedures were employed for all CBNC surveys in terms of field supervision and data management.

Data analysis

The analysis presented here is based on two dimensions of the CBNC programme:

1. Health system response, which includes system support: changes in infrastructure, supplies and availability of job aids from baseline survey (2013) to follow-up survey (2017), and system linkages: changes in supportive supervision for HEWs from baseline survey (2013) to follow-up survey (2017).

2. Quality of care delivered by HEWs, which includes the theoretical capacity of HEWs to deliver CBNC services as reflected by their functional health literacy and their actual performance in correctly managing the illness of young children. In our study context, functional health literacy is ‘the degree to which individuals have the capacity to obtain, process and understand basic health information and services needed to make appropriate health decisions’. This can be estimated using an index, categorized into inadequate, marginal and adequate (Nafradi et al., 2019). We studied two dimensions of functional health literacy: knowledge and competence. Data on the knowledge dimension were collected through a questionnaire, and data on the competence dimension, including problem solving ability, were collected through clinical vignettes. For each of the two dimensions, functional health literacy indices (range 0–100%) were created by the summation and percentage conversion of four theme-specific items (PSBIs case management, PSBIs case follow-up, diarrheal case management and general wellbeing advice). Furthermore, each index was divided into three categories of health literacy: inadequate (0–59%), marginal (60–74%) and adequate (75–100%) (Nafradi et al., 2019). To assess the performance of HEWs, we used the observations of case management of sick young infants at least 1 year after CBNC implementation (midline survey, 2015). For HEW diagnoses of key young-infant illness (PSBIs, local infection, dehydration and jaundice), sensitivity and specificity with their 95% confidence intervals (CIs) were calculated (for further details on the components, scoring and interpretation of the functional health literacy of health workers, see Supplementary Appendix 1). Analysis involved descriptive statistics: means and standard deviations for continuous variables and percentages for categorical variables. Chi-squared and two-sample t-test were used to assess change, with 5% as a cut-off to define statistical significance, between baseline and follow-up. The analysis accounted for clustering at the health post level, as multiple infants were assessed by the same HEW. Analysis was carried out using SPSS 27 (IBM Corp, USA).

This study was approved by the authors’ institutes. All respondents provided informed, voluntary consent and referral arrangements were in place in the event of an infant being diagnosed as very sick.

Results

Overall, response levels and completeness were almost 100% across all three surveys (Table 1), except that in the 2017 survey, when we planned to interview a second HEW at each selected health post, some were unavailable due to outreach activities, and only 81% of the targeted HEWs were interviewed. We assessed and provided feedback to all infants who presented at the health post on the day of the data collection. We therefore achieved a sample size of 893 infants as compared with our sample size estimate of 720. The results section follows the order of the theoretical framework (Figure 1). The section ‘health system response to CBNC programme needs’ presents the changes in system support and functional linkages between health centres and health posts from 2013 to 2017 (baseline and follow-up surveys). The section ‘quality of care delivered by HEWs’ presents results from all three surveys: (1) the functional health literacy of HEWs—(a) changes in CBNC knowledge from 2013 to 2017 and (b) the status of clinical problem-solving ability using the clinical vignettes assessed in 2015. And (2) the performance of HEWs assessed through direct observation of neonate case management during 2015 (midline survey).

Section: health system response to CBNC programme needs

Health system support

Changes in system support for CBNC services from baseline (2013) to follow-up surveys (2017) are presented in Table 2. We found evidence of improved availability of medical supplies and job aids.

Table 2.

Changes in infrastructure, supplies and job aids availability from baseline (2013) to follow-up survey (2017)

	Health centre			Health post
	Baseline (N: 206)	Follow-up (n: 206)		Baseline (n: 205)	Follow-up (n: 201)
	% (n)	% (n)	P-value	% (n)	% (n)	P-value
Drugs and vaccines
TTC eye ointment	85 (176)	82 (168)	0.155	38 (77)	39 (78)	0.767
Chlorhexidine	23 (48)	62 (127)	<0.001	4 (8)	15 (30)	<0.001
Injection gentamicin	64 (131)	94 (193)	<0.001	4 (8)	35 (70)	<0.001
Oral amoxicillin	99 (203)	100 (206)	0.248	26 (53)	79 (158)	<0.001
BCG	86 (178)	94 (193)	0.036	24 (50)	24 (48)	0.926
Polio	86 (177)	96 (197)	0.002	26 (54)	23 (47)	0.509
Equipment
Ambu bag (full size 0 and 1)/face mask	85 (175)	96 (197)	<0.001	15 (31)	17 (34)	0.609
Clinical thermometer—digital	76 (157)	80 (165)	0.81	68 (139)	79 (159)	0.008
Infant scale	95 (195)	97 (200)	0.303	64 (132)	69 (138)	0.328
Syringe with needles	97 (200)	100 (206)	0.060	–	–
Job aids
a) For diagnosis
Chart booklet (ICCM)	83 (171)	91 (188)	0.022	66 (134)	83 (166)	<0.001
IMNCI registration book for 0-under 2 months	80 (165)	97 (200)	<0.001	89 (182)	97 (195)	0.002
b) For reporting
Family health cards	81 (167)	89 (183)	0.036	69 (141)	80 (161)	0.011
Vaccination cards	87 (177)	85 (174)	0.51	79 (162)	85 (171)	0.136
Stock card/bin card	82 (166)	93 (192)	0.01	43 (87)	76 (152)	<0.001
HMIS forms (monthly and quarterly reporting)	90 (184)	97 (199)	0.009	67 (137)	83 (167)	<0.001
Request and re-supply form	75 (153)	87 (179)	0.007	33 (68)	56 (113)	<0.001
Supervision checklist	78 (158)	91 (187)	<0.001	29 (59)	41 (83)	0.009
Family folder	–	–	–	78 (160)	89 (179)	0.004
General infrastructure
Water	78 (158)	72 (149)	0.269	70 (143)	57 (114)	0.005
Electricity	55 (111)	67 (137)	0.010	21 (41)	19 (39)	0.783
Functional sterilizer	73 (150)	72 (148)	0.844	20 (42)	12 (25)	0.031
Functional fridge	93 (192)	86 (178)	0.023	20 (42)	15 (31)	0.192
Patient toilet	93 (187)	96 (201)	0.243	81 (166)	81 (163)	0.896

Medicine and vaccines for neonatal health services.The availability at health centres of drugs essential for CBNC services was better in 2017 than in 2013. A high proportion of health centres had stocks of injectable gentamicin 80 mg/2 ml (94%) and oral amoxicillin (100%) in the 2017 survey. Although a statistically significant improvement in drugs was observed at health posts since 2013, availability in 2017 was still unsatisfactory. Injectable gentamicin was available in around three out of ten health posts and oral amoxicillin in eight out of ten health posts. Overall, over 90% of health centres in the 2017 survey had Bacillus Calmette–Guérin (BCG) and polio vaccines required for within six weeks of birth; however, less than a quarter of health posts had a stock of these vaccines.

Equipment for neonatal health services.Availability of clinical equipment needed for newborn and young infant care was consistently better over time at the health centre than at the health post level. Achievement was less coherent at health posts: two-thirds had infant-weighing scales in 2017 and only 80% had a thermometer, an essential tool for detecting signs of PSBIs.

Job aids for neonatal health services.Availability of paper-based job aids for clinical guidance and record-keeping increased over the study period in both health centres and health posts. However, there was still room for improvement, e.g. chart booklets (a reference guide for assessing, classifying and treating young infants) were still not available in 2017 in around one-fifth of health posts.

PHCU infrastructure.The infrastructure of health facilities was little improved; in some cases, it had deteriorated—e.g. a relative 22% reduction in health posts with a regular water supply and a relative reduction of 8% in functional refrigerators at health centres.

Health system linkages

Supportive supervision. Table 3 shows changes in the frequency and content of supportive supervision of CBNC service delivery at the health post level in the last 3 months, from the perspective of both health centre and health post staff. No statistically significant change was observed in terms of the reported frequency of supportive supervision visits over time. 86% of HEWs reported receiving supportive supervision from the health centre level in the 3 months preceding the follow-up survey, while health-centre staff reporting supervision visits to health posts in the same period was higher (94%). Additionally (data not shown in the table), it is important to highlight that 80% of HEWs reported that they had received written feedback during their supervisory visit in the last 3 months. Survey field staff were able to verify 70% of these feedback forms.

Table 3.

Changes in supportive Supervision for Health Posts/ HEWs from baseline (2013) to follow-up survey (2017)

	Health centre			Health post
	Baseline (n: 206)	Follow-up (n: 206)		Baseline (n: 205)	Follow-up (n: 201)
	%(n)	%(n)	P-value	%(n)	%(n)	P-value
Supervision
Supportive supervisory visit of health post in last 3 months	94 (193)	94 (194)	0.900	82 (168)	86 (173)	0.210
Among: Yes If a visit in the past 3 months occurred, did discussion include:
Reporting of early identification of pregnancy	100 (193)	100 (193)	1.00	84 (141)	91 (157)	0.101
Provision of focused ANC	99 (191)	98 (190)	0.449	85 (142)	92 (159)	0.063
Promotion of institutional delivery	100 (193)	98 (189)	0.122	88 (148)	92 (159)	0.387
Safe and clean delivery	93 (179)	93 (180)	0.850	77 (130)	73 (127)	0.247
Immediate newborn care including cord care (chlorohexidine)	47 (90)	59 (114)	0.013	30 (50)	32 (55)	0.739
Recognition of asphyxia, initial stimulation and resuscitation of newborn babies	72 (138)	72 (139)	0.848	35 (58)	39 (67)	0.470
Prevention and management of hypothermia	77 (148)	69 (133)	0.106	40 (67)	41 (71)	0.899
Management of pre-term and/or low birth weight neonates	70 (136)	77 (148)	0.146	35 (58)	43 (75)	0.123
Management of very serious disease in newborns	83 (161)	84 (162)	0.805	41 (69)	57 (98)	0.005

With respect to the quality of supervisory visits, two distinct patterns can be observed in Table 3:

1. antenatal and childbirth components of CBNC were the most frequently discussed topics, with marginal improvement over the years;

2. postnatal components were consistently less-discussed topics, with marginal improvement over the years. Immediate newborn care was the least discussed aspect in supportive supervisory visits (27%) at health post level. Despite a relative increase of 44% from 2013 to 2017, the management of newborn and infant conditions, including diagnosis of very serious disease, was discussed with only half of all HEWs in 2017.

Section: quality of care delivered by HEWs

Theoretical capacity of HEWs for delivery of CBNC services

Two dimensions of functional health literacy were assessed: first, a knowledge dimension, from interviewing HEWs to assess their basic understanding of the core components of CBNC services, and second, a competence dimension, reflecting HEWs’ potential to apply health service and clinical knowledge to make appropriate health decisions in response to clinical vignettes representing various scenarios of health conditions of a young infant.

Figure 3 illustrates the knowledge dimension of functional health literacy, measured by HEWs’ knowledge of delivering CBNC services, assessed at baseline and follow-up surveys. CBNC service components are organized across four key thematic areas along the continuum of care: focused ANC, danger signs in pregnancy, post-partum care and Postnatal care (PNC). For each of these four themes, less than 10% of HEWs had adequate health literacy. The knowledge dimension of functional health literacy deteriorated from 2013 to 2017, with relative reductions of over 70%. The worst category was PNC (i.e. relative reduction over time of 77% of adequate literacy for maternal PNC, and 80% for newborn PNC). Disturbingly, this was a primary component of the CBNC service.

Figure 3.

Functional Health Literacy of HEW with 95% CIs: change in unprompted CBNC knowledge assessed at baseline-2013 and follow-up-2017 surveys)

Figure 4 shows the distribution of the competence dimension of functional health literacy, measured by HEWs’ competence in the management of four specific health conditions of the young infant, based on clinical vignette assessment at the midline survey 2015: diarrhoea case management, general wellbeing and breastfeeding advice, and management and follow-up of PSBIs. For the competence dimension, adequate functional health literacy levels ranged from 11 to 28%. This means that less than one in four HEWs had problem-solving capacity to adequately manage at least one of these conditions with a reduced likelihood of making a major error of clinical judgment. 55% had an inadequate level of the competence dimension of functional health literacy for PSBIs, meaning more than half of HEWs were very likely to make a major clinical error while dealing with PSBIs in a young infant. Overall, the vignettes suggested that HEWs lacked the skills to recognize symptoms and correctly diagnose a sick young infant. However, once alerted to the symptoms, they could classify and deliver appropriate treatment accurately.

Figure 4.

Functional Health Literacy of HEW with 95% CIs: clinical vignette assessment based on competence in CBNC-related clinical scenarios at the midline survey-2015

Performance of HEWs

Case classification.In this section, we present observation of HEWs’ consultations with, and independent re-examination of, sick young infants. Based on clinical diagnoses made by health officers upon re-examination, local bacterial infection was the most common illness (24%), followed by PSBI s in 15% of young infants.

Young-infant-illness clinical classifications, comparing diagnoses made by health officers with those made by HEWs, are shown in Table 4. Overall, compared with diagnoses made by health officers, HEWs case management showed good specificity: over 90% infants without disease were correctly identified by HEWs, except for local bacterial infection where 19% of young infants were incorrectly diagnosed as having this condition. Despite having good specificity, HEWs’ skills in correctly identifying sick infants as having a disease (sensitivity) were poor. HEWs missed 70% of PSBIs diagnosed by health officers. HEWs had relatively better sensitivity for local bacterial infections (55%).

Table 4.

Clinical case classification using the iCCM chart booklet assessed at the midline survey-2015

	Young infant cases diagnosed by health officers		Comparability of neonatal cases diagnosed by health officers (standard) vs HEWS
Condition	n: 893 % (n)	95% CI	Specificity	95% CI	Sensitivity	95% CI
Local bacterial infection	24 (214)	(21, 27)	81%	(76, 85)	55%	(47, 62)
Possible serious bacterial infection	15 (134)	(12, 18)	96%	(95, 97)	30%	(22.9, 39)
Any dehydration	4 (36)	(2, 5)	95%	(93, 97)	45%	(38, 53)
Jaundice	0	–	95%	(93, 97)	–	–

Discussion

The findings of the 4-year evaluation of CBNC implementation show important gains in tangible aspects of service provision for young infants, including the widespread availability of relevant antibiotics and job aids. However, human resource capacity remained very low, in terms of the quality of management of health-related conditions, particularly PSBIs. Certain aspects of health workers’ functional health literacy deteriorated over the period since inception of the CBNC programme. Supportive supervision forms a critical link between tangible elements and quality of delivery of health services. The frequency of 3-monthly supportive supervision was consistently high during the evaluation period. However, while there was strong emphasis on supervision of antenatal and facility-based delivery, there was relative neglect of the postnatal period—especially supervision elements to support the management of sick young infants.

We employed WHO’s functionality model as a health systems-thinking approach for assessing adaptive changes that were necessary for CBNC. Task-shifting is a key strategy of CBNC, with HEWs directly delivering service based on newly acquired skills of identification and management of newborn illness. According to the model, effective service delivery by HEWs is dependent on regular supportive supervision; access to adequate resources, supplies and medicines; capacity-building and performance-benchmarking. Contrary to the initial service-development plan, scaling-up of CBNC implementation in Ethiopia was expedited immediately after its inception (Berhanu and Avan, 2019). Efforts were concentrated primarily on the availability of medicines and job aids and on the initial training of health workers in CBNC protocols. Two major assumptions were made. First, HEWs would be capable of delivering new health services to young infants after their initial training, which was introduced without adequately assessing gaps in abilities. There was no plan for structured follow-up training, and no criteria were established to affirm competence to manage PSBIs in young infants. The second assumption was that maternal and child health supervisors at the district and PHCU level would support health workers across all aspects of antenatal, delivery and postnatal care, including PSBIs in young infants. In reality, robust national and political drives to enhance the coverage of facility-based delivery took priority over other aspects of maternal and newborn health (Marchant et al., 2019; Hill et al., 2019). Furthermore, PSBIs are a relatively uncommon event, with seasonal variations. Within the CBNC programme, there was no mechanism to update the discourse between supervisors and health workers, even during periods with few PSBI cases.

Global health targets like the Millennium Development Goals and Sustainable Development Goals have pushed the agendas of Health for All and Universal Health Coverage (Amouzou et al., 2020; Oleribe et al., 2015). Scaling up expanded packages of health services has been central to achieving health for all and universal health coverage goals (Mangham and Hanson, 2010). However, the effectiveness of such scaling-up is multidimensional, and coverage gains within the components of the health services package over time are just one dimension. Equally important are other considerations such as effective coverage, including quality of care, and strengthening health systems to enable the suitability of services (World Bank, 2007; World Health Organization, 2007). To the best of our knowledge, ours is the first study to use functional health literacy to assess the knowledge and competence of health workers. We found the approach useful for assessing and comparing aspects of the health workforce’s potential to deliver specific health services. We would recommend its further use in health system and programme evaluations.

The decision to expedite the scale-up of CBNC in Ethiopia stemmed from a cluster-randomized trial of task-shifting newborn-infection management from health officer to the HEW level in rural Ethiopia. The intervention achieved only 50% coverage, and the marginal reduction in neonatal mortality was consistent with that low level of coverage (Degefie Hailegebriel et al., 2017). Instead of taking a long-term system perspective by making coordinated changes in all components of the health system, CBNC implementation followed the public health services model commonly practised across LMICs whereby intervention scale-up is planned as a linear process with a focus on the service delivery level (Subramanian et al., 2011). This led to short-term implementation arrangements—including an intense inaugural phase of staff capacity-building and monitoring—in an effort to be able to demonstrate success in a defined period. However, less consideration was given to how these arrangements might be integrated into all health system blocks without compromising other routine services. It is important to highlight that improvements in infrastructure, supplies and job aids might be due to the longer term existence of most components of CBNC within the heath system. The exception is for the management of PSBIs, which still needs more systemic embedding and regulation.

A cost-effectiveness analysis of the CBNC prototype reported that transition of the programme from research to integrated regular service would need to be strengthened with more intense routine HEW supervision. Otherwise, inferior quality of support and supervision would be likely to result in lower impact (Mathewos et al., 2017).

Our evaluation highlights that resources alone are not sufficient for the effective scaling-up and integration of neonatal health services: those resources must be strongly coupled with supportive supervision. Systems-thinking perspectives and approaches are increasingly used to assess and guide programmatic and service scaling-up (Adam and de Savigny, 2012), leading to the examination of the linkages and interactions within the overall health system and its components (Leischow et al., 2008). Neonatal health managers, supervisors and service providers, and several other stakeholders, work in coordination with each other. Their linkages are potentially influenced by sub-system dynamics, including the general functioning of the health system, continuous professional development, benchmarking of care and institutional social network norms. In ideal circumstances, scaling up a neonatal health service should actively acknowledge and engage health system relationships in an all-encompassing manner (Paina and Peters, 2012).

Supportive supervision is a frequently neglected facet of the health system. In contrast with infrastructure, supplies and health service providers, supervision operates in the background, making it liable to be neglected and to receive inadequate attention in programme planning and resource allocation. Sustaining the skills, knowledge and practices acquired through task-shifting among CHWs is challenging in LMICs, as competencies remain only with rigorous training and refreshers and with functional supervision systems (Lehmann and Sanders, 2007; Rowe et al., 2005). Scaling up neonatal health services in health systems of poor quality will likely produce small improvements in health outcomes, unless scaling up is robustly supported across all components of the health system. Supervision coupled with training, along with strengthened infrastructure and supplies, has measurable advantages for health-worker performance (Rowe et al., 2018).

The study’s limitations include its lack of economic and sustainability assessments. However, the programme implementers and funders deemed these not to be research priorities due to the urgent and essential nature of the programme. Furthermore, as mentioned above in the methodology section, earlier implementation of CBNC services in the comparison arm led to study design modifications. Overall, our study conducted a series of surveys and assessments over time to provide evidence on before-and-after changes to enable course correction and programme improvement for the programme implementers. Therefore, owing to the lack of a conventional comparison arm, outcomes cannot be directly attributed to the CBNC programme. The study’s findings should therefore be interpreted with caution.

Judicious scientific evidence exists about examining context and reasons for adequate health-worker performance. However, limited studies have explored programme performance by comparing available resources and supervision (Rowe et al., 2010). Supportive supervision has the potential to improve CHW performance through the pathways of precise guidance, monitoring and accountability, provided other components of health systems are also in place (Aftab et al., 2018).

Future research

Despite the global push to improve neonatal survival, the evidence around implementation, scaling-up and effective integration into the health system is limited in scope, quality and the extent to which information is available. There is insufficient evidence of the role of supervision in primary health care services, especially that which would identify and address maternal and newborn service-delivery gaps in LMICs. Currently in LMICs, there is a pronounced public health interest in implementing WHO guidelines on the treatment of young infants with PSBIs by frontline workers within the public health system (Wammanda et al., 2020; Ahmed et al., 2019; Khanal et al., 2011). This trend will greatly benefit from further implementation research with an explicit systems perspective—especially research that establishes the roles and pathways of supportive supervision of health workers, and the direct effects of such supervision on health outcomes.

Data availability Statement

Study datasets for the Community Base Newborn Care Evaluation (CBNC) are made available on LSHTM Data Compass (https://doi.org/10.17037/DATA.00001979), a research data repository operated by the London School of Hygiene and Tropical Medicine.

To protect participant confidentiality, the datasets are made available through a controlled access approach. Although CBNC datasets do not contain direct identifiers (such as names), repository staff indicate there is a risk that participants may be indirectly identifiable in some circumstances, due to the large number of variables collected from each participant and the accuracy of measures.

Researchers wishing to access datasets are asked to apply for access via the repository’s data request form, providing information on the variables they wish to access and details of their analysis plan. The request will be sent to the study team (that performed the research and have greatest understanding of the data) and the LSHTM Research Data Manager (who acts as an independent advisor). If the data analysis can be performed in compliance with the study’s ethical and legal requirements, the study team will produce a derived dataset that contains the requested variables and work with the applicant to help them to understand the data. If there remains a recognisable risk that the derived dataset contains potentially identifiable information, the applicant will be asked to sign a Data Sharing Agreement before being provided with the dataset.

The study questionnaires and other supporting documents are made openly available under a Creative Commons Attribution (CC BY) licence. These can be found alongside the relevant datasets listed in https://doi.org/10.17037/DATA.00001979.

Funding

The study was supported by Bill and Melinda Gates Foundation [grant code: INV-007644]. This paper was published as part of a supplement financially supported by International Development Research Centre.

Ethical approval

This study was approved by the Institutional Review Boards of the London School of Hygiene and Tropical Medicine (reference no: 8759- 4); the Ethiopian Science and Technology Ministry (reference no: 3/10-94/2010) and Oromia, Amhara, SNNP and Tigray regional health bureaux.

Conflict of interest statement

The authors declare that they have no conflict of interest.