Practical issues in the measurement of child survival in health systems trials: experience developing a digital community-based mortality surveillance programme in rural Nepal

Abstract

Child mortality measurement is essential to the impact evaluation of maternal and child healthcare systems interventions. In the absence of vital statistics systems, however, assessment methodologies for locally relevant interventions are severely challenged. Methods for assessing the under-5 mortality rate for cross-country comparisons, often used in determining progress towards development targets, pose challenges to implementers and researchers trying to assess the population impact of targeted interventions at more local levels. Here, we discuss the programmatic approach we have taken to mortality measurement in the context of delivering healthcare via a public–private partnership in rural Nepal. Both government officials and the delivery organisation, Possible, felt it was important to understand child mortality at a fine-grain spatial and temporal level. We discuss both the short-term and the long-term approach. In the short term, the team chose to use the under-2 mortality rate as a metric for mortality measurement for the following reasons: (1) as overall childhood mortality declines, like it has in rural Nepal, deaths concentrate among children under the age of 2; (2) 2-year cohorts are shorter and thus may show an impact more readily in the short term of intervention trials; and (3) 2-year cohorts are smaller, making prospective census cohorts more feasible in small populations. In the long term, Possible developed a digital continuous surveillance system to capture deaths as they occur, at which point under-5 mortality assessment would be desirable, largely owing to its role as a global standard.

Key questions.

What is already known about this topic?

• Assessing child mortality is essential to understanding healthcare systems performance.

• Many low-income and middle-income countries lack robust vital registration systems and censuses, which hinders the reporting of accurate mortality rates.

• Child mortality estimates among smaller catchment area populations lack the reliability and precision to inform local or subnational public health programming.

• Implementers in resource-limited settings have a limited toolbox for mortality assessment in real time and at local levels.

What are the new findings?

• We describe an approach to a digital system for measuring mortality using community health workers. This approach, as is typical for grassroots implementers in resource-limited settings, necessarily proceeds in steps.

• In the initial phase, the team chose the under-2 mortality rate in lieu of the under-5 mortality rate given the baseline lack of strong vital registration systems and censuses, and the greater feasibility and reliability of this metric in our setting.

• Calculating the under-2 mortality rate requires only 2-year birth histories, is subject to less recall bias and can more accurately estimate the distribution of mortality in populations where child mortality is concentrated in the first 2 years of life.

• In the longer term, the team is developing a community health worker-based continuous surveillance system built into the routine course of clinical care for reproductive age women. Active case search allows care to begin for pregnant women soon after conception through the first 2 years of a child's life, or the ‘Golden 1000 days’.

Recommendations for policy

• In the initial phases of deploying digital community-based methods of mortality assessment, the under-2 mortality rate may be an appropriate metric in some settings.

• This initial phase can then form the basis of a continuous surveillance system to track child mortality among small populations at the subnational level. Such information can inform public health programming and healthcare delivery.

• Developing geospatially and temporally sensitive population health surveillance systems can approximate ‘real-time’ vital registration that allows for mortality assessment of any age group with greater precision and accuracy.

Introduction

Improving child survival is a central function of healthcare systems globally. There remains a substantial implementation gap in translating evidence-based interventions into healthcare systems improvements. Trials that test healthcare systems innovations are central to meeting this gap. The routine measurement of key population health outcomes, such as child mortality, presents a major challenge in implementation research on complex healthcare systems interventions in resource-limited settings.

It is essential that countries and individual programmes develop methods to continuously assess deaths, preferably within formal vital statistics systems. Continuous surveillance systems relying on local community health workers (CHWs) are feasible, as has been shown in one severely resource-constrained environment in Mali.¹ In the modern healthcare era, these systems should be digitised and integrated within routine longitudinal healthcare delivery. Yet it is unclear how to best create digital surveillance systems from current paper registry and intermittent survey approach.

Here, we explore, from the grassroots view of public sector healthcare implementers in rural Nepal, the path to develop more robust, locally responsive and meaningful measures of child mortality. We describe the direction that the Government of Nepal and the non-profit organisation, Possible, has taken via a public–private partnership (PPP) to achieve the objective of feasible and rigorous outcomes assessments of child survival initiatives.

Possible is a non-profit organisation that works with the Ministry of Health of Nepal via a PPP to run an integrated hospital and CHW network. In this approach, the government provides facilities, staff, supplies and co-financing, and Possible assumes management authority and is held accountable for direct healthcare delivery. Through this PPP, the team is committed to developing an implementation research programme that can be used for local and national healthcare systems learning and improvement. Child mortality is a key indicator for their implementation research programme.

The PPP team has confronted the challenge of child mortality estimation in Achham, a rural district of 260 000 people in the Far-Western Development Region of Nepal. Lying 30 hours from the capital city, with some of the lowest development indicators in the country, and emerging from a civil war in which it was one of the key conflict centres, Achham poses great challenges to healthcare systems strengthening.² In Achham, like in much of the developing world, one of the major barriers to measuring accurate and spatiotemporally precise population-based metrics is infrequent census updates (usually every 10 years) and the absence of effective vital registration systems. In Achham, ∼35% of children under-5 have birth certificates, compared with a national average of 60%.^{3 4}

Given these challenges of working in resource-limited rural Nepal, the PPP team realised that it was important to take a stepwise approach to assess child mortality. In the first phase, the team felt that the under-2 mortality rate (U2MR) could be more feasibly used as the basis for similar healthcare systems trials with small population sizes. We describe the PPP team's experiences in assessing this measure in rural Nepal through a census survey and our work towards continuous surveillance methods that can approximate a vital registration system.

Child mortality estimation: practices and pitfalls

The under-5 mortality rate (U5MR) has been broadly applied as a metric to assess progress in reducing child mortality across countries and over time for many years. It was the basis of Millennium Development Goal (MDG) 4 and the recently outlined Sustainable Development Goal (SDG) 3, which aims to reduce the U5MR to <25 deaths per 1000 live births. While useful for cross-country comparisons, the U5MR poses problems in collection and analysis for intervention trials, particularly in small populations and for assessment over shorter (eg, yearly) time periods.

In the absence of vital registration systems, estimates of child mortality are created by surveying mothers about their children, both living and deceased, with either: (1) the direct method using full birth histories, which include birth and death dates for each child, or (2) the indirect method using summary birth histories (all children ever born and ever died) and some proxy measure of exposure (ie, mother's age or time since first birth) to impute the ages of those children.^{5 6} Most sources of national mortality estimates use the full birth history method as it most closely replicates a vital registration system, if one assumes that coverage and recall are not biased. Collection of full birth histories is time-consuming, however, and the results are susceptible to so-called ‘age heaping’ around 12 months, which biases estimates of infant mortality upwards.⁷ Similarly, birth transference to ages older than 5 to avoid completion of the under-5 data collection contributes to concerns over data quality for the full birth history method.⁸

The indirect method relies on life tables to impute ages and estimate overall mortality risk before age 5, and then to derive infant mortality.⁵ This is more vulnerable to inaccurate fertility and mortality assumptions.⁷ Indirect methods may exclude data from the youngest mothers (ages 15–24 years) because children born to these mothers are high-risk first births and are most likely from lower socioeconomic groups.⁷ Since these mothers have the highest proportion of young children, their exclusion means both that recent mortality estimates cannot be reliably calculated and, perhaps equally as important, the estimate no longer reflects the true mortality risk of children born to those women.⁹ When the goal is to measure the impact of specific interventions on reduction of mortality risk, introducing these known biases is problematic.

These challenges are less significant on a national or global scale, as the methods have been developed explicitly for the purpose of comparing general trends across countries and multiyear time frames rather than specific point estimates. When considering the role of mortality estimation in subnational interventions, however, particularly those in need of holistic measures of healthcare systems improvement, child mortality estimation becomes especially problematic. The U5MR derived via national Demographic Health Surveys does not typically offer the precision necessary to find statistically significant differences across time or subnational populations.^{10 11} Estimation of mortality for recent periods (ie, within the 5 years preceding the survey) requires either strong assumptions about underlying fertility and mortality trends for indirect methods or large sample sizes for indirect and direct methods that make it untenable in small populations of 15 000 households or less when the U5MR dips below 100.^{8 10 12} Furthermore, U5MR estimates do not provide information regarding age distribution of deaths relevant for targeted public health programming. However, estimating mortality for different age groups is possible from full birth history data.

Context of child mortality in Nepal

Since the MDG era has passed, the global focus has shifted from addressing under-5 mortality to infant and neonatal mortality.¹³ The infant mortality rate (IMR) and neonatal mortality rate (NMR) are defined as the number of deaths per 1000 live births occurring before year 1 and before 28 days, respectively. In 2011, Nepal reached its U5MR target of 54, but neonatal and infant mortality improvements continue to lag behind overall under-5 mortality success.¹⁴ This pattern largely reflects a stagnant NMR over the same period, suggestive of the many challenges in addressing perinatal and infant mortality due to complications such as birth asphyxia, low birth weight and malnutrition.

The pattern of under-5 mortality decline in Nepal is not unique. Historically, transitions in childhood mortality have imparted significant changes in the distribution of deaths. Since the overall U5MR has decreased, the proportion of deaths occurring in the perinatal and neonatal periods has increased. For example, when the U5MR estimate is <60, ∼75% of deaths are concentrated in the infant period, with an NMR/U5MR ratio of >0.5.⁷ Deaths caused by diarrhoea and lower respiratory tract illness decrease almost linearly with a decreasing U5MR, while deaths from vaccine-preventable illness, perinatal and other causes remain stable at 25–35% of total Under-5 deaths.¹⁵ Thus, the proportion of infant (<1 year) versus child ( 1–5 years) mortality reflects both programmatically and epidemiologically relevant trends regarding underlying causes of death and the current state of the healthcare system. The U5MR itself does not provide a breakdown of mortality by age, however, and could reflect many combinations of infant and child mortality. Focusing solely on infant mortality also has drawbacks as there is commonly under-reporting of infant deaths, age heaping around 12 months and the cohort represents yet a smaller population for which sampling error is more problematic.⁷

Considering that the prioritisation of health programming in Nepal includes newborn health and the ‘Golden 1000 days’ (conception to age 2),¹⁶ and the fact that over 80% of under-5 deaths occur under age 2,³ the PPP team determined that the U2MR was more appropriate than the U5MR. The PPP team decided that the U2MR would provide a more robust and relevant estimate for implementation research trials in rural Nepal. Two-year cohorts emphasise neonatal mortality risk but also include some more exposure time during which infectious diseases are a major cause of death. Like the U5MR, the larger cohort may attenuate problems with age heaping and sampling error associated with the IMR. With a relatively short estimation window, since children age out of the cohort faster, the U2MR may also be valuable for iterative improvement in intervention design. Whether for under-2 or under-5 mortality, in small populations, sample surveys are often underpowered and census methods more appropriate. For example, a traditional survey aiming to assess a 20% change in under-5 mortality requires a sample of nearly 7000 women which, in the context of the hospital's catchment area population, is essentially the entire eligible population.^{12 17}

Given the significant resource constraints in the first iteration, the PPP team chose a truncated birth history method, collecting the birth and death dates of all children born in the prior 2 years during the first phase of data collection. In May 2015, the PPP team completed this first census phase to establish baselines for the U2MR, IMR and NMR, as well as institutional birth rates, antenatal care coverage and postpartum contraceptive prevalence rates. They adapted the survey instrument from the Demographic and Health Surveys and Multiple Indicator Cluster Survey,¹⁸ for an Android-based mobile phone running an Open Data Kit application.¹⁹ The PPP team translated the tools into Nepali and stored the data on SurveyCTO's secure, cloud-based servers.²⁰ CHWs, known locally in Nepal as Female Community Health Volunteers, registered and mapped over 7500 households and collected targeted birth histories from all 1223 women reporting a birth in the previous 2 years. This approach achieved ∼70% coverage of the eligible population, as 15% of households and 20% of identified women were unavailable due to migration or work responsibilities.

In the initial data set of 1237 births, the number of reported under-2 deaths was surprisingly low. The estimated U2MR was 36.9 per 1000 live births, with all reported deaths before or at 1 year of age. That no deaths occurred after age 1 suggested either: (1) birth transference, in which women or interviewers deliberately report a child as older than the age cut-off in order to avoid answering questions, which may have biased the mortality estimate downwards, or (2) mortality is lower than previously thought. There is a dearth of data on the validity of truncated birth history methods, however, and it is unclear whether shortening the window of eligible births would improve recall or birth transference bias. Given the intent to transition to continuous surveillance, the PPP team did not formally validate these methods but may have the opportunity to compare the initial estimates with repeated measures in the coming years.

Moving towards continuous surveillance

Since one key aim of the PPP is to develop an implementation research programme that is responsive and sustainable for ongoing healthcare systems innovation and care delivery, mortality estimation methods should ultimately be designed for continuous surveillance. With continuous surveillance of the entire population, any combination of mortality indicators could theoretically be calculated. In the initial phases of mortality assessment for healthcare systems design, the PPP chose the U2MR as the highest value indicator both for applicability and practicability. Based on the Government of Nepal's projections for the hospital's catchment area population, ∼10% of women of childbearing age deliver a child every year.³ Given the recent data that Nepali women space births by a median of 2–3 years and have an average of three children, an estimated 16–18% of women have children under age 2, 35–40% have children under age 5 and 90% have ever had children.²¹ Continuous surveillance of all women with children under age 5 would thus more than double the women followed, overburdening healthcare workers who have primary patient care responsibilities with most likely a small impact on overall child survival.

Outside of Nepal, real-time monitoring of vital events has been proposed as an alternative method of child mortality assessment. Several demographic surveillance sites globally have demonstrated success through intensive routine measurement.²² Over the past 50 years, the INDEPTH network has developed health and demographic surveillance systems in 19 countries around the world, often leveraging CHWs for birth and death registration, as well as continuous surveillance of maternal and child health status and chronic disease morbidity.²³ Recent studies have demonstrated the method's success at generating data sets robust to yearly mortality estimates at the village level that can be used for spatiotemporal analysis of disparity in trends.^24–26

In non-health and demographic surveillance systems sites, vital events reporting has often been a responsibility of national CHW programmes, but studies validating these data sources are few. The Real-Time Monitoring of Under-Five Mortality Project recently reported mixed success with their five-country trial of strengthened CHW vital events reporting, suggesting that successful programmes were more resource-intensive than what would likely be sustainable in similar settings.^{1 27–30}

To improve sustainability and scalability, the PPP team thus endeavoured to develop continuous vital events surveillance within an electronic medical record. This is implemented through a CHW network and a mobile application for care coordination (CommCare, developed by Dimagi)^{31 32} integrated with an existing hospital-based electronic medical record (OpenMRS-based Bahmni, developed by Thoughtworks). The electronic medical record product integrates and strengthens the core surveillance and care functions of the CHW by embedding active pregnancy case-finding and mortality surveillance into community-based care, which already aims to touch every eligible household.²⁵

Integration of continuous surveillance and routine care through a mobile electronic medical records system

Before beginning continuous surveillance, the PPP team attempted to visit previously missing households and women to finalise the census roll. It quickly became clear, however, that inadequate identification of households in the first phase would make this largely impossible. Working with other local government officials, the PPP team re-enumerated all households (including truncated birth histories for eligible women) with durable markers (small tin plaques with unique household IDs attached to homes). Together with the organisation SimPrints,³³ the team improved on the case management application with biometrics integration for fingerprinting identification of mothers and children. This was successfully pilot-tested using the biometrics tool with community members, with promising preliminary results showing that the use of two-finger identification in the cohort correctly returned a person as ranked first in search results about 97% of the time. This accuracy is also demonstrated before optimisation using the household Global Positioning System coordinates, or searching with other fingers. The mobile surveillance tool was fully deployed in May 2016.

In the revised system, CHWs visit every household on a quarterly basis to screen eligible women (married, aged 15–49) for early pregnancy, based on an algorithm for identifying unexplained amenorrhoea of >6 weeks duration. If women screen positive, they are offered home pregnancy testing and ultimately enrolled in home antenatal visits and group antenatal care offered at local clinics. CHWs document births and deaths during routine visits where they provide pregnant women and children under age 2 with preventive care and referrals for group antenatal and paediatric care or urgent facility-based services. These data are documented with mobile forms integrated into the electronic medical record, as described above. Figure 1 is a geotopographical map that visualises under-2 deaths collected through the continuous surveillance system. Through this routine outreach, the goal is to create a relatively real-time vital registration system that enables mortality assessment with spatiotemporal precision and accuracy. Over time, a verbal autopsy programme will be implemented to assess the cause of death through standardised family interviews,^34–36 further deepening an understanding of healthcare system gaps.

Figure 1.

Under-2 (U2) deaths in the catchment area.

Conclusion

Entering the post-MDG era and moving into the SDGs, the focus of mortality measurements must shift towards the development of robust impact assessments that are relevant to changing mortality patterns and feasible for community-level programme implementation. Given the relatively rare nature of childhood mortality and the absence of effective vital registration systems in much of the developing world, the precision of child mortality estimation is greatly affected by sample size and estimates cannot easily be disaggregated to district or subdistrict levels. As overall childhood mortality declines, deaths concentrate in the neonatal and infant periods, and there is substantial geographic clustering and inequities that only fine-grained data can capture.

The PPP team's initial experience with a census-based truncated birth history highlights the challenges of retrospective data collection when counting every death. Using an integrated, biometrics-enabled mobile application, the continuous surveillance system follows children through age 2, thus feasibly measuring under-2 mortality to inform public health programming and implementation research studies on child survival. Eventually, children under-5, under-10 or even the entire population (eg, for chronic diseases primarily affecting adults) can be continuously followed up and monitored in this system. Key questions will be around affordability, training, privacy and government and community buy-in. We aim to explore the validity, costs and sustainability of such a system in the coming years. In the mean time, we encourage others to explore how measurement of vital events might be integrated into routine community healthcare delivery in other settings and the effect of similar or unique technologies on data quality and efficiency.