Scale Matters: A Cost-Outcome Analysis of an m-Health Intervention in Malawi

Abstract

Background: The primary objectives of this study are to determine cost per user and cost per contact with users of a mobile health (m-health) intervention. The secondary objectives are to map costs to changes in maternal, newborn, and child health (MNCH) and to estimate costs of alternate implementation and usage scenarios. Materials and Methods: A base cost model, constructed from recurrent costs and selected capital costs, was used to estimate average cost per user and per contact of an m-health intervention. This model was mapped to statistically significant changes in MNCH intermediate outcomes to determine the cost of improvements in MNCH indicators. Sensitivity analyses were conducted to estimate costs in alternate scenarios. Results: The m-health intervention cost $29.33 per user and $4.33 per successful contact. The average cost for each user experiencing a change in an MNCH indicator ranged from $67 to $355. The sensitivity analyses showed that cost per user could be reduced by 48% if the service were to operate at full capacity. Conclusions: We believe that the intervention, operating at scale, has potential to be a cost-effective method for improving maternal and child health indicators.

Introduction

Despite recent gains, improving maternal, newborn, and child health (MNCH) remains an essential health priority for Malawi. Malawi has some of the highest maternal and neonatal mortality rates in the world, with an estimated 360 maternal deaths per 100,000 live births¹ and 22 neonatal deaths per 1,000 live births.² Many maternal and neonatal deaths can be averted using evidence-based interventions such as utilization of antenatal care, early initiation of breastfeeding, mosquito net use, and timely access to healthcare. However, in Malawi, intractable barriers such as lack of access to expedient health information, poor infrastructure, and long travel distances to health facilities prevent women and children from enacting positive health behaviors and seeking life-saving health services.

Persistent challenges in achieving MNCH outcomes and the rapid proliferation of mobile phone technologies in low- and middle-income countries have prompted the advent of mobile health (m-health) innovations. In recent years, m-health tools have been used to successfully facilitate obstetric care, ameliorate human resource challenges, and provide information for health promotion.^3,4 For example, the U.S.-based program text4baby sent text messages to pregnant mothers encouraging healthy attitudes, beliefs, and behavior during pregnancy. Although the program did not lead to significant behavior change, it did increase feelings of being prepared for motherhood among participants.⁵ Similar programs in low- and middle-income countries, such as Wired Mothers in Zanzibar and the Better Border and Healthcare Program in Thailand, have led to positive outcomes.^6,7

Despite promising evidence on the benefits of such technologies,^3,4,8 considerable gaps exist in the economic evaluations of m-health interventions.^8–13 In a global survey on m-health solutions conducted by the World Health Organization, lack of cost-effectiveness data was identified as one of the top four barriers in the implementation of m-health solutions.¹³

We were able to identify one cost analysis of a health intervention in a low-income country with an m-health component: an intervention in Rakai, Uganda that aims to improve AIDS care outcomes through the use of peer health workers. The m-health portion of the intervention used peer health workers' mobile phones to text clinical patient information to central staff, as well as to improve communication. The m-health portion was estimated to cost $1,046 annually or $2.35 per patient. The authors also estimated that the overall intervention cost $189 per virologic failure averted and $1,025 per patient lost to follow-up averted; however, it was not possible to determine the cost per outcome associated with the m-health portion of the intervention specifically.¹¹

In order to strengthen the knowledge base for the assessment of m-health technology, we have undertaken a cost-outcome analysis of an m-health intervention launched in Malawi in 2011. Chipatala cha pa Foni (CCPF), or “health center by phone,” consists of a toll-free hotline and a mobile phone–based tips and reminders (T&R) service seeking to improve MNCH in Balaka District, Malawi. The service targets caregivers of children under 5 years of age, pregnant women, and women of reproductive age. The hotline, staffed by hotline workers trained at similar levels as community health workers, provides callers with timely information and advice on issues across the reproductive and MNCH spectrum and refers callers displaying symptoms or danger signs to the nearest community health worker or health center for further care. The T&R messaging service provides weekly text or voice messages to callers who elect to enroll in the service. Messages are personalized for clients based on their week of pregnancy or the child's age and provide accurate information on key health practices.

Users can access the hotline for free from any phone with an Airtel SIM card. Airtel is the largest cellular communications provider in Malawi, and a SIM card costs about $0.30. The T&R messaging service can be accessed for free in one of three ways:

• 1. Users can receive text messages sent automatically to their phones.

• 2. Users can receive prerecorded voice messages sent automatically to their phones.

• 3. Users can retrieve prerecorded voice messages through an interactive voice response (IVR) system. To access CCPF tips and reminders, users call the toll-free IVR system, follow the menu prompts, and then enter the appropriate access code to hear their message. Pregnant women use their estimated due date as their code, and caregivers of children use their child's birth date.

There are three main software applications that support CCPF's services. Hotline workers are guided through health protocols and record information about each call using a customized version of Malawi's electronic health records software created by the Baobab Health Trust (baobabhealth.org/). A customizable software called IntellIVR supports the IVR system. T&R messages for subscribers are managed through a software application created for CCPF by VillageReach (Seattle, WA).

During the pilot, CCPF was marketed in four health center catchment areas with a population of approximately 150,000 people, including 32,000 women of childbearing age, 24,000 children under 5 years of age, and 7,000 expected pregnancies per year. In order to encourage utilization of the CCPF service, one or two volunteers were chosen in each village to promote the hotline service through one-on-one and small group outreach, distributing flyers, and talking about the CCPF project at community events. Each volunteer was given a low-cost phone in order to provide access to the service to those without personal phones. More detailed results from an independent evaluation of the effect of CCPF on the utilization of home- and facility-based MNCH practices have been reported elsewhere.¹⁴

The primary objective of this study is to determine cost per user and cost per contact with users of the CCPF service. The secondary objectives are to map costs to statistically significant changes in MNCH and to estimate costs of alternate implementation and usage scenarios to model future costs per users.

Materials and Methods

Analysis Methodology

The pilot phase of CCPF was implemented over a 2½-year period from January 2011 to June 2013, with CCPF services launching in July 2011. The authors undertook a cost-outcome analysis from the programmatic perspective by calculating the programmatic cost of implementing CCPF from January 1, 2011 to May 31, 2013, as well as the average cost per user and average cost per contact. Service users could access CCPF through any of the following modes of contact: calling the hotline, receiving text or voice messages on their mobile phones, or accessing voice messages by calling an automated IVR system. Only messages successfully sent to or retrieved by users were considered contacts, but the cost of all attempts was factored into the programmatic cost and sensitivity analysis. For example, a text message sent to a phone that was turned off would not be considered a successful contact, but the associated charge for attempting to send the message was included in the programmatic cost. The programmatic cost was linked to changes in intermediate health outcomes reported in the independent quantitative evaluation.¹⁴

Intervention Costing Methodology

A cost analysis was completed using data on program expenditures to estimate total cost of the CCPF pilot. Cost data were taken from program financial records, service level agreements, and the program budget. Costs were classified as recurrent or capital (defined as inputs lasting more than 1 year). Capital costs associated with hardware, as well as equipment, were annuitized over the lifetime of the asset using a social discount rate of 3%.¹⁵ All recurrent costs were categorized as follows: administrative, management and oversight, travel and transport, mobilization (demand generation), monitoring, technology-related fees and support, hotline, and T&R costs. Travel and transport costs included fuel, costs associated with vehicle maintenance, and per diem allowances. Administration costs predominantly included operational expenses such as utilities, rent, and supplies. Expenditures related to the hotline service comprised airtime, hotline worker salaries, and per diem expenses. T&R service charges primarily included charges for text messages and voice messages sent to subscribers and the cost associated with the IVR system. Both mobilization and monitoring included labor costs, costs associated with training and meeting with volunteers, and cost associated with community events to publicize the service. Management and oversight expenses consisted of staff salaries and associated fringe benefits.

Only capital costs that would be needed to scale-up or replicate the service were included, such as training of hotline workers and volunteers. Most recurrent costs were included, with the exception of international travel and staff time, because these were specifically for the pilot project and are not required for scale-up. Staff was categorized by the location of their work, and salaries and benefits of expatriate staff living in Malawi during the pilot were included in the base model. Because CCPF is currently transitioning program operations to domestic staff, the scale-up of the program should not require significant management and oversight by international staff. Research costs were also excluded from the costing analysis.

All costs in this evaluation have been expressed in U.S. dollars ($). To ensure comparability across different years, costs were adjusted to the same base year (2013) using the U.S. Consumer Price Index. The cost analysis was conducted using Microsoft (Redmond, WA) Excel^® version 14.2.3.

Quantitative Evaluation

The quantitative effects of CCPF on selected MNCH outcomes were determined through a counterfactual analysis using two different sets of comparisons. Details of the statistical analysis have been reported elsewhere,¹⁴ but this article provides a brief summary of the quantitative methodology. The impact of CCPF on self-reported users was estimated using a treatment-on-the-treated approach. The effect of CCPF on knowledge and behavior of MNCH practices was measured through seven outcomes assessed through baseline and end-line surveys. Each outcome was composed of multiple indicators (2–16, depending on the outcome). Outcomes were compared among CCPF users (from the treatment group) and nonusers (from the control group who were predicted as likely to use CCPF had the service been offered in their area) using an instrumented difference-in-difference regression analysis.¹⁶ The first stage of regression was used to estimate the degree to which treatment assignment (living in an area where CCPF is available) predicted treatment uptake (use of CCPF). In the second stage of regression, the predicted likelihood of CCPF use was regressed on each of the indicators. Results of the regression analysis were estimated in percentage point changes. The quantitative analysis was conducted using Stata version 12.0 software (StataCorp, College Station, TX).

Cost-Outcome Mapping Methodology

Total programmatic costs from the base cost model were mapped to statistically significant changes in MNCH intermediate indicators. This provides the average cost per user who experienced an improvement in the knowledge and/or behavior of MNCH practices. In the quantitative evaluation, separate indicators were used to measure maternal and child health. Hence the total program costs were split evenly between maternal health and child health. This was undertaken in order to minimize the overestimation of cost per additional user.

Sensitivity Analysis

Sensitivity analyses were conducted to estimate costs in alternate implementation scenarios. In the first model (Model 1) we explored the effect of increased use of CCPF on total cost and cost per contact. Specifically, Model 1A examined a scenario wherein hotline and T&R service operated at the highest rate observed during the pilot over a 2-year period. Model 1B examined costs for the scenario where the hotline operated at maximum capacity. The T&R has no maximum capacity limitations so the use of the T&R service was scaled in proportion to the percentage of hotline users who use the T&R service. In addition, we explored a second model (Model 2) by combining the increased service use in Model 1 with the removal of costs associated with unsuccessful message T&R attempts. The cost associated with voice messages was higher than those associated with the text messages. Consequently, in the third model (Model 3), the voice message component (through the IVR) was removed, and its effect was assessed on the total cost and cost per contact.

Results

There were 9,798 unique users during the CCPF pilot. Of these, 2,699 users were pregnant women, 3,747 were caregivers of children under 5 years of age, and the remaining 3,352 users included nonpregnant women, caregivers of older children, men, or callers with missing data on pregnancy status (Table 1). Because all attempts at reaching users were not successful, Table 1 presents the number of contact attempts, the number of successful attempts, and the proportion of the attempts that were successful.

Table 1.

Frequency of Contacts by Service Users of Chipatala cha pa Foni

TYPE OF CONTACT	TOTAL NUMBER OF SUCCESSFUL CONTACTS	TOTAL NUMBER OF ATTEMPTS	% OF CONTACT ATTEMPTS THAT WERE SUCCESSFUL
Number of calls	12,210	—	—
Number of voice messages retrieved from IVR	27,157	48,481	56%
Number of text messages (sent to personal phones)	23,880	39,118	61%
Number of voice messages (sent to personal phones)	3,193	5,685	56%
Total	66,440	103,002	—

IVR, interactive voice response.

Costs

Between January 2011 and May 2013, the total programmatic cost included in the base model was $287,357. This is equivalent to $29.33 per user and $4.33 per successful contact. Table 2 summarizes the cost profile included in the base cost model.

Table 2.

Chipatala cha pa Foni Cost Profile (January 2011–May 2013)

COST CATEGORY	INTERVENTION COST IN 2013 DOLLARS (PERCENTAGE OF TOTAL COST)
Recurrent costs
Domestic travel and transporta	27,505 (10)
Administrationb	19,321 (7)
Technology fees and supportc	30,085 (10)
Hotlined servicee	20,549 (7)
Tips and reminders servicef	18,875 (7)
Mobilizationg	37,104 (13)
Monitoringg	41,161 (14)
Domestic management and oversight	80,905 (28)
Capital costs
Annualized one-time costsh	11,852 (4)
Total	287,357

^aSubcategories include fuel, maintenance, and others.

^bBalaka office-related like rent and utilities, phone and Internet, and others.

^cLocal and international.

^dAirtime, labor, and others.

^eDue to systems issues on the side of the telecom provider, invoices from the provider often underestimated the total airtime used in a given month, and hence these costs may not reflect the true airtime related expenditures in the future.

^fShort message service (texting), voice/interactive voice response, and general.

^gLabor and general.

^hAnnualized cost included for the entire 2.5-year study period.

It should be noted that hotline use varied considerably during the 22-month intervention period, with some of the highest call volumes observed in the months following the hotline launch in July 2011. Hence, hotline-related costs varied over time and were largely driven by service utilization because labor costs were stable (Table 3).

Table 3.

Chipatala cha pa Foni (CCPF) Labor Costs Post-CCPF launch (July 2011–May 2013)

COST CATEGORY	LABOR COST IN 2013 U.S. DOLLARS (PERCENTAGE OF TOTAL BASE MODEL COST)
Management and oversight	80,905 (28%)
Hotline service	10,497 (4%)
Mobilization	21,488 (7%)
Monitoring	36,038 (13%)
Total	148,928 (52%)

Intervention

Of the 30 MNCH indicators assessed in the treatment-on-the-treated analyses, 10 showed a statistically significant improvement among users of CCPF services (Table 4). For these 10 indicators, CCPF had an effect of 15–80 percentage points, ranging from an estimated 562 to 2,159 users depending on the indicator. For example, after accounting for baseline difference, the proportion of CCPF users who knew the recommended number of times a pregnant women should go for an antenatal care visit was 25 percentage points higher than that of similar nonusers. These changes in percentage points were used to estimate the effect of CCPF on the 2,699 pregnant women who called the hotline during the pilot.

Table 4.

Estimated Effect of Chipatala cha pa Foni on Hotline Users During the 2-Year Pilot

OUTCOME, INDICATOR	PERCENTAGE POINT INCREASE OF REPORTED INDICATOR AMONG USERSa	ESTIMATED NUMBER OF ADDITIONAL USERS EXPERIENCING INDICATOR AFTER USING CCPFb	ESTIMATED TOTAL PROGRAMMATIC COST PER ADDITIONAL USER EXPERIENCING INDICATORc
1.1. Knowledge of home and facility-based care
Knowledge of 4+ antenatal care visits	25	675d	$213
Knows to drink more water when pregnant	40	1,080d	$133
Knows pregnant women should lift less	15	405d	$355
Knows pregnant women should not take traditional medicines	50	1,359d	$106
Can list average number of items to take to a health center birth	49	1,323d	$109
Knows a baby born at home should go to a health center immediately	80	2,159d	$67
Behavioral use of home-based practices
Used bednet during pregnancy	25	675d	$212
Breastfed within 1 h of birth	15	562e	$256
Child sleeps under bednet	30	1,124e	$128
Behavioral use of facility-based practices
Started antenatal care in first trimester	30	1,124e	$128

^aEstimates from the quantitative analysis.

^bCalculated from (a × d) or (a × e) depending on the indicator.

^c$43,629/d.

^dEstimated change in indicator applied to the subsample of 2,699 pregnant women who called the Chipatala cha pa Foni (CCPF) hotline.

^eEstimated change in indicator applied to the subsample of 3,747 callers who were caregivers of children under 5 years of age.

CCPF use had a notable effect of 80 percentage points on the knowledge of care-seeking practices following home births. Specifically, an estimated 2,159 additional users experienced this indicator compared with nonusers. On average, improvements in knowledge-related indicators were greater than those in behavioral indicators.

Mapping Costs and Outcomes

Among knowledge-based indicators, the average cost incurred by the program for each additional pregnant user reporting a gain in knowledge, ranged from $67 to $355, whereas the cost for behavioral indicators ranged from $128 to $256 per user experiencing the outcome. The cost per additional user was lower for indicators such as knowledge of care-seeking practices for home births that experienced substantial improvements compared with those that changed slightly such as breastfeeding practices and knowledge that pregnant women should lift less.

Sensitivity Analysis

Sensitivity analysis showed that total program cost and cost per contact varied significantly across various alternate implementation scenarios (Table 5). Because all scenarios assume increased service use compared with the base model, the total program cost increased in all scenarios. However, the cost per contact declined in both of the higher-volume scenarios, with a substantial reduction of 48% at full-capacity operation. Improvements in the text message and IVR success rate significantly lowered the cost per contact only at full-capacity operation (Model 2B), as opposed to a marginal drop of roughly 1% if the hotline had operated at its highest observed volume consistently during the pilot (Model 2A). Likewise, removal of the voice component from the T&R service was found to be beneficial only at full operational capacity for the hotline, with a slight reduction of 4% in cost per contact in lower-usage scenarios (Models 3A and 3B).

Table 5.

Sensitivity Analysis Examining Different Scenarios of Chipatala cha pa Foni Program Operation

				SENSITIVITY INDEX
MODEL	MODEL DESCRIPTION	TOTAL COST (2013 U.S. DOLLARS)	COST/CONTACT (2013 U.S. DOLLARS)	PERCENTAGE CHANGE IN COST/CONTACT	PERCENTAGE CHANGE IN TOTAL COST
Base cost	Total programmatic costs (primarily recurrent costs)	287,357	4.33	Reference	Reference
1Aa	Hotline and T&R services operating at the highest observed capacity during the pilot; mobilization costs scaled proportionate to call volume	323,182	2.86	−34%	+13%
1Bb	Hotline and T&R services operating at estimated full capacity; mobilization costs scaled proportionate to call volume	576,983	2.23	−48%	+100%
Reference: Model 1A
2Ac	Model 1A with a success rate of 100% for SMS and IVR messages	321,080	2.84	−0.7%	−0.7%
3Ad	Model 1A with IVR and voice message components removed from T&R service	321,671	4.17	+46%	−0.5%
Reference: Model 1B
2Be	Model 1B with a success rate of 100% for SMS and IVR messages	442,141	1.71	−23%	−23%
3Bd	Model 1B with IVR and voice message components removed from T&R service	444,295	2.29	+3%	−23%
4f	Total program costs including all one-time and initial start-up costs	604,911	9.10	—	—

All estimations used an exchange rate of 1 U.S. dollar = 334.5 Malawi kwacha (MWK), and mobilization costs were scaled at a rate of U.S. $3.039/call. Airtime and short message service (SMS) (text) message costs scaled at a rate of 30.75 MWK/min and U.S. $0.04/SMS sent.

^aModel based on highest observed call volume (939 calls/month), outgoing SMS (1,896 texts/month), and voice messages (258/month) and highest number of retrieved interactive voice response (IVR) messages (2,047/month) during the CCPF pilot. Costs incurred due to unsuccessful contact attempts (SMS and interactive voice response) were included in this model.

^bModel based on program operation at maximum capacity estimated at 2,160 hotline calls/month. All other contacts (SMS and voice) scaled proportionate to hotline call volume.

^cSimilar to Model 2A. Excludes costs incurred from unsuccessful contact attempts (SMS and interactive voice response).

^dIVR and voice contacts excluded from the model.

^eSimilar to Model 2B. Excludes costs incurred from unsuccessful contact attempts.

^fIncludes both annuitized and nonannuitized one-time expenses. Social discount rate of 3% used.

T&R, tips and reminders.

Discussion

This economic evaluation provides a detailed description of the costs and benefits identified during the pilot phase of the CCPF program in Balaka, Malawi. The end-line quantitative evaluation showed that CCPF users were significantly more likely than comparable nonusers to experience some intermediate MNCH outcomes, with the estimated difference ranging from 15 to 80 percentage points. On average, improvements in knowledge-related indicators were greater than those in behavioral indicators. We estimate the total cost of the program to be $29.33 per user and $4.33 per successful contact. The average cost for each user experiencing a change in an MNCH indicator ranged from $67 to $355. This large variation in cost per user experiencing an indicator is due to the large variation in CCPF's impact on the indicators themselves, ranging from 15 to 80 percentage point improvements.

Sensitivity analysis examined various scenarios of higher service demand and showed that pilot projects such as CCPF are likely to benefit from economies of scale. There are two ways that CCPF could potentially increase utilization: (1) increase the proportion of people living in CCPF's target areas that use CCPF and/or (2) increase CCPF's target area. During the pilot, the uptake of CCPF in treatment communities was roughly 20%, even though more than 75% of the target population had heard of the hotline service. Qualitative data from an independent evaluation¹⁴ suggested low perceptions of need, cultural barriers, and limited access to phones as barriers to accessing CCPF. Since the pilot, CCPF has expanded its target area to a total of four districts. Not surprisingly, this has led to a corresponding increase in call volume. In February 2014, the Reproductive Health Unit of Malawi's Ministry of Health endorsed CCPF and stated its desire to see the program scale nationally. CCPF is expected to double the number of districts where it is offered by 2016, and it is likely that as CCPF continues to expand its user base, it will begin to operate at lower cost per contact. Thus CCPF will likely reach the capacity explored in Model 1B simply by increasing its target area. As the sensitivity analyses showed, this increased utilization will likely reduce the cost per contact.

To our knowledge, this is the first cost-outcome evaluation of an m-health technology targeting maternal and child health outcomes. As in any economic evaluation, the current analysis has certain limitations. Our cost estimates are likely exaggerated because total programmatic costs could not be disaggregated by service type so we had to divide costs 50-50 between maternal and child health services. We were also unable to account for users who may have experienced more than one outcome as a result of using CCPF. Another limitation was the exclusion of volunteer time from total costs. Volunteers primarily contributed to demand generation. However, monetization of their time was not feasible because it was challenging to delineate their share of the output or direct contributions. They were, however, compensated with drinks and snacks at monthly meetings and other small incentives, the costs for which have been included under mobilization activities. Additionally, in the sensitivity analyses, costs incurred from mobilization events have been proportionately scaled—to account for the increased support that would need to be provided to volunteers.

As interest in m-health interventions increases at the local, national, and global levels, it is becoming increasingly important to assess the cost-effectiveness of these types of interventions in relation to similar interventions. In and of itself, cost-outcome analyses such as ours do not monetize the benefits of the program to allow a cost-benefit analyses or easy comparison with the cost-effectiveness of other programs. This analysis takes into consideration costs only from the program provider perspective; comparing costs and cost savings from the perspective of users and the health system would paint a fuller picture of economic implications of CCPF.

Conclusions

The CCPF pilot led to significant changes in knowledge and behavior among those who use it. We believe that CCPF, operating at scale, has potential to be a cost-effective method for improving maternal and child health indicators. M-health projects have the potential to reduce time, distance, and cost of health education and healthcare.⁹ As CCPF scales up, further research is needed to understand the economic implications of CCPF from different perspectives, including potential cost savings to the national health system and individual users and comparative cost-effectiveness in relation to other maternal and child health programs.

Acknowledgments

The CCPF pilot was funded through Concern Worldwide's Innovations for Maternal, Newborn & Child Health Initiative. Implementation was led by VillageReach, in partnership with the Malawi Ministry of Health and the Baobab Health Trust. The authors would like to thank Aimee Desrochers, for her edits and suggestions that were helpful in creating a clear and concise manuscript, and the reviewers, whose comments led to a stronger article.

Disclosure Statement

No competing financial interests exist.