Table 5.

Job aid application studies and reports

Study	Design	Intervention	Target outcomes	Key results
Intervention studies – published literature:
Martinez-Fernandez et al, 2015 [83]	Retrospective observational study of population level baseline to endline changes in mortality comparing intervention communities (n = 125) with non-intervention areas in rural Guatemala	FLWs equipped with cell phone-based tools, including job aid app, distance learning modules, and access to supervisors for phone consultation.	Maternal mortality, Infant mortality	Maternal mortality rate decreased in intervention areas (309 to 254) but increased in control areas (338 to 558) (P < 0.05 between groups). Infant mortality rate decreased in both groups (25-13; 27-20) (P = 0.054 between groups).
McNabb et al, 2015 [84]	Pre/post evaluation of pregnant mothers (n = 267) in Nigeria; baseline data collection at first ANC visit, endline 12 months later	Job aid app to guide facility-based health workers through ANC protocols and track client data in real time; 13 BCC audio files embedded	Improve quality of ANC care and client satisfaction	Quality score increased from 13.33 (baseline) to 17.15 out of 25 (P < 0.0001); greatest improvements in BCC message delivery
Battle et al, 2015 [85]	Mixed methods evaluation in Zanzibar, Tanzania. Quantitative: system generated monitoring data; N = 13 231 registered mothers who gave birth in the project period. Qualitative: semi-structured interviews with mothers (n = 27), FLWs (n = 25) and health facility staff (n = 12)	Job aid app supporting safe deliveries through client registration, monitoring, BCC, communication with health facilities and money transfer for transport	Increase uptake of facility-based delivery (FBD) and postnatal care	FBD: 75% vs 35% in most recent Demographic & Health Survey. Postnatal care attendance: 88%. Interview findings: referral and communication functions increased FLW confidence; frequent contact with FLWs and transport service increased FBD but cost of FBD was a barrier
Shiferaw et al, 2016 [86]	Non-randomized prospective controlled evaluation in 10 health facilities (5 intervention, 5 control) of Amhara region, Ethiopia. Exit interviews with mothers attending ANC: n = 933 (baseline); n = 1037 (endline). Chart review of ANC clients: n = 1224	Job aid app for facility-based health workers supporting client registration, ANC and postnatal care visit reminders, decision support, and pregnancy health information. Health workers contacted clients on receiving visit reminders.	Increase in mothers with 4 ANC visits, FBD and postnatal care at each facility	Results (I. vs C.): 4+ ANC visits: 27% vs 23.4% (AOR = 1.31; 95% CI = 1.00-1.72). FBD: 43.1% vs 28.4% (AOR = 1.98; 95% CI = 1.53-2.55). Postnatal care attendance: 41.2% vs 21.1% (AOR = 2.77; 95% CI = 2.12-3.61).
Hackett et al, 2018 [87]	Cluster randomized trial in 32 villages in rural Tanzania; N = 572 mothers selected for postnatal survey	Job aid app supporting FLWs with data management and real-time guidance for prenatal home visits	Increase uptake of FBD	FBD significantly more likely in I-group (74% vs 63%; OR = 1.96; 95% CI = 1.21-3.19; P = 0.01).
Ilozumba et al, 2018 [88]	Quasi-experimental study in 3 sub-districts of rural Jharkhand, India; N = 2200 mothers with infants <12 months old	2 sub-districts received NGO-led MNCH activities. Mobile for Mothers (MfM) app was added in 1 sub-district to support FLW home visits with data management and embedded multimedia BCC messages	Improve maternal health knowledge, ANC attendance and FBD	Mothers in the MfM group were significantly more likely than NGO or control groups to attend ≥4 ANC visits (OR = 1.23; 95% CI = 1.17-1.29; OR = 1.36; 95% CI = 1.30-1.42) and to have FBD (OR = 1.19; 95% CI = 1.13-1.25; OR = 1.34; 95% CI = 1.28-1.41)
Prinja et al, 2017 [89]	Pre/post quasi-experimental study in rural Uttar Pradesh, India. Annual Health Survey data served as pretest; matched with posttest household survey sample (N = 3106 mothers equally divided between intervention & control areas).	Job aid app supporting FLWs with data management and real-time guidance for home visits during pregnancy and infancy (ReMiND project)	Improve uptake of MNCH services	Large increases in both groups for 4 of 8 MNCH indicators. Significantly greater increases in the Intervention group for self-reporting illness during pregnancy (13.2 p.p.; P = 0.04) and after delivery (19.5 p.p.; P = 0.01)
Intervention studies – grey literature:
Borkum et al, 2015 [90]	Cross-sectional survey in Bihar, India comparing mothers of infants <12 months old (n = 1550) in randomly selected intervention and control communities. Qualitative process evaluation: interviews with FLWs (n = 23) and project staff (n = 4)	Job aid app for home visits during pregnancy and infancy, with multimedia BCC messages embedded	Improve coverage and quality of FLW services	Intervention group more likely to receive home visits prenatally, in first week postnatal, and for complementary feeding (P < 0.05), and more likely to report 3+ antenatal care visits, birth preparedness, and timely initiation of both breastfeeding and complementary feeding (P < 0.05).
BBC Media Action, 2016 [91]	Observational study in Bihar, India, comparing mothers exposed to the Mobile Kunji intervention (n = 2543) vs unexposed (n = 956). Qualitative process evaluation: 4 focus groups and 28 in-depth interviews with FLWs	Counseling tool (Mobile Kunji) combining visual aid with IVR messages accessed through FLWs’ personal phones	Improve quality and engagement with BCC for key MNCH practices	Exposed mothers more likely to have saved the FLW’s phone number (OR = 2.72) and to have fed infants 6-11 months old from at least one food group in past 24 h (OR = 1.72) but no effect on family planning practices.
World Vision, 2018 [92]	Randomized controlled trial in Niger. N = 126 FLWs randomized equally to intervention and control groups were evaluated for their assessment of 544 sick children age 2-59 months	Job aid app supporting Integrated Community Case Management for childhood illnesses	Improved assessment of childhood danger signs and counseling for caregivers	Preliminary results show no significant difference between groups for identification of cough, diarrhea or fever. Mixed results for counseling. Overall no added benefit from the app for management of priority illnesses.
Implementation studies – published literature:
Balakrishnan et al, 2016 [93]	Observational study comparing monitoring data from FLWs in intervention districts of Bihar, India with government statistics for the rest of the state and for the intervention district in the previous year	Job aid app for home visits during pregnancy and infancy, with multimedia BCC messages embedded	Improve quality, equity and efficiency of FLW delivery of 8 core MNCH services: pregnancy registration; registration in first trimester; 3 ANC visits; ≥1 tetanus toxoid vaccine; >90 iron folic acid tablets; delivery in health facility; early initiation of breastfeeding; and ≥1 postnatal home visit.	Coverage of all MNCH services was higher in implementation areas but statistical significance was not assessed. Utilization of MNCH services was similar between scheduled castes and others except for facility-based delivery. Instant data upload from app eliminated delay in data capture from paper forms.
Ilozumba et al, 2018 [94]	Mixed methods assessment of factors influencing outcomes of Mobile for Mothers (MfM) study in Jharkhand, India. Quantitative surveys with mothers (N = 740) and FLWs (n = 57) participating in the intervention. Qualitative interviews with FLWs (n = 28), mothers (n = 32) and men (n = 31).	Mobile for Mothers (MfM) app supporting FLW home visits with data management and embedded multimedia BCC messages	Improve maternal health knowledge, ANC attendance and FBD	MfM app increased FLWs’ knowledge, confidence and efficiency. Main barriers to ANC and FBD uptake were women’s workload, finances, household power dynamics and access to health services.
Feasibility studies – published literature:
McConnell et al, 2016 [95]	Randomized trial in Kiambu county, Kenya. N = 104 postnatal mothers recruited from one private maternity hospital and individually randomized to home visit (n = 32), phone call (n = 41) and control (n = 31) groups	3-d postnatal checklist administered by FLWs either by mobile phone or home visit, compared with standard of care	Improve knowledge and care-seeking for postnatal danger signs	Postnatal checklist administered to 76% of phone group and 59% in home visit group. Infant care-seeking occurred earlier in home visit (2.0 d, P = 0.014) and phone call groups (1.8 d, P = 0.034) compared with control.
Amoah et al, 2016 [96]	Pilot study in 4 communities of rural Ghana; N = 323 pregnant women. Outcomes compared with preliminary survey (N = 100 women in project communities who were pregnant in past 5 years)	Job aid app supporting FLWs with registration, data management and point-of-care guidance for prenatal clients; ultrasound scans conducted in project communities for mothers unable to attend hospital.	Increase ANC attendance and FBD	40 births in study period; 30 (75%) had >3 ANC visits; 25 (62.5%) had FBD compared with 54% and 33% in the preliminary survey.
Formative studies – published literature:
Modi et al, 2015 [97]	Descriptive report of design and feasibility testing of ImTeCHO app for FLWs (n = 45) in Gujarat, India. Feasibility study: Interviews with 6 FLWs and 2 medical officers; focus groups with 9 FLWs and 6 Auxiliary Nurse Midwives; home visit observations	Job aid app for FLWs with work flow scheduling, task monitoring, videos on key BCC messages; link to supervisory support for complex cases and ongoing monitoring	Improve coverage of key MNCH services assigned to FLWs	Job aid app well accepted & considered feasible but ongoing NGO support required for facilitation and technical support.
Kaphle et al, 2015 [98]	Demographics questionnaire completed by FLWs (n = 15); home visit observations (n = 14)	Job aid app for home visits	To develop methods to analyze 1) the effects of app adoption on the quality and experience of care; and 2) personal factors influencing app usage by FLWs	Quality scores were 33.4% higher for high users of the app (P = 0.04). No significant associations with individual factors were found.
Qualitative studies – published literature:
Hackett et al, 2018 [99]	Qualitative study in rural Tanzania. In-depth interviews with FLWs (n = 60) and two rounds of focus group discussions with mothers (n = 56) participating in an mHealth trial	Job aid app supporting FLWs with data management and real-time guidance for prenatal home visits	Women’s reproductive health is a private matter in rural Tanzania, due to fear of exposure to witchcraft. FLWs are trusted confidants. FLWs believed smartphones enhanced data privacy but some mothers expressed concerns about data storage and who could access the phones.
Pimmer & Mbvundula, 2018 [100]	Interpretive case study within the Millennium Village Project in rural Malawi. Focus group discussions with FLWs (n = 29) and interviews with supervisors (n = 3).	Job aid app for FLWs with embedded audio counseling messages related to health topics including MNCH and nutrition.	FLWs perceived the audio messages to support their work in three ways: i) legitimize the use of phones during home visits; ii) assist the FLW to deliver a comprehensive message; iii) support FLWs to persuade communities to adopt health practices.
Qualitative study – grey literature:
Treatman & Lesh, 2012 [101]	Interviews with implementers (n = 8) of CommCare app deployments in India	Job aid tool with embedded audio messages for BCC	Improve quality of counseling by FLWs	Audio messages improved FLW credibility and eased discussion of sensitive topics but design challenges with localization.
Reviews & case studies – grey literature:
Chamberlain, 2014 [102]	Case study	Counseling tool (Mobile Kunji) combining visual aid with IVR messages in Bihar, India	User-centred design process: extensive formative research, prototype development; iterative testing & refinement. Partnerships built with 6 major mobile network operators who subsidize 90% cost of Mobile Kunji calls. >35 000 FLWs using the service with their own basic phones.
Ramachandran, 2013 [103]	Case study	Cell phone-based audio-visual counseling tool for FLWs in Orissa, India	Iterative design process informed by qualitative data and concepts from theories of persuasion. Final tool used locally filmed video with built-in questions and pauses to encourage discussion.
Manthan Project, 2013 [104]	Case study of mSAKHI app with summary findings from 2 pre/post quasi-experimental feasibility studies: i) self-learning and counseling (n = 86 FLWs); ii) postnatal care delivery (n = 57 FLWs)	Job aid app with multimedia functions	i) improved knowledge & counseling delivery by FLWs; ii) Improved coverage and quality of newborn care	i) greater frequency & completion of BCC message delivery (P < 0.05 for 6/9 messages); ii) intervention group more likely to correctly assess newborn health (P < 0.05 for 5 of 7 skills)
Chatfield & Javinski, 2015 [105]	Narrative review of CommCare evidence base, both published & grey literature (n = 40 papers)	CommCare job aid tool for FLWs	Documented evidence related to CommCare acceptability; contribution to improved access to care; quality, experience, and accountability of care; and changes in client knowledge & practices.
Flaming et al, 2016 [106]	Narrative review of CommCare evidence base, both published & grey literature (n = 51 papers; 22 papers address MNCH)	CommCare job aid tool for FLWs	Descriptive review of evidence related to CommCare contribution to improvements in FLWs’ knowledge; performance and credibility; client knowledge and practices; and quality of care. Implementation challenges include technical difficulties, decrease in usage over time, and integrating data with local health systems.
Keisling, 2014 [107]	Project case studies: i) pre/post evaluation (n = 206) with comparison group in Afghanistan; ii) supervisory app monitoring data in India	CommCare job aid apps for FLWs	Improved MNCH knowledge and practices	i) changes in 7 of 15 indicators including birth preparedness, ANC and FBD; ii) 136% increase in clients asking questions during home visits; 78% of mothers served by app gave birth in facilities
World Vision, 2015 [108]	Narrative overview of World Vision’s mHealth projects with country project examples from India, Indonesia, Sierra Leone & Uganda	MOTECH Suite with apps supporting five MNCH project models including home visits by FLWs	Strengthen community health systems	16 projects in 21 countries; 7 projects have >100 users. Implementation lessons: high acceptability of apps; need adequate technology training and support to FLWs
MIRA Channel, 2015 [109]	Project brief summarizing MIRA concept and results of pilot test with 50 FLWs	Mobile phone “channel” integrating health communication and management functions for rural women in India	Increase rural women’s access to health knowledge and care	59% increase in ANC attendance, 49% increase in FBD and 41% increase in immunization coverage with MIRA use
World Vision, 2018 [92]	Narrative description of evaluation and research findings related to World Vision’s mHealth deployments for FLWs in 11 countries	CommCare job aid apps for FLWs	Process evaluation surveys in Sierra Leone and Uganda found high acceptance of job aid apps, particularly by beneficiaries with greater exposure to mHealth. Formative research in Mauritania and Tanzania guided program adaptions related to technology, contextualization and integration with Ministry of Health systems. Analysis of projects in Uganda, Sierra Leone and India using the mHealth Assessment and Planning for Scale (MAPS) toolkit identified site-specific strengths to leverage and challenges to address in ongoing program plans.

FLWs – frontline health workers, ANC – antenatal care; BCC – behaviour change communication, FBD – facility-based delivery, MNCH – maternal, newborn and child health, IVR – interactive voice response