Testing mHealth solutions at the last mile: insights from a study of technology-assisted community health referrals in rural Kenya

Abstract

Background

mHealth technologies are already disrupting conventional healthcare delivery by making innovative solutions more accessible in terms of reach and price across reach and price across the developing world. However, much less has been documented on the process of mHealth innovation introduction in the context of rural communities of Africa. Pending still is the widespread adoption of standards and the removal of barriers to introduction, testing and scale. This paper documents the innovation process of technology introduction, results and lessons learned through a case study of two mHealth initiatives: closed-loop referrals for maternal and child health; and HIV self-testing. Both initiatives were implemented and evaluated in Kisii County, Kenya by Living Goods.

Methods

Living Goods applied an innovation framework to introduce and evaluate two interventions integrated into the Living Goods Smart Health app, a smartphone-based digital health application designed to carry out household registration, assessment, and diagnosis at community level. Community health workers (CHWs) used digitally assisted, standardized Ministry of Health algorithms to assess and refer clients to the nearest health facility for diagnosis confirmation and treatment as appropriate. Routine data as well as periodic household surveys were captured to incorporate performance data and outcomes into activity management. A quasi-experimental evaluation was carried out using a Propensity Score Matching (PSM) methodology to evaluate intervention arms for each intervention.

Results

Findings suggest that the initiatives increased the frequency of visits to households with participants in the treatment groups being more likely to have been visited more than six times within the last six months. The interventions contributed in part to an increase in the frequency of CHW follow-up visits within the treatment group. Attitudes of trust and confidence in CHWs were high but limited to referral services and not to diagnostic and curative services.

Conclusions

The innovation process effectively positioned and tested at community level the two interventions to address key barriers to service delivery acceptance and uptake. Despite extensive pre-testing and field iterations to adapt the solutions to the local context, behavioral and technology barriers persisted. The study highlights important implications for both innovators and service providers: technology introduction and adaptation at community level requires multiple, rapid iteration loops to ensure product refinement and user-acceptance; behavioral assessments of acceptability require a wholistic approach to ensure effective alignment of senders, receivers and trusted intermediaries of novel services.

Introduction

Increasingly service providers and consumers in low-income countries have adopted mHealth solutions for the diagnosis and assisted management of various communicable and non-communicable diseases (1-3). Due to the rapid growth in mobile phone ownership (2) and shortage of health workforce (4), low-income communities can enjoy the benefits, accessibility (5) and reliability of mHealth with minimum training and investment. In the context of mHealth, no expensive equipment or laboratory facilities is required (6) in identifying and monitoring health concerns and outcomes (2).

An abundance of research on mHealth technologies has been conducted in Africa and elsewhere (2,3,6,7). In an assessment of health workforce needs for universal health coverage in Kenya, Miseda et al. (4), note that the use of mHealth can catalyze achievement of immediate, medium, and long-term national health objectives. Watterson et al. (8) corroborate these findings and suggest that there are significant improvements over existing service delivery models in low-and middle-income countries attributable to mHealth. They further note that adoption of mHealth has increased the coverage and care-seeking of antenatal care (ANC) and postnatal care (PNC), childhood immunizations, appointment and medication adherence, and patient records and identification.

As such, it is widely accepted in the emerging field of digital health that the adoption of mobile phone-based ‘mHealth’ technologies have substantially disrupted conventional healthcare delivery by making it more accessible in terms of reach and price across resource poor countries (2,5). In Bangladesh for example, Khatun et al. (1), found over 20 mHealth initiatives currently underway despite the low awareness of mHealth and its advantages among product users.

A review of literature (9) thus points towards far-reaching impact (2) and latent capability of mHealth to facilitate self-management actions, such as providing health educational content, and self-regulation of diseases particularly for people living in rural communities (10).

While evidence for mHealth interventions has been widely established, much less has been documented on the process of introducing and testing mHealth solutions in the context of rural communities of Africa (1). Moreover, the lack of standards and the removal of barriers to the introduction, testing and scale inhibits widespread adoption of these solutions. Living Goods sought to address this ‘test and scale’ gap, to get mHealth solutions into a testing pathway and then to the communities where millions of lives can potentially be saved.

The Living Goods Innovation Network

Living Goods’ vision is that by 2030 every mother and child will have access to basic healthcare in their community. We do this by supporting networks of performance-driven community health workers (CHWs), recognized as a part of the government public sector health workforce, to go door-to-door teaching families how to improve their health and distributing life-saving medicine. Living Goods’ approach blends the best practices from public health with effective technology and performance management from the private sector to deliver community health programs. Living Goods currently supports over 10,000 CHWs reaching nearly 8 million people across Kenya and Uganda, with limited presence in Sierra Leone and Burkina Faso. Living Goods is headquartered in Nairobi, Kenya.

The Innovation and Testing Process Overview

The Network combines all the elements necessary to bring promising technologies from concept to reality: a field-test site, research and delivery infrastructure, tools, resources, and expertise. The Network supports a cadre of 180 digitally enabled CHWs based in Kisii County, Kenya. The CHWs are trained in government-approved integrated community case management (iCCM)—a set of curative services for major childhood illnesses (diarrhea, malaria, and pneumonia) along with supplemental services for family planning and routine immunization. These preventive, promotive and curative services reach approximately 144,000 households in Kisii County alone.

An essential element of the Network is the digital infrastructure provided by Living Goods and technology partner Medic Mobile. The Smart Health App is an open-source platform based on Medic Mobile’s Community Health Toolkit (CHT) (11). Building on an open and extensible toolkit supports the testing of new software and hardware innovations, while ensuring the platform is available to the broader community of innovators and developers. Medic Mobile provides dedicated developers and designers for the Network. The Network collaborates closely with government partners to ensure integration with the broader health system, human subjects protection and review, data sharing and analysis. Strong partnership with the Kisii County Government ensures that successful technology innovations can make their way into the primary care system quickly and sustainably.

Innovation pipeline

Figure 1 shows the shared inclusion criteria used to ensure selection of the most promising and best-fitting new partners and ideas for the Network. Inclusion decisions are guided by a carefully defined set of criteria, including the potential to deliver high impact, feasibility (in terms of usability and cost) for CHWs to implement, health system acceptability, and replicability and scale.

Figure 1.

Inclusion Decision-Tree. Source: Living Goods, 2017.

Once a concept gets past the inclusion stage, it moves on to Idea Development and Concept Proposal Stage where the full concept is reviewed and approved. The central underlying methodology applied throughout the subsequent “Proof of Concept” phase is Human Centered Design (HCD) (9). HCD features prominently across the length and breadth of the Innovation and Testing Process: from Concept Design and Research (Phase 0), Feasibility and Planning (Phase 1), Design and Development (Phase 2), Validation and Launch Readiness (Phase 3), and lastly Launch and Lifecycle Management (Phase 4) stages. Figure 2 visually represents a funnel and accelerated concurrent pipeline process. The selection of new health technology solutions is sourced from the vibrant ecosystem of diverse technologies and partners found in East Africa and elsewhere.

Figure 2.

Innovation and Testing Process. Source: Living Goods (2019).

Phase 0—Concept Design and Research: Here user research explores current health seeking and delivery scenarios, defining problems and opportunities for improvement. User personas are developed using HCD techniques. Literature reviews and landscape analyses are undertaken. Stakeholders perspectives are considered at this stage and buy-in secured. Research is undertaken to establish and document baseline and key metrics of success. Finally, a regulatory compliance review is conducted to ensure alignment with current health and research protocol norms.

Phase 1—Feasibility and Planning: Design tools are deployed to document user stories and requirements. Workflows are developed. Prototyping is undertaken to assess feasibility, desirability and usability. If applicable, discreet choice experiments are conducted (such as A/B testing). Requirements and specification documentation are developed. Government and community partners are then engaged for validation once a feasible prototype is shared. Finally, the evaluation design and a monitoring and evaluation (M&E) plan is finalized.

Phase 2—Design and Development: Hardware and software prototypes are completed. User interface design and testing is conducted. A minimum viable product (MVP) is defined, and integrated (or made interoperable) with Living Goods’ Smart Health App. Field program requirements for test and evaluation are put in place, including logistics such as equipping CHWs, training, support and financial and non-financial incentives.

Phase 3—Validation and Launch Readiness: Field validation entails training and equipping CHW testers and conducting small scale pilots. Data collection and analysis are completed. User acceptability testing is repeated, and changes fed back into the MVP. Bugs are fixed and necessary software updated. The baseline data collection is conducted.

Phase 4—Launch and Lifecycle Management: At this “go-to-market” phase, the product is rolled out at large scale to a broad user-base. Training and on-going support is provided to ensure product viability. Ongoing monitoring and evaluation is undertaken to ensure product utilization, quality, efficiency and effectiveness.

Prior to launching an innovation, the Network conducts a last due diligence step by engaging the broader community of innovators to validate the implementation infrastructure required (12).

“Lessons from the Field”—a case study from two mHealth initiatives

The Network tested two mHealth initiatives with a sub-group of CHWs in Ogembo sub-County, Kisii, Kenya. They consisted of: (I) establishing Closed Loop for Referrals & Follow-up (CLE)¹ of Maternal, Neonatal and Child Health (MNCH) & Integrated Community Case Management (iCCM) to ensure that all health referrals and follow-ups by CHWs and confirmed via verifiable digital data; (II) establishing a Closed Loop for HIV Self Testing (HIV-ST, Referral & follow up, a technology-assisted protocol for the community distribution of HIV rapid self-tests kits (http://www.oraquick.com/). We collected pre- and post-intervention data, as well as routine and periodic monitoring data over 12 months from August 2018 to July 2019.

The two studies evaluated household attitudes, perception of the quality of community health services, health seeking practices associated with the use of public health facilities and referral, attitudes and knowledge about HIV and HIV testing, and knowledge and awareness of HIV self-test kits. The Network adapted Living Goods’ Smart Health App to carry out household assessments, diagnosis, and referral tracking to the nearest health facility for diagnosis confirmation and treatment as appropriate. In the case of a severe disease such as childhood pneumonia or reactive HIV self-test, the Smart Health App generated a detailed confidential referral summary which the CHW transferred in triplicate to the Ministry of Health’s paper referral note Form MOH 100. The original and the first copy were given to the client to take the health facility while the CHW retained the second copy. This mixed use of digital tools and paper was designed to enable rapid and reliable information sharing with facilities, even in areas where CHWs have very poor connectivity. After service provision at the health facility, the healthcare worker signed and stamped the two documents, filing the original, while the CHW went back to the household with a copy. The “closed loop” innovation enabled digital linkage between community health services and health facilities, enabling 360º view of patient management at the community level.

Methods

Living Goods commissioned an independent evaluator, The Busara Center for Behavioral Economics (Busara), to capture performance and outcome data. Routine data collection as well as periodic surveys and evaluations were undertaken to capture intervention outcomes and lessons learned. A combination of qualitative and quantitative studies was undertaken in close collaboration with the Kisii County Health Directorate. The qualitative component sought to understand how CHWs are integrated into the health services ecosystem and how they interact with and impact household health-seeking behaviors. Semi-structured interviews were conducted with 50 households to collect information on decisions and behaviors regarding relevant health outcomes. Process mapping with an additional 10 CHWs was undertaken to understand decision process flows and key decision points during a CHW household consultations. Finally, 20 “mystery client” interviews were collected in eight facilities in Ogembo sub-County by trained enumerators who simulated patient care-seeking roles at selected health facilities and recorded detailed notes from those experiences through an exit interview. Subsequently a quantitative baseline was initiated prior to the implementation of the interventions (NON-KEMRI Protocol No. 630).

Quantitative evaluation

A quantitative evaluation sought to answer the following research questions:

1. What is the impact of introducing the CLE intervention on households’ perception of CHWs, CHW services, referrals, health status, health facilities, follow-up experience and recovery speed?

2. What is the impact of the HIV-ST intervention on households’ perception of CHWs, CHW services, referrals, health facilities and follow-up experience?

3. What is the impact of introducing HIV self-test kits through CHWs on households’ attitudes and knowledge about HIV, HIV testing and HIV self-test kits?

Using constructs of health status, household perceptions of health services, health facility experience, follow-up experience as well as knowledge and attitudes towards HIV, we evaluated the effects of the CLE and HIV-ST initiatives. Because the two initiatives were purposively assigned to separate catchment areas (community units) by Living Goods, randomization of treatment assignment for the evaluation was not feasible. Instead we employed Propensity Score Matching (PSM), a statistical matching method that allows us to find a comparison group using the average characteristic of individuals in the treatment group. Based on the observed characteristics of people in the treatment group we estimated the likelihood that individuals where the initiative was not administered would likely to be eligible participants if they happened to be in the target communities.

The evaluation was undertaken in six community units within Ogembo sub-County: Mangere, Keragia, Mogambi, Bosoti, Nyabioto, and Buyonge. Research participants (households) were categorized into 3 groups: Closed Loop Existing (CLE), HIV Self-test (HIV-ST), and control. Field activities took place over 23 days from 11 September, to 9 October, 2019. Four units were assigned treatment initiative and two as control areas. 133 households were listed in each of the community units; in total we surveyed 798 households, 262 respondents per group.

A pre- and post-intervention household survey investigated outcomes of interest, this included healthcare-seeking and referral practices, priority referral treatment, as well as awareness and use of HIV self-test kits. Busara developed the survey instruments in collaboration with Living Goods and Medic Mobile, and validity and reliability tests were conducted. Enumerator training was conducted on research ethics and good practice, quality standards, key concepts and interviewing skills.

The study employed a stratified random sampling technique by community unit. A total of 68 CHWs from the 6 community units were randomly selected an allocated evenly by study arm. Each CHW served a population of around 100 households, providing this study with a total household population of approximately 6800. A random sample of households was interviewed through in-person and/or phone surveys to explore households’ attitudes, preferences, and practices around healthcare seeking and medical referrals, as well as awareness and past use of HIV self-test kits.

For the quantitative data analysis, we calculated difference of means for our outcome variables for the treatment and the control group. To do this, we ran regression analyses using statistical software (STATA). We also measured the statistical significance which we assumed at 95% to justify the external validity of our results.

A propensity score was calculated based on pre-selected variables including: gender, age, education, marital status, income, number of children, distance to the referral health facility, distance to preferred health facility, distance to the main road, health care preference options when sick. Participants were then matched in each treatment group to the control group, based on propensity score, using the nearest neighbor approach. After matching, the sample size for the treatments and control was:

1. Matched sample 1: CLE—107 participants; Control—107 participants.

2. Matched sample 2: HIV-ST—96 participants; Control—96 participants.

The control group consisted of two community units (Mogambi and Nyabioto) linked to two health facilities (Sengera Mission Health Center & Moogi Dispensary).

While propensity scoring does not grant the same depth of rigor as that of randomized treatment assignment, reliability tests indicate the propensity score and accompanying matching model developed were highly predictive. In addition, the study relied upon self-reported responses that may limit measurement validity and reliability due to biases associated with question framing and response limitations.

Results

Household perceptions towards CHWs

All respondents in both CLE and HIV-ST initiatives, including the control group, received at least one CHW visit in the past 6 months. Respondents in the CLE and HIV-ST treatment samples receive more frequent visits compared to the control group, receiving more than 6 visits in the last six months (P<0.05). Figure 3 shows that 57% of the participants in our treatment group across the two samples received the MOH 100 referral form. The difference in the proportions that received the form is more pronounced in the CLE sample.

Figure 3.

Receipt of MOH 100 Forms.

A larger proportion of the CLE and HIV-ST clientele consider the health recommendations of CHWs as important. Compared to the control group, this indicates a 2.8% and 6.14% difference for the CLE and HIV-ST groups respectively. Figure 4 shows that 50% prefer calling a CHW when they feel sick, rather than going to the hospital or purchasing medicines directly from the pharmacy. Likewise, respondents believe that other people in their communities have a preference for a CHW in the event that they fall sick.

Figure 4.

Household perceptions towards community health workers (CHWs).

Client comfortability, trust and satisfaction with CHW services

Respondents were asked to rank how comfortable they felt with diagnostic, medication and referral services performed by a Living Goods’ CHW. Table 1 shows comparisons of comfortability scores between the control group and treatment groups under the CLE and HIV-ST initiative. Participants under CLE and HIV-ST groups were more comfortable with referral services from the CHW compared to diagnostic and medication services. Taking the different comfortability components together, we find suggestive evidence that if a respondent is in the CLE treatment group, the more comfortable they are with CHW services as opposed to those in the control group. Conversely, our results do not show any significant difference in the HIV-ST sample.

Table 1. Study participant’s average comfortability score by initiative.

Comfortability questions	Control (mean)	CLE (mean)	Control (mean)	HIV-ST (mean)
How comfortable were you to receive a medical diagnosis by a Living Goods CHW?	4.55	4.46	4.59	4.69
How comfortable were you to be given medication by a Living Goods CHW from your home?	4.6	4.54	4.67	4.6
How comfortable were you to be referred to a health facility by a Living Goods CHW from your home?	4.45	4.52	4.64	4.75

CLE, Closed Loop Existing; HIV-ST, HIV Self-test; CHW, community health worker.

A proxy variable of trust and attitudes towards CHWs was created using a series of questions on satisfaction and feeling of respondents towards the CHWs services. Respondents ranked a series of statements on a scale of 1 to 5, with 1 indicating strongly disagrees and 5 complete agreement.

No significant impact was observed between CLE and HIV-ST treatment on respondent trust and attitudes towards CHWs as shown in Table 2. However, heterogeneous effects of these initiatives were found on a subsample of the study participants across different demographic segments. Among both the CLE and HIV-ST sample groups, older and better educated respondents have an increased level of trust towards CHWs.

Table 2. Average satisfaction score by initiative.

Satisfaction questions	Control (mean)	CLE (mean)	Control (mean)	HIV-ST (mean)
Receiving Living Goods CHW services at home saves me time	4.28	4.45	4.35	4.54
Receiving Living Goods CHW services at home saves me money	4.21	4.32	4.21	4.47
The CHWs’ explanations for my diagnosis, treatment and referrals are usually comprehensive and adequate	4.41	4.41	4.44	4.56
The CHWs’ services are very important for my health and the health of my family	4.41	4.45	4.47	4.53
I think all counties in Kenya should adopt the Living Goods system	4.29	4.27	4.36	4.45

CLE, Closed Loop Existing; HIV-ST, HIV Self-test; CHW, community health worker.

Furthermore, no significant difference was found in the rate of CLE referrals or compliance to referrals between treatment samples. However, for the CLE sample, women participants were more likely to adhere to referred facilities compared to male participants. Only a small proportion failed to comply to the referred facilities.

Facility and follow-up visits

Approximately, half of the participants in the CLE initiative went to the health facilities immediately after referral, compared to one-third of the control sample respondents for the CLE initiative that adhered immediately to referrals. Figures 5,6 show how long after the referral it took to visit the facility for both initiatives.

Figure 5.

How long after the referral did it take to visit facility (CLE).

Figure 6.

How long after the referral did it take to visit facility (HIV-ST).

Ninety percent of respondents in both samples confirmed to have received a follow-up visit. Nearly all follow-ups were done in-person with some done by phone or by another CHW. Respondents in the treatment groups received more follow-up visits from CHWs than the control groups in both samples, however the difference between the groups was not significant. All respondents who received follow up visits by CHWs reported them to be useful. Respondents in the HIV-ST group were significantly more likely to seek faster treatment after first symptoms compared to people in the control group. The same pattern was found for the CLE group, although the difference is not significant.

We determined the health facility experience of the respondents using statements that capture timeliness of service, courtesy of health staff, information provided, etc. Respondents were asked to rank each statement on a scale of 1 to 5, with 1 being the lowest and 5 the highest. No significant impact of the treatments on the health facility experience of both the CLE and HIV-ST groups was found. Table 3 shows that participants in the HIV-ST treatment group report a better experience across different health facility experience questions, these differences were not significant.

Table 3. Average health facility experience score by initiative.

Health facility experience questions	Control (mean)	CLE (mean)	Control (mean)	HIV-ST (mean)
The courtesy of admission staff was good	4.39	4.45	4.4	4.61
I had to wait for long before I saw a doctor	2.45	2.65	2.36	2.19
I had enough time to tell the doctor about my symptoms	4.21	4.31	4.35	4.6
I would recommend the facility to my friends/family	4.23	4.17	4.25	4.4
I was referred to a higher level of care	1.6	1.5	1.45	1.44
I was told about medication side effects	1.47	1.32	1.6	1.73
The health facility had all the medicines the doctor prescribed for me	1.24	1.23	1.75	1.77

CLE, Closed Loop Existing; HIV-ST, HIV Self-test.

HIV testing

More than 90% of the respondents have tested for HIV. Slightly more participants in the HIV-ST group have taken an HIV test before compared to the control group. Major reasons cited by respondents for getting tested as ‘need to know their status’ and ‘a requirement from a health facility’. Respondents in the HIV-ST group were more likely to have been tested for HIV, at least once, compared to people in the control group and this was statistically significant.

As compared to the control group, we found a significant difference in the proportion of people who have tested for HIV using the self-test kits. Overall, 37% prefer conducting their HIV tests at public health facilities rather than at home with the CHW. Only 16% of the respondents within the HIV-ST group had done their test with the HIV self-test kits provided from the Living Goods CHW. 31% of the tests occurred within the last 3 months, with more people in the HIV-ST group having tested for HIV for the last time within the same period as compared to people in the control group. However, inclusion in the HIV-ST group had no effect on whether one had tested for HIV within the last 3 months or not.

Table 4 shows respondents’ perception about HIV-AIDS. For 49% of respondents, knowing one's HIV status brought comfort to them. However, 14% of the respondents felt that people who test HIV positive should hide it from their friends and families. Respondents in the HIV-ST group held more negative beliefs about people living with HIV-AIDS. When asked whether respondents would rather not know their HIV status, 9% of the respondents from both groups strongly disagreed. Almost all respondents in both groups knew at least 1 correct way of how HIV-AIDS is transmitted.

Table 4. Negative beliefs about people with HIV-AIDS.

Statement	Control (mean)	HIV-ST (mean)
HIV is a punishment from God	3.3	3.7
Some people think HIV is a punishment for bad behavior	2.8	3.1
Most people with HIV deserve what they get	2.8	3.2
It is sex workers who spread HIV in the community	2.9	3.3
People with HIV are promiscuous.	2.9	3.2
People with HIV should be ashamed of themselves	3.2	3.2
Men who have sex with men deserve to get AIDS	2.9	3.2
HIV+ women should not have children	3.5	3.7
AIDS persons should be avoided	3.3	3.9
There is no cure for AIDS	3.1	3.2

At the time of data collection, 62% of the respondents had heard of HIV self-test kits, though 11% of the respondents had not heard of HIV self-test kits. Among those who have heard of HIV self-test kits, 43% learned of self-testing from a Living Goods CHW.

Thirty-six respondents in the HIV-ST group reported that they had received a testing kit from a Living Goods CHW and that:

• ❖ All 36 received it for free;

• ❖ 28 were tested with their partner, while 8 tested alone;

• ❖ All 36 received an education before the test;

• ❖ 28 self-administered the test.

Our analysis of this rich dataset yielded two key findings. First, both CLE and HIVST initiatives lead to significant improvements in CHW visits as measured by completed referrals and frequency of visits to households. Participants in the treatment groups were more likely to have been visited more than 6 times within the last six months (P<0.05). It thus appears that the CLE initiative contributed to an increase in the frequency of CHW follow-up visits within the treatment group.

Second, we noted that age and education of respondents are positively correlated with the likelihood of having positive attitudes towards CHW services. Still, HIV testing increased significantly for the HIV-ST group versus the control group irrespective of the source of the test. It thus appears that attitudes of trust and confidence in CHWs is high but limited to referral services and not to diagnostic and curative services. Additional analysis data can be found in Tables S1-S13.

Table S1. Summary of Propensity Score Matching on CLE.

	Control	Treated (CLE)
Original sample size	152	188
Matched sample	107	107
Unmatched	23	75
Discarded	22	6

CLE, Closed Loop Existing.

Table S2. Summary of balance before matching (CLE).

	Mean treated (n=107)	Mean control (n=107)	Mean diff
Distance	0.62	0.46	0.16
Gender, female	0.85	0.82	0.04
Age	39.19	38.13	1.06
Income
10,001–20,000 Kshs	0.06	0.06	0
20,001–30,000 Kshs	0.02	0.03	−0.02
30,001–50,000 Kshs	0	0.01	−0.01
50,001–100,000 Kshs	0	0	0
Marital status
Cohabiting but not married	0.01	0	0.01
Widowed	0.11	0.12	−0.02
Divorced/separated	0.03	0.02	0.01
Never married & not cohabiting	0.04	0.01	0.02
Children (No)	0.05	0.03	0.02
Education
Some primary	0.31	0.26	0.05
Primary school completed	0.24	0.15	0.09
Post primary technical training	0.01	0.01	0
Some secondary school	0.2	0.21	−0.01
Secondary completed	0.13	0.2	−0.07
Buy medicine from the pharmacy	0.07	0.07	−0.02
Buy herbal remedies	0.01	0	0.01
Make home remedies	0	0.03	−0.03
Calls the CHW	0.51	0.52	−0.01
Waits for the CHW to come	0	0.03	−0.03
Does nothing and wait	0.01	0.01	0
Travel time to nearest main road	7.75	15.88	−8.13
Travel time to facility	24.29	24.38	−0.09
Travel time to referral facility	23.58	25.28	−1.7

CLE, Closed Loop Existing; CHW, community health worker.

Table S3. Summary of balance after matching (CLE).

	Mean treated (n=107)	Mean control (n=107)	Mean diff (P values)
Distance	0.57	0.56	0
Gender, female	0.87	0.81	0.06
Age	36.83	37.97	−1.14
Income
10,001–20,000 Kshs	0.05	0.05	0
20,001–30,000 Kshs	0.03	0.02	0.01
30,001–50,000 Kshs	0	0	0
50,001–100,000 Kshs	0	0	0
Marital status
Cohabiting but not married	0	0	0
Widowed	0.08	0.09	−0.01
Divorced/separated	0.03	0.02	0.01
Never married & not cohabiting	0.02	0.01	0.01
Children (No)	0.03	0.03	0
Education
Some primary	0.28	0.3	−0.02
Primary school completed	0.16	0.18	−0.02
Post primary technical training	0.01	0.02	−0.01
Some secondary school	0.24	0.23	0.01
Secondary completed	0.19	0.15	0.04
Buy medicine from the pharmacy	0.05	0.05	0
Buy herbal remedies	0	0	0
Make home remedies	0	0	0
Calls the CHW	0.56	0.57	−0.01
Waits for the CHW to come	0	0	0
Does nothing and wait	0	0	0
Travel time to nearest main road	8.94	9.39	−0.44
Travel time to facility	23.06	24.64	−1.57
Travel time to referral facility	23.1	25.24	−2.14

CLE, Closed Loop Existing; CHW, community health worker.

Table S4. Summary of Propensity Score Matching on CLE.

	Control	Treated (CLE)
Original sample size	152	167
Matched sample	96	96
Unmatched	55	53
Discarded	1	18

CLE, Closed Loop Existing.

Table S5. Summary of balance before matching (HIV-ST).

	Mean treated (n=96)	Mean control (n=96)	Mean diff (P values)
Distance	0.58	0.46	0.12
Gender, female	0.89	0.82	0.08
Age	38.18	38.13	0.05
Income
10,001–20,000 Kshs	0.12	0.06	0.06
20,001–30,000 Kshs	0.04	0.03	0
30,001–50,000 Kshs	0.01	0.01	0.01
50,001–100,000 Kshs	0.01	0	0.01
Marital status
Cohabiting but not married	0	0	0
Widowed	0.14	0.12	0.02
Divorced/separated	0.03	0.02	0.01
Never married & not cohabiting	0.04	0.01	0.02
Children (No)	0.08	0.03	0.06
Education
Some primary	0.34	0.26	0.08
Primary school completed	0.15	0.15	0
Post primary technical training	0.01	0.01	−0.01
Some secondary school	0.16	0.21	−0.05
Secondary completed	0.25	0.2	0.05
Buy medicine from the pharmacy	0.04	0.07	−0.03
Buy herbal remedies	0	0	0
Make home remedies	0.02	0.03	−0.01
Calls the CHW	0.48	0.52	−0.04
Waits for the CHW to come	0.03	0.03	0
Does nothing and wait	0	0.01	−0.01
Travel time to nearest main road	11.81	15.88	−4.07
Travel time to facility	21.84	24.38	−2.54
Travel time to referral facility	21.03	25.28	−4.25

Table S6. Summary of balance after matching (HIV-ST).

	Mean treated (n=96)	Mean control (n=96)	Mean diff (P values)
Distance	0.53	0.53	0
Gender, female	0.86	0.86	0
Age	38.47	38.96	−0.48
Income
10,001–20,000 Kshs	0.07	0.07	0
20,001–30,000 Kshs	0.04	0.04	0
30,001–50,000 Kshs	0	0.01	−0.01
50,001–100,000 Kshs	0.01	0	0
Marital status
Cohabiting but not married	0	0	0
Widowed	0.15	0.13	0.02
Divorced/separated	0.01	0.02	−0.01
Never married & not cohabiting	0.02	0.02	0
Children (No)	0.04	0.04	0
Education
Some primary	0.31	0.31	0
Primary school completed	0.18	0.18	0
Post primary technical training	0.01	0.01	0
Some secondary school	0.14	0.14	0
Secondary completed	0.26	0.22	0.04
Buy medicine from the pharmacy	0.04	0.03	0.01
Buy herbal remedies	0	0	0
Make home remedies	0.03	0.03	0
Calls the CHW	0.55	0.52	0.03
Waits for the CHW to come	0.01	0.03	−0.02
Does nothing and wait	0	0	0
Travel time to nearest main road	14.05	12.03	1.02
Travel time to facility	22.02	24.63	−2.61
Travel time to referral facility	23.69	25.35	−1.66

Table S7. Impact on frequency of CHW visits in treatment one (CLE), regression model results.

	Dependent variable (treatment)
	−1	−2	−3	−4
Have you ever received a CHW visit (Yes)	0.171 (0.208)	0.199 (0.217)
Female		0.127 (0.102)		0.127 (0.102)
Age		−0.003 (0.003)		−0.003 (0.003)
Income 10,001–20,000 Kshs		0.057 (0.174)		0.097 (0.174)
Income 20,001–30,000 Kshs		0.326 (0.286)		0.412 (0.286)
Widowed		−0.038 (0.130)		−0.031 (0.129)
Separated		0.102 (0.240)		0.094 (0.238)
Not cohabiting		0.094 (0.326)		0.090 (0.323)
Children		0.048 (0.225)		0.323 (0.057)
Some primary education		−0.038 (0.142)		−0.064 (0.141)
Primary completed		−0.053 (0.161)		−0.052 (0.161)
Primary technical training		−0.205 (0.332)		−0.231 (0.329)
Some secondary school		−0.025 (0.158)		−0.038 (0.158)
Secondary completed		−0.007 (0.163)		−0.033 (0.163)
University or higher education		−0.219 (0.238)		−0.262 0.237()
3-5 visits in the last 6 months			0.024 (0.085)	0.021 (0.089)
More than 6 visits			0.190 (0.085)**	0.203 (0.088)**
Constant	0.333 (0.205)	0.286 (0.399)	0.433 (0.061)***	0.409 (0.338)
Observations	214	214	208	208
R2	0.003	0.03	0.028	0.059
Adjusted R2	−0.001	−0.044	0.014	−0.025
Residual Std. Error	0.502 (df =212)	0.512 (df =198)	0.498 (df =204)	0.507 (df =190)
F statistic	0.682 (df =1; 212)	0.407 (df =15; 198)	1.983(df =3; 204)	0.703 (df =17; 190)

**, P<0.05; ***, P<0.01.

Table S8. Impact on frequency of CHW visits in treatment two (HIV-ST), regression model results.

	Dependent variable (treatment)
	–1	–2	–3	–4
Have you ever received a CHW visit (Yes)	–0.172 (0.208)	–0.153 (0.224)
Female		–0.011 (0.117)		–0.054 (0.120)
Age		0.003 (0.004)		0.003 (0.004)
Income 10,001–20,000 Kshs		–0.037 (0.141)		–0.020 (0.141)
Income 20,001–30,000 Kshs		–0.042 (0.228)		–0.008 (0.228)
Income 30,001–50,000 Kshs		–0.678 (0.592)		–0.640 (0.592)
Widowed		0.106 (0.131)		0.048 (0.136)
Separated		–0.161 (0.312)		–0.143 (0.316)
Not cohabiting		0.100 (0.281)		–0.006 (0.286)
Children		–0.166 (0.227)		–0.186 (0.228)
Some primary education		0.033 (0.186)		0.037 (0.193)
Primary completed		0.060 (0.203)		0.058 (0.207)
Primary technical training		0.659 (0.599)		0.681 (0.599)
Some secondary school		0.045 (0.211)		0.021 (0.251)
Secondary completed		0.137 (0.198)		0.105 (0.204)
University or higher education		0.040 (0.269)		–0.010 (0.273)
3–5 visits in the last 6 months			0.005 (0.087)	0.029 (0.094)
More than 6 visits			0.152 (0.091)*	0.188 (0.100)*
Constant	0.677 (0.205)	0.636 (0.439)	0.433 (0.061)***	0.484 (0.405)
Observations	192	192	186	186
R2	0.004	0.037	0.019	0.052
Adjusted R2	–0.002	–0.051	0.008	–0.044
Residual Std. Error	0.502 (df =190)	0.512 (df =175)	0.498 (df =183)	0.507 (df =168)
F statistic	0.683 (df =1; 212)	0.420 (df =15; 198)	1.741(df =3; 204)	0.543 (df =17; 190)

*, P<0.1; ***, P<0.01.

Table S9. Impact on household knowledge and attitude towards CHWs (CLE), regression model results.

	Dependent variable (agree scale)
	−1	−2
Treatment one (CLE)	0.458 (0.491)	0.497 (0.477)
Age		0.067 (0.023)***
Female		0.193 (0.686)
Income 10,001–20,000 Kshs		−4.501 (1.170)
Income 20,001–30,000 Kshs		−0.798 (1.931)
Widowed		−0.261 (0.870)
Separated		−0.859 (1.611)
Not cohabiting		0.204 (2.189)
Children		−0.896 (1.511)
Some primary education		−0.013 (0.951)
Primary completed		0.795 (1.709)
Primary technical training		3.623 (2.233)
Some secondary school		1.637 (1.059)
Secondary completed		2.178 (1.097)**
University or higher education		3.745 (1.601)
Constant	30.421 (0.341)***	25.987 (2.278)***
Observations	214	214
R2	0.004	0.147
Adjusted R2	−0.001	0.082
Residual Std. Error	3.592 (df =212)	3.441 (df =198)
F statistic	0.870 (df =1; 212)	0.420 (df =15; 198)

**, P<0.05; ***, P<0.01.

Table S10. Impact on household knowledge and attitude towards CHWs (HIV-ST), regression model results.

	Dependent variable (agree scale)
	−1	−2
Treatment two (HIV-ST)	1.052 (0.553)*	0.853 (0.537)
Age		0.076 (0.027)***
Female		1.256 (0.829)
Income 10,001–20,000 Kshs		−3.733 (1.003)***
Income 20,001–30,000 Kshs		0.549 (1.620)
Widowed		−0.625 (0.913)
Separated		−0.475 (2.222)
Not cohabiting		2.876 (1.998)
Children		−0.094 (1.620)
Some primary education		0.833 (1.322)
Primary completed		2.576 (1.430)*
Primary technical training		4.548 (4.274)
Some secondary school		3.386 (1.490)
Secondary completed		3.397 (1.406)**
University or higher education		3.735 (1.906)
Constant	30.615 (0.391)***	24.914 (2.847)***
Observations	192	192
R2	0.019	0.178
Adjusted R2	−0.014	0.1303
Residual Std. Error	3.833 (df =190)	3.5656 (df =175)
F statistic	0.870 (df =1; 190)	2.364 (df =16; 175)

*, P<0.1; **, P<0.05; ***, P<0.01.

Table S11. Impact on HIV testing (HIV-ST), regression model results.

	Dependent variable (agree scale)
	−1	−2
Treatment two (HIV-ST)	0.073 (0.039)*	0.067 (0.039)
Age		−0.006 (0.002)***
Male		−0.015 (0.063)
Income 10,001–20,000 Kshs		−0.013 (0.070)
Income 20,001–30,000 Kshs		−0.188 (0.116)
Income 30,001–50,000 Kshs		0.098 (0.305)
Married		−0.184 (0.137)
Widowed		−0.044 (0.144)
Not cohabiting		−0.241 (0.198)
Children		−0.028 (0.115)
Some primary education		−0.015 (0.085)
Primary completed		0.009 (0.094)
Primary technical training		0.190 (0.303)
Some secondary school		0.026 (0.098)
Secondary completed		0.070 (0.093)
University or higher education		0.126 (0.135)
Constant	0.885 (0.027)***	1.312 (0.241)***
Observations	192	192
R2	0.018	0.137
Adjusted R2	0.013	0.059
Residual Std. Error	0.267 (df =190)	0.261 (df =175)
F statistic	3.573 (df =1; 190)	1.742 (df =16; 175)

*, P<0.1; ***, P<0.01.

Table S12. Distribution of people within the HIV-ST group that use self-test kits to test for HIV.

Services	Control	Treated (HIV-ST)
Public health facility	69	72
At a VCT center	4	2
In a private clinic near me	16	10
With HIV self-test kits gotten from a health facility	1	0
With HIV self-test kits gotten from a Living Goods CHW	0	16
Other	2	2

Total count exceeds the matched sample [96] because participants were allowed to enter multiple services.

Table S13. Impact on testing with CHW (HIV-ST), regression model results.

	Dependent variable (agree scale)
	−1	−2
Treatment two (HIV-ST)	0.157 (0.038)***	0.154 (0.040)***
Age		0.001 (0.002)***
Male		−0.094 (0.070)
Income 10,001–20,000 Kshs		0.035 (0.068)
Income 20,001–30,000 Kshs		−0.124 (0.121)
Income 30,001–50,000 Kshs		0.063 (0.251)
Married		0.076 (0.142)
Widowed		−0.026 (0.149)
Not cohabiting		0.247 (0.195)
Children		−0.145 (0.131)
Some primary education		−0.038 (0.092)
Primary completed		−0.008 (0.100)
Primary technical training		−0.311 (0.313)
Some secondary school		−0.061 (0.104)
Secondary completed		−0.061 (0.097)
University or higher education		−0.182 (0.134)
Constant	0.000 (0.028)	0.068 (0.257)
Observations	192	192
R2	0.081	0.125
Adjusted R2	0.076	0.046
Residual Std. Error	0.265 (df =192)	0.261 (df =177)
F statistic	16.940 (df =1; 192)	1.582 (df =16; 177)

***, P<0.01.

Discussion

The Living Goods Innovation Network serves as a model of applied public health research in a low-resource setting. From an organizational perspective, the Network permits Living Goods to manage the complexities of incorporating new solutions into an existing intervention approach.

Nonetheless, the two studies demonstrate the complexity of carrying out field experiments through existing operational models in terms of time constraints, programmatic alignment, and utilization of evidence. The implications of both innovations for Living Goods and partners is however ambiguous. While linked referral to health facilities is a necessary precursor to drive task-shifting of preventive and curative services at the community level, the technologies and referral technologies tested failed to yield significant improvements. Evidence from these two studies indicates important behavioral and technology barriers to service delivery acceptance and uptake. At the same time, the technology solutions deployed in CLE and HIV-ST did not adequately address community concerns.

The reasons for these gaps will be reflected upon as these innovations are returned to Phase 1 or Phase 2 of the Innovation Process for further consideration. Additional qualitative research may uncover the drivers of these barriers. At the same time, the potential misalignment of testing an HIV referral intervention in the context of community health services requires further reflection. Historically Living Goods focused on high burden MNCH interventions including malaria, pneumonia, and diarrhea. The extension of services to HIV self-testing placed the CHW in a novel role vis-à-vis community perceptions, as well as with health facilities. While the CLE methodology successfully contributed to increased referral follow-up at household level and HIV testing (irrespective of test location) was higher in the HIV-ST treatment group, little is known regarding attitudes and confidence health of facility staff towards CHWs taking on such diagnostic tasks outside the facility.

There are a number of important limitations associated with this study. Firstly, quasi-experimental studies face the challenge of lack of randomization, thus limiting the study’s ability to conclude a causal association between an intervention and an outcome. The adoption of a multi-site approach to the study generally strengthened external validity. Secondly, propensity scoring technique is not the most reliable method to evaluate impact. While we are confident that a good propensity score model was built, it does not grant us the depth of rigor that a randomized treatment assignment would have provided. Accordingly, we are cautious in making definitive impact claims. However, the technique gives us suggestive evidence of the initiatives’ effects on health outcomes. Thirdly, we recognize the inherent limitations of self-report measures. Self-reporting often raise questions concerning measurement validity and reliability, mostly because of the effects of biases, question framing and response bias. While we cannot control for all these effects, all efforts were made to avert these biases including extensive piloting of the survey instruments to identify and limit sources of measurement error, and rigorous training and quality control of enumerators.

Conclusions

The limitations of the Innovation Network platform were identified through the field testing of two mHealth interventions. The closed-loop referral technology linking client to facility proved to be an important functional capability of the testing platform for one of the two ‘use-cases’—iCCM. However, further refinement is needed for HIV self-testing at community level. The acceptability and health-seeking behaviors of community members, as well as perceptions of health facility staff on the appropriate role of CHWs in delivering diagnostic services at the community level were not adequately captured. Further investigation using qualitative insights is required.

Ethical considerations

Throughout this research, we adhered to the ethical principles of research with human subjects. We adhered to informed consent principles by informing participants about the identity of the interviewer, the nature of the research project, their right to choose not to participate, to refuse to answer, to drop out of the study at any point, data confidentiality, and we provided contact information for questions about the study. Once these conditions were met, individuals were invited to participate in the intervention and a written consent was obtained. Finally, we adhered to confidentiality by assigning participants with ID numbers and ensuring that no information presented in any public documents could be traced back to an individual.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Written informed consent was obtained from the patient for publication.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.