Skip to main content
NCBI home page
PLOS Global Public Health logoLink to PLOS Global Public Health
. 2024 Apr 29;4(4):e0002786. doi: 10.1371/journal.pgph.0002786

Improving care pathways for children with severe illness through implementation of the ASPIRE mHealth primary ETAT package in Malawi

Nicola Desmond 1,2,*, Marc Y R Henrion 1,3, Mtisunge Gondwe 1, Thomasena O’Byrne 1,3, Pui-Ying Iroh Tam 1,3, Deborah Nyirenda 1, Louisa Pollock 1,¤, Maureen Daisy Majamanda 4, Martha Makwero 4,5, Marije Geldof 6, Queen Dube 7,8, Chimwemwe Phiri 9, Chimwemwe Banda 10, Rabson Kachala 7, Prof Robert S Heyderman 11, Clemens Masesa 1,3, Norman Lufesi 7, David G Lalloo 3
Editor: Marie A Brault12
PMCID: PMC11057765  PMID: 38683833

Abstract

Providing emergency care in low resource settings relies on delivery by lower cadres of health workers (LCHW). We describe the development, implementation and mixed methods evaluation of a mobile health (mHealth) triage algorithm based on the WHO Emergency, Triage, Assessment, and Treatment (ETAT) for primary-level care. We conducted an observational study design of implementation research. Key stakeholders were engaged throughout implementation. Clinicians and LCHW at eight primary health centres in Blantyre district were trained to use an mHealth algorithm for triage. An mHealth patient surveillance system monitored patients from presentation through referral to tertiary and final outcome. A total of 209,174 children were recorded by ETAT between April 2017 and September 2018, and 155,931 had both recorded mHealth and clinician triage outcome data. Concordance between mHealth triage by lower cadres of HCW and clinician assessment was 81·6% (95% CI [81·4, 81·8]) over all outcomes (kappa: 0·535 (95% CI [0·530, 0·539]). Concordance for mHealth emergency triage was 0.31 with kappa 0.42. The most common mHealth recorded emergency sign was breathing difficulty (74·1% 95% CI [70·1, 77·9]) and priority sign was raised temperature (76·2% (95% CI [75·9, 76·6]). A total of 1,644 referrals out of 3,004 (54·7%) successfully reached the tertiary site. Both providers and carers expressed high levels of satisfaction with the mHealth ETAT pathway. An mHealth triage algorithm can be used by LCHWs with moderate concordance with clinician triage. Implementation of ETAT through an mHealth algorithm documented successful referrals from primary to tertiary, but half of referred patients did not reach the tertiary site. Potential harms of such systems, such as cases requiring referral being missed during triage, require further evaluation. The ASPIRE mHealth primary ETAT approach can be used to prioritise acute illness and support future resource planning within both district and national health system.

Introduction

In 2020, 5 million children under five years old died worldwide [1]. Almost half of these deaths occurred in sub-Saharan Africa, where, despite recent improvements in child mortality, one in twelve children still die before their fifth birthday [2]. The majority of these deaths were secondary to infections, primarily pneumonia, malaria and diarrhoea, and could have been prevented with prompt access to effective healthcare [3–5]. Improving recognition and case management of childhood illness is therefore an important part of the global strategy to reduce preventable child death. The WHO Integrated Management of Childhood Illness (IMCI) programme was designed to address this by improving case management at primary levels. However, implementation challenges and significant limitations in the programme’s scope are well-recognised [6–9].

One important aspect that is not adequately addressed in IMCI care in low resource settings is triage and emergency care: the prioritisation of sick children according to clinical need and the appropriate treatment of the critically ill. Failure to identify, prioritise, and treat children presenting with illness may lead to significant delays in not only accessing but also receiving appropriate care [10]. In fact, the poor quality of care delivered, rather than challenges with access, was estimated to contribute to 60% of global preventable deaths in 2018 [11]. Further, in some low resource settings, such as in Malawi, tertiary hospitals receive referrals from primary facilities where staff have limited experience and capacity to adequately identify, prioritise and treat sick children. The WHO Emergency, Triage, Assessment, and Treatment (ETAT) programme was developed as an adjunct to IMCI within primary settings in low resource contexts to address this gap [12–14].

Mobile health (mHealth) technologies, which have the capacity to overcome geographic, temporal and organisational barriers, and can promote monitoring and alerting systems, data collection, and record maintenance [15], have been used effectively to improve access to and quality of care in low resource health system settings [16]. Importantly, mHealth technologies are increasingly being used to support healthcare worker (HCW) performance, from providing clinical updates, to feedback on practice, learning materials, reminders, decision-making, and adherence to standards and guidelines in care [17]. mHealth has been increasingly recognised as a health systems strengthening tool [18], but there has been comparatively less progress in the evaluation and scale-up of effective mHealth solutions [19]. In settings where a large proportion of delivery at primary care facilities is performed by HCWs with limited exposure to formal medical training, the role of mHealth in supporting decision making may be particularly useful for resilient health system strengthening [20]. To date, this has not been evaluated at scale in Malawi.

In Malawi, despite major gains in child survival in recent years, the under-five mortality rate remains high at 43 per 1,000 live births and falls short of the sustainable development goal of 25 per 1,000 live births by 2030 [21]. The country has only one doctor for every 53,000 patients [22] and many primary health centres (PHCs) face staff shortages. One consequence is that community health workers, known as health surveillance assistants (HSAs), have been reassigned from the community to the clinic, despite the fact that the cadre was originally developed to provide community-based, preventive health services [23]. HSAs now comprise about 30% of the total health workforce in Malawi [24], and are tasked increasingly to provide diagnosis and treatment, including IMCI. Primary level care is generally provided by clinicians: either clinical officers (who receive three years of training) or medical assistants (who receive two years of training). Clinical care is supported by nursing staff, who focus primarily on maternity services at the primary level, and HSAs, who are often required to fill the gaps in service provision. HSAs need to have a minimum of only two years of secondary schooling, and receive only 12 weeks of training [25]. Given human resource challenges, both clinically trained HCW (clinicians, nurses, medical assistants) and non-clinically trained health workers (HSAs, security guards, and other health centre staff, referred to as lower cadre health workers (LCHW)) are involved in informal prioritisation and care delivery at PHCs. This means that the quality of diagnostic services, especially in response to severe illness, is often suboptimal.

The Achieving Sustainable Primary Improvement and Engagement in health (ASPIRE) project was developed in 2013 and implemented from 2013 to 2018 to use mHealth to address primary level care and develop a sustainable approach to ETAT implementation under such resource constraints. Undertaken in close collaboration with partners from the Ministry of Health, the project had six main objectives, of which three are the focus of this paper: 1) to improve the capacity of primary level staff to diagnose and prioritise severe illness consistently; 2) to improve primary level capacity to stabilise patients and make appropriate referral decisions; and 3) to establish a surveillance system to track patients and outcomes.

This paper describes the development and implementation of ASPIRE using mHealth technology to optimise the care delivered to children with severe illness. ASPIRE was undertaken in collaboration with district, national and international stakeholders and acknowledged the need for a health systems strengthening approach to reinforce and sustain progress. We conducted an observational study and report on the key outcomes of a pathways approach using mHealth technology to optimise the response to severe illness in children, including the feasibility and acceptability of implementation of full primary ETAT, the quality of prioritisation by varying cadre of HCW, taking clinician triage outcomes at primary level as the gold standard, and referral impact.

Materials and methods

Study setting

The ASPIRE mHealth primary ETAT project was developed in collaboration with national and district level partners as a pilot and feasibility implementation study. The project was undertaken across two phases within the Blantyre district of southern Malawi. The Blantyre district government health system is comprised of 29 PHCs, and Queen Elizabeth Central Hospital (QECH), functioning as both the secondary level hospital for the district, and as a tertiary hospital for the southern region of Malawi. Blantyre district had a population of 785,000 in 2014, about half over 15 years of age, and the majority of whom are resident in urban and densely populated townships. Each facility selected for the study served a catchment population ranging between 9,900 and 209,000 (personal communication, District Health Office, 2017).

Resilient health systems strengthening and policy-focused approach

In line with a comprehensive and sustainable health systems approach [26], and recognising the human resource constraints, the ASPIRE implementation project developed primary targeted training manuals for both clinicians (full ETAT) and LCHWs (triage only) through the establishment of a national primary ETAT advisory group and a technical working group. Both groups were comprised of members from key national and international stakeholder institutions including the Ministry of Health, WHO Malawi, UNICEF, and emergency and primary health care experts from across institutions in Malawi. The revised primary ETAT manuals have since been adopted as key resources for emergency care within the Ministry of Health and integrated into clinical training.

Study design

This work reports on the implementation research study, with the key focus being to develop a system integrated within the existing primary level health care in Malawi to improve care pathways for children with severe illness. Therefore, in this study, we followed the observational study design, following study designs of other implementation research [27,28]. We used a mixed methods approach with both quantitative and qualitative data collection and analysis with the goal of assessing and understanding the implementation context, process and acceptability (S1 Checklist).

The implementation study itself was designed across two phases: Phase 1 (2013 to 2015) focused on the development and piloting of a digital mobile phone algorithm for primary level triage, and Phase 1b introduced a comprehensive ETAT training package for both clinicians and LCHWs in the five busiest PHCs in Blantyre District and three in Chikwawa District, including Chikwawa District Hospital. Phase 2 (2016 to 2018), on which this paper reports, focused on introducing the full ETAT package in the five PHCs and a further three PHCs in Blantyre District. The study implementation relied on the HCWs already working within the government health system and no additional HCW were recruited.

The ASPIRE mHealth primary ETAT system

In Phase 1, we developed an mHealth primary ETAT algorithm for triage (S1 Fig), in collaboration with D-Tree International, which was based on the WHO recommended ETAT protocol for tertiary-level ETAT. This protocol was developed to take into account both the resource limitations and epidemiology of severe childhood illness in countries such as Malawi [13] and the algorithm was programmed on to Samsung J3 smartphones which are relatively low-cost tools at approximately $100 per unit. Initially, clinic-driven spatial and pathway adaptations were implemented, including separation of children’s outpatients from adults, introduction of consistent definitions of children from 0 to 14 years of age across primary to tertiary settings, identification of a stabilisation room, a three-day training programme on ETAT and the mHealth ETAT system, a process of raising awareness in the broader community and among HCWs, and introducing the Chipatala Robots (Chip) system (meaning: clinic traffic lights), and provision of a package of triage equipment such as thermometers to health centres. All participating PHCs then introduced a three-tiered prioritisation system (emergency (E), priority (P), queue (Q)) using the phones in the children’s clinic. This mHealth approach facilitated movement and faster identification of sick patients as staff were able to easily move between benches to access waiting patients in overburdened clinics. Following prioritisation, patient-carers were given a coloured strip of red, yellow or green laminated card which led them either directly for emergency care (red: emergency), to the front of the consulting room queue (yellow: priority), or to wait for their turn (green: queue). Over the period of the project, we identified only a few cases where a coloured strip was exchanged between patient-carers to bypass the system.

In Phase 2, we introduced and evaluated the full mHealth primary ETAT package with an integrated data surveillance system supported through an in-house data platform. This package, implemented fully in April 2017, included the digital triage approach, the collaborative development of revised ETAT manuals for both HCWs and LCHWs at PHCs nationally, a revised peer and simulation-based training programme (2·5 days), and the development of a minimum treatment package including essential equipment and supplies to support stabilisation and treatment, for all PHCs.

Data monitoring and surveillance to track patient pathways

In parallel, a patient surveillance system was established to monitor patients from initial presentation through referral to tertiary and final outcome (Fig 1). Participants were entered into the surveillance system as they were triaged by mHealth (Point A), which was done by all cadres of staff working in PHCs. In addition to the outcome data from the triage algorithm, data were collected on sex and age of patient with a unique identification number (ID) and clinic, time and date triaged automatically assigned through the algorithm. The patient health record book was stamped with the unique ID number extracted from the phone as well as the triage outcome. Initial triage by mHealth was carried out by LCHWs, predominantly HSAs. After seeing the clinician (Point B1), either a medical assistant or clinical officer, data were entered on clinician triage outcome and outcome of consultation (referral, admitted or sent home) as well as date and time and laboratory test results. If a child was admitted at B1 further data were collected on the outcome post-admission at primary care (Point B3). For all referrals to QECH, either at B1 or B3 referral, diagnosis data were collected and entered retrospectively through a standard referral stamp and patients were tracked to monitor time and date of arrival at the referral centre (Point C). At this stage, data were also collected on means of transport and primary referring centre. For those admitted to hospital, signs and symptoms on admission, where relevant, were documented (Point D1), in addition to laboratory test results, final diagnosis, outcome (discharged, self-discharged, died), and date of discharge (Point D2).

Fig 1. Flowchart of data collection points.

Fig 1

Open in a new tab

Qualitative evaluation

As part of an iterative and context-informed approach [29], we used qualitative research to evaluate the real-life impact and acceptability of the implementation. Patient journey modelling was conducted at baseline with primary facility HCW to identify context-specific barriers and facilitators to diagnosis and treatment and develop context-driven pathway solutions. A total of 70 semi-structured interviews with primary care providers (37) and users (33) were undertaken to assess challenges, acceptability and unintended impacts before and after full mHealth primary ETAT implementation.

Endpoints

We hypothesised that the implementation of primary ETAT through an mHealth algorithm would provide a systematic and timely approach to recognising severe illness in children 0 to 14 years of age at PHCs. Our primary endpoint was defined as the rate of concordance between triage assignment amongst lower cadres of health workers and clinician assessment. The secondary endpoint included the proportion of E, P, and Q referred following consultation, and the proportion of successful referrals, defined as arrival at QECH following clinician referral from the PHC.

Data analysis

Records with invalid patient IDs were removed prior to analysis, as were records with duplicated patient IDs that could not be matched unambiguously between data collection points. Records from the different data collection points were matched by patient IDs to allow patient journey monitoring. Data for the different collection points were extracted as csv files from the database, then loaded into R. Data analysis and visualisation were done in the R environment for statistical computing (v 4·2·2, R Foundation for Statistical Computing, Vienna, Austria).

Concordance was measured as the proportion of triages that were identical between mHealth and clinician triage assessments. We calculated 95% exact binomial confidence intervals and reported results both overall, by facility, by mHealth E, P, and Q triage, and by clinician E, P, and Q triage. While these concordance proportions are easiest to interpret, the overall concordance results are dominated by the concordance among Q triages as these were by far the most frequent triage assessment. For this reason, we also reported results stratified by E, P, and Q triage and we further computed Cohen’s kappa for inter-rater agreement as metrics of concordance. We reported concordance based on the kappa standard suggested by Cohen [30]. We similarly reported proportions and 95% binomial confidence intervals for the frequencies of emergency and priority signs, the number of successful referrals, and the use of ambulance transport from primary to tertiary health centre. Not all participants had recorded mHealth or clinician triage data. For frequency of emergency signs, tabulated numbers are conditional on mHealth triage data having been recorded (denominator 195,151) whereas for referral data, numbers are conditioned on both mHealth and clinician triage (denominator 155,931).

Qualitative data from the evaluation were transcribed and translated into English and entered into NVIVO 11·0 (QSR, Melbourne, Australia) for data management. First order descriptive codes were developed deductively from the topic guides (S1 Table) and inductively from the transcripts. Thematic content analysis with constant comparison was used to explore the data across clinics and pre-and post-phases of the implementation of the project. Second-order codes were then developed to explore factors impacting on acceptability and the intended and unintended consequences of introducing mHealth ETAT to primary-level clinics.

Ethical approval for this study was obtained from the College of Medicine (P·09/16/2021). Written informed consent was obtained from all participants in structured or semi-structured interviews and group discussions. Facility-level consent was obtained for data collection through the mHealth tool and all data were anonymised and analysed in aggregate.

Results

From April 2017 to September 2018, ASPIRE mHealth primary ETAT was implemented across eight PHCs in Blantyre district in southern Malawi. A total of 209,134 children were recorded by ETAT with 204,914 triaged by either mHealth or clinician. Of these, 195,151 had recorded mHealth triage outcome data, 165,694 had recorded clinician triage outcome data, and 155,931 had both. Of children with both an mHealth and clinician triage record, a total of 3,004 referrals were made, representing 1·9% of all PHC presentations with both mHealth and clinician triage. The ETAT data include 518 emergency cases identified through mHealth triage and 212 by clinician triage (out of 165,694 records with clinician triage only v. 123/155,931 records with clinician and mHealth triage and identified as E by both mHealth and clinicians). There were 60,021 presentations assessed as priority cases by mHealth, and 35,752 by clinicians (26,692 identified as P by both; Fig 2). By mHealth assessment, the most common emergency sign was breathing (384/518; 74·1% (95% CI [70·1, 77·9]); Table 1) and the most common priority sign was raised temperature (45,748/60,021; 76·2% (95% CI [75·9, 76·6])).

Fig 2. Triage outcomes by clinic 2017 to 2018.

Fig 2

Open in a new tab

Table 1. Emergency and priority signs across all primary health centres, by mHealth and clinician triage.

Characteristic mHealth triage Clinician triage
No. of cases with sign Proportion of cases with sign (95% CI) No. of cases with sign AND clinician emergency triage Proportion of cases with sign AND clinician emergency triage (95% CI)
Emergency signs N = 518 N = 186a
Breathing 384 74·1% (70·1, 77·9) 77 41·4% (34·2, 48·8)
 Obstructed breathing 203 39·2% (35·0, 43·5) 38 20·4% (14·9, 26·9)
 Blue gums 9 1·7% (0·8, 3·3) 3 1·6% (0·3, 4·6)
 Respiratory distress 174 33·6% (29·5, 37·8) 36 19·4% (13·9, 25·8)
Circulation 1 0·2% (0·0, 1·1) 0 0·0% (0·0, 0·2)
 Capillary refill >3s 1 0·2% (0·0, 1·1) 0 0·0% (0·0, 0·2)
 Weak and fast pulse 0 0·0% (0·0, 0·7) 0 0·0% (0·0, 0·2)
Consciousness 123 23·7% (20·1, 27·7) 44 23·7% (17·7, 30·4)
 Coma 27 5·2% (3·5, 7·5) 11 5·9% (3·0, 10·3)
 Convulsions 97 18·7% (15·5, 22·4) 33 17·7% (12·5, 24·0)
Dehydration 10 1·9% (0·9, 3·5) 3 1·6% (0·3, 4·6)
 Lethargic or unconscious 7 1·4% (0·5, 2·8) 1 0·5% (0·0, 3·0)
 Sunken eyes 10 1·9% (0·9, 3·5) 3 1·6% (0·3, 4·6)
 Very slow skin pinch 3 0·6% (0·1, 1·7) 1 0·5% (0·0, 3·0)
Priority signs N = 60,021 N = 33,633b
Temperature 45,748 76·2% (75·9, 76·6) 19,547 58·1% (57·6, 58·6)
Tiny Baby 5,085 8·5% (8·3, 8·7) 2,472 7·3% (7·1, 7·6)
Trauma 4,192 7·0% (6·8, 7·2) 2,455 7·3% (7·0, 7·6)
Severe Pain 2,501 4·2% (4·0, 4·3) 1,145 3·4% (3·2, 3·6)
Respiratory Distress 1,193 2·0% (1·9, 2·1) 509 1·5% (1·4, 1·6)
Burns 492 0·8% (0·7, 0·9) 246 0·7% (0·6, 0·8)
Severe Pallor 366 0·6% (0·5, 0·7) 93 0·3% (0·2, 0·3)
Restlessness 193 0·3% (0·3, 0·4) 106 0·3% (0·3, 0·4)
Oedema 94 0·2% (0·1, 0·2) 45 0·1 (0·1, 0·2)
Malnutrition 77 0·1% (0·1, 0·2) 36 0·1% (0·1, 0·1)
Poisoning 42 0·1% (0·1, 0·1) 24 0·1% (0·0, 0·1)
Urgent Referral 38 0·1% (0·0, 0·1) 14 0·0% (0·0, 0·1)
Open in a new tab

aNumbers assessed by mHealth and triaged as emergency by a clinician.

bNumbers assessed by mHealth and triaged as priority by a clinician.

Of a total of 3,004 referrals made to QECH across all eight PHCs, 1,644 were recorded as arriving at QECH (successful referrals 54·7% (95% CI [52·9, 56·5]; Table 2). Of the 1,644 who arrived at QECH following referral, 53 were recorded as having been referred by ambulance (3·2%).

Table 2. Referral success by emergency, priority, and queue, and by primary health centres.

No. triageda No. referred No. successfully referred Proportion successfully referred (95% CI) No. used ambulance Proportion used ambulance (95% CI)
Emergency 186 97 52 53·6% (43·2, 63·8) 4 7·7% (2·1, 18·5)
Priority 33,633 2,126 1,216 57·2% (55·1, 59·3) 29 2·4% (1·6, 3·4)
Queue 122,112 781 376 48·1% (44·6, 51·7) 20 5·3% (3·3, 8·1)
Bangwe 23,389 411 232 56·4% (51·5, 61·3) 2 0·9% (0·1, 3·1)
Chileka 13,266 201 81 40·3% (33·5, 47·4) 10 12·3% (6·1, 21·5)
Chilomoni 14,342 206 103 50·0% (43·0, 57·0) 7 6·8% (2·8, 13·5)
Limbe 23,099 969 540 55·7% (52·5, 58·9) 4 0·7% (0·2, 1·9)
Mpemba 12,388 98 44 44·9% (34·8, 55·3) 24 54·5% (38·8, 69·6)
Ndirande 38,243 843 518 61·4% (58·1, 64·7) 6 1·2% (0·4, 2·5)
South Lunzu 18,990 152 67 44·1% (36·0, 52·4) 0 0·0% (0·0, 5·4)
Zingwangwa 12,214 124 59 47·6% (38·5, 56·7) 0 0·0% (0·0, 6·1)
Total 155,931 3,004 1,644 54·7% (52·9, 56·5) 53 3·2% (2·4, 4·2)
Open in a new tab

CI, confidence interval; QECH, Queen Elizabeth Central Hospital.

aClinician triage for participants with both mHealth and clinician triage records (155,931 records with both data).

Overall, concordance between triage on arrival by lower cadres of staff trained and using the mHealth phone algorithm and clinician outcome was 81·6% (127,303/ 155,931; 95% CI [81·4, 81·8]; Fig 3) with kappa values of 0·535 (95% CI [0·530, 0·539]; Fig 4). Concordance for mHealth emergency triage was 0.31 with kappa 0.42.

Fig 3. Concordance proportions between mHealth triage and clinician assessment overall, by E/P/Q triage and by facility.

Fig 3

Open in a new tab

Since proportions depend on a denominator, which are different for mHealth and clinician triages, we distinguish between concordance for mHealth E/P/Q triages and clinician E/P/Q triages. E, emergency; P, priority; Q, queue.

Fig 4. Kappa statistic values for mHealth and clinician triage agreement, overall, by E/P/Q triage and by facility.

Fig 4

Open in a new tab

Since kappa is independent of which triage is considered to be the reference and which the comparator, we do not distinguish between mHealth and clinician E/P/Q in this figure. E, emergency; P, priority; Q, queue.

The intervention was well received at both district and primary health levels because the project used the National ETAT manuals that were finalised through high level policy engagement with the Ministry of Health. A total of 877 HCW and support staff were trained in the mHealth ETAT system both within the implementation clinics and across Malawi health training institutions, including Kamuzu College of Nursing and the Kamuzu University of Health Sciences [31]. We found overall high levels of both patient-carer and provider satisfaction with the implementation of mHealth primary ETAT across all PHCs. Both carers and providers felt that the intervention improved the quality of care and the ability to recognise and respond to severe illness in PHCs, and helped to improve patient flow through the clinic (Table 3).

Table 3. Qualitative feedback on quality of care and patient pathways.

Improved recognition of severe illness ‘I am so thankful because of what has happened today. My baby was identified among others that he was an emergency and he was taken in front of the queue to be seen immediately by the clinician and he is now better’ Carer of an infant attending South Lunzu primary health centre
‘Triage is being done systematically and children with critical illnesses are being identified and treated on time’ Health worker, Bangwe primary health centre
‘Children could be seriously ill and they would faint or maybe die right there (on the queue), but nowadays children who are critically ill are treated immediately compared to others.’ Health worker, Mpemba primary health centre
Improved patient flows ‘At Bangwe we are now working together as a team. It is helping us manage the children so much better. We are seeing them far more quickly than before’ Health worker, Bangwe primary health centre
‘In the past even if you come with a child who is very sick your fellow carers could not give you a chance to go in front of a queue for your child to be helped immediately but now things have improved because when a child is very sick s/he is put in front of a queue’ Carer of an under 5 child attending Ndirande primary health centre
‘There is now improvement, children don’t take long to be attended to’ Health worker, Chilomoni primary health centre
Open in a new tab

Discussion

A key feature of resilient health systems strengthening in low resource settings is a sustainable primary level ETAT system to ensure that good quality emergency care is delivered within PHCs [32]. In a setting with high burden of disease and severe human resource constraints, the mHealth primary ETAT approach that we used improved prioritisation within the triage system, so that patients presenting with severe illness in busy clinics can be recognised with use of a systematic algorithm and managed in a timely fashion. We established that a mHealth triage algorithm can be used by LCHWs and had moderate concordance with clinician triage. We demonstrated that mHealth data can provide a snapshot of the PHC setting, including the emergency and priority signs with which patients present. We showed that mHealth technologies can be used for a dual purpose: for both the development and application of diagnostic and clinical decision-making tools, and the collection of patient pathway surveillance data that can monitor and evaluate the impact on standard of care. Finally, we have shown that mHealth ETAT at primary level was well received amongst health workers (clinicians and LCHWs) and patient-carers, and perceived to improve the quality of care.

Primary health systems are complex environments where vertically-driven health interventions may be effective in the short-term, with increased, externally provided resources, but post-intervention the impact is often less evident, particularly in contexts where the already overburdened health staff are required to adopt additional roles. Rather than increasing workloads in such settings, this mHealth-based primary-level ETAT approach mitigates many of these constraints by creating ownership and facilitating practice from the outset. We did note that the concordance between E, P, and Q were different, that mHealth appeared to err more often on the side of caution with a 31·4% (123 out of 392) mHealth triages confirmed by a clinician (specifically, out of 392 mHealth E triages, 123 were confirmed by clinician triage), but the majority of clinician E triages were captured by mHealth (66·1%, 123 out of 186). In our kappa-adjusted analyses, the agreement between mHealth and clinician triage of E was lower than for P and Q. In a health system with immense resource limitations, we believe the findings demonstrate that the expansion and adoption of primary-level ETAT has the potential to identify children with severe illness faster, and improve care pathways, but that this must be balanced with the potential harm of missed cases at triage.

The relatively low proportion of children who were referred could be due to some emergency referrals not being recorded in the system, as acutely ill patients are likely to be identified immediately and bypass the triage system but we were unable to quantify this. The introduction of equipped stabilisation and treatment facilities in all participating clinics during the implementation study may have led to an increase in those stabilised and sent home following treatment, and may also have contributed to the increase in the number of successful referrals across all PHCs.

Inclusion of a surveillance system at the PHC level is also valuable in providing data to reflect both temporal and environmental impacts on child health. With a real-time surveillance system, the geographic precision of symptom and diagnostic data by PHCs could highlight local disease outbreaks as they occur, enabling district and national level policymakers to make relevant and timely decisions on health financing and respond to needs as they arise. The applications of this type of surveillance system can go beyond response to localised disease outbreaks and be extrapolated to identify longer-term environmental impacts on health as a longitudinal surveillance system.

This study had some limitations to note. This was not a randomised controlled trial but rather implementation research that was responsive to context-driven needs. We therefore do not have control data. Also, ETAT was not operational after hours, and this may contribute to missed or incomplete patient pathways. Participants may also have skipped or bypassed steps in the patient pathway, and data recording issues in a busy clinical care setting may lead to undercounting. While Phase 1 was focused on ETAT development and data were not consistently collected during this phase, successful referral rates were below 40%. As the delivery of care was constantly evolving, we cannot claim that the increase in the proportion of successful referrals from Phase 1 to Phase 2 was solely the result of improved emergency care at PHCs. However, the data support the fact that the implementation of prioritisation, stabilisation, and treatment through a PHC treatment package was feasible and well received. In low resource settings where there is very little clinical capacity, the focus is on stabilisation for emergency cases, followed by referral pathways so that sick patients can be transferred to higher level care in a timely manner. As such, implementation of ETAT within the primary health system led to an increase, from Phase 1 to Phase 2, in the likelihood of those referred reaching tertiary care but rates of successful referral remained less than optimal at 54·7%. Both economic and social barriers to the referral pathway have been identified across sub-Saharan Africa including Malawi[33] and are likely to have impacted here. While we initially aimed to measure referral times from presentation at PHC to the referral clinician through the data monitoring system, our analysis suggested potential problems with data reliability. This could have been due to internet connectivity problems, resetting of digital clocks within the phones, or retrospective input of some data by triage staff. We have thus not included times of presentation either within the PHC or on the referral pathway.

Our implementation approach was developed in close collaboration with district and primary health level stakeholders and national policymakers. This intervention was embedded within the current system and integrated with policy from the outset, to optimise ownership and ensure that sustainable improvements in the provision of quality care can be achieved. Due to the establishment of such close working relationships, we recognised that continued engagement with the system and thus overall sustainability was dependent on the buy-in from the clinical officer in charge of the particular facility, as well as the support of the district level management team. Through recognition of these influencing factors, we emphasise the need for additional strategies to promote sustainability, such as the development of leadership, accountability, and the strengthening of capacity at different levels of the health system [34,35].

Conclusion

The ASPIRE mHealth primary-level ETAT system was successfully implemented in eight of the busiest clinics in urban Blantyre district, southern Malawi, through a collaborative effort between research, implementation, and policy partners, and improved care pathways for children with severe illness in urban Blantyre. The aim from the outset was to ensure a sustainable implementation approach that did not require the recruitment of additional staff but drew on the skills and needs of those working within the government health system. Despite these constraints, the system was responsive and functioned well, highlighting its utility for HCWs of all cadres in these settings. A higher number of emergency cases identified by LCHW through the mHealth algorithm indicates a greater likelihood of timely referral and appropriate treatment, which was one of the goals. That only half of referred patients reached the tertiary site indicates an opportunity for interventions to close the referral gap. We suggest that supporting capacity at the primary level to stabilise and treat patients and improving pathways and linkages from primary through to tertiary for patients requiring referral, are essential components to improved health systems and require further focus, particularly in tackling dropouts along the referral pathway itself. Close attention must also be paid to potential harms of such systems, for example, cases requiring referral being missed during triage.

Supporting information

S1 Checklist. STROBE statement—Checklist of items that should be included in reports of observational studies.

(DOCX)

pgph.0002786.s001.docx (35KB, docx)
S1 Fig. mHealth primary ETAT algorithm.

(DOCX)

pgph.0002786.s002.docx (15.9KB, docx)
S1 Table. Topic guide for semi-structured interviews with parents or carers.

(DOCX)

pgph.0002786.s003.docx (18.2KB, docx)

Acknowledgments

The ASPIRE project was the result of collaborative efforts from many individuals, organisations, and funders. The Meningitis Research Foundation provided additional support throughout the project and we especially thank Vinny Smith, and Rachel Perrin, D-Tree International including Marc Mitchell and Chris Kulanga. The project was conducted in collaboration with the Malawi Government through the Ministry of Health and Population. The development and translation of the primary-level ETAT package for national implementation was the work of a number of individuals who sat on both the Primary ETAT Advisory Group and the Technical Working Group, we acknowledge their effort, commitment, and technical expertise. The District Health Office in Blantyre and the former District Health Officer, Dr Medson Matchaya. Key partners in Blantyre PHCs including all in charges, HSAs and OPD personnel. The A&E and paediatrics departments at Queen Elizabeth Central Hospital. We thank the ASPIRE implementation team and all the patients and their carers who benefited from the system. In memory of Clemens Masesa, Head of Data at Malawi Liverpool Wellcome Trust, 1966–2022.

Data Availability

As data contain potentially identifying or sensitive patient information, an anonymised, de-identified version of the dataset can be made available on request to allow all results to be reproduced. All requests should be directed to the Malawi-Liverpool Wellcome Research Programme (datamanagers_mlw@mlw.mw).

Funding Statement

This study was supported by the Meningitis Research Foundation (CSF 19-17 to ND), the Scottish Government (M/15/H/005 to ND), Irish AID, and Wellcome (206545/Z/17/Z to ND). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

  • 1.Liu L, Oza S, Hogan D, Chu Y, Perin J, Zhu J, et al. Global, regional, and national causes of under-5 mortality in 2000–15: an updated systematic analysis with implications for the Sustainable Development Goals. Lancet 2016, 388(10063):3027–3035. doi: 10.1016/S0140-6736(16)31593-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.UNICEF, WHO, World Bank Group, United Nations: Levels and Trends in Child Mortality. In New York, Geneva, Washington DC: United Nations; 2021.
  • 3.Bhutta ZA, Das JK, Walker N, Rizvi A, Campbell H, Rudan I, et al. Pneumonia Interventions Study G: Interventions to address deaths from childhood pneumonia and diarrhoea equitably: what works and at what cost? Lancet 2013, 381(9875):1417–1429. [DOI] [PubMed] [Google Scholar]
  • 4.Rutherford ME, Mulholland K, Hill PC: How access to health care relates to under-five mortality in sub-Saharan Africa: systematic review. Trop Med Int Health 2010, 15(5):508–519. doi: 10.1111/j.1365-3156.2010.02497.x [DOI] [PubMed] [Google Scholar]
  • 5.Chopra M, Mason E, Borrazzo J, Campbell H, Rudan I, Liu L, et al. Ending of preventable deaths from pneumonia and diarrhoea: an achievable goal. Lancet 2013, 381(9876):1499–1506. doi: 10.1016/S0140-6736(13)60319-0 [DOI] [PubMed] [Google Scholar]
  • 6.Armstrong Schellenberg JR, Adam T, Mshinda H, Masanja H, Kabadi G, et al. Effectiveness and cost of facility-based Integrated Management of Childhood Illness (IMCI) in Tanzania. Lancet 2004, 364(9445):1583–1594. doi: 10.1016/S0140-6736(04)17311-X [DOI] [PubMed] [Google Scholar]
  • 7.Bryce J, Victora CG, Conference Organizing Group: Child survival: countdown to 2015. Lancet 2005, 365(9478):2153–2154. [DOI] [PubMed] [Google Scholar]
  • 8.Nguyen DT, Leung KK, McIntyre L, Ghali WA, Sauve R: Does integrated management of childhood illness (IMCI) training improve the skills of health workers? A systematic review and meta-analysis. PLoS One 2013, 8(6):e66030. doi: 10.1371/journal.pone.0066030 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Walter ND, Lyimo T, Skarbinski J, Metta E, Kahigwa E, Flannery B, et al. Why first-level health workers fail to follow guidelines for managing severe disease in children in the Coast Region, the United Republic of Tanzania. Bull World Health Organ 2009, 87(2):99–107. doi: 10.2471/blt.08.050740 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Desmond NA, Nyirenda D, Dube Q, Mallewa M, Molyneux E, Lalloo DG, et al. Recognising and treatment seeking for acute bacterial meningitis in adults and children in resource-poor settings: a qualitative study. PLoS One 2013, 8(7):e68163. doi: 10.1371/journal.pone.0068163 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Kruk ME, Gage AD, Arsenault C, Jordan K, Leslie HH, Roder-DeWan S, et al. High-quality health systems in the Sustainable Development Goals era: time for a revolution. Lancet Glob Health 2018, 6(11):e1196–e1252. doi: 10.1016/S2214-109X(18)30386-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Ayieko P, Ntoburi S, Wagai J, Opondo C, Opiyo N, Migiro S, et al. A multifaceted intervention to implement guidelines and improve admission paediatric care in Kenyan district hospitals: a cluster randomised trial. PLoS Med 2011, 8(4):e1001018. doi: 10.1371/journal.pmed.1001018 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Gove S, Tamburlini G, Molyneux E, Whitesell P, Campbell H: Development and technical basis of simplified guidelines for emergency triage assessment and treatment in developing countries. WHO Integrated Management of Childhood Illness (IMCI) Referral Care Project. Arch Dis Child 1999, 81(6):473–477. doi: 10.1136/adc.81.6.473 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Tamburlini G, Di Mario S, Maggi RS, Vilarim JN, Gove S: Evaluation of guidelines for emergency triage assessment and treatment in developing countries. Arch Dis Child 1999, 81(6):478–482. doi: 10.1136/adc.81.6.478 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.WHO: Compendium of new and emerging technologies. In. Geneva, Switzerland: World Health Organization; 2011. [Google Scholar]
  • 16.Nigussie ZY, Zemicheal NF, Tiruneh GT, Bayou YT, Teklu GA, Kibret ES, et al. Using mHealth to Improve Timeliness and Quality of Maternal and Newborn Health in the Primary Health Care System in Ethiopia. Glob Health Sci Pract 2021, 9(3):668–681. doi: 10.9745/GHSP-D-20-00685 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Braun R, Catalani C, Wimbush J, Israelski D: Community health workers and mobile technology: a systematic review of the literature. PLoS One 2013, 8(6):e65772. doi: 10.1371/journal.pone.0065772 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Labrique A, B, Vasudevan L, Kochi E, Fabricant R, Mehl G: mHealth innovations as health system strengthening tools: 12 common applications and a visual framework. Global Health: Science and Practice 2013, 1(2):160–171. doi: 10.9745/GHSP-D-13-00031 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.de la Torre-Diez I, Lopez-Coronado M, Vaca C, Aguado JS, de Castro C: Cost-utility and cost-effectiveness studies of telemedicine, electronic, and mobile health systems in the literature: a systematic review. Telemed J E Health 2015, 21(2):81–85. doi: 10.1089/tmj.2014.0053 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Lehmann U: Strengthening human resources for health systems resilience to care for mothers and children. BMC Health Serv Res 2015, 15 Suppl 1:S6. doi: 10.1186/1472-6963-15-S1-S6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.UNICEF WB, UN-DESA Population division: Levels & Trends in Child Mortality Report 2018. In. Geneva, Switzerland: UNICEF; 2018.
  • 22.Mandeville KL, Hanson K, Muula AS, Dzowela T, Ulaya G, Lagarde M: Specialty training for the retention of Malawian doctors: A cost-effectiveness analysis. Soc Sci Med 2017, 194:87–95. doi: 10.1016/j.socscimed.2017.10.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Chikaphupha KR, Kok MC, Nyirenda L, Namakhoma I, Theobald S: Motivation of health surveillance assistants in Malawi: A qualitative study. Malawi Med J 2016, 28(2):37–42. doi: 10.4314/mmj.v28i2.2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Smith S, Deveridge A, Berman J, Negin J, Mwambene N, Chingaipe E, et al. Task-shifting and prioritization: a situational analysis examining the role and experiences of community health workers in Malawi. Hum Resour Health 2014, 12:24. doi: 10.1186/1478-4491-12-24 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Kok MC, Namakhoma I, Nyirenda L, Chikaphupha K, Broerse JEW, Dieleman M, et al. Health surveillance assistants as intermediates between the community and health sector in Malawi: exploring how relationships influence performance. BMC Health Serv Res 2016, 16:164. doi: 10.1186/s12913-016-1402-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.WHO: Framework for health systems development towards universal health coverage in the context of the sustainable development goals in the African region. In.; 2017.
  • 27.Hwang S, Birken SA, Melvin CL, Rohweder CL, Smith JD: Designs and methods for implementation research: Advancing the mission of the CTSA program. J Clin Transl Sci 2020, 4(3):159–167. doi: 10.1017/cts.2020.16 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Harrison MB, Graham ID, van den Hoek J, Dogherty EJ, Carley ME, Angus V: Guideline adaptation and implementation planning: a prospective observational study. Implement Sci 2013, 8:49. doi: 10.1186/1748-5908-8-49 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Gaglio B, Henton M, Barbeau A, Evans E, Hickam D, Newhouse R, et al. Methodological standards for qualitative and mixed methods patient centered outcomes research. BMJ 2020, 371:m4435. doi: 10.1136/bmj.m4435 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Cohen J: A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 1960, 20(1):37–46. [Google Scholar]
  • 31.Majamanda MD, Joshua Gondwe M, Makwero M, Chalira A, Lufesi N, Dube Q, et al. Capacity Building for Health Care Workers and Support Staff in Pediatric Emergency Triage Assessment and Treatment (ETAT) at Primary Health Care Level in Resource Limited Settings: Experiences from Malawi. Compr Child Adolesc Nurs 2021:1–16. doi: 10.1080/24694193.2021.1916127 [DOI] [PubMed] [Google Scholar]
  • 32.WHO: Building health systems resilience for universal health coverage and health security during the COVID-19 pandemic and beyond. In. Geneva: WHO; 2021. [Google Scholar]
  • 33.Kaunda W, Umali T, Chirwa ME, Nyondo-Mipando AL: Assessing Facilitators and Barriers to Referral of Children Under the Age of Five Years at Ndirande Health Centre in Blantyre, Malawi. Glob Pediatr Health 2021, 8:2333794X211051815. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Marsh RH, Plyler C, Miller M, Klar R, Adeiza M, Wachekwa I, et al. Facing COVID-19 in Liberia: Adaptations of the Resilient and Responsive Health Systems Initiative. Ann Glob Health 2021, 87(1):101. doi: 10.5334/aogh.3245 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.McAlearney AS: Using leadership development programs to improve quality and efficiency in healthcare. J Healthc Manag 2008, 53(5):319–331; discussion 331–312. [PubMed] [Google Scholar]
PLOS Glob Public Health. doi: 10.1371/journal.pgph.0002786.r001

Decision Letter 0

Marie A Brault

31 Oct 2023

PGPH-D-23-01714

Improving care pathways for children with severe illness through implementation and mixed methods evaluation of a primary level mHealth ETAT package in Malawi: the ASPIRE project

PLOS Global Public Health

Dear Dr. Iroh Tam,

Thank you for submitting your manuscript to PLOS Global Public Health. After careful consideration, we feel that it has merit but does not fully meet PLOS Global Public Health’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Editor comments:

  • The reviewers and I agree that this manuscript addresses important questions in emergency care in Malawi, with implications for triage implementation in other resource-constrained settings.

  • However, as noted by the reviewers below there are a number of areas in the methods, results, and discussion that need to be addressed. Greater clarity on the objectives of the study, the measures and methods used to assess and compare the ASPIRE approach with clinician triage, and outcomes are needed.

Please submit your revised manuscript by Dec 15 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at globalpubhealth@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pgph/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Marie A. Brault, PhD

Academic Editor

PLOS Global Public Health

Journal Requirements:

1. Please provide separate figure files in .tif or .eps format.

For more information about figure files please see our guidelines:  LINK

https://journals.plos.org/globalpublichealth/s/figures 

https://journals.plos.org/globalpublichealth/s/figures#loc-file-requirements 

Additional Editor Comments (if provided):

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Does this manuscript meet PLOS Global Public Health’s publication criteria? Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe methodologically and ethically rigorous research with conclusions that are appropriately drawn based on the data presented.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: No

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available (please refer to the Data Availability Statement at the start of the manuscript PDF file)?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS Global Public Health does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Dear Authors,

The paper is very interesting and focusses new concepts and practices in the health and medical field. The paper is good but some adjustments are needed. Please see below some of the issues that need your attention.

1. Consider adjusting the title of the paper. In its current state it does not really capture the main objective of the paper. The paper does not examine improvements on care pathways for children with severe illness but rather examines the implication of primary level mHealth ETAT package through the ASPIRE project in Malawi and how that has resulted in improved care pathways for children with severe illness in the broader sense (see line 142 – 146) . You could modify the title to read like “Implication of The ASPIRE mHealth primary ETAT approach on improving care pathways for children with severe illness in Malawi” or something along those lines that clearly elucidates the main purpose and objective of the study.

2. The paper does not have a clear study design. From lines 82-84, 91-97, 142-146 it is clear that the paper is trying to address the issue of failure to adequately optimize responses and recognition as well as management of childhood illnesses that affect quality of childhood illness care through the ASPIRE mHealth Primary ETAT system. If this is the case then, the study design should focus on this and not the implementation phases of the ASPIRE mHealth Primary ETAT system. Not that the implementation phases are not relevant, the implementation phases can be presented as part of the introduction with the study design focusing on the main approach of the study that guides the overall quantitative and qualitative analysis process of the study.

3. In line 400-401 it is mentioned that the study is an implementation research. Expanding or providing an elaborate justifications for the selection of this approach as well as taking into account the relevance of the various components of implementation research should suffice as an adequate study design if done within the context of the study. Consider expounding on this and use it as the overall study design.

Reviewer #2: This is an important paper on a critical area for primary care in Malawi, a high burden context. However the way the results are currently presented and the focus on the discussion and conclusion on successful implementation rather than the results achieved, particularly in terms of missed Emergency cases, is concerning and requires significant revisions. I have made a number of suggestions as follows.

Abstract, conclusion: is a Kappa of 0.535 “moderate concordance”? Figure 2 also indicates low concordance of emergency cases (518 mHealth, 212 clinician, and only 123 with both mHealth and clinician triage outcome as Emergency). Indeed, the emergency cases should be the focus of your results given the ETAT intervention and bad health outcomes being concentrated in emergency cases. Please report the mHealth E results in the abstract (from Figure 4, concordance is 0.31, and from Figure 5, Kappa is 0.42), and modify your conclusion accordingly both in the abstract and the main paper.

Also is Kappa appropriate given you have a gold standard (Clinician triage) – might calculating sensitivity and specificity be more appropriate?

“lower cadre health workers” – please define which cadres you mean by “lower cadre”. Perhaps ‘non-physician clinician’ would be more appropriate?

Introduction, good background provided, though please can you explain in one sentence how exactly ASPIRE uses mHealth? this sentence near the end of the introduction could be explained: “using mHealth technology to optimise the response to severe illness in children”. Related to this the third paragraph in your background section on mHealth could more specifically cover what exactly you are doing with mHealth in ASPIRE.

Introduction, end of 4th paragraph: “However, HSAs need to have a minimum of only two years of secondary schooling, and receive only 12 weeks of training, which means that the quality of diagnostic services, especially in response to severe illness, is often suboptimal.” please provide reference(s) to back up these points.

Methods, line 220: “triaged by mHealth” – which cadres of health worker were doing the triaging with the mHealth tool? Please make it clear that health workers were inputting data into the mHealth tool to do the triage – also how accurate was this data entry into the smart phones? In your discussion (lines 418-419) you say: “retrospective input of some data by triage staff” – this is quite concerning and suggests the triage was not always done correctly. Please explain and add an estimate to the paper of how often this may have occurred.

Figure 2: please add points D1 and D2.

Methods, qualitative evaluation: please add more details on your facility observations – how many observations were done? over what time period? what was recorded? and how was it analysed? and used in your study?

Methods, qualitative analysis – please add reference for the method you used. Also why did you use grounded theory rather than thematic analysis? was there no relevant prior qualitative or theoretical work you could build on? and did your qualitative analysis inform the design or implementation of your interventions at all? if so, how specifically? From the limited qualitative results presented it seems not as only positive things are mentioned and nothing about the potential for the mHealth intervention to miss emergency cases that Clinicians would catch for example, or over-diagnose emergency cases and the implications of that? Were these issued explored in your qualitative evaluation? Please add your topic guide as supplementary material. Also was your topic guide iterated at all from baseline to implementation phases based on emerging quantitative results?

Results – Figure 3 - the log scale makes it difficult to visually compare mHealth and Clinician triage outcomes. Please redo the figure as separate Figures 3A, B and C for Emergency, Priority, and Queue respectively, with mHealth and Clinician side by side for each clinic, with normal y-axis scales (not log). Could exclude “All” so the scale is not too large, and you just show for each clinic separately.

Results – Table 1 – please add columns to show Clinician results as well, so that it’s possible for the reader to compare mHealth and Clinician triage by emergency and priority sign.

Results – Table 2 – please clarify the denominator – it’s not clear why there are 186 Emergency triaged for example, when the number of Emergency triage is 123 for both mHealth and Clinician, 212 for Clinician, and 518 for mHealth.

Results: “Since kappa is independent of which triage is considered to be the reference and which the comparator, we do not distinguish between mHealth and clinician E/P/Q in this figure.” – is this appropriate given Clinician triage is your gold standard? might it be more appropriate to report sensitivity and specificity of mHealth in relation to Clinician triage? This would also enable a clearer picture to be seen of under and overdiagnosis of Emergency cases by mHealth, which should be a focus of your paper. From your discussion: “but the majority with clinician triage of E were captured by mHealth (66·1%, 123 out of 186)” it’s apparent that 34% of E cases were missed by mHealth, which is surely a major shortcoming and could have serious consequences if the mHealth intervention was more widely adopted?

Discussion – as per above comments, please also discuss under- and over-diagnosis of Emergency cases by the mHealth intervention in comparison to Clinician triage, and the implications of this. Perhaps also considering the coverage of Clinician triage in routine settings and the potential for mHealth to cover more patients as a counterpoint? what might the net benefits, or harms, of the mHealth intervention be?

Discussion, first paragraph: “the implementation of ETAT through a mHealth algorithm approach increased the proportion of successful referrals from primary to tertiary.” – this increase is not shown in your results – please add to your results and explain how it was calculated.

Discussion, last sentence: “we emphasise the need for additional strategies to promote sustainability, such as the development of leadership, accountability, and the strengthening of capacity at different levels of the health system.” – this sentence and this whole paragraph is written on the assumption that the intervention should continue. Should this intervention be sustained though given it could potentially lead to harm by missing Emergency cases? (see above comments)

Conclusion – The conclusion is currently written as if the intervention were a success purely because it was implemented – the results of the implementation need to be the focus of your conclusion - please revise in light of the above comments, considering the potential for harm of the intervention.

Reviewer #3: I am not sure if this was a study or a program. One of the major purpose and utility of a scientific article is its validity and reliability. For this, a detailed description of all variables and the tests performed needed to explain with greater detail. It is also essential to talk about the tools. For qualitative data, a detail description of the tools, purpose, themes and analysis technique is needed. The submitted manuscript lacks all of these. It is rather a detailed description of an intervention and program data in the current form. I suggest, the authors should revise it according to standard format of a scientific public health article.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

Do you want your identity to be public for this peer review? If you choo

PLOS Glob Public Health. doi: 10.1371/journal.pgph.0002786.r003

Decision Letter 1

Marie A Brault

1 Feb 2024

PGPH-D-23-01714R1

Improving care pathways for children with severe illness through implementation of the ASPIRE mHealth primary ETAT package in Malawi

PLOS Global Public Health

Dear Dr. Iroh Tam,

Thank you for submitting your manuscript to PLOS Global Public Health. After careful consideration, we feel that it has merit but does not fully meet PLOS Global Public Health’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Mar 17 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at globalpubhealth@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pgph/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Marie A. Brault, PhD

Academic Editor

PLOS Global Public Health

Journal Requirements:

Additional Editor Comments (if provided):

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: (No Response)

**********

2. Does this manuscript meet PLOS Global Public Health’s publication criteria? Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe methodologically and ethically rigorous research with conclusions that are appropriately drawn based on the data presented.

Reviewer #1: Yes

Reviewer #2: No

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

4. Have the authors made all data underlying the findings in their manuscript fully available (please refer to the Data Availability Statement at the start of the manuscript PDF file)?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS Global Public Health does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors have addressed all my comments. I have no other issues regarding the paper.

Reviewer #2: Thanks for your revision. Unfortunately it appears to have been rushed as many of my comments from my first review were not addressed. I hope you submitted the final tracked changes version as many of the line numbers in your response have no changes. In fact as you'll see below I've stopped half way through as it seems most the revisions were not done - please check thoroughly and re-submit and I will review then.

My comment about focusing on the Emergency (E) cases was not addressed: “Please report the mHealth E results in the abstract (from Figure 4, concordance is 0.31, and from Figure 5, Kappa is 0.42), and modify your conclusion accordingly both in the abstract and the main paper.”

There needs to be a focus on the Emergency cases as they have the most severe outcomes i.e. getting them wrong has the most harmful consequences. Please highlight in the abstract that concordance was only 0.31 “fair” for Emergency cases, and that this has implications in that XX% of emergency cases requiring referral may be missed if mHealth triage is done instead of clinician assessment.

Your response below to my comment below is also not clear – there are no tracked changes in the line numbers indicated and you haven’t explained the mHealth intervention as far as I can tell – what does it add to existing prioritisation methods? – please quote the added sentences in your response. To clarify I was asking you to explain what your sentence “using mHealth technology to optimise the response to severe illness in children” means, not repeat that sentence in your paper:

“ Introduction, good background provided, though please can you explain in one sentence how exactly ASPIRE uses mHealth? this sentence near the end of the introduction could be explained: “using mHealth technology to optimise the response to severe illness in children”. Related to this the third paragraph in your background section on mHealth could more specifically cover what exactly you are doing with mHealth in ASPIRE.

Author comments: We have included this sentence [L157-158] and added some background on the third paragraph related mHealth and linking this to ASPIRE [L112-125-126]. “

Another comment that is not in the revised paper (there is no tracked changes in lines 254-255 and I searched as well is): “Author comments: We have clarified that all cadres of staff working in PHCs, including security guards and primarily HSAs, were conducting the triage with the mHealth tool [L254-L255].

Your next response is also not done in the paper, and in any case to cover all of the points in your response (copied below) would require more than one added line of text (you say L457-458):

“With regards to the concern raised by the reviewer regarding the imprecise nature of triage in the study setting, we acknowledge this as a limitation. Given the management of large numbers of patients, and that the goal of this implementation study was to integrate processes into the existing health system, staff had a heavy workflow and it was not always possible for them to input data in real time. The fundamental basis of implementation research is accepting that there is a trade-off between fidelity to the protocol and the pragmatism of real world conditions. This therefore results in issues such as that described above. We have explained this further in the text [L457-458]. Given the implementation nature of this work, we do not have estimates as to how often this may have occurred.”

I’m going to stop here as the revision needs to be thoroughly checked and re-sent, and then will require re-review.

Last sentence of conclusion needs re-writing to be clear as it’s very long, and the new additions are not clear – probably best to split into at least two sentences e.g. the original last sentence + a new additional sentence: “We suggest that supporting capacity at the primary level to stabilise and treat patients and improving pathways and linkages from primary through to tertiary for patients requiring referral, are essential components to improved health systems and require further focus, particularly in tackling dropouts along the referral pathway itself. Close attention must also be paid to potential harms of such systems, for example cases requiring referral being missed during triage.”

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

Do you want your identity to be public for this peer review? If you choose “no”, your identity will remain anonymous but your review may still be made public.

For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Tim Colbourn

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLOS Glob Public Health. doi: 10.1371/journal.pgph.0002786.r005

Decision Letter 2

Marie A Brault

11 Mar 2024

PGPH-D-23-01714R2

Improving care pathways for children with severe illness through implementation of the ASPIRE mHealth primary ETAT package in Malawi

PLOS Global Public Health

Dear Dr. Iroh Tam,

Thank you for submitting your manuscript to PLOS Global Public Health. After careful consideration, we feel that it has merit but does not fully meet PLOS Global Public Health’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

Editor comments:

  • The reviewer and I thank the authors for making their revisions clearer. The manuscript is much-improved, but there are few areas where additional edits are needed before this manuscript can be accepted.. 

  • In addition to the wording suggestions below, please pay special attention to Reviewer 2's question concerning the emergency triage aspect of the ETAT tool (points 2 and 3 below). 

==============================

Please submit your revised manuscript by Apr 10 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at globalpubhealth@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pgph/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Marie A. Brault, PhD

Academic Editor

PLOS Global Public Health

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments (if provided):

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: (No Response)

**********

2. Does this manuscript meet PLOS Global Public Health’s publication criteria? Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe methodologically and ethically rigorous research with conclusions that are appropriately drawn based on the data presented.

Reviewer #2: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available (please refer to the Data Availability Statement at the start of the manuscript PDF file)?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: No

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS Global Public Health does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: Thanks for clearly showing all tracked changes and for your additional revisions to the paper, which is much improved. I still have a few comments below related to prior points and your revisions.

1. Line 399-400 “mHealth appeared to err more often on the side of caution with 31.4% (123 out of 390) concordance between mHealth and clinician triage”. Should the end of this sentence say “between clinician triage and mHealth” to make it clear that the 123 is the clinician triage and 390 is the mHealth? Or even say explicitly that the 123 is clinician triage and the 390 mHealth? Otherwise it could be interpreted as mHealth identifying 123 of the 390 identified by clinicians.

2. Author response: “While we take your point and considered your comment about focusing on emergency cases, the purpose of ASPIRE was to develop and evaluate whether the mHealth primary ETAT triage algorithm can be used by lower cadre health care workers and improve care pathways, and this focus on emergency triage was neither the intent nor focus of ASPIRE”

Given ETAT is Emergency Triage Assessment and Treatment, why is “this focus on

emergency triage [i.e. the ET in ETAT] neither the intent or focus of ASPIRE?” ?

3. Following from point 2 the above, I still think the emergency results should be a focus in the main paper, abstract, and conclusion. Also because it is the emergency cases that contribute the most to the bottom line of poor health outcomes and mortality, which as you explain in your introduction section should be the main focus of interventions in high-mortality contexts like Malawi. It still remains unclear to me from the results in your paper that expanding ETAT to lower-cadre health workers using mHealth would have net benefits on mortality in Malawi (I’m also saying this as someone who has been involved in child health research in Malawi for 17 years). Although Clinicians with 3 year BSc training may not be gold standards, they are surely more accurate than the lower-cadre health workers using the mHealth.

4. Author comments: We have clarified that all cadres of staff working in PHCs, including security guards and primarily HSAs, were conducting the triage with the mHealth tool with the sentence “Given human resource challenges, both clinically trained HCW (clinicians, nurses, medical assistants) and non-clinically trained health workers (HSAs, security guards, and other health centre staff, referred to as lower cadre health workers (LCHW)) are involved in delivering care at PHCs.” [L130-133].

Please change the end of this sentence from “are involved in delivering care in PHCs” to “were conducting triage with the mHealth tool” to clarify this.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

Do you want your identity to be public for this peer review? If you choose “no”, your identity will remain anonymous but your review may still be made public.

For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLOS Glob Public Health. doi: 10.1371/journal.pgph.0002786.r007

Decision Letter 3

Marie A Brault

1 Apr 2024

Improving care pathways for children with severe illness through implementation of the ASPIRE mHealth primary ETAT package in Malawi

PGPH-D-23-01714R3

Dear Dr Iroh Tam,

We are pleased to inform you that your manuscript 'Improving care pathways for children with severe illness through implementation of the ASPIRE mHealth primary ETAT package in Malawi' has been provisionally accepted for publication in PLOS Global Public Health.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they'll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact globalpubhealth@plos.org.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Global Public Health.

Best regards,

Marie A. Brault, PhD

Academic Editor

PLOS Global Public Health

***********************************************************

Reviewer Comments (if any, and for reference):

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Checklist. STROBE statement—Checklist of items that should be included in reports of observational studies.

    (DOCX)

    pgph.0002786.s001.docx (35KB, docx)
    S1 Fig. mHealth primary ETAT algorithm.

    (DOCX)

    pgph.0002786.s002.docx (15.9KB, docx)
    S1 Table. Topic guide for semi-structured interviews with parents or carers.

    (DOCX)

    pgph.0002786.s003.docx (18.2KB, docx)
    Attachment

    Submitted filename: ASPIRE_overview_PLoS GPH-response.docx

    pgph.0002786.s004.docx (30KB, docx)
    Attachment

    Submitted filename: ASPIRE_overview_PLoS GPH-response2.docx

    pgph.0002786.s005.docx (50KB, docx)
    Attachment

    Submitted filename: ASPIRE_overview_PLoS GPH-response3.docx

    pgph.0002786.s006.docx (22.4KB, docx)

    Data Availability Statement

    As data contain potentially identifying or sensitive patient information, an anonymised, de-identified version of the dataset can be made available on request to allow all results to be reproduced. All requests should be directed to the Malawi-Liverpool Wellcome Research Programme (datamanagers_mlw@mlw.mw).

    Articles from PLOS Global Public Health are provided here courtesy of PLOS

    RESOURCES