Performance of Health Workers Using an Electronic Algorithm for the Management of Childhood Illness in Tanzania: A Pilot Implementation Study

Abstract

In low-resource settings, where qualified health workers (HWs) are scarce and childhood mortality high, rational antimicrobial prescription for childhood illnesses is a challenge. To assess whether smartphones running guidelines, as compared with paper support, improve consultation process and rational use of medicines for children, a pilot cluster-randomized controlled study was conducted in Tanzania. Nine primary health-care facilities (HFs) were randomized into three arms: 1) paper algorithm, 2) electronic algorithm on a smartphone, and 3) control. All HWs attending children aged 2–59 months for acute illness in intervention HFs were trained on a new clinical algorithm for management of childhood illness (ALMANACH) either on 1) paper or 2) electronic support; 4 months after training, consultations were observed. An expert consultation was the reference for classification and treatment. Main outcomes were proportion of children checked for danger signs, and antibiotics prescription rate. A total of 504 consultations (166, 171, and 167 in control, paper, and phone arms, respectively) were observed. The use of smartphones versus paper was associated with a significant increase in children checked for danger signs (41% versus 74%, P = 0.04). Antibiotic prescriptions rate dropped from 70% in the control to 26%, and 25% in paper and electronic arms. The HWs–expert agreement on pneumonia classification remained low (expert's pneumonia identified by HWs in 26%, 30%, and 39% of patients, respectively).Mobile technology in low-income countries is implementable and has a potential to improve HWs' performance. Additional point-of-care diagnostic tests are needed to ensure appropriate management. Improving the rational use of antimicrobial is a challenge that ALMANACH can help to take up.

Introduction

In the context of the global rapid spread of resistance to antimicrobials, there is an urgent need to reduce the overuse of these life-saving medicines worldwide. In low-income countries, where childhood mortality due to infectious diseases is high,¹ rational use of antimicrobials in children is a challenge for health workers (HWs) lacking accurate tools and skills to make appropriate diagnoses.² Providing HWs with guidelines allowing them to identify patients in need or not of antibiotics and antimalarials may help them to rationalize their antimicrobials prescription. For these guidelines to reach their goal, high HWs' compliance is necessary. The current reference standard for the management of children in primary care setting in low-income countries is the Integrated Management of Childhood Illness (IMCI). In countries where IMCI was implemented, an important overprescription of antimicrobials remain, due to two main limitations. First, the IMCI algorithm, designed in the 1990s to tackle the high childhood mortality,³ had to rely on poorly specific diseases classification that leads to overprescription of both antibiotics and antimalarials. Antimalarials overprescription that resulted from the presumptive treatment recommended for all febrile children in endemic areas has recently been addressed by IMCI that now recommends test-based malaria treatment.^4,5 However, regarding bacterial diseases, both the low specificity of the criteria used to classify pneumonia^6–8 and the lack of clear recommendations for the management of nonmalaria febrile illnesses prompt clinicians to prescribe antibiotics to be “on the safe side.”^5,9,10 The second limitation lies in the low compliance of HWs to IMCI recommendations.^7,11

To address these limitations, a new algorithm for the management of sick children aged 2–59 months (algorithm for management of childhood illnesses [ALMANACH]) was developed, both on paper and electronic support.¹² Derived from IMCI, ALMANACH integrates evidence-based diagnostic procedures targeting the main infectious diseases, including the use of a rapid diagnostic test for malaria diagnosis (mRDT), a urine dipstick for the classification of urinary tract infection (UTI), and abdominal tenderness for the classification of typhoid fever. For children with fever or history of fever, a new disease classification, namely “likely viral infection,” is proposed at the end of the child's assessment when no cause of fever has been identified, and bacterial disease and malaria have been ruled out, which encourages HWs to withhold unnecessary antibiotic prescription. The electronic version of ALMANACH was developed using a set of open-source software from Open Data Kit¹³ and OpenMRS,¹⁴ to be run on android mobile devices (smartphones and tablets).¹² The impact of ALMANACH's use on children's health outcome and antibiotic prescriptions has been demonstrated in controlled conditions in Tanzanian primary care health facilities (HFs). The use of the new algorithm achieved a better cure rate at day 7 and a dramatically lower rate of antibiotic prescription than routine practice (15% with ALMANACH versus 84% in the control; P < 0.001).¹⁵ Used in controlled conditions, ALMANACH was thus able to improve the rational use of antibiotics in a safely manner.

In this article, we present the results of a comparative study to assess the impact of the electronic ALMANACH on HWs' performance and antimicrobials prescription when used in programmatic conditions. Performance, in terms of completeness of assessment, appropriateness of diseases classification, and treatment prescription, was assessed for HWs trained on and provided with paper or electronic ALMANACH, and compared with that for HWs without specific training or tool.

Materials and Methods

Study setting.

The study took place between April and December 2011 in Dar es Salaam, the economic capital of the United Republic of Tanzania, with 4,360,000 inhabitants in 2012.¹⁶ Dar es Salaam has three municipalities with a public health system organized in four levels (reference hospitals, district hospitals, health centers, and dispensaries). Tanzania adopted the IMCI protocols as national policy in 1998.¹⁷ In 2011, Tanzania was in the process of mRDT implementation at national scale, but the administrative region of Dar es Salaam was not yet covered. For this study, we conveniently selected nine public HFs (three health centers and six dispensaries) that had been using mRDT routinely since a previous research project (IMALDIA 2006–2009).⁵ The three selected HFs per municipality were randomly allocated to 1) electronic ALMANACH (electronic arm), 2) paper ALMANACH (paper arm), and 3) standard practice (control arm), by a block stratified randomization.¹⁸ In each arm, a total of three HFs were thus enrolled (Table 1).

Table 1.

Characteristics of the nine public HFs involved in the survey and detailed list of HWs trained and supervised

HF	Arm	District	Type	Total number of HWs	Number of HWs in OPD	Number of HWs trained	Number of HWs supervised
1	Control	Kinondoni	HC	17	6	–	–
2	Control	Ilala	D	8	7	–	–
3	Control	Temeke	D	10	6	–	–
4	Paper	Kinondoni	D	5	4	4	4
5	Paper	Ilala	D	9	6	6	5†
6	Paper	Temeke	HC	12	8	8	8
7	Electronic	Kinondoni	D	17	11	11	9†
8	Electronic	Ilala	HC	13	8	7*	6†
9	Electronic	Temeke	D	16	12	12	11†

D = dispensary; HC = health center; HF = health facility; HW = health worker; OPD = outpatients department.

^*Clinicians in OPD not trained: HF8: one transferred out.

^†Clinicians trained not supervised: HF5: one transferred out; HF7: one maternity leave and one long leave; HF8: one long leave; HF9: one transferred out.

Intervention: ALMANACH pilot implementation.

In April and May 2011, a total of 48 HWs (18 in the paper, 30 in the electronic arm) involved in children's management in the six enrolled intervention HFs received a 2-day training. The training was conducted by the study team (C. Rambaud-Althaus, A. Shao, J. Samaka, N. Swai, S. Perri), with the support of external experts (acknowledged), in both English and Kiswahili (both Tanzanian official languages). Clinicians from both intervention arms were trained together, through lectures and practical exercises, on the problem of antibiotic resistance and on the ALMANACH's content and rationale. Each participant received his/her personal copy of the ALMANACH booklet. In the afternoon of the second day, HWs were trained separately on the use of ALMANACH, with clinical case studies, on either the booklet or the smartphone application, according to their HF's arm.

In both arms, clinicians received a paper version of ALMANACH,¹² written in Kiswahili. The electronic version contains all and only the information available in the booklet, both in English and Kiswahili.

After the training, three to five smartphones (according to the number of working places) were brought in the electronic arm HFs. During the following 3 months, one day of face-to-face supervision was delivered in the six intervention HFs to 43 HWs (17 paper, 26 electronic) by one of the study supervisors (C. Rambaud-Althaus, A. Shao, J. Samaka, N. Swai, S. Perri) while the clinician was attending real patients. After all HWs had been trained and supervised, monthly supervision visits were carried out in the six HFs. Throughout the study period, the study team made sure that antimalarials and mRDTs were always available both in intervention and control HFs.

Study design and subjects.

A cross-sectional comparative survey was conducted in September and October 2011 in the nine involved HFs to assess HWs' performance 4 months after the initial training and face-to-face supervision. All consecutive children aged 2–59 months presenting during the working hours of the study team were recruited if they fulfilled the following inclusion criteria: 1) first consultation for an acute medical problem, 2) absence of severe illness requiring immediate life-saving procedures, 3) main complaint(s) not related to injury or trauma, and 4) written informed consent by the caretaker. Because they valued the intervention and the access to an expert consultation, none of the caretaker of eligible children refused to participate. A total of 150 children were enrolled per arm, 50 per HF. This sample size allowed the detection of a 25% point difference in the proportion of patients checked for danger signs, from an expected proportion of 90%, with 80% power and 5% significance, assuming 0.02 intracluster correlation.

Procedures.

After oral consent was given by the HW and written consent by the child's caretaker, the consultation was observed by an external study–trained clinical officer (hereafter referred to as the observer) who recorded details of the HW's assessment of the child, including symptoms and signs found, laboratory investigation(s) performed, classification(s) reached, advice to caretakers, and treatment(s) prescribed, without intervening. Following the HW's consultation, each enrolled child underwent a second full clinical assessment by an experienced clinician following strictly the electronic ALMANACH (expert). During the study, the classification, treatment, and advice of the expert were the basis for the final treatment of the child. The study protocol and related documents were approved by the Ethikkommission beider Basel in Switzerland, and by the Institutional Review Board of the Ifakara Health Institute and the National Institute for Medical Research Review Board in Tanzania.

Data collection, management, and analyses.

For each enrolled child, the observer completed a standardized paper case report form (CRF) where he/she recorded HW and child's demographics, tasks performed by the HW, and findings. The CRFs were adapted from the Health Facility Survey checklist questionnaire developed by the World Health Organization (WHO).¹⁹ Data from the expert consultation were collected through the smartphone-run electronic ALMANACH. All paper-collected data were double entered in EpiInfo (version 3.5.3, Center for Disease Control, Atlanta, GA). Data cleaning, management, and analyses were conducted using Stata software version 10 (StatCorp LP, College Station, TX). A set of indicators was derived from the Health Facility Survey tool in IMCI multicountry evaluation¹⁹ to assess HWs' performance outcomes in terms of 1) completeness of assessment, 2) accuracy of disease classification, and 3) appropriateness of antimicrobial prescription, as compared with the expert assessment. The list of indicators and their definition is presented in Table 2. The unit of analysis was the patient. Proportions were compared by calculating odds ratios using a multilevel mixed-effects logistic regression model to account for HF clustering. Adjusted risk ratios (aRR) were calculated from the fitted values for each cell of the 2 × 2 tables, for each intervention arm (paper and electronic) as compared with the control arm, and then to compare electronic to paper arm. Venn diagrams were computed using Venn Diagram Plotter software²⁰ to illustrate the accuracy of the HWs' bacterial and viral classifications as compared with the expert's classifications.

Table 2.

List and definition of HWs' performance outcomes indicators

Performance indicator	Definition
Completeness of assessment
Assessment tasks requested for all children
Danger signs	Proportion of children asked for all of the three following danger signs: “difficulty to drink,” “vomits everything,” and “history of convulsion”
Main symptoms	Proportion of children asked for all of the three following main symptoms: “fever,” “cough,” and “diarrhea”
Palmar pallor	Proportion of children checked for palmar pallor
Index of integrated assessment	Proportion of the 11 assessment tasks required for each child actually performed (three danger signs, three main symptoms, other problem, weight, temperature, vaccination card, palmar pallor; total = 11 tasks)
Assessment tasks requested for a subset of children
Respiratory rate	Proportion of children with fever and cough who had their respiratory rate counted
Malaria test	Proportion of febrile children who had a malaria rapid test ordered for
Classifications appropriateness
Children appropriately classified	Proportion of children with all classifications of HWs matching the expert classifications
Bacterial classifications identified	Proportion of children with at least one expert classification potentially due to a bacteria (pneumonia, dysentery, ear infection) identified by the HW
Viral infection identified	Proportion of children with expert classifications likely due to a virus (cough or cold, likely viral) identified by the HW
Treatment appropriateness
Children appropriately managed	Proportion of children who were prescribed antimalarial, antibiotics, oral rehydration solution, and zinc when needed and no antibiotic nor antimalarial when not needed.
Children in need of antibiotic for whom it was prescribed	Proportion of children in need of an antibiotic as per expert classifications for whom at least one antibiotic was prescribed by the HW
Children not in need of antibiotic for whom it was not prescribed	Proportion of children not in need of an antibiotic as per expert classification for whom no antibiotic was prescribed by the HW
Children in need of antimalarial for whom it was prescribed	Proportion of children in need of an antimalarial as per expert classification for whom an antimalarial was prescribed by the HW
Children not in need of antimalarial for whom it was not prescribed	Proportion of children not in need of an antimalarial as per expert classification for whom no antimalarial was prescribed by the HW

HW = health worker.

Results

Characteristics of the study population.

From September 2 to November 23, 2011, 504 children with a median age of 15 months (interquartile range: 8–29) were enrolled (166 in the control, 171 in the paper, and 167 in the electronic arms). Fever was reported in 71% (95% confidence interval [CI]: 67–75%) and cough in 66% (CI: 62–70%) of the 504 children. The most frequent classification given by the expert was “cough or cold” reached in 55% (CI: 51–60%) of the children, followed by “acute diarrhea without dehydration” in 19% (CI: 16–23%) and “pneumonia” in 10% (CI: 8–13%). According to the expert, 63 (13%) of these 504 children had a condition requiring antibiotics, and 7 (1%) antimalarials. The detailed characteristics of the 504 enrolled children as per expert assessment are presented in Table 3.

Table 3.

Demographic characteristics, symptoms, and disease classification as per expert's evaluation of the 504 enrolled children

	Electronic (N = 167)	Paper (N = 171)	Control (N = 166)	Overall (N = 504)
	Proportion % (three HFs range†)	Proportion % (three HFs range†)	Proportion % (three HFs range†)	Proportion %
Demographics
Children aged 2–11 months	43 (41–47)	36 (28–40)	32 (25–41)	37
Girls	48 (38–57)	46 (43–52)	51 (44–55)	48
Symptoms
Fever*	75 (69–80)	64 (59–67)	73 (72–75)	71
Cough	69 (65–72)	64 (57–74)	65 (63–69)	66
Fever* and cough	55 (51–57)	44 (39–51)	46 (42–49)	49
Diarrhea	23 (22–24)	17 (12–24)	20 (17–24)	20
Skin lesion	7 (2–13)	12 (7–21)	5 (2–7)	8
Ear problem	0	4 (3–4)	2 (0–4)	2
Disease classification (expert)
Severe disease	1 (0–2)§	2 (0–4)‡	0	1
Malaria	1 (0–2)	1 (0–4)	2 (0–5)	1
Pneumonia	14 (12–15)	6 (4–7)	11 (4–20)	10
Cough or cold	55 (51–57)	57 (54–63)	53 (47–59)	55
Acute diarrhea	22 (20–24)	17 (12–24)	19 (17–24)	19
Dysentery	1 (0–2)	0	1 (0–4)	1
UTI	1 (0–2)	1 (0–2)	1 (0–2)	1
Likely viral	11 (7–15)	10 (4–18)	16 (11–21)	12

HF = Health facility; UTI = urinary tract infection.

^*History of fever as mentioned by the caretaker.

^†Range of proportions in the three health facilities.

^‡Three severe classifications: one severe malnutrition, one severe anemia, one “vomits everything”.

^§Severe anemia.

Completeness of integrated assessment.

In the control arm, some of the assessment tasks that should be systematically performed in every child were rarely performed by the clinicians. Dangers signs were checked in only 3% of the children (range: 2–4% among three HFs) and palmar pallor in 14% (range: 6–20%). In this arm, only 21% of the children (range: 18–24%) had an index of integrated assessment ≥ 50% (meaning that only one child of five had at least half of the systematic assessment tasks performed) and 11% (range: 9–14%) an index above 75% (Table 4). In the paper arm, danger signs were checked in 41% of the children (range: 16–71% among three HFs, aRR as compared with control arm [CI]: 14.4 [3.4–69.7]) and palmar pallor in 30% (range: 11–50%, 2.1 [0.8–5.7]). In this arm, 58% of the children (range: 33–85%, 2.9 [1.4–6.3]) had an index of integrated assessment ≥ 50%, and 16% (range: 7–27%, 1.5 [0.7–3.1]) an index above 75%. In the electronic arm, danger signs were checked in 74% of the children (range: 63–94%, 30.9 [9.2–120.2]) and pallor in 69% (range: 56–80%, 5.2 [3.1–9.1]). In this arm, the proportion of children with an index ≥ 50% was 87% (range: 80–98%, 4.2 [3.1–5.6]) and with one ≥ 75% was 43% (range: 40–46%, 4.0 [2.5–6.4]) (Table 4). Main symptoms were checked in two-thirds of the children in the control (77%, range: 64–91%) and paper arm (75%, range: 68–82%, 1.0 [0.8–1.2]), and in almost all children in the electronic arm (99%, range: 98–100%, 1.3 [1.2–1.3]).

Table 4.

Indicators of completeness of assessment and appropriateness of classifications and treatments

Indicators	Arm	Total number of children (N)	Indicator value % (three HFs range)	Risk ratio accounting for clustering aRR (95% CI)
				Versus control	P value	Versus paper	P value
Assessment
Danger signs	Control	165	3 (2– 4)	–	–	–	–
	Paper	168	41 (16–71)	14.4 (3.4–69.7)	< 0.001	–	–
	Electronic	152	74 (63–94)	30. 9 (9.2–120.2)	< 0.001	2.0 (1.0–3.5)	0.04
Main symptoms	Control	163	77 (64–91)	–	–	–	–
	Paper	168	75 (68–82)	1.0 (0.8–1.2)	0.68	–	–
	Electronic	158	99 (98–100)	1.3 (1.2–1.3)	0.001	1.3 (1.2–1.3)	< 0.001
Palmar pallor	Control	166	14 (6–20)	–	–	–	–
	Paper	171	30 (11–50)	2.1 (0.8–5.7)	0.12	–	–
	Electronic	167	69 (56–80)	5.2 (3.1–9.1)	< 0.001	2.5 (1.5–4.3)	0.001
Proportion of children with an index of integrated assessment ≥ 50%	Control	166	21 (18–24)	–	–	–	–
	Paper	171	58 (33–85)	2.9 (1.4–6.3)	0.004	–	–
	Electronic	167	87 (80–98)	4.2 (3.1–5.6)	< 0.001	1.6 (1.1–2.0)	0.027
Proportion of children with an index of integrated assessment ≥ 75%	Control	166	11 (9–14)	–	–	–	–
	Paper	171	16 (7–27)	1.5 (0.7–3.1)	0.273	–	–
	Electronic	167	43 (40–46)	4.0 (2.5–6.4)	< 0.001	2.7 (1.7–4.1)	< 0.001
Respiratory rate measured	Control	69	4 (0–7)	–	–	–	–
	Paper	73	53 (25–70)	12.8 (3.6–53.5)	< 0.001	–	–
	Electronic	79	41 (32–56)	9.4 (3.1–30.9)	< 0.001	0.8 (0.5–1.3)	0.34
Malaria test ordered	Control	123	84 (75–88)	–	–	–	–
	Paper	112	92 (88–95)	1.1 (1.0–1.2)	0.06	–	–
	Electronic	116	72 (56–88)	0.9 (0.7–1.1)	0.16	0.8 (0.7–0.9)	0.07
Classifications
Children appropriately classified	Control	166	34 (22–56)	–	–	–	–
	Paper	171	39 (35–42)	1.1 (0.7–1.9)	0.60	–	–
	Electronic	167	53 (47–59)	1.6 (1.0–2.5)	0.045	1.6 (1.0–2.5)	0.07
Bacterial classifications identified	Control	22	27 (0–38)	–	–	–	–
	Paper	11	36 (20–100)	1.3 (0.4–3.6)	0.59	–	–
	Electronic	26	46 (0–70)	1.8 (0.5–8.4)	0.41	1.0 (0.3–8.7)	0.949
Viral classifications identified	Control	114	36 (17–74)	–	–	–	–
	Paper	115	33 (10–43)	0.9 (0.3–2.8)	0.81	–	–
	Electronic	110	70 (59–77)	2.1 (1.0–4.0)	0.04	2.2 (1.4–3.5)	0.001
Treatment
Children appropriately managed	Control	166	37 (29–44)	–	–	–	–
	Paper	171	62 (55–74)	1.7 (1.3–2.2)	< 0.001	–	–
	Electronic	167	63 (52–72)	1.7 (1.3–2.2)	< 0.001	1.0 (0.8–1.3)	0.91
Children prescribed antibiotic by the health worker	Control	166	70 (60–85)			–	–
	Paper	171	26 (14–37)	0.4 (0.2–0.6)	< 0.001	–	–
	Electronic	167	25 (17–33)	0.3 (0.2–0.5)	< 0.001	1.1 (0.6–1.6)	0.825
Children in need of antibiotic for whom it was prescribed	Control	22	100 (100–100)	–	–	–	–
	Paper	14	36 (20–67)	0.4 (0.3–0.7)	0.002	–	–
	Electronic	27	48 (0–80)	0.5 (0.4–1.0)	0.04	1.2 (0.4–6.7)	0.823
Children not in need of antibiotic for whom it was not prescribed	Control	144	34 (18–47)	–	–	–	–
	Paper	157	75 (63–87)	2.3 (1.5–3.4)	< 0.001	–	–
	Electronic	140	80 (72–90)	2.4 (1.7–3.3)	< 0.001	1.1 (0.9–1.0)	0.474
Children in need of antimalarial for whom it was prescribed	Control	3	100 (100)	–	–	–	–
	Paper	2	50 (50)	NA	–	–	–
	Electronic	2	50 (0–100)	NA	–	NA
Children not in need of antimalarial for whom it was not prescribed	Control	163	93 (87–98)	–			–
	Paper	169	100 (100–100)	1.1 (1.0–4.0)	0.03		–
	Electronic	165	96 (89–100)	1.1 (0.9–1.1)	0.37	1.0 (1.0–1.0)	0.466

aRR = adjusted risk ratio; CI = confidence interval; NA = not applicable, due to a too small number of children.

Assessment tasks to be performed in a subset of children (such as malaria test, or respiratory rate count) were not included in the index of integrated assessment. Respiratory rate was counted in only 4% (range: 0–7%) of eligible children in the control arm, and in 53% (range: 25–70%, 12.8 [3.6–53.5]) and 41% (range: 32–56%, 9.4 [3.1–30.9]) in the paper and electronic arms, respectively. The proportion of febrile children tested for malaria was not significantly different in the three arms (84%, range: 75–88% in the control; 92%, range: 88–95%, 1.1 [1.0–1.2] in the paper; and 72%, range: 56–88%, 0.9 [0.7–1.0] in the electronic arms) (Table 4).

Appropriateness of disease classification.

The proportion of children with all HWs' disease classifications matching the expert's ones was low in both control (34%, range in the three HFs: 22–56%]) and paper arms (39%, range: 35–42%, 1.1 [0.7–1.9]) and slightly higher in the electronic one (53%, range: 47–59%, 1.6 [1.0–2.5]). This was due to a higher proportion of viral conditions identified by HWs in the electronic arm (70%) than in the control arm (36%; 2.1 [1.0–4.0]). The proportions of bacterial conditions identified by HWs was however not significantly different between the three arms and highly heterogeneous between the HFs (27%, range: 0–38% in control; 36%, range: 20–100% in paper; and 46%, range: 0–70% in electronic arm) (Table 4). Figure 1 presents Venn diagrams illustrating the agreement between HWs and expert in terms of viral and bacterial classifications, and the proportion of children prescribed antibiotics by HWs.

Figure 1.

Venn diagram illustrating the level of agreement (in %) between the expert and the health workers' bacterial and viral classifications, as well as the total proportion (in %) of children prescribed an antibiotic in each study arm.

Appropriateness of treatment.

The proportion of children appropriately managed (antimalarials, antibiotics, zinc, and rehydration prescribed when needed only) was similar in the two intervention arms and significantly higher than that in the control arm (37%, range: 29–44% in control; 62%, range: 55–74%, 1.7 [1.3–2.2] in paper; and 63%, range: 52–72%, 1.7 [1.3–2.2] in the electronic arm). The proportion of children prescribed antibiotics was much lower in the interventions than in the control arm (70%, range 60–85% in the control; 26%, range 14–37%, 0.4 [0.2–0.6] in the paper; and 25%, range: 17–33%, 0.3[0.2–0.5] in the electronic arm). Thereby, the proportion of children not in need of antibiotics and for whom it was not prescribed was higher in the paper (75%, 2.3 [1.5–3.4]) and the electronic HFs (80%, 2.4 [1.7–3.3]) than in the control HFs (34%). However, the proportion of children in need of antibiotics and for whom it was not prescribed was also higher in the intervention arms: only five of the 14 children in need of antibiotic were prescribed it by the attending HW in the paper arm, and only 13 of the 27 children in the electronic arm. In the control arm, where 70% of all children were prescribed an antibiotic, all the 22 children in need of an antibiotic were prescribed one by the HW; however, more than half of the antibiotic prescriptions in the control arm were inappropriate (Table 4). Among the seven malaria cases, five were appropriately treated and two were not prescribed any antimalarial by the HW (one in the paper and one in the electronic arm). Some children received an unnecessary antimalarial treatment (11 in the control and six in the electronic arm).

Identification of serious conditions by HWs.

Table 5 presents for each HF the list of serious conditions, that is, either severe diseases or infections requiring antimalarial (malaria) or antibiotic treatment (pneumonia, dysentery, UTI), identified in the three arms compared with that identified by the expert. Overall only 30 (43%) of 70 conditions were appropriately identified by the HWs. Pneumonia, that was the most frequent serious condition identified by the expert, was also the condition most often missed by the HWs in all three arms: only about one-third of pneumonia was identified by the HWs (5/19, 3/10, and 9/23 in control, paper, and electronic arms, respectively). These proportions varied a lot among the different HFs involved. The other conditions requiring antibiotics or antimalarials, though rare, were more often recognized by the HWs. When looking at the 35 expert pneumonia that were missed by HWs (14, 7, and 14 in the control, paper, and electronic arms, respectively), cough was not identified in two children in the control arm, and therefore pneumonia was not considered (2/14,0/7, and 0/14, respectively) and the others were classified as “cough or cold,” either without respiratory rate measured (9/14, 1/7, and 9/14, respectively) or with a respiratory rate below the threshold (2/14, 5/7, and 5/14, respectively). For the latter children, the median respiratory rate measured by HWs was 45/minute (range: 38–49) for eight children < 1 year and 43/minute (38–49) for children ≥ 1 year, whereas it was 53/minute (52–57) and 54/minute (51–58), respectively, when the child was assessed by the expert around half an hour later. One child in the paper arm had a respiratory rate above the threshold that was misinterpreted, and one child in the control arm had a cough attributed by the HW to worm infestation.

Table 5.

Detailed list of serious^* conditions identified by health workers among conditions validated by the expert

	Control arm			Paper arm			Electronic arm			Total
	HF1	HF2	HF3	HF1	HF2	HF3	HF1	HF2	HF3
Severe diseases	–	–	–	2/2† (100%)	1/1‡ (100%)	–	–	–	0/1‡ (0%)	3/4 (75%)
Malaria	3/3 (100%)	–	–	–	2/2 (100%)	–	–	1/1 (100%)	0/1 (0%)	6/7 (86%)
Pneumonia	2/6 (33%)	0/2 (0%)	3/11 (27%)	2/2 (100%)	1/4 (25%)	0/4 (0%)	0/8 (0%)	5/8 (63%)	4/7 (57%)	17/52 (33%)
Dysentery	1/2 (50%)	–	–	–	–	–	–	1/1 (100%)	1/1 (100%)	3/4 (75%)
UTI	–	0/1 (0%)	–	–	–	1/1 (100%)	–	1/1 (100%)	–	2/3 (67%)
Total/HF	6/11	0/3	3/11	4/4	4/7	1/5	0/8	8/11	5/10
Total/arm	9/25 (36%)			9/16 (56%)			13/29 (45%)			31/70 (0.44)

HF = health facility; UTI = urinary tract infection.

^*Serious conditions: either severe diseases or nonsevere bacterial infections or malaria.

^†One severe malnutrition and one very severe disease (vomits everything).

^‡Severe anemia.

Direct comparison of paper and electronic arms' results.

The indicators of the completeness of the assessment were significantly higher in the electronic arm compared with paper, with aRR electronic versus paper (aRR-e/p) above two for proportion of children checked for danger signs (aRR-e/p: 2.0 [P = 0.04]), checked for palmar pallor (aRR-e/p: 2.5 [P = 0.001]), and with an index test of integrated assessment > 75% (Table 4). However, for respiratory rate measure (aRR-e/p: 0.8 [P = 0.34]) and malaria test (aRR-e/p: 0.8 [P = 0.07]), the proportion of children assessed in the electronic arm was lower than in the paper, although not significantly. Except for viral classifications that were more often correctly identified in electronic arm (aRR-e/p: 2.2 [P = 0.001]), there was no difference in the identification of bacterial infections, and overall classifications. With regard to treatment, similar outcomes were observed in the two interventions arms.

Discussion

Our findings show that the implementation of ALMANACH on either paper or electronic support has the potential to tackle some of the difficulties of HWs when managing childhood illnesses. In the control arm, HWs' compliance to integrated assessment tasks was low, and antibiotics were prescribed to more than two-thirds of the children, even when the classification did not require antibiotics according to IMCI. The use of ALMANACH (in both paper and electronic arms) was associated with a significantly higher proportion of children having an index of integrated assessment ≥ 50%, being checked for danger signs, having their respiratory rate measured when necessary, and being appropriately managed. A significantly lower prescription of unnecessary antibiotics as compared with the control was equally observed in paper and electronic arms. Observed in both intervention arms, these results were likely due to the recent training received by clinicians on ALMANACH's content. The use of the electronic ALMANACH was further associated with a significantly higher proportion of children assessed for danger signs, main symptoms, and palmar pallor, children having an index of integrated assessment ≥ 75% and having an appropriate classification than in the control. The proportion of children identified both by the expert and the HW as viral infection was significantly higher in the electronic arm than in the control and paper arms. Although improved in intervention arms, the respiratory rate remained infrequently measured in children with cough, and the proportion of febrile children tested for malaria was lower in the electronic than in the control and paper arms. In this pilot study, the small number of HFs involved and the limited number of consultations observed did not allow to observe significant differences in the impact of the electronic versus paper intervention compared with control (the 95% CI of the aRR always overlapped, Table 4); however, the results show that the use of handheld technology can improve the training's impact on simple easy-to-perform tasks, such as asking for danger signs and main symptoms, or looking at a child's palm. However, in this survey, the use of mobile device was insufficient in itself to convince HWs to perform more time-consuming tasks systematically, such as measuring respiratory rate over a full minute, or sending a child to the laboratory for a malaria test.

The most frequently missed serious condition was pneumonia. In previous studies assessing HWs' performance after IMCI training, pneumonia identification was always a challenge, with only 41% of pneumonia correctly identified in South Africa,²¹ and only 40–50% of pneumonia cases identified who were prescribed the recommended treatment by HWs in Benin.²² This may be explained by problems in pneumonia definition that were not addressed by our intervention. In IMCI and in ALMANACH, the pneumonia classification relies on respiratory rate. Respiratory rate fluctuates over time, and HWs have low motivation to measure it. Indeed, in our study, most of missed pneumonia had either no respiratory rate measured, or a rate below the expert one, the difference being either due to the variability over time or to the low inter-rater reproducibility of the measurement. Respiratory rate that need to be counted over a full minute is perceived as time consuming by HWs both in low- and high-income countries.²³ Moreover, the low performance of fast breathing to predict bacterial pneumonia render this sign in fact not very useful,⁸ and efforts are ongoing to find better clinical ways to diagnose pneumonia. Both issues call for novel diagnostic assays or devices to identify this serious condition.

For malaria, the availability of mRDT, improves HWs' diagnoses. In this survey, all malaria cases were identified by the HWs, except one (Table 5). This missed malaria case, occurred in a 21-month-old boy, who had neither fever nor history of fever, but for whom the expert identified a nonsevere anemia, based on palmar pallor, and therefore ordered an mRDT. The HW in this case did not assess the child's palm, and missed both anemia and malaria. The very few cases of anemia and malaria in our survey do not allow assessing the impact of our intervention on this critical issue. However, the trends in the proportion of children assessed for palmar pallor (14%, 30%, and 69% in control, paper, and electronic facilities, respectively; Table 4) suggest that ALMANACH could help improving anemia detection.

The use of ALMANACH resulted in an important decrease in the unnecessary antibiotic prescription. However, because of the inability of HWs to properly classify pneumonia, some of the children in need of antibiotics according to the WHO clinical pneumonia definition were not prescribed such treatment, even in the intervention HFs. In the present study, all these children actually received antibiotics from the expert, but this could be a matter of concern if real bacterial pneumonia would remain untreated by HWs in programmatic conditions. One study on the health outcome of untreated nonsevere pneumonia in children in a low-income country showed no difference in clinical outcome with the treated control group,⁶ which suggest that, in young children, a high number of clinical pneumonia episodes are of viral origin, as it is the case in northern countries. Therefore the low specificity of the present criteria used to diagnose pneumonia, even when using a higher threshold for respiratory rate,⁸ together with the fact that most children can be followed up to look at the clinical course, must balance the concern of missing a true bacterial pneumonia case. It has often been argued that follow-up visits are difficult to organize in low-resource settings due to transport and cost constraints. That was one of the reasons for encouraging presumptive treatment of malaria and having a low clinical threshold to give antibiotics for pneumonia. There is now more and more awareness that this strategy leads to overprescription of antimicrobials and the development of resistance. While using the electronic ALMANACH, the HWs were more likely to appropriately deliver messages on when to bring back the child in case his conditions would not improve or even worsen (data not shown). Follow-up is an important part of outpatient management of nonsevere children, and if this strategy is not promoted, there is little hope to tackle the antibiotic overuse and the spread of drug resistance.

Strengths and limitations.

To our knowledge, this study was the first implementation in programmatic conditions of an electronic algorithm for the management of childhood illness. A previously published study also reported improved HW's performance when using an electronic IMCI compared with the paper IMCI²⁴ and a positive user's experience,²⁵ but these results were observed within a short study period of 3–6 days, and HWs using the electronic tool only over these few survey days.

The present project has demonstrated the feasibility of using innovative technology in resource-limited settings, and its high potential to improve the quality of health care delivered to children. The use of this technology also allowed data collection and activity monitoring that are essential assets to monitor and improve quality of health care.

The small number of HFs involved, their disparities in size, and the relatively small number of consultations observed limits the power of the analyses, but the results still show the potential of such a tool that deserves further assessment.

In the present survey, a high proportion of children were prescribed an antibiotic in the control HFs. This is in line with observation made 4 years earlier in the same HFs, after the introduction of mRDT.⁵ In Dar es Salaam, where the prevalence of malaria is rather low, antimalarial drugs, formerly prescribed presumptively for fever, were indeed replaced by presumptive antibiotics, when the malaria test was negative. In a setting where malaria prevalence would be higher, we can expect the level of antibiotic prescriptions to be lower (lower proportion of non-malaria fever for which antibiotics are too often prescribed), but in the absence of appropriate patient assessment, not more appropriate.

The choice of routine practice as a control and the deliberate short training (2 days) in the intervention HFs were decided to be as close as possible to programmatic conditions. Although IMCI is the standard of care for children in Tanzania since the 1990s, the HWs involved in the control arm had no recent IMCI refresher training. This control group does not allow comparison of ALMANACH versus IMCI algorithms, but rather the difference in HW's performance when using a new tool on paper or on electronic support, compared with the previous practice.

Potential to scale-up.

This pilot intervention comprised a short training followed by 1 day of HW face-to-face supervision, and monthly HF supervision visits. This short duration mitigates the resources needed for the initial training and may facilitate scale-up. It raises the need for resources in the immediate aftermath that may be a bottleneck to scale-up. However, close follow-up and regular supportive supervision are acknowledged as a key for sustainable improvement of the quality of care.

Conclusion

The use of mobile technology to support health-care delivery in low-income countries is feasible and has a huge potential for improvement of quality of care and monitoring. As long as no automated devices are available to replace the key signs required for clinical diagnosis or selection of patients for testing, training of HWs on the basic clinical skills is still essential. As any other clinical support system, it also requires continuous supportive supervision to encourage HWs using it appropriately. Improving the rational use of antimicrobial is a challenge that ALMANACH can help to take up. It has to be acknowledged and accepted that, because no diagnostic tool is fully sensitive, the reduction in antibiotic prescriptions will always be accompanied by a small risk of withholding them in a child that would have benefited from them. Proposing the best available accurate diagnostic procedures to HWs, and enforcing good adherence as well as the delivery of appropriate counseling to caretakers on when to bring the child back, are needed to reach a rational use of antibiotics and slow down the development of resistance.

Disclaimer: The funders played no role in study design, collection, analysis, interpretation of data, writing the report, or in the decision to submit the paper for publication.