Usability and acceptability of a multimodal respiratory rate and pulse oximeter device in case management of children with symptoms of pneumonia: A cross‐sectional study in Ethiopia

Abstract

Aim

Pneumonia is the leading infectious cause of death among children under five globally. Many pneumonia deaths result from inappropriate treatment due to misdiagnosis of signs and symptoms. This study aims to identify whether health extension workers (HEWs) in Ethiopia, using an automated multimodal device (Masimo Rad‐G), adhere to required guidelines while assessing and classifying under five children with cough or difficulty breathing and to understand device acceptability.

Methods

A cross‐sectional study was conducted in three districts of Southern Nations, Nationalities, and Peoples' Region, Ethiopia. Between September and December 2018, 133 HEWs were directly observed using Rad‐G while conducting 599 sick child consultations. Usability was measured as adherence to the World Health Organization requirements to assess fast breathing and device manufacturer instructions for use. Acceptability was assessed using semi‐structured interviews with HEWs, first‐level health facility workers and caregivers.

Results

Adherence using the Rad‐G routinely for 2 months was 85.3% (95% CI 80.2, 89.3). Health workers and caregivers stated a preference for Rad‐G. Users highlighted a number of device design issues.

Conclusion

While demonstrating high levels of acceptability and usability, the device modifications to consider include better probe fit, improved user interface with exclusive age categories and simplified classification outcomes.

Abbreviations

ARI

acute respiratory infection

ARIDA

acute respiratory infection diagnostic aid

CHW

Community Health Worker

CI

confidence interval

FLHFW

first‐level health facility worker

HEW

health extension worker

IFU

instructions for use

iCCM

integrated community case management

IMCI

integrated management of childhood illnesses

RR

respiratory rate

SNNPR

Southern Nations, Nationalities, and Peoples' Region

SD

standard deviation

UNICEF

United Nations Children's Fund

WHO

World Health Organization

Key notes.

• Community health workers (CHWs) manually count respiratory rate (RR) to detect fast breathing as a sign of pneumonia, often finding accuracy difficult.

• A new multimodal device (Masimo Rad‐G), which measures RR, oxygen saturation and heart rate, was tested for feasibility and acceptability as a pneumonia diagnostic aid in children under five in Ethiopia.

• While Rad‐G was found to be feasible and acceptable to CHWs in Ethiopia, modifications to the design could further improve it.

1. INTRODUCTION

Acute respiratory infections (ARIs), primarily pneumonia, are the leading infectious causes of death among children under 5 years of age globally, accounting for an estimated 800,000 pneumonia‐related deaths in 2017. ¹ Deaths from pneumonia in children result mostly from delayed presentation to appropriate healthcare providers, wider issues of quality of care such as referral challenges and access to oxygen and inappropriate treatment. ² Classification of fast breathing, as a sign of pneumonia, by CHWs and first‐level health facility workers (FLHFWs; collectively known as frontline health workers) is based on manually counting the number of breaths in 60 s in children under five (U5) years of age with cough and/or difficulty breathing. This allows the health worker to assess whether the RR is high enough for a particular age to prescribe antibiotics, and treat suspected pneumonia, as defined by the World Health Organization (WHO) integrated management of childhood illnesses (IMCI) guidelines ³ for FLHFWs and the iCCM guidelines ⁴ for CHWs. In practice, frontline health workers indicate that counting respiratory rate (RR) can be difficult because children breathe irregularly and faster than adults; the child may not be calm and still for a full minute; and it is difficult to define what is and is not a breath. ⁵ Misclassification of the observed rate remains high ⁶ , ⁷ and often leads to inappropriate treatment. ⁸

Low blood oxygen saturation, or hypoxaemia, is a sign that can be observed among a variety of different diseases, including pneumonia, that has been identified as a marker of severity and a predictor for morbidity and mortality in children with respiratory illness. ⁹ However, hypoxaemia, if characterised at all in low‐ and middle‐income country (LMIC) settings, is typically identified based on clinical findings alone, ¹⁰ and the inability of healthcare workers to promptly detect and refer these children, whose lives are in danger, leads to an adverse prognosis in many of these children. While pulse oximetry is a reliable and non‐invasive method for identifying children with hypoxaemia, pulse oximeters are rarely available outside higher‐level facilities in resource‐constrained countries. ¹¹ A modified iCCM algorithm would therefore be required to include oxygen saturation (SpO₂) less than 90% as a sign of hypoxaemia and as a referral and treatment sign. This can then be used by the HEWS, in combination with the current RR cut‐offs, in the sick child consultations.

United Nations International Children's Emergency Fund (UNICEF)’s Acute Respiratory Infection Diagnostic Aid (ARIDA) project ¹² was initiated as a response to the calls for better devices that diagnose symptoms of pneumonia. ¹³ , ¹⁴ Two ARIDA devices have been developed and commercialised in response to UNICEF’s Request for Proposals, ¹⁵ and field trials were conducted in Ethiopia and Nepal to assess usability and acceptability of devices whose attributes meet the specifications outlined in the UNICEF ARIDA target product profile (TPP) ¹⁶ for use by frontline health workers at the lowest levels of health care. The study reported in this paper is one of the field trials in Ethiopia funded through UNICEF’s partnership with ‘La Caixa’ Foundation and conducted by Malaria Consortium.

Ethiopia was selected to host the ARIDA acceptability field trial due to the high burden of ARIs in children U5 (16% in 2016), ¹⁷ availability of a CHW cadre (locally called health extension workers [HEWs]) who deliver community‐based management of pneumonia ¹⁸ and availability of first‐line amoxicillin dispersible tablets (Amox DT). Since 2010, Ethiopia has scaled up iCCM in most regions following a national policy change supporting community‐based management of childhood illnesses through HEWs. ¹⁹ All HEWs are women, educated to at least tenth grade, trained for 1 year in iCCM and other health preventive and curative interventions. They are government paid and equipped to assess, classify and manage uncomplicated pneumonia, malaria, diarrhoea and severe acute malnutrition and provide preventive and curative health services. ¹⁸ Trained HEWs are deployed to a health post to work at the sub‐district (kebele) level and serve a population of approximately 5000 people. Through iCCM, any child 2–59 months with fast breathing pneumonia and no danger signs can be treated at the health post; sick children under 2 months are referred to a health centre. ¹⁸ The next level of the Ethiopian health system is the health centre, which is staffed by around 20 health professionals (FLHFWs), provides preventative and curative services to approximately 25,000 people and serves as a referral centre and practical training institution for HEWs. In 2016, the Ethiopian oxygen roadmap was published, which has a vision for pulse oximeters to be available and used at all levels of the Ethiopian health system. ²⁰

This paper reports findings of a cross‐sectional study to determine usability, through exploring whether HEWs in Ethiopia using Rad‐G could adhere to the required steps to assess and classify pneumonia in children under five with cough and/or difficult breathing.

2. METHODS

Full details of the methods have been described elsewhere. ²¹

Masimo developed the Rad‐G (Figure 1) which uses differential light absorbance technology to measure SpO₂ and derives respiration rate (RR), pulse rate (PR) and perfusion index (Pi) from the photo plethysmography waveform or Pleth (RRp™) in children 0 to 59 months and classifies the RR and SpO₂ of the child according to WHO IMCI/iCCM guideline cut‐offs. ³ , ⁴ The device, designed for LMIC use, is comprised of a handheld unit with a liquid crystal display touchscreen user interface and a detachable universal sensor probe suitable for children above 3 kg and which is placed on the child's finger or toe. The device also has an animated display to help keep the child calm during use (Figure 1).

Figure 1.

Masimo Rad‐G

2.1. Study setting and participants

The study was conducted in community settings and first‐level health facilities in Sodo Zuria, Damote Sore and Damote Gale districts of Wolaita zone in Southern Nations, Nationalities, and Peoples' Region (SNNPR), Ethiopia. SNNPR and the Wolaita zone were selected because of the high burden of ARIs, sufficient number of HEWs with experience, relative political stability (no insecurity that might have affected the implementation of the study) and availability of oxygen at the district hospital in case a child was referred. Wolaita zone has logistic and operational feasibility for data collection and quality assurance, as the health posts and health centres are accessible to road transport. In addition, Malaria Consortium has strong relationships with SNNPR health bureau and the Wolaita zonal health department through implementing different projects in Wolaita zone in the past, such as the TRAction study. ²² All available HEWs (134) in the three districts were selected for the study. All selected HEWs were trained in and using iCCM at their health posts (65). For the acceptability element of the study, a convenience sample of twenty FLHFWs (one per health centre) who were trained in IMNCI and working in the under five clinic of their health centre was selected to participate in consultation with the heads of the three district health offices.

2.2. Data collection methods and sampling

This was a cross‐sectional study using direct observation of HEW consultations and semi‐structured interviews with frontline health workers and caregivers. The study group decided to directly observe HEWs only, with the assumption that FLHFWs would be able to use Rad‐G as well as, or better than FLHFWs due to their higher education level and longer training.

2.2.1. Direct observation

One hundred and thirty‐four HEWs and 20 FLHFWs were provided with a 2‐day training on the WHO requirements to assess fast breathing plus the measurement of SpO₂ as a sign of hypoxaemia, detailed in a job aid (see Appendix S1), trained on how to use Rad‐G and completed pre‐ and post‐training assessments to test their knowledge of iCCM/IMNCI and Rad‐G. A pass mark of 75% was required of all HEWs participating in the study. Immediately after the training 133 HEWs were directly observed using Rad‐G (observation 1) to assess and classify pneumonia during two consecutive child consultations in their usual workplace (the health post). HEWs were observed completing nine assessment, classification, treatment and referral steps from the WHO requirements to assess fast breathing and the device manufacturer instructions for use (IFU; Table 1). HEWs were then left to use the device routinely for 2 months before being directly observed a second time (observation 2), again for two consecutive child consultations. The sample size was calculated assuming 71% of HEWs (calculated as 75% complete correct RR assessment, of which 95% complete correct RR classification) would adhere to the required assessment and classification steps for the primary outcome, with 95% confidence and 7.5% precision, including an adjustment of 10% for non‐participation or exclusion for any other reason and a design effect to account for clustering at HEW level of 1.7. ²³

Table 1.

Steps of the child consultation that health extension workers using Rad‐G were observed completing

	Consultation step	Definition	Source of step
1	Child calm before Rad‐G attempt	Calm: not actively crying or moving	WHO requirements to assess fast breathing WHO requirements to assess fast breathing
2	Correct mode selected	Screening mode	Device manufacturer instructions for use
3	Correct age group	Age group recorded by HEW on Rad‐G device matches screening checklist	WHO requirements to assess fast breathing WHO requirements to assess fast breathing
4	Correct probe position	Fully inserted	Device manufacturer instructions for use
5	Correct probe direction	Picture on top of finger or toe	Device manufacturer instructions for use
6	Child not eating/feeding during Rad‐G attempt	No eating/breastfeeding	WHO requirements to assess fast breathing WHO requirements to assess fast breathing
7	Child calm during Rad‐G attempt	Calm: not actively crying or moving	WHO requirements to assess fast breathing
8	Correct classification using Rad‐G	According to iCCM guidelines, based on screening age group and breathing status of the child	WHO requirements to assess fast breathing
1‐8	Correct assessment and classification (steps 1–8) – primary outcome	HEW correctly completed all steps 1–8	Device manufacturer instructions for use and WHO requirements to assess fast breathing
9	Correct treatment and referral guidance using RAD‐G classification and HEW's assessment of other symptoms (yes/no?)	According to iCCM guidelines, based on age group recorded during child screening, and breathing status of the child. N.B HEW will not be marked as 'incorrect' if there was stock‐out of antibiotics, caregiver refused treatment or other valid reason for no treatment recorded	WHO requirements to assess fast breathing

To ensure there were sufficient children visiting the health post on the day of the HEWs’ observation, HEWs were encouraged to mobilise caregivers of sick children aged U5. Observations were conducted by twelve research assistants (either medical officers or degree‐qualified nurses). Research assistants were trained to screen U5 children for inclusion criteria and then silently observe the HEW assessing a child under five presenting for illness (aged 0 to <2 months) or for cough and/or difficult breathing (aged 2–59 months) according to the WHO requirements to assess fast breathing and device manufacturer IFU. HEWs had up to three attempts to obtain a RR classification with Rad‐G; if these attempts were unsuccessful, the HEW reverted to using standard practice (ARI timer, smartphone timer or watch). Each step of the HEWs’ assessment, classification and treatment and/or referral (Table 1) of the child was independently recorded by two research assistants in a digital tablet‐based data collection form using CommCare (version 2.38.1, Dimagi). Research assistants photographed Rad‐G with the RR and SpO₂ result, classification and age displayed, to provide source documents for verification purposes. Data were synced daily to a protected cloud server and validated and cleaned by the data manager. Between the first and second observations, HEWs were encouraged to use Rad‐G during routine practice, but were allowed to revert to standard practice if required, and instructed to record which device they used in their patient register using coloured stickers (one patient register per health post). FLHFWs were not directly observed conducting a sick child assessment using Rad‐G but were encouraged to use Rad‐G at the health centre for the 2 months prior to their semi‐structured interview. A half‐day focused training was provided to all HEWs participating in the study before observation 2, as it was felt by the research team that participants needed further clarification on the importance of probe fit and signal strength in achieving a successful reading. This was based on their experience during observation 1, where health workers needed further support to get a stronger signal by fitting the probe correctly.

2.2.2. Semi‐structured interviews

We interviewed a purposive sub‐sample of 14 HEWs immediately after observation 2, making sure to include HEWs with a range of years’ experience practising as a HEW. We also interviewed 14 caregivers of children who were assessed by these HEWs, to explore their reactions and experience of the device use on their child. A convenience sample of five FLHFWs who were available on the day we visited health facilities was also interviewed. Topic guides were developed using a comprehensive conceptual framework of acceptability of healthcare interventions. ²⁴ Questions relating to seven facets of acceptability (attitude, burden, perceived effectiveness, ethicality, intervention coherence, opportunity costs and self‐efficacy) were included to prompt caregivers to evaluate their experience of the device and health workers to evaluate their experience of using the device. Six research assistants (all Ethiopian nationals with some prior experience of qualitative methods) were trained over 3 days to conduct interviews with health workers and caregivers. The topic guides were translated into the local languages and pilot tested with HEWs and caregivers at three health posts. Following pilot testing, some questions were revised and minor amendments made to the translation of specific words and phrases. All semi‐structured interviews were conducted in the local language and audio‐recorded with participants’ consent. Interviews were conducted with HEWs, FLHFWs and caregivers until data saturation was reached.

2.3. Data analysis

Descriptive information about the HEWs is presented in frequencies and percentages, including numbers trained and numbers completing first and second observation, sex, district, number of years qualified as a HEW, last integrated refresher training and last supervision. Descriptive information about the number of children enrolled, number of evaluations that started, number of evaluations completed by Rad‐G and standard practice and child age and sex was also presented. The primary outcome was calculated as the proportion of U5 children consultations where HEWs using Rad‐G adhered to WHO requirements to assess fast breathing and device manufacturer IFU after 2 months of routine use. This analysis was disaggregated by age group, respiration rate and SpO₂ classification. Secondary outcomes including the proportion of HEWs correctly performing steps reflecting the device manufacturer IFU and the steps that reflect the WHO requirements to assess fast breathing are also presented and the difference in the proportion of consultations that were completed correctly between observation one and observation two. For the main outcomes, the most conservative estimates were used, that is if the two research assistants disagreed on how the HEW performed a step in the assessment, the one that recorded an inconsistency/error for that step was used over the one who recorded that the step was performed correctly. The mean time taken to complete the full assessment was calculated for Rad‐G from the time when the HEW turned on the Rad‐G (prior to probe placement) to when the Rad‐G displayed both SpO₂ and RR readings. The number of children who were assessed for signs of respiratory illness by HEWs with Rad‐G or standard practice during routine care was also presented. Univariable logistic regression analyses were performed to assess the relationship between the time since (i) a HEW’s last routine iCCM integrated refresher training; (ii) a HEW’s last routine supervision; or (iii) qualification as a HEW and the proportion of consultations where HEWs adhered to WHO case management and device manufacturer IFU.

For the qualitative analysis, data collected via semi‐structured interviews with caregivers and frontline health workers were analysed separately. Transcripts were created from the recordings of the interviews, translated and checked for accuracy. We carried out a thematic analysis of the qualitative data using MAXQDA (VERBI Software, 2016) to manage data coding, searching and retrieval. One author reviewed all transcripts to identify possible codes; the list of codes was refined in discussion with the research team, and final coding frames were developed for the HEW/FLHFW interviews and the caregiver interviews. After coding all data, matrices were used to display data for each emerging theme and further explore similarities and differences within the data. Each theme was critically analysed by the research team until the final themes were agreed upon.

2.4. Ethical approval

The study was approved by the Liverpool School of Tropical Medicine Research Ethics Committee (ref. 18‐026) and the SNNPR Regional Health Bureau Ethics Committee (ref. 6‐19/10426). Written consent for observations and semi‐structured interviews was obtained from each HEW prior to the observation and from each caregiver whose under five child was assessed by a HEW during an observation.

3. RESULTS

3.1. Participant characteristics

One hundred and thirty‐four HEWs completed the training, of whom 133 were directly observed using Rad‐G between September and December 2018. Those observed had an average of 7.7 (S.D 4.3)‐year experience as a HEW, 78.2% had received iCCM integrated refresher training within 3 years, and 52.6% had received their last supervision within 3 months prior to the study (Table 2). Fifteen HEWs from Sodo Zuria, Damote Sore and Damote Gale participated in the semi‐structured interviews (SSI). Twenty female FLHFWs completed the training, of whom eight participated in a semi‐structured interview (SSI) from Sodo Zuria (38%), six from Damote Sore (46%) and six from Damote Gale (15%). Fifteen caregivers (33%) in each of the three districts participated in SSIs.

Table 2.

Characteristics of frontline health workers, by district (where available)

	District
	Sodo Zuria	Damote Sore	Damote Gale	Overall
Number of HEWs completed 2‐day training on iCCM and Rad‐G	49	38	47	134
Average pre‐training score (%)	63.6	56.9	64.5	61.7
Average post‐training score (%)	87.9	88.7	88.5	88.7
Number of HEWs participating in observation 1 using Rad‐G	49	38	46	133
Mean years' experience as a HEW (SD)	7.0 (4.1)	8.3 (4.2)	8.1 (4.5)	7.7 (4.3)
Received iCCM Integrated Refresher Training ≤3 years ago (%)	98.0	97.4	41.3	78.2
Received last supervision ≤3 months ago (%)	40.8	65.8	54.3	52.6
Number of FLHFWs completed training (N)	8	6	6	20
Average pre‐training score (%)	54.8	64.1	56.4	58.3
Average post‐training score (%)	80.8	79.5	71.8	77.3

Two hundred and sixty‐six children were enrolled for the first observation and 260 children for the second observation (Figure 2). Of these, 264 (observation 1) and 259 (observation 2) Rad‐G consultations were started; the reasons for not starting were as follows: (1) the child not being calm enough to start, and (2) the Research Assistants (RAs) recorded different age data, and therefore, these data were excluded during analysis. For observations at the first time point, just over half the children were male, and for the second time point, 54% were female (Figure 2). For observation 1 49.2% of consultations were completed on the first attempt. This increased to 81.1% for the second observation. The main reasons for unsuccessful attempts during observation 1 were that the child was not calm or was moving (44.6%) and the Rad‐G displayed an error or ‘motion detected’ message (52.3%). While the overall number of unsuccessful attempts was less for observation 2, the main reasons remained the same (Table 3). During observation 1, 19.7% and 1.5% of children were determined to have fast breathing and hypoxaemia by Rad‐G, respectively; these percentages were similar during observation 2 (20.1% and 1.9%).

Figure 2.

Participant study flow for observations one (after training) and two (after 2 months routine Rad‐G use)

Table 3.

Reasons for failed attempts during observation 1 and 2 for Rad‐G

Reason for Rad‐G unsuccessful attempt a	Observation 1					Observation 2
	1st attempt	2nd attempt	3rd attempt	Total	%	1st attempt	2nd attempt	3rd attempt	Total	%
Child not calm or moving	99	70	49	218	44.6	44	28	16	88	51.8
Child pulls off Rad‐G	0	1	0	1	0.2	0	0	0	0	0.0
Rad‐G displays error message	66	45	26	137	28.0	20	11	8	39	22.9
Child starts feeding stops assessment	0	2	1	3	0.6	0	0	0	0	0.0
HEW could not start Rad‐G	0	1	0	1	0.2	0	0	0	0	0.0
Other	0	1	0	1	0.2	1	0	0	1	0.6
Rad‐G signal strength low	0	0	2	2	0.4	1	1	1	3	1.8
Probe attached with picture facing down	4	3	0	7	1.4	0	0	0	0	0.0
Rad‐G displays 'motion detected' symbol	61	37	21	119	24.3	19	12	8	39	22.9

^aRAs could select multiple reasons per attempt.

3.2. Rad‐G Usability for HEWs

After 2 months using Rad‐G, HEWs adhered to the WHO requirements to assess fast breathing and device manufacturer IFU in 85.3% (95% CI 80.2–89.3) of consultations with children under five. This was an absolute increase of 8.9% from the first observation immediately after training (Table 4). When broken down by assessment steps, initially the HEWs found it more difficult to calm the child (84.8%; 95% CI 79.9, 88.7) and select the right age group (89.8%; 95% CI 85.4, 92.9) but both proportions improved in observation 2 (95.8%; 95% CI 92.5, 97.6 and 95%; 95% CI 91.5, 97.1, respectively). After 2 months, HEWs gave the correct treatment and referral in 95.8% (95% CI 92.3, 97.7) of consultations based on the classification results shown by Rad‐G (Table 4).

Table 4.

Number and proportion of child consultation steps correctly performed by HEW with Rad‐G after training (observation 1) and after 2 months of routine use (observation 2)

	Consultation steps	Observation 1 (after training)			Observation 2 (after 2 months)
		n	%	95% CI	n	%	95% CI
1	Child calm before Rad‐G attempt	224	84.8	79.9, 88.7	248	95.8	92.5, 97.6
2	Correct mode selected	264	100.0	98.6, 1.0	258	99.6	97.3, 99.9
3	Correct age group	237	89.8	85.4, 92.9	246	95.0	91.5, 97.1
4	Correct probe position	253	95.8	92.6, 97.7	258	99.6	97.3, 99.9
5	Correct probe direction	257	97.3	94.5, 98.7	254	98.1	95.4, 99.2
6	Child not eating/feeding during Rad‐G attempt	244	92.4	88.5, 95.1	257	99.2	96.9, 99.8
7	Child calm during Rad‐G attempt	249	94.3	90.8, 96.6	251	96.9	93.9, 98.5
1–7	Cumulative assessment (steps 1–7)	180	68.2	62.3, 73.6	227	87.6	83.0, 91.1
1–7: Denominator = Children who were assessed (assessment started)		264			259
8	Correct classification using Rad‐G	183	93.8	89.4, 96.5	229	96.2	92.9, 98.0
1–8	Correct assessment and classification (steps 1–8) – primary outcome	149	76.4	69.9, 81.9	203	85.3	80.2, 89.3
9	Correct treatment and referral guidance using Rad‐G classification and HEW's assessment of other symptoms (yes/no?)	185	94.9	90.7, 97.2	228	95.8	92.3, 97.7
Denominator = Children with RR and SpO2SpO2 classification (Rad‐G)		195			238
	Intention to treat
8	Correct classification using Rad‐G	183	69.3	63.5, 74.6	229	88.4	83.9, 91.8
1–8	Correct assessment and classification (steps 1–8) – primary outcome	149	56.4	50.4, 62.3	203	78.4	72.9, 83.0
	Further analysis
2, 4, 5	Manufacturer instructions for use correctly performed (steps 2, 4, 5)	248	93.9	90.3, 96.3	252	97.3	94.4, 98.7
1–3, 6–8	Adherence to WHO requirements to assess fast breathing (steps 1, 3, 6, 7, 8)	151	57.2	51.1, 63.1	208	80.3	75.0, 84.7
Denominator = Children who were assessed (assessment started)		264			259

There was no significant difference (p = 0.58) in the proportion of HEWs adhering to each of the steps after 2 months for the 2‐ to <12‐month‐olds (88.3%; 95% CI 78.8, 93.9) compared to the 12‐ to 59‐month‐olds (87.4%; 95% CI 81.6, 91.6; Table 5). The largest variation in assessment steps between age groups was seen for both ‘child calm in advance’ (94.8% vs. 96.6%) and ‘child calm during assessment’ (94.8% vs. 98.3%) for the 2‐<12 months and 12‐ to 59‐month groups, respectively; the difference was not significant (p = 0.93). HEWs completed all eight steps correctly more frequently for normal breathing children (89.9%; 95% CI 84.9, 93.3) compared to fast breathing children (78.8%; 95% CI 65.2, 88.1), but again was not found to be significant (p = 0.97). The logistic regression analysis found no significant association between correct assessment and classification of children after 2 months of routine use and the number of months since last iCCM integrated refresher training (p = 0.28), nor the number of months since last supporting supervision (p = 0.17). There was a statistically significant positive association between number of years since qualification as HEW and ability to adhere to relevant guidelines using the Rad‐G (OR 1.13; 95% CI 1.04–1.22, p < 0.001). The mean performance time, that is from the moment the Rad‐G was turned on to when the reading was displayed (inclusive of up to three attempts) was 06:58 min (range 1 min 32 s to 36 min 32 s) for observation 1, which decreased to 5 min 42 s (range 1 min 30 s to 22 min 31 s) for observation 2. The proportion of correct assessments relating to the manufacturer's IFU after 2 months of routine utilisation was 97.3% (95% CI 94.4–98.7), representing an increase of 4.3% from when HEWs were assessed for observation 1 immediately after training (Table 4). This compares to adherence to the steps included in the revised iCCM guidelines, which was lower at 80.3% (95% CI 75.0–84.7), but which had also increased by 23.1% from observation 1 (Table 4).

Table 5.

Number and proportion of child consultation steps correctly performed by HEW with Rad‐G after 2 months of routine use, by child age group and breathing status

	Consultation steps	Observation 2 (0–<2 months) N = 7			Observation 2 (2–<12 months) N = 77			Observation 2 (12–59 months) N = 175			Observation 2 (fast breathers) N = 52			Observation 2 (normal breathers) N = 207
		n	%	95% CI	n	%	95% CI	n	%	95% CI	n	%	95% CI	n	%	95% CI
1	Child calm before Rad‐G attempt	6	85.7	25.7, 99.0	73	94.8	86.7, 98.1	169	96.6	92.5, 98.5	49	94.2	83.0, 98.2	199	96.1	92.4, 98.1
2	Correct mode selected	7	100.0	59.0, 1.0	76	98.7	91.0, 99.8	163	93.1	88.2, 96.1	52	100.0	96.4, 1.0	206	99.5	96.9, 99.9
3	Correct age group	7	100.0	59.0, 1.0	76	98.7	91.0, 99.8	175	100.0	97.9, 1.0	47	90.4	78.4, 96.1	199	96.1	92.4, 98.1
4	Correct probe position	7	100.0	59.0, 1.0	76	98.7	91.0, 99.8	175	100.0	97.9, 1.0	52	100.0	96.4, 1.0	206	99.5	96.6, 99.9
5	Correct probe direction	7	100.0	59.0, 1.0	75	97.4	89.9, 99.4	172	98.3	94.8, 99.5	50	96.2	85.3, 99.1	204	98.6	95.6, 99.5
6	Child not eating/feeding during Rad‐G attempt	7	100.0	59.0, 1.0	77	100.0	95.3, 1.0	173	98.9	95.5, 99.7	50	96.2	85.3, 99.1	207	100.0	98.2, 1.0
7	Child calm during Rad‐G attempt	6	85.7	25.7, 99.0	73	94.8	86.7, 98.1	172	98.3	94.8, 99.5	49	94.2	83.0, 98.2	202	97.6	94.3, 99.0
1–8	Cumulative assessment (steps 1–7)	6	85.7	25.7, 99.0	68	88.3	78.8, 93.9	153	87.4	81.6, 91.6	41	78.8	65.2, 88.1	186	89.9	84.9, 93.3
1–8	Correct classification using Rad‐G (yes/no?)	6	100.0	54.1, 1.0	61	96.8	87.8, 99.2	162	95.9	91.5, 98.0	43	84.3	71.1, 92.1	186	99.5	96.2, 99.9

A total of 74 health posts reported data on device use using stickers in their patient registers, of which 71 reported data correctly. A total of 579 pneumonia consultations for children U5 were completed between October–December 2018 (median = 63 days) at these 71 health posts. Of these, 486 (83.9%) were completed with Rad‐G (a mean of 6.6 Rad‐G consultations per health post), 57 (9.8%) were completed with their standard practice device (ARI timer, phone timer or watch), and 36 (6.2%) were completed with an unrecorded device. Of the assessments completed with a known device, 89.5% were completed with Rad‐G.

3.3. Usability and acceptability of Rad‐G

We identified three overarching themes relating to the usability and acceptability of Rad‐G among HEWs and FLHFWs and acceptability to caregivers: (1) adherence to modified iCCM guidelines and manufacturer IFU, (2) usability of Rad‐G and (3) acceptability of Rad‐G.

3.3.1. Adherence to modified iCCM guidelines and manufacturer instructions for use

When discussing the results displayed on Rad‐G after assessments, many HEWs and FLHFWs expressed no difficulty in reading the oxygen and RR levels and that the display was ‘readable’, ‘clear’ and the colours helped to interpret positive and negative readings. Some FLHFWs and just two HEWs reported being able to refer cases based on oxygen level, which they were unable to do before. This indicates that the SpO₂ function on the Rad‐G, when taken in isolation, was still seen as having value over no device. Many HEWs and FLHFWs mentioned using the job aid to help with classification and referral, specifically mentioning how easy or straightforward it was to identify classification categories using the charts in the job aid. Some HEWs mentioned using the job aid to compare the findings on the screen with the charts to decide on the treatment. Notably, some health workers talked about how Rad‐G supported them to improve their antibiotic prescribing behaviour. For example, some FLHFWs reported that they had previously prescribed ‘drugs’ or ‘treated’ children ‘without counting the RR’. Some HEWs said that previously they would treat cases ‘without proper diagnosis’, based on observation only or based on breath counting which ‘isn't accurate’. (See Appendix S1 for quotes).

3.3.2. Usability of Rad‐G

Many HEWs and FLHFWs expressed initial hesitation or fear of the device, explaining how they expected it be ‘confusing’ or ‘difficult to operate’, that they ‘hesitated’ due to lack of knowledge about the device and how they ‘feared’ to use it at first. These initial concerns were alleviated with practice, and most health workers said their ‘skills’, ‘knowledge’ and ‘ability’ to use the device had improved with time and experience. There was considerable discussion about the user interface and errors that impacted health workers’ ability to use the device. For example, many HEWs and FLHFWs mentioned that the device displays an ‘error’ message if the child moves or ‘there is motion’ and this ‘distorts’ the assessment, ‘affects the result’ and makes it difficult to ‘get the correct result’. Some mentioned having to start the assessment again, while others resorted to the ‘previous’ or ‘manual’ method with a few saying they ‘give up’ using it until the next day. Another problem with the interface related to selecting age categories; some FLHFWs explained that the categories are too close to each other resulting in other categories ‘being tapped’ by mistake, while HEWs and FLHFWs mentioned difficulty selecting the age of the child because mothers sometimes did not know the actual age of the child, which then led to inaccurate results.

Some FLHFWs and many HEWs said having the results displayed in the device made it easier to communicate, share or explain the results to caregivers. Across all districts, caregivers described feeling happy about the result of the assessment with the Rad‐G device, because the child ‘has no disease’ or because their child was ‘healthy’. Caregivers from all three districts indicated they would recommend the device to others, and the main reason was because the device ‘identifies disease’; others were keen to express that they would recommend the device based on the fact the it shows the ‘correct result’ or helps ‘rule out whether the child is positive or negative’.

There was widespread agreement among HEWs and FLHFWs and strong views against working on other tasks while Rad‐G is attached to the child and measuring the RR. Many health workers expressed the need to maintain their attention on the child, that ‘concentration’ is needed and that your ‘business should be only with the device and the child being assessed’. They felt a lot of care was needed to control the motion of the child and to ‘focus on the measurement’ and follow the assessment process. HEWs and FLHFWs agreed that older children were generally more easily distracted with the animal images or the ‘game’ on the device screen, although the specific age group varied from those over 1 year, over 2 years or older. Several HEWs talked more generally about children being distressed when they saw the device or the red light, but this was transient, and many calmed down when shown the pictures or the ‘game’ on the device screen (see Appendix S2 for quotes).

3.3.3. Acceptability of Rad‐G

There was consensus among FLHFWs that they preferred Rad‐G compared with previous manual counting methods. The main reason stated was that Rad‐G provided an ‘accurate’ result, despite this study not measuring device accuracy. Others suggested it was easier than counting, that it was better than ‘simply observing’. A few mentioned that it enabled them to provide ‘better treatment’ and the fact it measured oxygen saturation was helpful. A few mentioned the durability of the device and that it does not rely on batteries that can easily run out.

HEWs also preferred Rad‐G compared with the previous method of counting RR, with most reporting Rad‐G was ‘easier’ than counting sometimes ‘repeatedly’ and the potential for ‘missing the correct rate’. Some emphasised the device was ‘faster’ than manual counting and gave the result more ‘quickly’ therefore reducing the assessment time. In contrast to FLHFWs, only very few HEWs mentioned the accuracy of the result as a reason for their preference for Rad‐G, rather they reported that their ability to classify pneumonia cases had improved, and this was important to avoid ‘guessing based on respiratory count’ and the ‘difficulties’ associated with classifying based on counting breaths.

HEWs and FLHFWs talked a lot about children's and mother's fear of the device and particularly the red light within the finger probe. According to the health workers, children would cry and become ‘afraid’ or ‘scared’ of the red light, thinking it is ‘fire’, while mothers also feared that the red light was fire or could burn they were also concerned that the red light might ‘suck blood’ from their child. FLHFWs described how caregiver's initial fear was usually allayed when they took time to ‘explain’ or ‘convince’ them.

When discussing their trust in the device and the results, many FLHFWs talked about checking the result several times before being confident in the result. Others mentioned checking the result with their own clinical observations and comparing the device result with their own ‘estimation’, and a few had ‘total trust’ in the device ‘regardless of drawbacks’ because it is ‘modern’ or the ‘latest technology’. Both HEWs and FLHFWs perceived that caregivers generally accepted Rad‐G, and many said this was because it is a ‘new device’ which had generated a lot of interest in the community. Across all three districts, caregivers reported broad acceptance of the Rad‐G. A common view among caregivers was that the ‘government brings only good things’ that do not cause harm and ‘what government brings does not worry’. FLHFWs and HEWs talked a lot about how having the device available at the health post or health centre enhanced either their own credibility or that of the service provided. Health workers explained that credibility was linked to caregivers’ preference for assessment with the device, and their belief that their children recover quickly if assessed with a ‘medical device like Rad‐G rather than giving treatment without a device’. HEWs were more likely to reflect that their own ‘motivation to work more’ had increased because people ‘appreciated my work’ or the device enabled them ‘to deliver an effective service’. Another common view among HEWs and FLHFWs was that the availability of the device for diagnosis combined with free treatment encouraged caregivers to the health centre and the health post, and those caregivers whose children recovered after ‘treatment using Rad‐G’ also encouraged others to attend. (See Appendix S3 for quotes).

4. DISCUSSION

This is the first study to examine usability and acceptability of a multimodal device with RR counting capabilities and pulse oximetry at the community level. The results are encouraging, demonstrated by high levels of usability, with HEWs adhering to the required steps in assessing and classifying U5 children using Rad‐G in more than three‐quarters of assessments. Furthermore, the data show that through adequate training and continued practice, using Rad‐G over a 2‐month period the HEWs’ ability to adhere to the required guidelines improved, as did their ability to calm the child before the attempt and get successful readings with the device, both for RR and SpO₂. Furthermore, we saw a significant association between the number of years since being trained and the adherence of the user. Therefore, it is key to provide adequate training, including clear instruction on using both clinical signs and device readings to correctly diagnose patients and adequate focus on device specific training, particularly on probe fit and signal strength. This was highlighted by HEWs when interviewed, who also stated that they found the support of the job aid valuable when classifying children using Rad‐G. Given the number of failures and difficulties expressed by HEWs using the device during observation 1 the research team conducted a half‐day focused training to support HEWs further on probe fit and signal strength issues with using Rad‐G. This reflects the findings in other recent studies which highlight the need for a task‐based active learning approach to training when introducing pulse oximetry in new settings. ²⁵

High levels of acceptability were reported in this study, with both HEWs and FLHFWs expressing desire to keep using the device and feeling it improved their self‐efficacy and encouraged care seeking behaviours when combined with free treatment. Again, this reflects the findings of previous studies, which requested the development of these types of multimodal devices, offering both RR and SpO₂ measurements. ²⁵ , ²⁶ Health workers also explained that they needed to provide a lot of explanation to alleviate children's fear of the red light emitted by the probe. This could have contributed to the high proportion of children who were not calm before and during assessments. Health workers also reported that trust in the device was low at first and that HEWs said they knew not to trust the result if the child was not calm.

A number of issues with the usability of Rad‐G were highlighted. Firstly, further work needs to focus on a potential redesign of the probe to ensure better fit, especially in the younger children. Such a probe has recently been developed and tested by Lifebox ²⁶ and Masimo have also planned further development of their probe for the Rad‐G. Alternatively, multiple probes may need to be developed to allow these types of devices to be used effectively on all ages of children, as argued by King et al. ²⁷ Secondly, the age cut‐offs need to be mutually exclusive to avoid any confusion when HEWs select the age group of the child. Finally, the classification options need to be simplified for the device to be effectively used by HEWs at the community level.

In this study, the average time to obtain a reading with the Rad‐G device for HEWs immediately after training was close to 7 min, ranging from 1 min 32 s to 36 min to 32 s. This reduced by just over a minute after 2 months of practice and still with a large range. While Masimo has developed the ‘Signal Extraction Technology’, or Masimo SET, which has been shown to better account for movement artefacts, ²⁸ the manufacturer has confirmed that this technology has not been applied to the RR measurements from Rad‐G. This could explain the longer times this device was taking to get a successful reading. Further developments by the manufacturer to increase ease of use and reduce reading errors due to motion disturbance are therefore recommended to shorten this time period, as this may otherwise become a major hurdle for its wider use if rolled out in a busy health post or health centre.

A strength of our study is that data quality was maintained as source documents were used to verify the age selected by the HEW and the RR/SpO₂ and classification of Rad‐G. A limitation of this study is that it did not compare the usability and acceptability of Rad‐G versus standard practice, nor did we use a ‘gold standard’ to review the HEWs’ management of the child, as done in similar studies of other RR diagnostic aids. ⁷ , ²⁹ A further limitation is that a second refresher training was deemed necessary by the research team in the period between the two observations, in order to improve skills on probe fit and signal strength. This may have influenced results positively during observation 2. It also proved challenging to recruit children less than 2 months old from the health post, given that cultural barriers discourage women from taking their newborns out of the home when the babies are young. Instead, newborns were seen by HEWs during home‐based post‐natal check‐ups. A recruitment drive towards the end of data collection increased the proportion of young infants in the study, but the numbers were still small, and the study was not sufficiently powered to detect a difference in adherence to assessment and classification steps between age groups. While the research team took steps to minimise the possibility that HEW’s changed their behaviour when being observed, by silently observing child assessments and not interfering with the assessments, it is possible that the results are influenced by the ‘Hawthorne effect’. There is potential for courtesy bias in the qualitative data; it is possible that some interviewees may have responded in ways they felt were appropriate rather than reflecting their own experience or views. To minimise this, the interviewers ensured the HEW was not present for the caregiver interview and were trained to build rapport with the interviewee to make them feel comfortable.

5. CONCLUSIONS

The findings suggest that with adequate training and a comprehensive job aid, Rad‐G supports HEWs in SNNPR, Ethiopia to adhere to the required modified iCCM guidelines and manufacturer's IFU in children under five. HEWs and caregivers both found the device acceptable and usable. Specific modifications to the user interface and the probe could improve the usability and also the time taken to get a reading. The findings from this study support the rationale for further studies on performance, cost‐effectiveness and implementation, of this and other multimodal devices to inform policy decisions in countries with a high burden of childhood pneumonia.

CONFLICT OF INTEREST

We declare there is no conflict of interest as these are discrete pieces of work.

Funding information

This study is conducted in partnership with UNICEF and was funded through a grant from the ‘La Caixa’ Foundation. The manufacturer of the devices (Philips and Masimo) did not contribute any funding for this work and did not participate in any data analysis or interpretation. The Malaria Consortium research team has subsequently received funding from the Philips Foundation to conduct future work on developing an improved RR reference standard.

Supporting information

Supplementary Material

Baker K, Ward C, Maurel A, et al. Usability and acceptability of a multimodal respiratory rate and pulse oximeter device in case management of children with symptoms of pneumonia: A cross‐sectional study in Ethiopia. Acta Paediatr.2021;110:1620–1632. 10.1111/apa.15682