Accuracy and reliability of a wireless vital signs monitor for hospitalized patients in a low-resource setting

Abstract

Objective

The purpose of this study was to evaluate the accuracy and reliability of neoGuard in comparison to a conventional bedside monitor on patients in a low-resource clinical setting.

Design

This was a single-arm methods comparison study involving the use of a wearable vital signs monitor (neoGuard^TM) versus a conventional bedside monitor (Edan iM8).

Setting

The study was conducted at Jinja Regional Referral Hospital, a tertiary care hospital situated in Eastern Uganda.

Participants

Thirty patients (10 male, 20 female) were enrolled from the adult recovery ward at JRRH. Participants were eligible for the study if they were at least 18 years of age, had 2 sets of normal vital sign measurements obtained 1 h apart, and were able and willing to provide informed consent.

Main Outcome and Measures

The primary outcome measures were (i) bias (mean deviation) and (ii) limits of agreement [95% CI]. Bland-Altman plots were generated to illustrate the level of agreement between the neoGuard^TM technology and the Edan iM8 monitor.

Results

Bland-Altman analysis was performed for 24 participants; datasets from six participants were excluded due to missing or invalid measurements. Findings showed a moderate level of agreement for measurement of SpO₂, PR, and RR, with >80% of subject means falling within the predefined acceptability limits. However, there was also notable variation in accuracy between subjects, with large standard deviations observed for measurement of all four parameters. While the level of agreement for measurement of temperature was low, this is partly explained by limitations in the comparison method.

Introduction

Continuous vital signs monitoring is a basic tenet of specialized care in the developed world, but has limited availability in clinical care settings in most low-and-middle income countries. Despite the positive outcomes associated with continuous vital signs monitoring, including early detection of complications, lower complication rates, shorter length of stay in hospital, and a reduced risk of severe complications and mortality,^1,2 the prohibitive costs of conventional patient monitors and the difficulty in maintaining complex medical equipment ³ limit its utilization in resource-constrained settings. Medical practitioners in these settings instead rely on intermittent vital signs monitoring, a practice that is more likely to yield suboptimal patient outcomes as it leaves greater room for missed detection of early warning scores of patients in distress. Research shows that delays in recognizing clinical deterioration can prevent clinicians from providing more timely and effective interventions, leading to a higher risk of complications and failure-to-rescue.^4–6

In recent years, advancements in technology have given rise to the continuous evolution of wearable health devices (WHDs) with potential application for both personal health tracking and clinical monitoring. ⁷ In addition to being more affordable than traditional bedside counterparts, other key features that are drawing more health providers to WHDs include: wireless mobility; interactivity and intelligence; sustainability and durability; simple operation and miniaturization; and of course, wearability and portability. ⁸ With a growing list of technologies that fit this bill, it is not surprising that the global health care industry has shown considerable interest in adopting wearable devices in clinical settings.

Presently, there are several wearables that have received Conformitè Europëenne (CE) mark and/or United States Food and Drug Administration (FDA) clearance, including ViSi Mobile® (Sotera Wireless, San Diego, CA, USA), VitalPatch® (Vital Connect, San Jose, CA, USA) and Sensium® vitals patch (The Surgical Company, Oxford, United Kingdom (UK)). These devices are supported by data that suggests that WHDs perform optimally in hospital settings, have the potential to improve the timely detection of early warning scores, and are positively received by both clinical staff and patients.^9–14

A recent addition to the growing list of WHDs approved for hospital use is the neoGuard^TM technology (Neopenda, PBC, Chicago, IL, USA); a wireless vital signs monitor that continuously measures four parameters: pulse rate (PR), respiratory rate (RR), peripheral oxygen saturation (SpO₂), and temperature. The wearable device is worn on the patient's forehead and wirelessly transmits real-time measurements to a central monitoring application (visible on a tablet) via Bluetooth Low Energy® (Figure 1). Health care staff are able to simultaneously monitor up to 15 patients wearing the devices from just one tablet and receive immediate alerts when a vital sign measurement exceeds the preset alarm limits, an action that can trigger a timely intervention to a patient's changing status.

Figure 1.

The neoGuard vital signs monitoring system.

In 2021, the neoGuard^TM technology received CE mark clearance and has subsequently been introduced to health facilities in Uganda, Kenya, Tanzania, and Nigeria. This study is among the first to examine the performance of the neoGuard^TM technology in hospitalized patients in a low-resource setting. The primary objective was to measure the level of agreement between the neoGuard^TM technology and a conventional bedside monitor, Edan iM8 (Edan, Shenzen, China). The secondary objectives were to evaluate the reliability and feasibility of a wearable sensor attached to the patient's forehead.

Methods and materials

Study design and setting

This was a single-arm methods comparison study involving a wearable vital signs monitor attached to the patient's forehead (neoGuard^TM) and a conventional bedside monitor (Edan iM8) connected to the patient via 5-lead Electro Cardio Gram (ECG) electrodes, an axillary temperature probe, and a pulse oximetry finger sensor. The Edan iM8 monitor and neoGuard^TM devices used in this study were obtained specifically for research purposes. The Edan iM8 was chosen as a comparison monitor because Edan is one of the most common brands of patient monitors available in this market and the iM8 model measures all the same parameters as the neoGuard^TM technology.

The study was conducted over a 4-week period at Jinja Regional Referral Hospital (JRRH), a tertiary care hospital situated in Jinja Town in Eastern Uganda. The recovery ward was specifically chosen for selection of study participants because it admits low-acuity patients. Data collection was completed in February 2021.

Study population

A total of 30 consecutive patients were screened from the adult recovery ward and invited to participate in the study. To be eligible for the study, participants met the following inclusion criteria: at least 18 years of age, had two sets of normal vital sign measurements obtained 1 h apart, and were able and willing to provide informed consent. Participants were excluded from the study if they had been admitted for management of a thyroid condition, pre-eclampsia, or eclampsia, as these conditions are characterized by sudden changes in vital signs. All 30 participants approached expressed interest in participating in the study and provided written informed consent. The study participants included 10 male patients and 20 female patients.

Study procedures

Study participants were sampled consecutively by study nurses during routine admissions. If a patient expressed interest in participating in the study after the study objectives and procedures had been explained to them, a study physician was notified to perform a screening exam to confirm participant eligibility. Upon confirmation of eligibility and obtaining informed consent, a study nurse enrolled the patient into the study by assigning a unique participant ID and administering a short demographic questionnaire. The study nurse then retrieved one neoGuard^TM device from storage and affixed it to the patient's forehead. Simultaneously, the patient was connected to the Edan iM8 bedside monitor via 5-lead ECG electrodes, an axillary temperature probe, and a pulse oximeter finger sensor.

For each participant, the neoGuard^TM device and Edan iM8 monitor were left to log vital sign measurements concurrently for a 30-min observation period. At the end of the observation period, data logs were extracted from the system memory and saved to an external hard drive. Since this was an accuracy validation study, vital sign data collected during study procedures were not used to influence patient care. However, research personnel were instructed to immediately halt study procedures and notify the attending health staff if the observed vital sign readings prompted any patient safety concerns. No such incident occurred during the study.

Outcome measures

The primary outcome measures were (i) measurement bias (mean deviation) and (ii) limits of agreement (LOA) [95% CI] for measurement of SpO₂, PR, RR, and temperature. Bland-Altman plots were also generated to illustrate the level of agreement between the neoGuard^TM technology and the Edan iM8 monitor for all four parameters.

Two secondary measures were used to describe the reliability of neoGuard^TM: (i) the proportion of participants with adequate signal quality to obtain measurements for the neoGuard^TM device and the (ii) the overall percentage of data points excluded due to artifacts (missing or invalid measurements).

Statistical analysis

Applying Lu et al.'s method for sample size estimation for Bland-Altman analysis, ¹⁵ we pre-specified the expected/target value for each vital sign parameter and determined that a minimum of 19,152 paired measurements would be sufficient to estimate accuracy for SpO₂, PR, and temperature, α = 0.05, β = 0.20. Similarly, we estimated that a minimum sample size of 226 paired measurements would be sufficient to estimate accuracy for RR, α = 0.05, β = 0.20.

For practical significance, we collected 30 min of data from 30 participants, to allow for ample stabilization of readings and mitigate any data collection missed due to unexpected technical or physical interruptions, such as the device falling off the patient or delayed connectivity between the device and the central monitor. Participant characteristics including age, sex, ethnicity, and clinical disposition were summarized using descriptive statistics.

Vital sign log files from the neoGuard^TM and Edan iM8 monitors were extracted and copied to Excel for pre-processing. Pre-processing involved linking data points by their timestamps, checking for incomplete and invalid measurements. Missing and invalid data were excluded from analysis.

Pre-processed files were pooled into a single dataset and loaded to MedCal statistical software where we applied the Bland-Altman approach for evaluating agreement between two methods of clinical measurement, correcting for the multiple measurements per individual subject. ¹⁶ For each vital sign, the measurement bias was obtained by averaging the pair-wise differences of data points, while the upper and lower LOA were calculated using the formula:

$$\mathrm{LOA}=\mathrm{Bias}\pm 1{.96}^{*}\mathrm{total}\mathrm{SD}.$$

The expected/target or predefined values for LOA that we aimed to achieve were based on the acceptable clinical standards and the anticipated limitations of field testing. For SpO₂, we aimed to obtain a predefined LOA ≤ ± 4%; for PR, a predefined LOA ≤ ± 6 beats per minute (bpm), for RR, a predefined LOA ≤ ± 7 breaths per minute (brpm); and for temperature, a predefined LOA ≤ ± 0.5 ^°C (Table 1).

Table 1.

Summary statistics for Bland-Altman plots for measurement of SpO2, PR, RR and temperature.

	SpO2	PR	RR	Temp
Expected values
Bias	0.0	0.0	0.0	0.0
Total SD	2.0	3.0	3.0	0.25
Upper LOA	4.0	6.0	7.0	0.50
Lower LOA	−4.0	−6.0	−7.0	−0.50
Actual values
Bias	1.73	0.56	0.90	0.06
Total SD	2.44	5.18	6.34	0.70
Upper LOA [95% CI]	6.51 [5.61, 7.85]	10.71 [9.98, 11.57]	13.33 [11.42, 16.04]	1.44 [1.17, 1.84]
Lower LOA [95% CI]	−3.06 [−4.40, −2.15]	−9.58 [−10.45, −8.86]	−11.53 [−14.24, −9.62]	−1.32 [−1.72, −1.06]

Results

Participant characteristics

A total of 30 participants (10 males and 20 females) met the inclusion criteria and were enrolled in the study. The median (IQR) age for male participants was 49 (35–53), while the median age for female participants was 27 (23–32). All study participants were of African ethnicity and determined to be clinically stable on examination.

Missing and invalid data

Comparison readings for SpO₂, PR, and temperature were captured at second-to-second intervals, generating a total of 34,296 paired measurements for SpO₂ and PR, and 30,034 paired measurements for temperature from 30 participants. Comparison readings for RR were captured at minute-to-minute intervals, generating a total of 794 paired measurements from 30 participants.

While all 30 participants completed the study procedures, Bland-Altman analysis was only performed for 24 participants, as neoGuard^TM readings from 6 participants (1 male and 5 females) were excluded due to unstable signal quality characterized by low red light absorption levels (<400,000) and a high percentage of missing or invalid measurements. The proportion of data points excluded due to missing or invalid measurements was: 3709/34296 (10.8%) for SpO₂, 3482/34296 (14.3%) for PR, 4018/30034 (13.4%) for temperature and 88/794 (11.1%) for respiratory rate. Overall, we generated approximately 10 h of performance data for analysis.

Bland-Altman analysis

The Bland-Altman plots in Figures 2 to 5 show the dispersion of pair-wise differences (y-axis) plotted against corresponding pair-wise means (x-axis) for each vital sign. Figure 2 shows that there was moderate agreement for SpO₂ readings, with a measurement bias of 1.73 and 87.5% of the subject means falling within the predefined LOA (±4%). Figure 3 shows that there was moderate agreement for PR readings, with a measurement bias of 0.56 and 100% of subject means falling within the predefined LOA (±6 bpm). Figure 4 shows that there was moderate agreement for RR readings, with a measurement bias of 1.2 and 80% of the subject means falling within the predefined LOA (±7 brpm). Figure 5 shows that there was low agreement for temperature readings, with a mean deviation of 0.06 and 52% of subject means falling within the predefined LOA (±0.5 °C).

Figure 2.

Bland-Altman plot for comparison of neoGuard (SpO2n) and Edan iM8 (SpO2e) oxygen saturation readings.

Figure 5.

Bland-Altman plot for comparison of neoGuard (TEMPn) and Edan iM8 (TEMPe) temperature readings.

Figure 3.

Bland-Altman plot for comparison of neoGuard (PRn) and Edan iM8 (PRe) pulse rate readings.

Figure 4.

Bland-Altman plot for comparison of neoGuard (RRn) and Edan iM8 (RRe) respiratory rate readings.

While all four figures depict that majority of the subject means were concentrated within the predefined LOA (green solid lines), they also indicate that the actual LOA (red dashed lines) and corresponding 95% CI (blue perpendicular bars) were considerably wider than the predefined LOA. This is further illustrated in Table 1, which shows the difference between expected (predefined) inputs and actual (observed) inputs for the Bland-Altman plots.

Reliability

The neoGuard^TM device was able to detect an optimal signal quality to transmit stable readings for 24/30 (80%) of the study participants. Potential explanations for the low signal quality observed in the remaining 6/30 (20%) of participants include: improper sensor placement, subject movement, skin pigmentation, sweatiness at the placement site or low blood perfusion, but the extent to which each factor may have contributed to the low signal quality cannot be quantified from the data available.

For the six subjects with low signal quality, the total number of data points observed to be missing or invalid were recorded. The total proportion of unreliable readings (defined as missing or invalid due to low signal quality) from these six subjects was: 3709/34296 (10.8%) for SpO₂, 3482/34296 (10.2%) for PR, 4018/30034 (13.4%) for temperature, and 88/794 (11.1%) for RR.

Discussion

While the idea of adopting WHDs in clinical settings is steadily gaining traction, there remain notable implementation challenges including limited accuracy and reliability, as well as mixed perceptions of usability, acceptability, and patient preferences.

With regard to accuracy and reliability of vital sign measurement, this study demonstrated that the overall performance of neoGuard^TM is very promising, but presently less optimal than desired due to the high level of variability between subjects. Similar studies on other wearable vital sign monitors have noted comparable challenges during field testing and early adoption of WHDs, specifically, the limited capacity to distinguish between true physiological changes and patient-based artifacts like motion.^17–21 This is a prominent concern because it can lead to over-detection of mostly benign vital sign changes or false positives which can create a feeling of burnout for nurses and clinicians who are constantly responding to invalid vital sign alarms, an experience known as alarm fatigue. ²²

One approach to minimizing the impact of motion artifacts is to design wearables for body parts other than the fingers, wrists, or arms. Body parts like the head, chest, and abdomen may be more suited for placement of wearable devices since they tend to display more subtle and less frequent movements. In the present study, a wearable vital sign device attached to the forehead demonstrated a moderate level of accuracy and reliability against a conventional bedside monitor; however, substantial motion artifacts and inadequate signal quality for 6/30 (20%) of study participants were still observed.

Based on the results of this and other field studies, the accuracy and reliability of neoGuard^TM readings are undergoing optimization. One area of advancement is the “smoothness” of the readings. Users reported that neoGuard readings fluctuate much more than that of other conventional monitors; this variability is apparent in these results and is due to the responsiveness of the algorithms to sensor inputs and new algorithm outputs every second. An adaptive smoothing algorithm will be added to the pulse rate and respiration rate readings in order to filter out aberrant outputs and provide a smoother, averaged reading. Advanced methods for improving resistance to motion artifacts shall also be investigated, such as utilization of accelerometer sensors to filter out movements from the sensor signal.

In light of the Coronavirus Disease 2019 (COVID-19) pandemic, researchers have also highlighted key concerns regarding the effect of skin pigmentation on pulse oximeter accuracy. As pulse oximetry relies on optical measurements, it has been suggested that accuracy may be decreased in darker skin patients.^23–27 The FDA issued a safety communication in 2021 highlighting the concern of potential inaccuracies of pulse oximeters in darker skin patients, and the need for further evaluation of the relationship between accuracy and skin pigmentation. ²⁸ The FDA provides guidance that at least 15% of the subjects tested during a desaturation study for accuracy validation for SpO₂ should have dark skin. In the present study, 100% of patients tested had dark skin; further evaluation of neoGuard^TM performance on dark skin patients is warranted and will help generate additional data to optimize neoGuard^TM performance in this population.

Despite the current limitations with WHDs, there remains a critical gap in continuous vital sign monitoring for patients in low resource settings. Moreover, nurses and clinicians who are the primary end-user of these products have shown a willingness to try out less conventional medical equipment and an eagerness to share feedback on the technological features and performance issues that can be improved on to make WHDs more attractive and acceptable to them. This response should therefore encourage innovators and researchers to continue making new strides towards realizing more feasible, accurate, reliable, and sustainable vital sign monitoring solutions for resource-constrained settings.

Study limitations

The study had some limitations.

1. The sample of participants consisted of only clinically stable adult patients whose vital signs were within clinically acceptable ranges. This limits the generalizability of these findings. Further research on other patient subpopulations like newborns and pediatrics or those with a high acuity clinical status is needed.

2. Ambient temperature and placement of temperature sensors on different body parts (e.g. chest vs forehead) can have a significant effect on observed temperature readings. This makes it very difficult to accurately assess the performance of temperature sensors outside of a controlled lab setting and may partly explain the low agreement for temperature readings.

3. Due to practical limitations and ethical considerations, researchers opted to use a commercially available bedside monitor as opposed to the same clinical validation instruments that were used to establish the expected accuracy of neoGuard^TM prior to CE mark clearance. For instance, oximetry was previously validated against arterial blood gas (ABG) analysis in healthy participants. However, since ABG analysis would have required repeated blood draws from a small sample of recovering patients, researchers opted for a less robust, but less invasive comparison method that would render the least discomfort to this group.

Study strengths

The study also had several strengths.

1. Measurement error was minimized by capturing vital sign readings from digital data logs, as opposed to having vital sign readings manually recorded by the study nurses.

2. The study sample consisted of dark-skinned participants of African ethnicity. Due to the concerns that other researchers have raised regarding the effects of skin pigmentation on the accuracy of pulse oximetry, this is a relevant subpopulation to focus on when evaluating the performance of clinical measurement tools that rely on optical sensors.

3. The research team was able to collect a large enough sample of data to generate precise estimates of variability both within and between subjects.

4. All data collection was observed by a research assistant who was trained to identify and limit the occurrence of user-based errors.

Conclusion

While the neoGuard^TM device had successfully completed clinical accuracy validation studies required for CE Mark regulatory approval, it is typical for performance in the field to vary from the clinical accuracy validation due to the introduction of new variables (patient movement, environmental conditions, device placement, diversity of subjects, diversity of users, etc.). It is an important objective of post-market clinical research studies such as this to monitor product performance in new environments and use cases, and collect data regarding deviations from laboratory accuracy results so that the product can be improved.

In theory, a wearable device connected to the forehead may be ideal for minimizing motion artifacts, but this design choice should be balanced with practical considerations such as user acceptability and patient preferences. Further modifications and additional research on the neoGuard^TM solution are ongoing.