Prediction of low pulse oxygen saturation in COVID-19 using remote monitoring post hospital discharge

Abstract

Background

Monitoring systems have been developed during the COVID-19 pandemic enabling clinicians to remotely monitor physiological measures including pulse oxygen saturation (SpO₂), heart rate (HR), and breathlessness in patients after discharge from hospital. These data may be leveraged to understand how symptoms vary over time in COVID-19 patients. There is also potential to use remote monitoring systems to predict clinical deterioration allowing early identification of patients in need of intervention.

Methods

A remote monitoring system was used to monitor 209 patients diagnosed with COVID-19 in the period following hospital discharge. This system consisted of a patient-facing app paired with a Bluetooth-enabled pulse oximeter (measuring SpO₂ and HR) linked to a secure portal where data were available for clinical review. Breathlessness score was entered manually to the app. Clinical teams were alerted automatically when SpO₂ < 94 %. In this study, data recorded during the initial ten days of monitoring were retrospectively examined, and a random forest model was developed to predict SpO₂ < 94 % on a given day using SpO₂ and HR data from the two previous days and day of discharge.

Results

Over the 10-day monitoring period, mean SpO₂ and HR increased significantly, while breathlessness decreased. The coefficient of variation in SpO₂, HR and breathlessness also decreased over the monitoring period. The model predicted SpO₂ alerts (SpO₂ < 94 %) with a mean cross-validated.

sensitivity of 66 ± 18.57 %, specificity of 88.31 ± 10.97 % and area under the receiver operating characteristic of 0.80 ± 0.11. Patient age and sex were not significantly associated with the occurrence of asymptomatic SpO₂ alerts.

Conclusion

Results indicate that SpO₂ alerts (SpO₂ < 94 %) on a given day can be predicted using SpO₂ and heart rate data captured on the two preceding days via remote monitoring. The methods presented may help early identification of patients with COVID-19 at risk of clinical deterioration using remote monitoring.

1. Introduction

The COVID-19 pandemic has resulted in significant morbidity and mortality, placing a major strain on healthcare systems worldwide. Symptoms of COVID-19 can persist for weeks or months, and a long lasting sequelae of COVID-19 infection, or ‘long covid’, is now a recognised diagnosis [1], [2], [3]. While initial COVID-19 symptoms are frequently mild, clinical deterioration to severe disease is common in certain populations and predicting those who will deteriorate can be difficult [4]. In patients with COVID-19 recently discharged from hospital, remote monitoring can enable early recognition of patient deterioration and facilitate reassessment and readmission if necessary [5], [6], [7]. Remote monitoring may also allow identification of patients experiencing low oxygen saturation levels in the absence of signs of respiratory distress [8], [9], [10], [11].

Previously reported systems for remote monitoring of patients with COVID-19 following hospital discharge have used mobile apps [6], [7] and daily surveys [5], [12] to monitor oxygen saturation (SpO₂), heart rate (HR), body temperature, and self-reported symptoms [5]. To enable early detection of worsening clinical status, the monitoring systems sent automatic alerts to the clinical team if SpO₂, HR, temperature or breathlessness score passed defined thresholds [5], [6], [7]. These systems enabled early identification of patients who required additional assessment or intervention, supporting hospital discharge during a period of intense demand on hospital capacity, and demonstrated high levels of patient adherence.

Clinical remote monitoring systems, however, have not yet been harnessed to investigate how symptoms and vital sign measures change over time in patients with COVID-19, or to develop methods to predict clinical deterioration which may enable more efficient remote monitoring of large patient groups. Machine learning models have been used to predict clinical deterioration in patients hospitalised with COVID-19 based on clinical and laboratory data [13], [14], [15], [16], [17], [18], [19], [20], and hospitalisation and illness in individuals with COVID-19 using wearable sensors [21], [22]. Models using clinical data from patients hospitalised with COVID-19 have also been used to predict intensive care unit admission [13], [14], in-hospital mortality [13], [14], [15], clinical deterioration defined as either intubation, intensive care unit admission, in-hospital mortality, or a high National Early Warning Score [16], [17], [18], [19], [20]. Low SpO₂ and elevated respiration rate have also been reported as risk factors for in-hospital death using multivariate regression [15], [23]. Prior to hospitalisation, models have been reported to predict COVID-19 infection [21], hospitalisation [22] and sickness on a given day [22] in individuals with COVID-19 based on respiration and HR measures using large datasets obtained from consumer wrist-worn devices. A method to predict clinical deterioration in patients with COVID-19 following discharge from hospital using objective remote monitoring data, such as SpO₂ and HR, has not yet been reported.

In this study, SpO₂, HR and breathlessness data recorded using a remote monitoring system in 209 patients with COVID-19 following discharge from hospital were retrospectively examined. A predictive model was developed to identify patients at risk of experiencing low SpO₂ (<94 %) requiring assessment and possible readmission. The associations between age and sex, and symptom presentation when SpO₂ was<94 % were also investigated.

2. Methods

2.1. Remote monitoring system

During 2020, a remote monitoring system was used by the Health Service Executive in Ireland as an early discharge pathway for patients with COVID-19 [6]. On discharge from hospital, patients with COVID-19 were provided with a Bluetooth Smart Pulse Oximeter (NoninConnect 3230, Nonin Medical Inc., Plymouth, MN, USA) linked to a mobile application to monitor SpO₂, HR and self-reported breathlessness (patientMpower ltd, Dublin, Ireland). Recorded data were encrypted and sent to a secure cloud database accessible only to the hospital-based COVID-19 monitoring teams. The mobile application sent a prompt to check SpO₂ at rest, four times daily for 14 days after hospital discharge, and patients could enter additional data/measurements at will. SpO₂ and HR were measured by the pulse oximeter, and data were automatically captured by the mobile application. A prompt also asked if the patient felt more breathless than usual. If the patient selected “yes”, they were asked to rate their breathlessness using a visual analogue scale, ranging from 1 (no symptoms) to 10 (worst ever). A clinical alert was generated if SpO₂ was < 94 %, with an SMS text sent to the hospital monitoring team for clinical follow-up. Age and sex were optional entries in the monitoring system.

2.2. Dataset

Data recorded by patients with COVID-19 who used the remote monitoring system following discharge from six Irish hospitals in the period 1st April – 30th June 2020 were examined retrospectively in this study. During this period, 209 patients (aged > 18 years) agreed that their de-identified data could be used for research purposes (via an ‘opt in’ statement in the mobile app), and their de-identified data were shared with the researchers. Ethical approval was obtained from the Human Research Ethics Committee at University College Dublin and a consent declaration for the retrospective analysis of these data was obtained from the Health Research Consent Declaration Committee.

2.3. Data analysis

SpO₂, HR and breathlessness data from the first ten days after hospital discharge were examined. For each patient, the mean and coefficient of variation (CV) of all SpO₂, heart rate and breathlessness measures were calculated for each day. The minimum SpO₂, and the maximum HR and breathlessness scores were also calculated daily for each patient. Linear mixed-effects models were used to examine the effect of time (days since discharge) on these nine variables, with a random intercept for each patient. The threshold for significance was p < 0.005, adjusted using Bonferroni’s method accounting for multiple comparisons [24].

For every patient, each day was categorised as an alert day (SpO₂ < 94 %), or a non-alert day (SpO₂ ≥ 94 %). An SpO₂ threshold of 94 % was chosen for consistency with the clinical remote monitoring system used to record the data [6], and with the threshold for clinical alerts used by other remote monitoring systems for patients with COVID-19 who have been discharged from hospital [5], [7]. However, it should be noted that in hospitalised patients with COVID-19, lower SpO₂ thresholds would be more appropriate to define clinical deterioration [25]. Alert days were further categorised as symptomatic or asymptomatic, using heart rate and breathlessness scores recorded while the patient was resting. On an alert day, if breathlessness (breathlessness score > 1) or an elevated heart rate (>100 bpm) were recorded, this day was considered a symptomatic alert day. If breathlessness was not reported (breathlessness score = 0) and an elevated heart rate was not detected (<=100 bpm) on a day when SpO₂ < 94 % was recorded, the day was considered an asymptomatic alert day.

To investigate how age and sex are associated with the occurrence of asymptomatic alerts, three patient groups were examined: patients who experienced at least one asymptomatic alert day (Asym), patients who experienced symptomatic alert days, and did not experience any asymptomatic alert days (Sym), and patients who did not experience any alert days during the ten days (NoAlert). Age and biological sex were compared across patient groups using linear mixed-effect models, with a random effect for patients, and fixed effects for patient group (Asym/Sym/NoAlert), age and sex. Differences in overall mean HR and overall mean SpO₂ between patient groups were also examined, with age and sex considered fixed effects with a random subject-specific intercept. The threshold for significance was p < 0.015, adjusted using Bonferroni’s method accounting for the four models developed [24].

2.4. Predictive model development

A random forest classifier was developed to predict SpO₂ alerts (SpO₂ < 94 %) on a given day (Day_A) in the period Day_3-10 (Day₁ was the day of hospital discharge), using SpO₂ and HR data from the discharge day and the two days preceding Day_A (Day_A-1, Day_A-2), and validated internally using tenfold cross-validation.

For the 196 patients who used the monitoring system for at least 3 days, 22 features were extracted and considered during sequential feature selection. For discharge day, the day before (Day_A-1) and two days before (Day_A-2) the day of interest (Day_A, where A = 3–10): mean, CV and minimum SpO₂, mean, CV and maximum HR (18 features). The difference in mean and CV SpO₂ (SpO₂ mean diff and SpO₂ CV diff), and difference in mean and CV HR (HR mean diff and HR CV diff), between Day_A-1 and Day_A-2 were also examined (4 features). Pearson’s correlation coefficient was calculated for each possible pair of features.

During the eight-day period (days 3–10), 70 patients experienced SpO₂ alerts across 149 monitoring days. To develop a model to predict SpO₂ alerts days, validated using cross-validation, data for these 149 alert days were used with data for 157 monitoring days with no SpO2 alerts, Fig. 1 . The 70 participants who experienced an alert during the monitoring period recorded data on 354 days where an SpO₂ alert did not occur. Ninety-seven of these 354 days were identified with no SpO₂ alerts on the two preceding or two subsequent days (from 34 participants). Data for 60 monitoring days (from 49 patients) were also randomly selected from the 126 patients (865 days) who did not experience any alerts, Fig. 1. The final model development dataset included 306 patient days, with data from 119 patients.

Fig. 1.

Flowchart illustrating breakdown of remote monitoring data, and selection of data for model training. Data used during cross-validation are shown using shaded blocks.

Tenfold cross-validation was used to develop the predictive model, and to assess model performance, with feature selection and model optimisation performed within each fold [26], [27]. Data were stratified into ten folds, ensuring that data for each individual participant were present in one fold only (9 folds included data for 12 participants, and 1 fold included data for 11 participants). The number of patients days in each fold ranged from 25 to 37 days, with a mean (standard deviation) of 30.6 (3.9) days.

Sequential forward feature selection was implemented within 10-fold cross-validation to select the optimum combination of five features to predict days with SpO₂ alerts. Bayesian optimisation was used to tune the number of learning cycles for each random forest classifier. These measures were implemented to increase the interpretability and robustness of the final model. A final model was then developed by resubstituting all data.

Model performance was assessed using the mean area under the receiver operating characteristic (AUC ROC), sensitivity and specificity across all folds. Sensitivity, or the true positive rate, was calculated as the number of correctly predicted alerts divided by the total number of alerts. Specificity, or one minus the false positive rate, was calculated as the number of correctly predicted non-alert days, divided by the total number of non-alert days. The specificity of the predictive model was additionally examined using the remaining data for 805 patient days recorded by participants who had no alerts during the monitoring period (Fig. 1).

Data analysis and model development were conducting using custom-developed scripts in Matlab (Mathworks, Natick, MA, USA).

3. Results

3.1. Data summary

Data for up to ten days following discharge were examined, resulting in 1368 patient days. On the day of hospital discharge, 209 patients used the remote monitoring system, with this number reducing over the ten-day period to 134 active users on day 10 (Fig. 2 ). Over the ten-day monitoring period, the total dataset included 1378 days with breathlessness scores entered by 190 patients, 1775 days with SpO₂ measurements by 209 patients, and 1763 days with HR measurements by 206 patients. 196 patients used the remote monitoring system for at least three days (Table 1 ), with SpO₂ and HR data for these patients used to develop a model to predict SpO₂ alerts.

Fig. 2.

The number of patients, N, who entered data for SpO₂, heart rate, and breathlessness, and who experienced an SpO₂ alert (SpO₂ < 94 %), in the 10 days following hospital discharge.

Table 1.

Summary of recorded SpO₂ alerts in the full cohort, and in the subset of the cohort where age and sex data are available. * This group consists of all the patients who experienced only symptomatic alerts, together with a subset of the patients who experienced asymptomatic alerts. ** This group consists of a subset of the patients who experienced asymptomatic alerts.

	Full cohort (N = 209)	Cohort with 3 + days (N = 196)	Cohort with age and sex (N = 114)
Days included	Discharge-Day 10	Day 3–10	Discharge—Day 10
Patients with no alerts	116	126	60 (NoAlert)
Patients with asymptomatic alerts	39	26	21 (Asym)
Patients with only symptomatic alerts	54	44	33 (Sym)
Patients with symptomatic alerts*	68	54	41
Symptomatic due to breathlessness	55	45	34
Symptomatic due to elevated HR	44	31	27
Symptomatic due to concurrent elevated HR and breathlessness	28	22	17
Patients with both symptomatic and asymptomatic alerts**	14	10	8
Symptomatic due to breathlessness	10	5	7
Symptomatic due to elevated HR	9	7	5
Symptomatic due to concurrent elevated HR and breathlessness	4	0	3

Age and sex were optional entries in the monitoring system, 114 patients reported both their age and sex (39.5 ± 13.9 years; 74 female), Table 1. The effects of age and sex on SpO₂ alerts were examined using data for these 114 patients.

3.2. Data analysis

Descriptive statistics of the daily mean and minimum SpO₂, and mean and maximum heart rate and breathlessness, for the ten days after hospital discharge are presented in Table 2 .

Table 2.

Descriptive statistics calculated using all monitoring data collected during the first ten days after hospital discharge: 1378 days with breathlessness scores entered by 190 patients, 1775 days with SpO₂ measurements by 209 patients, and 1763 days with HR measurements by 206 patients. Note that the percentage of days when SpO₂ < 94 % includes day 1 and day 2, which are not included in the predictive model development. SD = standard deviation of all calculated data points. Min = minimum value calculated. Max = maximum value calculated.

	Mean	Median	SD	Min	Max	Alert and symptomatic days (%)
Mean SpO2 (%)	96.99	97.33	1.56	86.25	100
Minimum SpO2 (%)	95.94	96.00	2.51	80.00	100.00	20.79 % (SpO2 < 94 %)
Mean HR (bpm)	78.15	77.25	12.21	51.00	121.00
Maximum HR (bpm)	85.47	84.00	14.99	52.00	130.00	16.28 % (HR > 100 bpm)
Mean breathlessness (/10)	0.50	0.00	1.20	0.00	8.33
Maximum breathlessness (/10)	0.71	0.00	1.59	0.00	9.00	19.30 % (Breathlessness > 1)

Mean and minimum SpO₂, and mean and maximum heart rate, increased significantly over the ten days following hospital discharge (p < 0.005 for all), while the mean and maximum breathlessness decreased over the ten days (p < 0.005 for both), Fig. 3 . The CV of SpO₂, HR and breathlessness decreased significantly over the ten days, Fig. 3.

Fig. 3.

Data for all patients for the first ten days after discharge from hospital. Mean, coefficient of variation (CV) of all SpO₂, heart rate and breathlessness data, and minimum SpO₂, maximum heart rate and maximum breathlessness, for each day across all patients are presented. The mean and standard error of each feature across all patients are presented.

The effect of patient group (Sym, Asym, NoAlert), age and sex on overall mean SpO₂ and overall mean heart rate are presented in Table 3 , Fig. 4 a and 4b. The effect of patient group on age and sex are also presented in Table 3, Fig. 4c and 4d.

Table 3.

The effect of patient group (Sym/Asym/NoAlert) on mean SpO₂, mean HR, age and sex. Mean SpO₂ and mean heart rate are calculated for each patient using all data for that patient during the monitoring period. Significant differences (p < 0.015) are indicated using bold font.

Response variable	Overall effect of group	NoAlerts vs Sym	NoAlerts vs Asym	Sym vs Asym	Effect of age	Effect of sex
Mean SpO2 (%)	< 0.001	< 0.001	< 0.001	0.03	< 0.001	0.51
Mean HR (bpm)	0.02	0.005	0.37	0.15	0.82	0.50
Age (years)	0.07	0.06	0.06	0.78	–	–
Sex (M/F)	0.72	1	0.44	0.49	–	–

Fig. 4.

(a) Mean SpO₂, (b) mean heart rate and (c) age data for each patient group. (d) Percentages of male and female patients in each group.

3.3. Predictive model

Sequential forward feature selection selected five features for inclusion in the final model (from highest to lowest feature importance): SpO₂ min Day_A-2, SpO₂ min Day_A-1, SpO₂ CV diff, HR CV Day_A-2, HR CV Day_A-1.

The random forest classifier predicted an SpO₂ alert on a given day with a mean sensitivity of 66.00 ± 18.57 % and mean specificity of 88.31 ± 10.97 %, and mean AUC ROC of 0.80 ± 0.11 across tenfold cross-validation, Fig. 5 . The mean specificity of the predictive model on all unseen non-alert days was 88.72 ± 1.99 %.

Fig. 5.

(a) Receiver operating characteristic (ROC) for the predictive model. The mean ROC across all cross-validation folds is illustrated using a solid line, and the standard deviation is illustrated as a shaded area. (b)-(f) The five features included in the final predictive model, for patient days when SpO₂ was >= 94 % (non-alert days) and days when SpO₂ < 94 % (alert days).

Seventy-two out of 101 symptomatic alert days were correctly predicted as alert days, 32 out of 48 asymptomatic alert days were correctly predicted as alert days, and 147 out of 157 non-alert days were classified correctly.

Correlation analysis revealed that several SpO₂ and HR features were strongly correlated with each other, with Rho > 0.8 or Rho < -0.8, Table 4 .

Table 4.

SpO₂ and heart rate features with strong positive (Rho > 0.8) or negative (Rho < -0.8) correlations. Features included in the final random forest model are in bold font.

Feature	Rho > 0.8	Rho < -0.8
SpO2 mean Discharge	SpO2 min Discharge
SpO2 mean DayA-2	SpO2 min DayA-2
SpO2 mean DayA-1	SpO2 min DayA-1
SpO2 CV Discharge		SpO2 min Discharge
SpO2 CV DayA-2,		SpO2 min DayA-2
SpO2 CV DayA-1		SpO2 min DayA-1
SpO2 min Discharge	SpO2 mean Discharge	SpO2 CV Discharge
SpO2 min DayA-2	SpO2 mean DayA-2.	SpO2 CV DayA-2
SpO2 min DayA-1	SpO2 mean DayA-1	SpO2 CV DayA-1
HR mean Discharge	HR max Discharge
HR mean DayA-2	HR mean DayA-1, HR max DayA-2
HR mean DayA-1	HR mean DayA-2, HR max DayA-1
HR max Discharge	HR mean Discharge
HR max DayA-2	HR mean DayA-2.
HR max DayA-1	HR mean DayA-1.

4. Discussion

A model to predict SpO₂ alerts (<94 %) in patients with COVID-19 in the ten days following hospital discharge was developed using random forest classification. The model utilised patient-recorded SpO₂ and HR features measured on the two preceding days, and the day of discharge. The random forest model provides a method to predict low pulse oxygen saturation a day in advance of the SpO₂ event occurring. Model performance was assessed using cross validation, with a mean ROC AUC of 0.80, true positive rate of 66 % and a low false positive rate of 11.69 %. The presented method could allow early intervention to prevent further clinical deterioration and assist clinical teams to remotely triage patients following hospital discharge.

Prediction of low SpO₂ in patients with COVID-19 using remote monitoring data after hospital discharge has not previously been reported. However, several studies have presented models to predict clinical deterioration in hospitalised patients with COVID-19 [13], [14], [15], [16], [17], [18], [19], [20], or in non-hospitalised individuals with COVID-19 [21], [22]. A range of machine learning techniques models using clinical and laboratory data (including SpO₂) obtained on hospital admission have been reported, including logistic regression [14], [16], [17], deep learning [13], random forests [16], [20] or gradient boosting models such as XGBoost [16], [18], [20]or CatBoost [19], [20]. The random forest algorithm was used in this study due to its interpretability, and the performance of previously reported models based on ensembles of decision trees to predict clinical decline in hospitalised patients with COVID-19 [18], [19], [20] and to detect COVID-19 infection in non-hospitalised individuals with COVID-19 [21]. The random forest model presented here, with a ROC AUC of 0.80, performed comparably to previous studies predicting clinical deterioration in patients or non-hospitalised individuals with COVID-19 [13], [14], [15], [16], [18], [19], [20], [21], [22], [23]. To demonstrate the generalisability of predicting low SpO₂ in COVID-19 patients following hospital discharge using the methods presented, three alternative machine learning algorithms were also implemented (Supplementary Material): logistic regression with LASSO regularisation, support vector machine, and adaptive boosting. These alternative models achieved cross-validated ROC AUCs of 0.75 ± 0.11 for logistic regression, 0.79 ± 0.06 for adaptive boosting and 0.83 ± 0.07 using support vector machines (Supplementary Material), indicating that the performance of the presented method would generalise to other classification algorithms.

Remote monitoring also provides an opportunity to examine how HR, SpO₂ and breathlessness vary in the period following hospital discharge. Mean HR significantly increased over the ten day period examined. Consistent with this, a previous study reported that heart rate in individuals with COVID-19 decreased until approximately 13 days after symptom onset, and increased after this point [22]. Mean SpO₂ also significantly increased, while breathlessness significantly decreased, consistent with symptom improvement following discharge for the cohort in general. The CV of daily SpO₂, HR and breathlessness measures decreased over the ten days, indicating increased stability in these measures over time.

No significant effects of age or sex on the occurrence of asymptomatic hypoxia in COVID-19 were observed. A previous study of 195 patients with COVID-19 at a hospital emergency department reported silent hypoxia in 25 (13 %) of patients [25]. In the current study, 39 patients (19 %) experienced asymptomatic SpO₂ alerts at least once during the ten days after discharge. The slightly higher preponderance of asymptomatic alerts is potentially due to differences in clinical setting, inpatient vs post discharge, and resulting differences in SpO₂ thresholds, 94 % for early identification of clinical decline after hospital discharge in this study, consistent with previous studies monitoring patients after hospital discharge [5], [7], while a lower threshold of 90 % was used in the previous hospital-based study [25]. The inclusion of elevated HR (>100 bpm) as a symptom of respiratory distress in this study, consistent with previous studies on post-discharge monitoring in COVID-19 [5], may also have contributed to this difference.

The generalisability of the presented predictive model is limited by the cohort size, particularly in the subset of the cohort who provided demographic information. However, the cross-validated classification results indicate potential as a clinical tool which warrants trialling in a larger cohort. It should be noted that several features were highly correlated, Table 4, indicating that alternative feature combinations may produce similar performance to those reported. There was a possibility for data entry errors for breathlessness scores, however, the Bluetooth connection of the pulse oximeter reduced the input errors in SpO₂ and HR measures. While the accuracy of SpO₂ compared to blood oxygen saturation in COVID-19 has been reported to be adequate, with a reported bias of 0.4 % [32], physiologic or mechanical artifacts with vital sign alerts have also been reported, together with methods to identify erroneous alerts [33].

This study demonstrates the feasibility and effectiveness of predicting SpO₂ < 94 % in recently discharged COVID-19 patients on a given day, based on data for the two previous days and the day of hospital discharge. The presented model may provide a useful tool to enhance remote monitoring systems, providing early identification of patients at risk of low pulse oxygen saturation.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary material

The following are the Supplementary data to this article: