Prediction of viral symptoms using wearable technology and artificial intelligence: A pilot study in healthcare workers

Pierre-François D’Haese; Victor Finomore; Dmitry Lesnik; Laura Kornhauser; Tobias Schaefer; Peter E Konrad; Sally Hodder; Clay Marsh; Ali R Rezai

doi:10.1371/journal.pone.0257997

. 2021 Oct 14;16(10):e0257997. doi: 10.1371/journal.pone.0257997

Prediction of viral symptoms using wearable technology and artificial intelligence: A pilot study in healthcare workers

Pierre-François D’Haese ^1,^2,^3,^‡,^*, Victor Finomore ^1,^2,^3,^‡, Dmitry Lesnik ⁴, Laura Kornhauser ⁴, Tobias Schaefer ⁴, Peter E Konrad ^1,^2,³, Sally Hodder ^1,^2,³, Clay Marsh ^1,^2,³, Ali R Rezai ^1,^2,³

Editor: Nizam Uddin Ahamed⁵

¹Rockefeller Neuroscience Institute, West Virginia University, Morgantown, West Virginia, United States of America

²West Virginia Clinical and Translational Science Institute, West Virginia University, Morgantown, West Virginia, United States of America

³Health Sciences Center, West Virginia University, Morgantown, West Virginia, United States of America

⁴Stratyfy, Inc, New York, New York, United States of America

⁵University of Pittsburgh, UNITED STATES

Competing Interests: The authors have read the journal’s policy, and the authors of the study have the following competing interests to declare: DL, LK, and TS are paid employees of Stratyfy. This does not alter our adherence to PLOS ONE policies on sharing data and materials. There are no patents, products in development or marketed products associated with this research to declare.

‡ PFD and VF are Co-first authors.

^✉

* E-mail: pd0033@hsc.wvu.edu

Roles

Pierre-François D’Haese: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

Victor Finomore: Conceptualization, Formal analysis, Funding acquisition, Methodology, Resources, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

Dmitry Lesnik: Data curation, Formal analysis, Software

Laura Kornhauser: Supervision

Tobias Schaefer: Data curation, Formal analysis, Methodology

Peter E Konrad: Project administration, Resources, Supervision, Validation, Writing – review & editing

Sally Hodder: Conceptualization, Formal analysis, Methodology, Project administration, Supervision, Writing – original draft, Writing – review & editing

Clay Marsh: Investigation, Methodology, Resources, Supervision, Writing – review & editing

Ali R Rezai: Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – review & editing

Nizam Uddin Ahamed: Editor

PMCID: PMC8516235 PMID: 34648513

Abstract

Conventional testing and diagnostic methods for infections like SARS-CoV-2 have limitations for population health management and public policy. We hypothesize that daily changes in autonomic activity, measured through off-the-shelf technologies together with app-based cognitive assessments, may be used to forecast the onset of symptoms consistent with a viral illness. We describe our strategy using an AI model that can predict, with 82% accuracy (negative predictive value 97%, specificity 83%, sensitivity 79%, precision 34%), the likelihood of developing symptoms consistent with a viral infection three days before symptom onset. The model correctly predicts, almost all of the time (97%), individuals who will not develop viral-like illness symptoms in the next three days. Conversely, the model correctly predicts as positive 34% of the time, individuals who will develop viral-like illness symptoms in the next three days. This model uses a conservative framework, warning potentially pre-symptomatic individuals to socially isolate while minimizing warnings to individuals with a low likelihood of developing viral-like symptoms in the next three days. To our knowledge, this is the first study using wearables and apps with machine learning to predict the occurrence of viral illness-like symptoms. The demonstrated approach to forecasting the onset of viral illness-like symptoms offers a novel, digital decision-making tool for public health safety by potentially limiting viral transmission.

Introduction

Virus transmission from asymptomatic or pre-symptomatic individuals is a key factor contributing to the SARS-CoV-2 pandemic spread. High levels of SARS-CoV-2 virus have been observed 48–72 hours before symptom onset. As high viral loads of SARS-CoV-2 may occur before the onset of symptoms, strategies to control community COVID-19 spread that rely only on symptom-based detection are often unsuccessful. The development of novel approaches to detect viral infection symptoms during this pre-symptomatic phase are critical to reducing viral transmission and spread by facilitating appropriate early quarantine before symptoms occur.

Once infected, the incubation period commonly ranges from 2–14 days (mean of 5.2 days), and infectious transmission starts around 2.5 days and peaks at 0.7 days before the onset of symptoms [1–4]. Of note, the loss of sense of smell and taste are more specific symptoms for COVID-19 [3]. Even when symptomatic COVID-19 occurs, the symptoms and signs of COVID-19 overlap with other viral illnesses such as influenza.

Today, 1 in 5 Americans use fitness tracking devices [5]. While these technologies can inform population-level data sharing to detect disease state [6–9], to our knowledge, they have not been used to forecast communicable infectious disease at the individual level. Outputs from wearable technology including heart rate (HR), heart rate variability (HRV), respiration rate (RR), temperature, blood oxygenation, sleep, and other physiological assessments are increasingly being explored in studies of health and disease [10–12]. Moreover, a variety of subject-reported symptoms captured on mobile apps transforms both surveillance and contact tracing management strategies for COVID-19 [13–15].

Machine-learning algorithms are becoming more popular and useful when collecting large amounts of disparate data to provide insight into otherwise complex relationships not easily determined with routine statistical methods. Using a machine learning model informed by self-reported symptoms, we demonstrate that the combination of physiological outputs from wearable technology and brief cognitive assessments can predict symptoms and signs of a viral infection three days before the onset of those symptoms. This forecasting model could be used to enhance conventional infection-control strategies for COVID-19 and other viral infections.

Methods

Study design

The Rockefeller Neuroscience Institute (RNI) team initiated a study approved by the institutional review board (IRB) at the West Virginia University Medical Center (#2003937069), Vanderbilt University Medical Center (#200685), and Thomas Jefferson University (#2004957109A001) to combine physiological and cognitive biometrics and self-reported symptoms information from individuals at risk for exposure to COVID-19 and potential contracture of a viral illness. We recruited study participants from each tertiary medical center by approaching front-line health care workers receiving regional referrals for COVID-19 patients. We asked each participant to 1) wear a smart ring device [16] with sensors that collect physiological measures such as body temperature, sleep, activity, heart rate, respiratory rate, heart rate variability; 2) use a custom mobile health app [17] to complete a brief symptoms diary [3], social exposure to potentially infected contacts, and measures of physical, emotional, and cognitive workload; (see S1 Table and S1 File) as well as the psychomotor vigilance cognitive task (PVT) [18] to measure attention and fatigue twice a day. All data are collected, structured, and organized into the RNI Cloud data lake for analysis. The RNI Cloud is a HIPAA compliant data platform hosted in Amazon Web Services (AWS) that supports all the security and legal requirements to protect the data’s privacy and integrity from the participants in the context of multi-center clinical studies [19].

We utilized a machine learning approach that combines features through probabilistic rules and provides a prediction. The training process consists of two steps. It combines subject reported symptoms (labeling model) to inform a predictive framework (forecast model) that uses physiological and cognitive signals to forecast suspicion of a viral illness (Fig 1). The dataset consisting of PVT and wearable data is split, 75% for training, and 25% reserved for testing the model [20] (see S4 File). The labeling and forecasting models are created from a set of rules combining one or more features. All rules are given a weight and combined to provide a final decision [21–23] (see S2 Table).

Fig 1 — Data Flow a) the labeling model, b) the forecast model. Each model takes as input three days of data (d, d-1, d-2).

Labeling model

We use a rule-based approach to create an AI model that labels an individual’s self-reported symptoms as suspicious (or not) for presenting symptoms consistent with a viral illness. The labeling model is created based on the expert knowledge manually translated into decision rules (see below). The purpose of this model is to define if a person is being suspicious of an infectious disease (below we just say suspicious) based on its self-reported symptoms. Rules are based on those symptoms commonly present in a diagnosed viral-like condition and those more specific for SARS-Cov-2 (e.g., loss of taste and smell) [2, 24, 25]. Resulting rules (Table 1) assign, for instance, higher confidence on suspicion of a viral-illness for self-reported fever with the persistence of symptoms for more than two consecutive days. In comparison, lower confidence is assigned for stuffy nose and swollen eyes without fever (see S5 File). The rules and weights in this model were establish from clinical subject matter experts. In particular, the weights associated with the rules were chosen to minimize the labeling error assessed by medical experts. We also fine-tuned some rule weights by fitting the model to a small synthetic data set, which contained typical symptom combinations. The actual calculations of the labeling model’s output score is based on the machinery of the probabilistic logic, described in more detail in the next paragraph.

Table 1. Labeling model rules.

Conditions that INCREASE suspicion of viral-like symptoms	THEN	Confidence	Comments
Are_you_Positive_for_COVID_19	suspicious of viral-like symptoms	high
Sense_of_Smell_Change	suspicious of viral-like symptoms	high
Fever	suspicious of viral-like symptoms	high
Cough	suspicious of viral-like symptoms	high
Shortness_of_Breath	suspicious of viral-like symptoms	high
Coughing_up_blood	suspicious of viral-like symptoms	medium
Nausea_or_vomiting	suspicious of viral-like symptoms	medium
Fatigue	suspicious of viral-like symptoms	medium	Fatigue combined with respiratory symptoms make high
Sinus_Pain	suspicious of viral-like symptoms	medium
Sore_throat	suspicious of viral-like symptoms	medium
Chills	suspicious of viral-like symptoms	medium
Phlegm	suspicious of viral-like symptoms	low	Make Low
Bone_or_joint_pain	suspicious of viral-like symptoms	low	Make low
Diarrhea	suspicious of viral-like symptoms	low	Make low if only one day
Stuffy_nose	suspicious of viral-like symptoms	low
Loss_of_appetite	suspicious of viral-like symptoms	low	Loss of appetite with other symptoms increase to medium (except fever, which is high)
Any persistent symptom for 2 or 3 days	suspicious of viral-like symptoms	low
Stuffy nose AND Swollen eyes AND Fever	suspicious of viral-like symptoms	high
Feel_Sick	suspicious of viral-like symptoms	low
Headache	suspicious of viral-like symptoms	low	Headache with other symptoms increase to medium (except fever make high)
Swollen_eyes	suspicious of viral-like symptoms	low
Any medium or low item with fever		high
Conditions that DECREASE suspicion of viral-like symptoms	THEN	Confidence
Stuffy_nose AND Swollen_eyes AND NOT Fever	NOT suspicious of viral-like symptoms	low

Open in a new tab

Confidence levels

• High = Condition is strongly indicative of risk

• Medium = Condition is somewhat indicative of risk

• Low = Condition is slightly indicative of increase/decrease of risk

Forecasting model

The forecasting model was used to associate a label of suspicion for viral illness from the Labeling model to the features extracted from the user’s cognitive function assessment and physiological signals. The physiological features include (1) single day and (2) rolling averages over 28 days of the heart rate, heart rate variability, respiration rate, activity, sleep latency, sleep duration, composition (light, REM, deep), skin temperature, and sleep efficiency. Physiological features to the exclusion of skin temperature are measured during the night to remove noise due to varying daily activities. The daily cognitive task (PVT) is a sustained-attention, a reaction-timed task that measures the speed with which subjects respond to a visual stimulus [18]. From this data set, the algorithm extracts rules using an information gain-based approach and combines them in a predictive model using a probabilistic graphical network as follows.

The set of probabilistic rules comprises a Markov network. The joint distribution defined by the Markov network can be written as $P (x) = \frac{1}{Z} \exp (\sum_{j} ω_{j} f_{j} (x))$ where x = (x₁,x₂,…,x_n,y) denotes a set of n+1 binary variables, out of which the first n are input variables, and y is the output variable. Here, f_j(x)∈{1,0} is a Boolean function corresponding to the rule, ω is a factor associated with the corresponding rule, Z is the normalization constant. In the current implementation, the relation between the rule’s factor ω and the weight ψ used in the supplementary materials is given by $ψ = \frac{\exp (ω)}{1 + \exp (ω)}$ . More details on the fundamentals of the probabilistic logic can be found in [21–23]. With the joined distribution defined above, the model prediction s for every observation vector r = (r₁,r₂,…,r_n) is computed as the conditional probability of the output variable y as s = P(y = 1|r).

If a training set is available, the model’s parameters can be determined by the calibration process, which minimizes the prediction error. Suppose, for the i-th training example the model’s prediction is s_i, and the observed (ground truth) output is y_i. We define the cross-entropy loss function as

L = - \sum_{i} y_{i} \log (s_{i}) + (1 - y_{i}) \log (1 - s_{i})

The calibration process uses the steepest gradient descent to find a combination of rules weights which minimizes the loss function. In our particular implementation we used Limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm (L-BFGS).

Our model was developed on Stratyfy’s Probabilistic Rule Engine, a commercial machine learning platform [26] (see S3 File). In the application to our study, this general framework for creating a rule-based predictive model was applied as follows: In a preprocessing step, the data from the wearable device (e.g., heart rate, temperature, etc.) and the information for the mobile app (e.g., symptoms, results of the PVT, etc.) were collected, checked for completeness, and engineered variables were extracted. We found that, for our study, large gaps in the data had a significant negative impact on the predictive power of the model and, therefore, our efforts were concentrated on cases where most of the required information was actually available. We identified a number of engineered variables (for instance, a ratio of heart rate to heart rate variability) which helped significantly improve the model’s predictive power. In order to be used with probabilistic rules, continuous variables are discretized, and then discretized and categorical variables are converted into binary variables by one-hot encoding. The labeling model described above was used to construct the binary output variable, marking for each case days of potential onset of a viral infection. At this point, the setup fit into the context of a standard supervised learning problem: We needed to train a classifier to predict the onset of a disease based on the information available before the actual onset. We opted for the rule-based system described above for several reasons. A main reason was the transparency and interpretability of our model. In this case, our rule-based system produced models that were fairly small in size (20–50 rules) and still highly accurate. We compared our approach to standard approaches, for example gradient boosting, and found the rule-based approach most promising. Note that, in this study, the rule-based models were used in two ways. In the labelling model, the rules, together with the confidences, were developed and specified by clinical experts. To create the forecasting model, the rules were extracted from the available data via rule mining. For this purpose, we used the Association Rule Mining algorithm [27], which is based on the co-occurrences frequency analysis. After extracting the rules, the weights of the rules were determined by the calibration process outlined above.

Validation of the model

Model performance was tested with K-fold cross-validation with in our case we perform four rounds of validation (K = 4). One round of cross-validation involves portioning the dataset into complementary subsets, performing the training on one subset and the validation on the other. To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are combined (averaged) over the rounds to give an estimate of the model’s predictive performance. The entire dataset is divided 4 times as 75% for training and 25% for validating the model. The results are then average across the 4 runs of training-validation. The model weights in the final model are obtained by using training dataset of the model. We measure the model’s performances at various threshold settings. We also used the area under the curve (AUC) of the receiver operating characteristic (ROC) curve as a threshold-invariant performance measure. Additionally, we report the model’s learning performances, i.e., how much data is required to reach the stability of the model. Learning is achieved when adding more data does not significantly impact the performance of the model.

Results

We enrolled 867 subjects in the study between Apr 7th, 2020, and Aug 1st, 2020 (age ranged from 20 to 76 years old) (Table 2). The data set includes 75,292 unique data points (median number of days of data per participant is 90 days) (see S2 File). 33% (289) unique participants were labeled (via the labeling model) as having symptoms consistent with a viral illness. The forecasting model’s inclusion criteria require at least three days of continuous data with no more than one feature missing due to compliance (Fig 1). Of the 767 participants that met the criteria, 276 had missing data for the wearable and 376 for the cognitive assessment. The remaining 115 participants were used to label the wearable and cognitive data as input for the three-day forecasting model. Each day of data was adjudicated by the labeling model, which predicted a 10% occurrence of symptoms consistent with a viral-like illness. The remaining days were labeled as negative or non-suspicious of viral-like illness. From the training dataset, the algorithm identified 45 probabilistic rules. These are combined to form the forecasting model (Table 3). The rules contributing to the high probability of developing symptoms within three days are related to low HRV, slower response time to cognitive testing, longer latency to get asleep combined with an increased REM sleep time, and an increased HR. The rules that contribute to a lower probability of developing symptoms are related to lower HR, increased HRV, increased sleep quality, and faster response rate to cognitive testing. Fig 2 provides the model performance as a function of the threshold. Fig 3 illustrates that the model reaches a plateau after about 1500 samples, and that much accuracy cannot be gained by adding more samples. Table 4 reports the precision, recall, and accuracy metrics obtained with a threshold = 0.1 for models with and without cognitive assessment data. The threshold was selected to maximize the balance between precision and recall. The overall accuracy of the model is 82%. The recall positive defined as the true positives (TP) over the total number of positive values (TP/(TP+FN)) is 79% (no PVT:67%). The accuracy of calling negatives (recall negative) defined as the true negatives (TN) over the total amount of negative values (TN/(TN+FP)) is 83% (no PVT:84%). AUC is 89% (no PVT: 83%).

Table 2. Study populations demographics.

Group	WVU (N = 698)	Vanderbilt (N = 97)	TJU (N = 69)	All (N = 867)
Sex–no (%)
Male	212 (30.3)	14 (14.4)	10 (14.5)	236 (27.2)
Female	252 (36.1)	40 (41.2)	21 (30.4)	313 (36.1)
Did Not Respond	234 (33.5)	43 (44.3)	38 (55.1)	318 (36.7)
Age (mean ± SD)	37.6 ± 11.6	37.8 ± 9.7	32.8 ± 10.0	37.6 ± 11.3
Diabetes Yes-No (%)	11 (1.6)	1 (1)	1 (1.4)	13 (1.5)
Hypertension Yes-No (%)	31 (4.4)	7 (7.2)	2 (2.9)	40 (4.6)

Open in a new tab

Table 3. Algorithm-derived rules list of 45 rules extracted by the algorithm and used in the model with their relative weights.

IF	lbl_score > = 0.5	THEN	suspicious	0.97
IF	lbl_score in [0.2.. 0.5)	THEN	suspicious	0.95
IF	(HRV in (30.. 43] AND lbl_score < 0.2)	THEN	suspicious	0.91
IF	(Breath_Average < = 14.5 AND MedianResponseTime_AM > 365)	THEN	suspicious	0.90
IF	(Age in (27.. 33] AND lbl_score < 0.2)	THEN	suspicious	0.87
IF	(Sex = Female AND lbl_score > = 0.5)	THEN	suspicious	0.84
IF	(Onset_Latency > 0.0417 AND REM > 1.62)	THEN	suspicious	0.83
IF	(Age in (27.. 33] AND lbl_score_t1 < 0.2)	THEN	suspicious	0.78
IF	(Breath_Average < = 14.5 AND HRV > 43)	THEN	suspicious	0.77
IF	(HR_delta < = -1.47 AND Light < = 3.42)	THEN	suspicious	0.75
IF	(Age > 46 AND Sex = Female)	THEN	suspicious	0.75
IF	(MedianResponseTime_AM in (322.. 365] AND lbl_score < 0.2)	THEN	suspicious	0.73
IF	(Onset_Latency in (0.00333.. 0.0417] AND Sleep_Score < = 72)	THEN	suspicious	0.73
IF	(Onset_Latency in (0.00333.. 0.0417] AND HRV_delta_t1 < = -4.11)	THEN	suspicious	0.73
IF	(AM_Readiness < = 5.28 AND Sex = Female)	THEN	suspicious	0.72
IF	(HR_delta in (-1.47.. 1.38] AND Sex = Male)	THEN	suspicious	0.70
IF	(E1 < = 0.274 AND HRV_base in (30.1.. 43.5])	THEN	suspicious	0.69
IF	(MedianResponseTime_PM in (326.. 375] AND Sex = Male)	THEN	suspicious	0.64
IF	(Onset_Latency in (0.00333.. 0.0417] AND lbl_score in [0.2.. 0.5))	THEN	suspicious	0.58
IF	True	THEN	suspicious	0.11
IF	(HR_Lowest in (55.. 61] AND TLX_Stress_Score in (88.. 163])	THEN	not suspicious	0.87
IF	(Age in (27.. 33] AND Sex = Female)	THEN	not suspicious	0.86
IF	(Light > 4.31 AND MedianResponseTime_PM > 375)	THEN	not suspicious	0.86
IF	(HRV in (30.. 43] AND lbl_score_t1 < 0.2)	THEN	not suspicious	0.84
IF	(HR_delta_t1 in (-1.45.. 1.35] AND REM > 1.62)	THEN	not suspicious	0.83
IF	(Sleep_Score > 82 AND HRV_delta > 2)	THEN	not suspicious	0.83
IF	(MedianResponseTime_PM in (326.. 375] AND Sleep_Score < = 72)	THEN	not suspicious	0.81
IF	(MedianResponseTime_AM in (322.. 365] AND lbl_score_t1 < 0.2)	THEN	not suspicious	0.80
IF	(E4_t2 in (1.44.. 2.32] AND Score_Efficiency in (83.. 96])	THEN	not suspicious	0.79
IF	(E1_t2 < = 0.27 AND Light < = 3.42)	THEN	not suspicious	0.78
IF	(HR_Lowest > 61 AND HRV in (30.. 43])	THEN	not suspicious	0.77
IF	(E5_t1 in (-0.753.. 0.724] AND HR < = 62)	THEN	not suspicious	0.77
IF	(E1 < = 0.274 AND HR_delta in (-1.47.. 1.38])	THEN	not suspicious	0.77
IF	(Onset_Latency > 0.0417 AND HRV_delta_t1 > 1.99)	THEN	not suspicious	0.76
IF	(E5 > 0.758 AND HRV_delta_t1 < = -4.11)	THEN	not suspicious	0.75
IF	(Breath_Average < = 14.5 AND Sex = Female)	THEN	not suspicious	0.74
IF	(E5_t2 in (-0.74.. 0.779] AND Temperature_Delta in (-0.1.. 0.08])	THEN	not suspicious	0.74
IF	(HR_delta_t2 in (-1.45.. 1.43] AND Temperature < = 97.6)	THEN	not suspicious	0.73
IF	(Duration_Integer_hr < = 5.8 AND E1 < = 0.274)	THEN	not suspicious	0.73
IF	(E4_t1 in (1.45.. 2.3] AND E5_t1 in (-0.753.. 0.724])	THEN	not suspicious	0.71
IF	(TLX_Stress_Score > 163 AND HRV in (30.. 43])	THEN	not suspicious	0.68
IF	(Light in (3.42.. 4.31] AND Sex = Male)	THEN	not suspicious	0.67
IF	(Age in (33.. 37.5] AND TLX_Stress_Score < = 88)	THEN	not suspicious	0.66
IF	(Sex = Male AND TLX_Stress_Score > 163)	THEN	not suspicious	0.63
IF	Age in (27.. 33]	THEN	not suspicious	0.61

Open in a new tab

Rules are aggregated to forecast suspicion of a viral disease in a participant.

Fig 2 — ROC (False Positive Rate vs True Positive Rate) and Precision/recall curves for the forecasting model with (A, C) and without cognitive assessment (B, D).

Fig 3 — The shade colors represent the variance of results created by the K-Fold(4) validation.

Table 4. Model performance with and without PVT.

A. With PVT. B. Without PVT.

A
Threshold	0.05	0.1	0.2	0.3
Recall Pos	0.89	0.79	0.61	0.46
Recall Neg	0.70	0.83	0.91	0.94
Precision Pos	0.24	0.34	0.43	0.47
Precision Neg	0.98	0.97	0.95	0.94
Accuracy	0.72	0.82	0.88	0.90
AUC	0.88
B
Threshold	0.05	0.1	0.2	0.3
Recall Pos	0.8	0.66	0.53	0.47
Recall Neg	0.62	0.83	0.92	0.95
Precision Pos	0.20	0.31	0.45	0.54
Precision Neg	0.97	0.95	0.94	0.94
Accuracy	0.64	0.82	0.88	0.91
AUC	0.83

Open in a new tab

Discussion

In this study, we measure daily changes in autonomic activity using a wearable device and cognitive assessments via a mobile app. Using machine-learning analytics, we then forecast the onset of symptoms consistent with a viral illness. Specifically, we describe our strategy of using an AI model in conjunction with a non-invasive and readily available technology, which predicts the likelihood of developing symptoms consistent with a viral infection three days before symptom onset with an accuracy of 82%. The model has a false positive rate of 21% (meaning the system would label a non-infected participant as suspicious) and a false-negative of 17% (meaning the system would not detect a suspicious participant). Due to the occurrence of disease in the population, our dataset is unbalanced with more negatives than positives to a ratio of about 4 to 1. The model would detect 79% of individuals who will develop symptoms (i.e., sensitivity) and correctly predicts, almost all of the time (97%, negative predictive value), individuals who will not develop viral-like illness symptoms in the next three days. Conversely, the model precision is 34%. That precision is defined as the ratio of true positives (TP) over positives (P). In other words, if the model flags someone to develop viral-like symptoms in the next three days, the model is correct 34% of the time. Finally, the very little difference in AUCs between each fold suggest that the model is consistently generalizable.

The current model parameters were chosen to provide a conservative framework that warns potentially pre-symptomatic individuals to socially isolate while minimizing warnings to individuals with a low likelihood of developing viral-like symptoms in the next three days. The individuals predicted to be positive (true or false positives) would undergo additional screening and precautions. This framework can be applied as a digital decision-making management tool for public health safety in addition to conventional infection-control strategies.

Other investigators have confirmed the relationship between autonomic activity and the inflammatory response [28–30]. This study suggests a time-dependent relationship between autonomic and cognitive activity and the forecasting of symptoms consistent with a viral illness. We observed consistent changes in the autonomic nervous system function preceding the onset of symptoms. Specifically, differences were observed in HRV, HR, and sleep indices three days before symptom onset. Importantly, this period corresponds to the pre-symptomatic phase of some viral illness such as COVID-19 that is estimated to be 2.5 days [1–4]. In addition to the autonomic changes measured by the wearables, our analyses demonstrate the additional value of cognitive assessments (PVT) to predict symptoms consistent with a viral illness.

There are several limitations to this study. First, we did not diagnose infection nor measure infection markers in each individual. Instead, we relied on self-reported symptoms known to be associated with the occurrence of a viral infection. Without definitive diagnostics, we cannot confirm the presence of viral infection among persons who self-report symptoms. In the next phase of the study, we plan to test specific viruses (e.g., influenza and SARS CoV-2). The participants in this study are limited to front-line health care workers. Our model would benefit from being extended to other populations. Finally, participant compliance to consistently use their wearable and the app remains a challenge. Non-compliance among our participants reduced the usable data set. We plan on developing additional models to impute the data in an efficient way in order to extend the usability of the forecasting model. While we have demonstrated that the dataset is sufficient to reach this model’s predictive stability, additional data will provide further insights and reinforce the conclusions.

Viral infections have physical, cognitive, behavioral, and environmental influences and stressors that impact infection risk [31, 32]. To our knowledge, this is the first study using wearables and apps with machine learning to predict symptoms consistent with viral infection three days before their onset. The demonstrated approach to forecasting the onset of viral illness-like symptoms offers a novel digital decision-making tool for public health safety by potentially limiting viral transmission.

Supporting information

S1 Table. Data dictionary.

Data dictionary of each data element used in the model.

(PDF)

Click here for additional data file.^{(61.8KB, pdf)}

S2 Table. Disease onset model rules.

The following table reports the probabilistic weights for each rule of the symptom onset forecasting model.

(PDF)

Click here for additional data file.^{(102.9KB, pdf)}

S1 File. List of questions.

List of questions asked to the participants.

(PDF)

Click here for additional data file.^{(37.1KB, pdf)}

S2 File. Dataset and inclusion/exclusion criteria.

Inclusion/exclusion criteria and description of data set.

(PDF)

Click here for additional data file.^{(2.1MB, pdf)}

S3 File. Probabilistic rule engine.

Detailed description of the Probabilistic Rule Engine.

(PDF)

Click here for additional data file.^{(51.2KB, pdf)}

S4 File. Validation approach.

Detailed description of the validation approach.

(PDF)

Click here for additional data file.^{(57.3KB, pdf)}

S5 File. Labeling model.

Detailed description of the labeling model.

(PDF)

Click here for additional data file.^{(38KB, pdf)}

Acknowledgments

We would like to thank the teams and clinical coordinators from Vanderbilt University and Thomas Jefferson University, who have provided the support to recruit and support the participants. We would like to thank the OURARing team for their partnership and integration of their data system.

Data Availability

Data cannot be shared publicly because of parts are being owned by a third party. Data are available from the RNI Institutional Data Access / Ethics Committee (contact via Dr Padmashree Tirumalai, padma.tirumalai@hsc.wvu.edu) for researchers who meet the criteria for access to confidential data.

Funding Statement

Stratyfy provided financial support for this study in the form of salaries for DL, LK, and TS. The specific roles of these authors are articulated in the ‘author contributions’ section. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. No additional external funding was received for this study.

References

1.Li Q, Guan X, Wu P, et al. Early Transmission Dynamics in Wuhan, China, of Novel Coronavirus–Infected Pneumonia. New England Journal of Medicine. 2020;382(13):1199–1207. doi: 10.1056/NEJMoa2001316 [DOI] [PMC free article] [PubMed] [Google Scholar]
2.He X, Lau EHY, Wu P, et al. Temporal dynamics in viral shedding and transmissibility of COVID-19. Nat Med. 2020;26(5):672–675. doi: 10.1038/s41591-020-0869-5 [DOI] [PubMed] [Google Scholar]
3.Spinato G, Fabbris C, Polesel J, et al. Alterations in Smell or Taste in Mildly Symptomatic Outpatients With SARS-CoV-2 Infection. JAMA. Published online Apr 22nd, 2020. doi: 10.1001/jama.2020.6771 [DOI] [PMC free article] [PubMed] [Google Scholar]
4.Guan W-J, Ni Z-Y, Hu Y, et al. Clinical Characteristics of Coronavirus Disease 2019 in China. New England Journal of Medicine. 2020;382(18):1708–1720. doi: 10.1056/nejmoa2002032 [DOI] [PMC free article] [PubMed] [Google Scholar]
5.[No title]. Accessed Sept 22nd, 2020. https://news.gallup.com/file/poll/269141/191206HealthTrackers.pdf
6.Izmailova ES, McLean IL, Hather G, et al. Continuous Monitoring Using a Wearable Device Detects Activity‐Induced Heart Rate Changes After Administration of Amphetamine. Clinical and Translational Science. 2019;12(6):677–686. doi: 10.1111/cts.12673 [DOI] [PMC free article] [PubMed] [Google Scholar]
7.Erb MK, Karlin DR, Ho BK, et al. mHealth and wearable technology should replace motor diaries to track motor fluctuations in Parkinson’s disease. NPJ Digit Med. 2020;3:6. doi: 10.1038/s41746-019-0214-x [DOI] [PMC free article] [PubMed] [Google Scholar]
8.Griffin B, Saunders KEA. Smartphones and Wearables as a Method for Understanding Symptom Mechanisms. Front Psychiatry. 2019;10:949. doi: 10.3389/fpsyt.2019.00949 [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Radin JM, Wineinger NE, Topol EJ, Steinhubl SR. Harnessing wearable device data to improve state-level real-time surveillance of influenza-like illness in the USA: a population-based study. The Lancet Digital Health. 2020;2(2):e85–e93. doi: 10.1016/S2589-7500(19)30222-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
10.Jarczok MN, Kleber ME, Koenig J, et al. Investigating the Associations of Self-Rated Health: Heart Rate Variability Is More Strongly Associated than Inflammatory and Other Frequently Used Biomarkers in a Cross Sectional Occupational Sample. PLOS ONE. 2015;10(2):e0117196. doi: 10.1371/journal.pone.0117196 [DOI] [PMC free article] [PubMed] [Google Scholar]
11.Bakken AG, Axén I, Eklund A, O’Neill S. The effect of spinal manipulative therapy on heart rate variability and pain in patients with chronic neck pain: a randomized controlled trial. Trials. 2019;20(1). doi: 10.1186/s13063-019-3678-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
12.Johnston BW, Barrett-Jolley R, Krige A, Welters ID. Heart rate variability: Measurement and emerging use in critical care medicine. Pediatr Crit Care Med. 2020;21(2):148–157. doi: 10.1177/1751143719853744 [DOI] [PMC free article] [PubMed] [Google Scholar]
13.Abeler J, Bäcker M, Buermeyer U, Zillessen H. COVID-19 Contact Tracing and Data Protection Can Go Together. JMIR Mhealth Uhealth. 2020;8(4):e19359. doi: 10.2196/19359 [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Show evidence that apps for COVID-19 contact-tracing are secure and effective. Nature. 2020;580(7805):563. doi: 10.1038/d41586-020-01264-1 [DOI] [PubMed] [Google Scholar]
15.Wang S, Ding S, Xiong L. A New System for Surveillance and Digital Contact Tracing for COVID-19: Spatiotemporal Reporting Over Network and GPS. JMIR Mhealth Uhealth. 2020;8(6):e19457. doi: 10.2196/19457 [DOI] [PMC free article] [PubMed] [Google Scholar]
16.Oura Ring: the most accurate sleep and activity tracker. Oura Ring. Accessed Jul 7th, 2020. https://ouraring.com
17.‎RNI Health. Accessed Aug 14th, 2020. https://apps.apple.com/us/app/rni-health/id1515732074
18.Roach GD, Dawson D, Lamond N. Can a Shorter Psychomotor Vigilance Task Be Used as a Reasonable Substitute for the Ten‐Minute Psychomotor Vigilance Task? Chronobiology International. 2006;23(6):1379–1387. doi: 10.1080/07420520601067931 [DOI] [PubMed] [Google Scholar]
19.D’Haese P-F, Konrad PE, Pallavaram S, et al. CranialCloud: a cloud-based architecture to support trans-institutional collaborative efforts in neurodegenerative disorders. Int J Comput Assist Radiol Surg. 2015;10(6):815–823. doi: 10.1007/s11548-015-1189-y [DOI] [PMC free article] [PubMed] [Google Scholar]
20.Stone M. Cross-Validatory Choice and Assessment of Statistical Predictions. Journal of the Royal Statistical Society: Series B (Methodological). 1974;36(2):111–133. doi: 10.1111/j.2517-6161.1974.tb00994.x [DOI] [Google Scholar]
21.Richardson M, Domingos P. Markov logic networks. Machine Learning. 2006;62(1–2):107–136. doi: 10.1007/s10994-006-5833-1 [DOI] [Google Scholar]
22.Nilsson NJ. Probabilistic logic. Artificial Intelligence. 1986;28(1):71–87. doi: 10.1016/0004-3702(86)90031-7 [DOI] [Google Scholar]
23.Wang C, Komodakis N, Paragios N. Markov Random Field modeling, inference & learning in computer vision & image understanding: A survey. Computer Vision and Image Understanding. 2013;117(11):1610–1627. doi: 10.1016/j.cviu.2013.07.004 [DOI] [Google Scholar]
24.Flu Symptoms & Diagnosis | CDC. Published Dec 5th, 2019. Accessed Jul 25th, 2020. https://www.cdc.gov/flu/symptoms/index.html
25.Eccles R. Understanding the symptoms of the common cold and influenza. Lancet Infect Dis. 2005;5(11):718–725. doi: 10.1016/S1473-3099(05)70270-X [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Stratyfy, Inc.. https://www.stratyfy.com
27.Jochen Hipp, Ulrich Güntzer, and Gholamreza Nakhaeizadeh. 2000. Algorithms for association rule mining—a general survey and comparison. SIGKDD Explor. Newsl. 2, 1 (June, 2000), 58–64. 10.1145/360402.360421 [DOI]
28.Pavlov VA, Tracey KJ. The vagus nerve and the inflammatory reflex—linking immunity and metabolism. Nature Reviews Endocrinology. 2012;8(12):743–754. doi: 10.1038/nrendo.2012.189 [DOI] [PMC free article] [PubMed] [Google Scholar]
29.Pereira MR, Leite PEC. The Involvement of Parasympathetic and Sympathetic Nerve in the Inflammatory Reflex. Journal of Cellular Physiology. 2016;231(9):1862–1869. doi: 10.1002/jcp.25307 [DOI] [PubMed] [Google Scholar]
30.Pal GK, Nanda N. Vagus Nerve: The Key Integrator of Anti-inflammatory Reflex. International Journal of Clinical and Experimental Physiology. 2020;7(1):01–02. doi: 10.5530/ijcep.2020.7.1.1 [DOI] [Google Scholar]
31.Elkington LJ, Gleeson M, Pyne DB, Callister R, Wood LG. Inflammation and Immune Function. Antioxidants in Sport Nutrition. Published online 2014:171–181. doi: 10.1201/b17442-11 [DOI] [Google Scholar]
32.Gleeson M. Immune function in sport and exercise. J Appl Physiol. 2007;103(2):693–699 doi: 10.1152/japplphysiol.00008.2007 [DOI] [PubMed] [Google Scholar]

PLoS One. doi: 10.1371/journal.pone.0257997.r001

Decision Letter 0

Nizam Uddin Ahamed

20 Jan 2021

PONE-D-20-36216

Prediction of viral symptoms using wearable technology and artificial intelligence: A pilot study in healthcare workers

PLOS ONE

Dear Dr. DHAESE,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Mar 06 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Nizam Uddin Ahamed, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Competing Interests section:

"The authors have declared that no competing interests exist"

We note that one or more of the authors are employed by a commercial company: Stratyfy, Inc.

2.1. Please provide an amended Funding Statement declaring this commercial affiliation, as well as a statement regarding the Role of Funders in your study. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. You can update author roles in the Author Contributions section of the online submission form.

Please also include the following statement within your amended Funding Statement.

“The funder provided support in the form of salaries for authors [insert relevant initials], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.”

If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement.

2.2. Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc.

Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests) . If this adherence statement is not accurate and there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include both an updated Funding Statement and Competing Interests Statement in your cover letter. We will change the online submission form on your behalf.

3. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

4. Please include your tables as part of your main manuscript and remove the individual files. Please note that supplementary tables should be uploaded as separate "supporting information" files.

5. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is an incredibly timely study tackling the most urgent issue facing us all right now. It presents a wearables-based approach for capturing physiological and physical behaviours, and using these to predict whether someone may potentially have coronavirus. Importantly, the work aims to predict this even before major symptoms develop. Should it work, this would be a significant tool in helping us to control the spread of this virus.

The study is, for a wearables-based work, extensive - with over 800 frontline workers tracked for around 90 days. Unfortunately, due to data collection issues, only about 115 participants are used in the final results.

Nonetheless, this gives us a preliminary and encouraging result that can be acted upon.

Important though this work is, I have some concerns regarding both the methodology used and the presentation of the results. These should be addressed as speedily as is possible to ensure the work is fit for publication - and, most importantly, replication.

First off, the machine learning component is not clear. The model is described as a probabilistic graphical network acting on a combination of binary features. The exact operation or preprocessing applied to these features, or indeed what these features are, is not specified in the main paper. Further information is supplied in the supplementary material, however it would enhance reader understanding considerably to give some clearer example on, for example, how measured heart rate is incorporated into the model. (Even in the supplementary material, the model description could be improved.) This whole section should be written in a clearer, fuller, way using appropriate terminology to allow easier replication. It would also be helpful to know why this model was chosen over other potential machine learning approaches.

The training and testing procedure of the algorithm is also unclear. The supplementary material goes some way to clarifying this, but the main text could be tightened up somewhat. As K-fold cross-validation is used, it would help to directly specify the value of K (=4 in this case). Specifying a 25% test set split is not quite enough because this could imply a single leave-one test set out evaluation. With the results being averaged over the K folds, then some additional measure of variance might be helpful. Following from this, the supplementary material provides a table of model weights - how are these arrived at from the K-folds?

Finally, the results presentation should be improved. The use of several complementary metrics makes sense, however the headline result in the abstract and discussion present misleading figures. The abstract reports a positive recall, or sensitivity, of 97%, when in fact it should be 79%. According to the tables, the negative recall, or specificity, should be around 83%, however the discussion states that "almost all of the time (97%), individuals who will not develop viral-like illness symptoms". There may be additional errors or confusions in the reporting of these results that I have not found, and I would recommend going through these in detail again.

Some specific presentation issues:

- Introduction, para 3: sentence structure on 'Outputs from wearable...health and disease'

- See also 'Besides, subject-reported ... COVID-19'

- Forecasting Model: unclear statement 'where x is a set of binary variables (among which there are n input and one output variable)...' What does n refer to? This whole section needs expanding and clarification.

- Results: 867 -> 767 -> 115. Was it not possible to salvage some of the missing data? For example, include some of the 376 missing cognitive data that had some wearable data towards training the forecasting model?

Reviewer #2: The authors present a rule based probabilistic analysis for determining the onset of symptoms for SARS-CoV-2 among the subjects in the study. They use different wearables to taking measurements of the people involved in their study and try to determine the people susceptible to infection with SARS-CoV-2. The work is interesting and is of value for everyone considering the entire world is grappling with the pandemic.

The concerns that I have about the manuscript in its current state are as follows:

1. The authors state that "The model is calibrated using an accepted method (cross-entropy loss function, see supplementary material) that finds the set of weights , which minimizes the prediction errors using the training model". After checking the supplementary material, I found the authors state that the cross-entropy function maximizes the maximum likelihood estimation (MLE). However, it is not clear to me what is the y in the process to calculate the conditional probability P(y=1|r), the y is not explained clearly in supplementary material (SM). Is it the subjects in the study who were eventually infected by COVID-19? Moreover the cross entropy loss function is Loss = -y*log(y). This term is then summed over all the possible classes in the dataset. How are the rules fit into this framework? If the authors are not using something like this how are weights calculated?

This calculation of weights is a very important detail in the machine learning (ML) model developed by the authors but it is a bit unclear. I would like the authors to explain it in greater detail and write about it in the main manuscript rather than SM.

2. The authors use the exact same line twice in the paper once in abstract and once in discussion

"Conversely, the model correctly predicts as positive, 34% of the time, individuals who will develop viral-like illness symptoms in the next three days. "

Details about the implication of the statement in the discussion section are missing

What does 34% signify in this case, does it mean that the accuracy for viral like symptoms is 34%?

3. Are the rules stated in the forecasting model created by experts after observing the data? If not, how were they created? The rules in Table 6.1 clearly state expert rule used in labelling model but there is no such mention of that in table 7.1. Assuming it was created by experts it would have been more useful to use some ML techniques that figure about the rules themselves like random forests and other decision tree based models.

4.Intuitively, the greater weight would imply greater influence of the rule in predicting the label. However, most of the weights presented in the study are fairly close. For example: Shortness of Breath, Coughing up blood, Sore throat, Chills, Phlegm, Diarrhea have almost the same weight ~0.82. No analysis of why this is happening has been presented, is it because of the nature of the data? some of the parameters presented in the example are distinctly different. Another thing that I found interesting was that persistence of symptoms had lesser weight than cough and fever. However, if the symptoms in the table are persistent for 2-3 days then the person would be more worried. It would be interesting to analyze the weights in greater detail. It will make the study more clearer and accessible

5.This is a minor concern: The very essence of ML lies in learning from the features and developing a mapping between labels (in this case the rules and labels). If the authors use some more advanced ML models in their future studies perhaps there will no need to draft the rules. One problem that I see that the designed rules may not be exhaustive and some of the important rules may be missing from the model in the current state.

Overall, I feel that the work and the methodology are of value great research community. However, the authors have some small gaps in their study and need to address a few things in greater detail and clarity.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: J. Ward

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2021 Oct 14;16(10):e0257997. doi: 10.1371/journal.pone.0257997.r002

Author response to Decision Letter 0

2 Jun 2021

Response to Reviewers

First, we would like to thank the Reviewers for the detailed reading of our manuscript and all the valuable comments. Please find below a list of changes in response to their comments.

Responses to the comments of the first Reviewer:

1. The Reviewer wrote:

“First off, the machine learning component is not clear. The model is described as a probabilistic graphical network acting on a combination of binary features. The exact operation or preprocessing applied to these features, or indeed what these features are, is not specified in the main paper. Further information is supplied in the supplementary material, however it would enhance reader understanding considerably to give some clearer examples on, for example, how measured heart rate is incorporated into the model. (Even in the supplementary material, the model description could be improved.) This whole section should be written in a clearer, fuller, way using appropriate terminology to allow easier replication. It would also be helpful to know why this model was chosen over other potential machine learning approaches.”

Our reply:

To address the reviewer’s comments, we rewrote the section about forecasting to clarify our approach. In particular, we incorporated some of the supplementary material into the main text to add more information about the underlying algorithm in this section and the relevant references to Markov Logic. We also added information about the data flow and the preprocessing to facilitate the replication of our approach.

2. The Reviewer wrote:

“The training and testing procedure of the algorithm is also unclear. The supplementary material goes some way to clarifying this, but the main text could be tightened up somewhat. As K-fold cross-validation is used, it would help to directly specify the value of K (=4 in this case). Specifying a 25% test set split is not quite enough because this could imply a single leave-one test set out evaluation. With the results being averaged over the K folds, then some additional measure of variance might be helpful. Following from this, the supplementary material provides a table of model weights - how are these arrived at from the K-folds?”

Our reply:

We made the appropriate changes to the main text; in particular, we specified the value of K=4:

“Model performance was tested with K-fold cross-validation using the reserved 25% portion of the data, i.e. K=4 in our case.”

As usual in supervised learning, the model weights in the final model are not found via averaging but instead obtained by using all available data to train the model. To clarify this in the main text, we added the following sentence:

“The model weights in the final model are obtained by using all available data to train the model.”

The differences in the AUCs of the different folds were too small to be of interest. Moreover, our samples are appropriately stratified, avoiding a single leave-one test set out evaluation.

3. The Reviewer wrote:

“Finally, the results presentation should be improved. The use of several complementary metrics makes sense; however the headline result in the abstract and discussion present misleading figures. The abstract reports a positive recall, or sensitivity, of 97%, when in fact it should be 79%. According to the tables, the negative recall, or specificity, should be around 83%; however, the discussion states that "almost all of the time (97%), individuals who will not develop viral-like illness symptoms". There may be other errors or confusions in the reporting of these results that I have not found, and I would recommend going through these in detail again.”

Our reply:

We agree with the Reviewer and modified our report of results to increase clarity. The numbers in the abstract refer to Table A, particularly to the column reporting results for the threshold chosen to be 0.1. For this case, the sensitivity is indeed 79%, and the 93% refer to the negative predictive value (TN/(TN+FN)). We made the following changes in the text to improve clarity:

“We describe our strategy using an AI model that can predict, with 82% accuracy (negative predictive value 97%, specificity 83%, sensitivity 79%, precision 34%)”

We also added the words “negative predictive value” and “precision” to the main text:

The model would detect 79% of individuals who will develop symptoms (i.e., sensitivity) and correctly predicts, almost all of the time (97%, negative predictive value), individuals who will not develop viral-like illness symptoms in the next three days. Conversely, the model precision is 34%. Remember that precision is defined as the ratio of true positives (TP) over positives (P). In other words, if the model flags someone to develop viral-like symptoms in the next three days, the model is correct 34% of the time.

4. The Reviewer wrote:

“Some specific presentation issues:

- Introduction, para 3: sentence structure on 'Outputs from wearable...health and disease'

- See also 'Besides, subject-reported ... COVID-19'”

Our reply:

We changed the sentences that the reviewer pointed out. The new version reads:

Outputs from wearable technology, including heart rate (HR), heart rate variability (HRV), respiration rate (RR), temperature, blood oxygenation, sleep, and other physiological assessments, are increasingly being explored in studies of health and disease 10–12. Moreover, various subject-reported symptoms captured on mobile apps transform both surveillance and contact tracing management strategies for COVID-19 13–15.

5. The Reviewer wrote:

“Forecasting Model: unclear statement 'where x is a set of binary variables (among which there are n input and one output variable)...' What does n refer to? This whole section needs expanding and clarification.”

Our reply:

We rewrote this section to address the Reviewer’s comments. We rephrased the paragraph as follows:

The joint distribution defined by the Markov network can be written as where denotes a set of n+1 binary variables, out of which the first n are input variables, and y is the output variable. Here, is a Boolean function corresponding to the rule, is a factor associated with the corresponding rule, is the normalization constant.

6. The Reviewer wrote:

“Results: 867 -> 767 -> 115. Was it not possible to salvage some of the missing data? For example, include some of the 376 missing cognitive data that had some wearable data towards training the forecasting model?”

Our reply:

In principle, we agree with the Reviewer that some of this data could be attributed appropriately to train the forecasting model. We did some research in this direction, but the preliminary results based on naïve imputation techniques were not very encouraging such that more research is necessary in order to salvage the missing data. In order to capture this in the text, we added the following sentence in the discussion:

We plan to develop additional models to attribute the data efficiently to extend the forecasting model’s usability.

Responses to the comments of the second Reviewer:

1. The Reviewer wrote:

“1. The authors state that "The model is calibrated using an accepted method (cross-entropy loss function, see supplementary material) that finds the set of weights , which minimizes the prediction errors using the training model". After checking the supplementary material, I found the authors state that the cross-entropy function maximizes the maximum likelihood estimation (MLE). However, it is not clear to me what is the y in the process to calculate the conditional probability P(y=1|r), the y is not explained clearly in supplementary material (SM). Is it the subjects in the study who were eventually infected by COVID-19? Moreover the cross entropy loss function is Loss = -y*log(y). This term is then summed over all the possible classes in the dataset. How are the rules fit into this framework? If the authors are not using something like this how are weights calculated?

Our reply:

To address the reviewer’s concerns, we rewrote the section on the forecasting model and included more material from the supplementary material in the main text. This also includes relevant references in the main text.

2. The Reviewer wrote:

“2. The authors use the same line twice in the paper, once in abstract and once in the discussion.

"Conversely, the model correctly predicts as positive, 34% of the time, individuals who will develop viral-like illness symptoms in the next three days. "

Details about the implication of the statement in the discussion section are missing.

What does 34% signify in this case? Doesit mean that the accuracy for viral-likesymptoms is 34%?”

Our reply:

To clarify the text, we added the word “precision” to the sentence in the abstract. Precision is defined as the ratio TP/P = TP/(TP+FP) (where TP are the true positives and FP are the false positives). We also made the following changes in the discussion:

Conversely, the model precision is 34%. Remember that precision is defined as the ratio of true positives (TP) over positives (P). In other words, if the model flags someone to develop viral-like symptoms in the next three days, the model is correct 34% of the time.

3. The Reviewer wrote:

“3. Are the rules stated in the forecasting model created by experts after observing the data? If not, how were they created? The rules in Table 6.1 clearly state expert rule used in labelling model but there is no such mention of that in table 7.1. Assuming it was created by experts it would have been more useful to use some ML techniques that figure about the rules themselves like random forests and other decision tree based models.”

Our reply:

The rules used in the forecasting model are found via rule mining from our algorithm. The rewritten section on forecasting now contains more information to clarify this point. In particular:

To create the forecasting model, however, the rules were extracted from the available data via rule mining and the weights of the rules were determined by the calibration process.

Also, note that we were focusing on developing an interpretable rule-based model consisting of a relatively small number of rules in our approach. Random forest usually produces a large number of rules, which makes this approach much less interpretable. In our research, the probabilistic rules yielded the most promising results when developing an accurate yet transparent model.

4. The Reviewer wrote:

“4.Intuitively, the greater weight would imply greater influence of the rule in predicting the label. However, most of the weights presented in the study are fairly close. For example: Shortness of Breath, Coughing up blood, Sore throat, Chills, Phlegm, Diarrhea have almost the same weight ~0.82. No analysis of why this is happening has been presented, is it because of the nature of the data? some of the parameters presented in the example are distinctly different. Another thing that I found interesting was that persistence of symptoms had lesser weight than cough and fever. However, if the stable’s symptoms are persistent for 2-3 days, the person would be more worried. It would be interesting to analyze the weights in greater detail. It will make the study clearer and more accessible.”

Our reply:

The rules and weights of the labeling were dictated from the input of medical and epidemiologists experts. This is different from the forecasting model in which the rules are found via rule mining by our algorithm and the weights set through algorithm calibration. To clarify the origin of the rules and weights of the labeling model, we added the following sentences:

The rules and weights in this model were found from expert input and reviewed by our medical team. In particular, the weights associated with the rules were chosen to minimize the labeling error assessed by medical experts. We also fine-tuned some rule weights by fitting the model to a small synthetic data set containing a few most typical symptom combinations. The actual calculation of the labeling model’s output score happens by applying the probabilistic logic machinery, described in more detail in the next paragraph.

We agree with the Reviewer that the labeling model itself, together with the rules and weights, would warrant a broader discussion. In this paper, however, the labeling model is mainly used to define the output variable and thus set the forecasting model stage, which is in this paper’s focus. However, we plan to extend the discussion with our medical team and look into the weights more in-depth to publish insights focusing on the labeling model itself.

5. The Reviewer wrote:

“5.This is a minor concern: The very essence of ML lies in learning from the features and developing a mapping between labels (in this case the rules and labels). If the authors use some more advanced ML models in their future studies perhaps there will no need to draft the rules. One problem that I see that the designed rules may not be exhaustive and some of the important rules may be missing from the model in the current state.”

Our reply:

We assume that the Reviewer’s comments are in regard to the labeling model. The Reviewer is correct in the sense that rule-based systems are most powerful when there is a combination of expert input and rule-mining from the data. In the particular case of the labeling model in this study, the output variable needed to be defined properly and the idea was to model this in the same way a doctor would define the onset of a viral infection looking at symptoms present. The Reviewer is correct in the sense that, with more data and more input, this definition of onset might be refined by data-driven rules in the future.

Attachment

Submitted filename: Response to Reviewers (1).docx

Click here for additional data file.^{(299.6KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0257997.r003

Decision Letter 1

Nizam Uddin Ahamed

7 Jul 2021

PONE-D-20-36216R1

Prediction of viral symptoms using wearable technology and artificial intelligence: A pilot study in healthcare workers

PLOS ONE

Dear Dr. DHAESE,

Please submit your revised manuscript by Aug 21 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Nizam Uddin Ahamed, PhD

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #1: Partly

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #1: No

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Reviewer #1: Following from the earlier review, this paper is both timely and extremely relevant. Frustratingly, the initial round of revisions did not fully address all of the concerns raised and so I would suggest a final round of revisions focusing on clarity and reproducibility of the work.

The cross-validation approach described in the main text remains unclear. The revision now states that "Model performance was tested with K-fold cross-validation using the reserved 25% portion of the data, i.e. K=4 in our case.” This reads as if K-fold CV is being applied to the 25% (the reserved test set), rather than the training set as would be expected. This should just be a matter of clarification in the text.

The fact that there was very little difference in the AUCs for each fold is a good thing, perhaps worth mentioning as it suggests that the model is consistently generalisable.

Rule mining is stated as the main method used to build the forecasting model, however very little detail is given as to what rule mining actually is, or how exactly it is implemented. There isn't even a reference given on the technique. Some further detail on this would be appreciated.

The revised statement about model weights is concerning. The use of 'all available data' would suggest that there is no separation of training and test data for evaluating the final model. Just to be clear, and avoid any suspicion of overfitting, can you clarify that this model is only then applied to previously unseen data?

The calibration process uses gradient descent, however there remains a lack of clarity on the exact implementation used or the various design choices made. The one-line description in the main text is vague and includes the statement, 'or any of its variants', which is too unspecific for a reproducible work. It would aid the reader to include further details on the implementation. The commercial implementation that is used should also be mentioned directly in the main text (rather than simply referenced). Ideally some implementation details on this could be included in the supplementary text, too. Considering that the commercial company which implemented this is included as an affiliate on the paper, it does not seem unreasonable to expect a bit more detail.

Supplementary material. There is a disconnect between some of the text and the provided tables and figures. Generally, all tables need to be clearly referenced (and include some caption). For example, p12 states, "the learning curve presented in the table,"... yet does not specify which table.

Further comments:

- The cross-entropy function does not render well on the PDF that this reviewer received - several variables were replaced by black rectangles.

- The references all link to a paperpile.com repository which is not accessible (to this reviewer at least)

- The ROC plots need labels on the axes (e.g. Fig2)

- All tables should include descriptive captions

- Figure 3 - please specify in the caption what the variance represents

Reviewer #2: The authors have improved the manuscript significantly. The clarity of the manuscript has been significantly enhanced.

I would like the authors to add one final detail in the manuscript .

1. What is the distribution of the labels in data used to train the model. Since, a lot of data is missing due non-compliance I think it is a relevant detail that needs to be added. For eg: If the dataset is skewed towards one class the numbers indicating the performance can be sometimes misleading. As a constant model that predicts a single class all the time can also have high scores on performance metrics. Addition of this data will help increase the confidence on this developed model

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: J Ward

Reviewer #2: No

PLoS One. 2021 Oct 14;16(10):e0257997. doi: 10.1371/journal.pone.0257997.r004

Author response to Decision Letter 1

12 Aug 2021

First, we would like to thank the Reviewers for the detailed reading of our manuscript and all the valuable comments. Please find below a list of changes in response to their comments.

Responses to the comments of the first Reviewer:

The Reviewer wrote:

“Following from the earlier review, this paper is both timely and extremely relevant. Frustratingly, the initial round of revisions did not fully address all of the concerns raised and so I would suggest a final round of revisions focusing on clarity and reproducibility of the work”

Our reply: We thank the reviewer and agree that these revisions will increase clarity of the

work. We have revised the manuscript following each review.

The Reviewer wrote:

“The cross-validation approach described in the main text remains unclear. The revision now states that "Model performance was tested with K-fold cross-validation using the reserved 25% portion of the data, i.e. K=4 in our case.” This reads as if K-fold CV is being applied to the 25% (the reserved test set), rather than the training set as would be expected. This should just be a matter of clarification in the text”

Our reply: We thank the reviewer and agree that these revisions will increase clarity of the

work. We have revised the manuscript following each review. We have modified the text as

follows:

The Reviewer wrote:

“The fact that there was very little difference in the AUCs for each fold is a good thing, perhaps worth mentioning as it suggests that the model is consistently generalizable. ”

Our reply: We agree with the reviewer that the AUCs variations across folds is a good thing. It is mentioned in the text already in the result section as a reference to figure 3 as “Figure 3 illustrates that the model reaches a plateau after about 1500 samples, and that much accuracy cannot be gained by adding more samples“

We have added a mention of it in the discussion as well as the following statement:

Finally, the very little difference in AUCs between each fold suggest that the model is consistently generalizable.

The Reviewer wrote:

“Rule mining is stated as the main method used to build the forecasting model, however very little detail is given as to what rule mining actually is, or how exactly it is implemented. There isn't even a reference given on the technique. Some further detail on this would be appreciated.”

Our reply: We agree and we have added some information about rule mining. The rule mining is done using the Assocation Rule Mining Algorithm published by Hipps et al in June 2000. We have added a reference to that paper as well for full reproducibility.

The paragraph added in the Method section now reads:

To create the forecasting model, the rules were extracted from the available data via rule mining. For this purpose, we used the Association Rule Mining algorithm [Jochen Hipp, Ulrich Güntzer, and Gholamreza Nakhaeizadeh. 2000. Algorithms for association rule mining — a general survey and comparison. SIGKDD Explor. Newsl. 2, 1 (June, 2000), 58–64. DOI:https://doi.org/10.1145/360402.360421], which is based on the co-occurrences frequency analysis. After extracting the rules, the weights of the rules were determined by the calibration process outlined above.

And reference 32 was added

32. Jochen Hipp, Ulrich Güntzer, and Gholamreza Nakhaeizadeh. 2000. Algorithms for association rule mining — a general survey and comparison. SIGKDD Explor. Newsl. 2, 1 (June, 2000), 58–64. DOI:https://doi.org/10.1145/360402.360421

The Reviewer wrote:

“The revised statement about model weights is concerning. The use of 'all available data' would suggest that there is no separation of training and test data for evaluating the final model. Just to be clear, and avoid any suspicion of overfitting, can you clarify that this model is only then applied to previously unseen data?”

Our reply: We have rewritten the statement to clarify that indeed the model is trained only on the training dataset.

It now reads as: The entire dataset is divided 4 times as 75% for training and 25% for validating the model. The results are then average across the 4 runs of training-validation. The model weights in the final model are obtained by using training dataset of the model.

The Reviewer wrote:

“The calibration process uses gradient descent, however there remains a lack of clarity on the exact implementation used or the various design choices made. The one-line description in the main text is vague and includes the statement, ‘or any of its variants', which is too unspecific for a reproducible work. It would aid the reader to include further details on the implementation. The commercial implementation that is used should also be mentioned directly in the main text (rather than simply referenced). Ideally some implementation details on this could be included in the supplementary text, too. Considering that the commercial company which implemented this is included as an affiliate on the paper, it does not seem unreasonable to expect a bit more details“

Our reply:

We agree and we have modified the text to add clarifications. We have added in the method section details on the gradient descent:

We have added to the supplementary material that now reads as:

The probability distribution is calibrated to the training set using a cross-entropy loss function. The calibration allows finding the set of weights ω_j which maximizes the likelihood of the observation. The model prediction s for every observation r=〖(r〗_1,r_2,and.,r_n) is computed as the conditional probability of the output variable y: s=P(y=1|r)

The numerical implementation of the calibration routine is based on the Limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm (L-BFGS). We chose this method for fast convergence and efficient use of computational resources.

We have added the reference to the commercial implementation in the text as well:

Our model was developed on Stratyfy’s Probabilistic Rule Engine, a commercial machine learning platform26.

We have added the section on the rule mining algorithm as well as mentioned previously:

To create the forecasting model, the rules were extracted from the available data via rule mining. For this purpose, we used the Association Rule Mining algorithm [32], which is based on the co-occurrences frequency analysis. After extracting the rules, the weights of the rules were determined by the calibration process outlined above.

We have added the reference to paper explaining the underlaying methods for full reproducibility:

The Reviewer wrote:

“Supplementary material. There is a disconnect between some of the text and the provided tables and figures. Generally, all tables need to be clearly referenced (and include some caption). For example, p12 states, "the learning curve presented in the table,"... yet does not specify which table.“

Our reply:

We reviewed all the tables and figures and made sure they connect with the text. All tables and figures have their captions submitted as per the guidelines of PLOS one and are shown on the first page of the figures and tables section.

The Reviewer wrote:

Further comments:

- The cross-entropy function does not render well on the PDF that this reviewer received - several variables were replaced by black rectangles.

Our reply:

The figures are also sent in high resolution and we believe this will be changed when the final paper is put together. The black rectangles have been removed.

- The references all link to a paperpile.com repository which is not accessible (to this reviewer at least)

Our reply: We have removed the hyperlinks.

- The ROC plots need labels on the axes (e.g. Fig2)

Our reply:

We have added the labels on the axis

- All tables should include descriptive captions

Our reply:

We have added the captions in the pdf of submission to address this problem.

The list is the following:

Figure 1: Data Flow a) the labeling model, b) the forecast model. Each model takes as input three days of data (d, d-1, d-2).

Figure 2: ROC (False Positive Rate vs True Positive Rate) and Precision/recall curves for the forecasting model with (A, C) and without cognitive assessment (B, D).

Figure 3: Model Performance (Learning and Stability). The shade colors represent the variance of results created by the K-Fold(4) validation.

Table 1: Labeling Model Rules

Table 2: Study Populations Demographics

Table 3: Algorithm-Derived Rules List of 45 rules extracted by the algorithm and used in the model with their relative weights. Rules are aggregated to forecast suspicion of a viral disease in a participant.

Table 4: Model Performance With and Without PVT.

- Figure 3 - please specify in the caption what the variance represents

Our reply:

The variance represents the variance from the 4 runs of validation as per the K-fold validation process. We will add this in the caption

It now reads as:

Figure 3: Model Performance (Learning and Stability). The shade colors represent the variance of results created by the K-Fold (4) validation.

Responses to the comments of the second Reviewer:

The Reviewer wrote:

“The authors have improved the manuscript significantly. The clarity of the manuscript has been significantly enhanced.”

Our reply: We thank the reviewer for its positive feedback.

The Reviewer wrote:

“I would like the authors to add one final detail in the manuscript.

Our reply: We agree with the reviewer. The distribution of the labels used are too big for

the main paper but are part of the supplementary materials. We have added text in the discussion about the label distribution of the two classes (positives vs negative) for clarity.

Attachment

Submitted filename: Rebutal letter 3 - PLOS final.docx

Click here for additional data file.^{(36.5KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0257997.r005

Decision Letter 2

Nizam Uddin Ahamed

16 Sep 2021

Prediction of viral symptoms using wearable technology and artificial intelligence: A pilot study in healthcare workers

PONE-D-20-36216R2

Dear Dr. DHAESE,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Nizam Uddin Ahamed, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #1: No

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Reviewer #1: All reviewer comments have been adequately addressed. I am happy to accept the manuscript in its current form.

Reviewer #2: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

PLoS One. doi: 10.1371/journal.pone.0257997.r006

Acceptance letter

Nizam Uddin Ahamed

28 Sep 2021

PONE-D-20-36216R2

Prediction of viral symptoms using wearable technology and artificial intelligence: A pilot study in healthcare workers

Dear Dr. D’Haese:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Nizam Uddin Ahamed

Academic Editor

PLOS ONE

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

S1 Table. Data dictionary.

Data dictionary of each data element used in the model.

(PDF)

Click here for additional data file.^{(61.8KB, pdf)}

S2 Table. Disease onset model rules.

The following table reports the probabilistic weights for each rule of the symptom onset forecasting model.

(PDF)

Click here for additional data file.^{(102.9KB, pdf)}

S1 File. List of questions.

List of questions asked to the participants.

(PDF)

Click here for additional data file.^{(37.1KB, pdf)}

S2 File. Dataset and inclusion/exclusion criteria.

Inclusion/exclusion criteria and description of data set.

(PDF)

Click here for additional data file.^{(2.1MB, pdf)}

S3 File. Probabilistic rule engine.

Detailed description of the Probabilistic Rule Engine.

(PDF)

Click here for additional data file.^{(51.2KB, pdf)}

S4 File. Validation approach.

Detailed description of the validation approach.

(PDF)

Click here for additional data file.^{(57.3KB, pdf)}

S5 File. Labeling model.

Detailed description of the labeling model.

(PDF)

Click here for additional data file.^{(38KB, pdf)}

Attachment

Submitted filename: Response to Reviewers (1).docx

Click here for additional data file.^{(299.6KB, docx)}

Attachment

Submitted filename: Rebutal letter 3 - PLOS final.docx

Click here for additional data file.^{(36.5KB, docx)}

Data Availability Statement

[pone.0257997.ref001] 1.Li Q, Guan X, Wu P, et al. Early Transmission Dynamics in Wuhan, China, of Novel Coronavirus–Infected Pneumonia. New England Journal of Medicine. 2020;382(13):1199–1207. doi: 10.1056/NEJMoa2001316 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref002] 2.He X, Lau EHY, Wu P, et al. Temporal dynamics in viral shedding and transmissibility of COVID-19. Nat Med. 2020;26(5):672–675. doi: 10.1038/s41591-020-0869-5 [DOI] [PubMed] [Google Scholar]

[pone.0257997.ref003] 3.Spinato G, Fabbris C, Polesel J, et al. Alterations in Smell or Taste in Mildly Symptomatic Outpatients With SARS-CoV-2 Infection. JAMA. Published online Apr 22nd, 2020. doi: 10.1001/jama.2020.6771 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref004] 4.Guan W-J, Ni Z-Y, Hu Y, et al. Clinical Characteristics of Coronavirus Disease 2019 in China. New England Journal of Medicine. 2020;382(18):1708–1720. doi: 10.1056/nejmoa2002032 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref005] 5.[No title]. Accessed Sept 22nd, 2020. https://news.gallup.com/file/poll/269141/191206HealthTrackers.pdf

[pone.0257997.ref006] 6.Izmailova ES, McLean IL, Hather G, et al. Continuous Monitoring Using a Wearable Device Detects Activity‐Induced Heart Rate Changes After Administration of Amphetamine. Clinical and Translational Science. 2019;12(6):677–686. doi: 10.1111/cts.12673 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref007] 7.Erb MK, Karlin DR, Ho BK, et al. mHealth and wearable technology should replace motor diaries to track motor fluctuations in Parkinson’s disease. NPJ Digit Med. 2020;3:6. doi: 10.1038/s41746-019-0214-x [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref008] 8.Griffin B, Saunders KEA. Smartphones and Wearables as a Method for Understanding Symptom Mechanisms. Front Psychiatry. 2019;10:949. doi: 10.3389/fpsyt.2019.00949 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref009] 9.Radin JM, Wineinger NE, Topol EJ, Steinhubl SR. Harnessing wearable device data to improve state-level real-time surveillance of influenza-like illness in the USA: a population-based study. The Lancet Digital Health. 2020;2(2):e85–e93. doi: 10.1016/S2589-7500(19)30222-5 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref010] 10.Jarczok MN, Kleber ME, Koenig J, et al. Investigating the Associations of Self-Rated Health: Heart Rate Variability Is More Strongly Associated than Inflammatory and Other Frequently Used Biomarkers in a Cross Sectional Occupational Sample. PLOS ONE. 2015;10(2):e0117196. doi: 10.1371/journal.pone.0117196 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref011] 11.Bakken AG, Axén I, Eklund A, O’Neill S. The effect of spinal manipulative therapy on heart rate variability and pain in patients with chronic neck pain: a randomized controlled trial. Trials. 2019;20(1). doi: 10.1186/s13063-019-3678-8 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref012] 12.Johnston BW, Barrett-Jolley R, Krige A, Welters ID. Heart rate variability: Measurement and emerging use in critical care medicine. Pediatr Crit Care Med. 2020;21(2):148–157. doi: 10.1177/1751143719853744 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref013] 13.Abeler J, Bäcker M, Buermeyer U, Zillessen H. COVID-19 Contact Tracing and Data Protection Can Go Together. JMIR Mhealth Uhealth. 2020;8(4):e19359. doi: 10.2196/19359 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref014] 14.Show evidence that apps for COVID-19 contact-tracing are secure and effective. Nature. 2020;580(7805):563. doi: 10.1038/d41586-020-01264-1 [DOI] [PubMed] [Google Scholar]

[pone.0257997.ref015] 15.Wang S, Ding S, Xiong L. A New System for Surveillance and Digital Contact Tracing for COVID-19: Spatiotemporal Reporting Over Network and GPS. JMIR Mhealth Uhealth. 2020;8(6):e19457. doi: 10.2196/19457 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref016] 16.Oura Ring: the most accurate sleep and activity tracker. Oura Ring. Accessed Jul 7th, 2020. https://ouraring.com

[pone.0257997.ref017] 17.‎RNI Health. Accessed Aug 14th, 2020. https://apps.apple.com/us/app/rni-health/id1515732074

[pone.0257997.ref018] 18.Roach GD, Dawson D, Lamond N. Can a Shorter Psychomotor Vigilance Task Be Used as a Reasonable Substitute for the Ten‐Minute Psychomotor Vigilance Task? Chronobiology International. 2006;23(6):1379–1387. doi: 10.1080/07420520601067931 [DOI] [PubMed] [Google Scholar]

[pone.0257997.ref019] 19.D’Haese P-F, Konrad PE, Pallavaram S, et al. CranialCloud: a cloud-based architecture to support trans-institutional collaborative efforts in neurodegenerative disorders. Int J Comput Assist Radiol Surg. 2015;10(6):815–823. doi: 10.1007/s11548-015-1189-y [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref020] 20.Stone M. Cross-Validatory Choice and Assessment of Statistical Predictions. Journal of the Royal Statistical Society: Series B (Methodological). 1974;36(2):111–133. doi: 10.1111/j.2517-6161.1974.tb00994.x [DOI] [Google Scholar]

[pone.0257997.ref021] 21.Richardson M, Domingos P. Markov logic networks. Machine Learning. 2006;62(1–2):107–136. doi: 10.1007/s10994-006-5833-1 [DOI] [Google Scholar]

[pone.0257997.ref022] 22.Nilsson NJ. Probabilistic logic. Artificial Intelligence. 1986;28(1):71–87. doi: 10.1016/0004-3702(86)90031-7 [DOI] [Google Scholar]

[pone.0257997.ref023] 23.Wang C, Komodakis N, Paragios N. Markov Random Field modeling, inference & learning in computer vision & image understanding: A survey. Computer Vision and Image Understanding. 2013;117(11):1610–1627. doi: 10.1016/j.cviu.2013.07.004 [DOI] [Google Scholar]

[pone.0257997.ref024] 24.Flu Symptoms & Diagnosis | CDC. Published Dec 5th, 2019. Accessed Jul 25th, 2020. https://www.cdc.gov/flu/symptoms/index.html

[pone.0257997.ref025] 25.Eccles R. Understanding the symptoms of the common cold and influenza. Lancet Infect Dis. 2005;5(11):718–725. doi: 10.1016/S1473-3099(05)70270-X [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref026] 26.Stratyfy, Inc.. https://www.stratyfy.com

[pone.0257997.ref027] 27.Jochen Hipp, Ulrich Güntzer, and Gholamreza Nakhaeizadeh. 2000. Algorithms for association rule mining—a general survey and comparison. SIGKDD Explor. Newsl. 2, 1 (June, 2000), 58–64. 10.1145/360402.360421 [DOI]

[pone.0257997.ref028] 28.Pavlov VA, Tracey KJ. The vagus nerve and the inflammatory reflex—linking immunity and metabolism. Nature Reviews Endocrinology. 2012;8(12):743–754. doi: 10.1038/nrendo.2012.189 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0257997.ref029] 29.Pereira MR, Leite PEC. The Involvement of Parasympathetic and Sympathetic Nerve in the Inflammatory Reflex. Journal of Cellular Physiology. 2016;231(9):1862–1869. doi: 10.1002/jcp.25307 [DOI] [PubMed] [Google Scholar]

[pone.0257997.ref030] 30.Pal GK, Nanda N. Vagus Nerve: The Key Integrator of Anti-inflammatory Reflex. International Journal of Clinical and Experimental Physiology. 2020;7(1):01–02. doi: 10.5530/ijcep.2020.7.1.1 [DOI] [Google Scholar]

[pone.0257997.ref031] 31.Elkington LJ, Gleeson M, Pyne DB, Callister R, Wood LG. Inflammation and Immune Function. Antioxidants in Sport Nutrition. Published online 2014:171–181. doi: 10.1201/b17442-11 [DOI] [Google Scholar]

[pone.0257997.ref032] 32.Gleeson M. Immune function in sport and exercise. J Appl Physiol. 2007;103(2):693–699 doi: 10.1152/japplphysiol.00008.2007 [DOI] [PubMed] [Google Scholar]

PERMALINK

Prediction of viral symptoms using wearable technology and artificial intelligence: A pilot study in healthcare workers

Pierre-François D’Haese

Victor Finomore

Dmitry Lesnik

Laura Kornhauser

Tobias Schaefer

Peter E Konrad

Sally Hodder

Clay Marsh

Ali R Rezai

Roles

Abstract

Introduction

Methods

Study design

Fig 1.

Labeling model

Table 1. Labeling model rules.

Forecasting model

Validation of the model

Results

Table 2. Study populations demographics.

Table 3. Algorithm-derived rules list of 45 rules extracted by the algorithm and used in the model with their relative weights.

Fig 2.

Fig 3. Model performance (learning and stability).

Table 4. Model performance with and without PVT.

Discussion

Supporting information

Acknowledgments

Data Availability

Funding Statement

References

Decision Letter 0

Nizam Uddin Ahamed

Roles

Author response to Decision Letter 0

Decision Letter 1

Nizam Uddin Ahamed

Roles

Author response to Decision Letter 1

Decision Letter 2

Nizam Uddin Ahamed

Roles

Acceptance letter

Nizam Uddin Ahamed

Roles

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases