A Smartwatch-Based Framework for Real-Time and Online Assessment and Mobility Monitoring

Abstract

Smartphone and smartwatch technology is changing the transmission and monitoring landscape for patients and research participants to communicate their healthcare information in real time. Flexible, bidirectional and real-time control of communication allows development of a rich set of healthcare applications that can provide interactivity with the participant and adapt dynamically to their changing environment. Additionally, smartwatches have a variety of sensors suitable for collecting physical activity and location data. The combination of all these features makes it possible to transmit the collected data to a remote server, and thus, to monitor physical activity and potentially social activity in real time. As smartwatches exhibit high user acceptability and increasing popularity, they are ideal devices for monitoring activities for extended periods of time to investigate the physical activity patterns in free-living condition and their relationship with the seemingly random occurring illnesses, which have remained a challenge in the current literature. Therefore, the purpose of this study was to develop a smartwatch-based framework for real-time and online assessment and mobility monitoring (ROAMM). The proposed ROAMM framework will include a smartwatch application and server. The smartwatch application will be used to collect and preprocess data. The server will be used to store and retrieve data, remote monitor, and for other administrative purposes. With the integration of sensor-based and user-reported data collection, the ROAMM framework allows for data visualization and summary statistics in real-time.

1. Introduction

As technology continues to evolve, more and more physical devices are being integrated with sensors and connectivity. This growing network of devices able to connect and exchange data has been termed the Internet of Things [1], and will ultimately lead to high-fidelity data collection on a variety of health-based outcomes at the population level. At present, such connected devices offer researchers the immediate benefit of free-living data collection in the absence of direct physical contact with the device or participant. In fact, modern mobile devices (e.g., smart devices) offer a convenient platform that includes power computational abilities, high-speed connectivity, adequate storage, and a wide array of sensors. Unlike more specialized devices for data collection (e.g. FitBit) that are focused on one domain (i.e. accelerometer-based activity monitors), smart devices combine power and flexibility with a variety of sensors that provide the necessary framework for a more comprehensive approach to remote personal health monitoring. Moreover, this approach also provides several major advantages over traditional methods including the ability to customize apps through the Application Program Interface (API), a screen interface for displaying information and interacting with participants, a physical input option (e.g. turn dial bezel), the large number of sensors, and the potential to have remote connectivity and control of sensors.

Smartwatches are convenient to wear and have the capability to collect data in a continuous manner given that the battery is charged periodically. Furthermore, they provide additional benefit to the participant including managing their calendar, text messaging and making phone calls which can have a signifi-cant impact on their acceptance and wear time. There has been a growing interest in adopting smartwatches for research on behaviors and mobility patterns [2]. While their convenience in wear and continuous data collection capability already make them a lucrative research tool, their ability for remote access and control of sensors along with the potential for direct communication with a user offers seemingly limitless possibilities of additional applications. For example, ecological momentary assessments (EMA) can be incorporated to describe health, symptoms and potential episodic health events that are challenging to capture in real-time (e.g., falls, hospitalizations). The concomitant collection of location information via GPS to understand community mobility patterns, physical activity data from an accelerometer and reported health symptoms or events is ideal for creating a narrative of personal health information in a remote and interactive manner.

Several smartphone-based frameworks have been introduced to monitor health conditions in free-living environment. These mobile healthcare or mHealth [3] frameworks rely on communication means to obtain vital information from patients in real time and provide warnings and guidelines remotely when data deviate from an expected value. While such frameworks bring merit, there are some gaps in phone-based ascertainment. Smartphones are usually carried in pocket, which is not an ideal location for activity recognition, or in hand-held bags (typically for women). Therefore, sensor data collected from these devices do not provide information required for activity recognition; especially, for cases where there are only hand movements, such as drinking water [4]. Smart-watches offer a more logical choice because they possess the same sensors and connectivity and are fixed to the body. Despite the benefits, the development of smartwatch apps for data collection has not progressed since their initial consumer release.

In sum, the purpose of this study is to develop the framework for a novel remote monitoring system through the integration of a smartwatch-based application and a remotely-connected server. Such framework will pave the way for additional applications that simultaneously collect data in the target domains of physical activity, mobility, EMA, patient-reported outcomes, and intervening health events. To accomplish this goal, we present the Real-time Online Assessment and Mobility Monitoring (ROAMM) framework that offers: 1) a convenient approach for long-term assessment in the context of varying health, 2) the ability to synchronize sensor data with reports of health events and symptoms (e.g., pain, fatigue), and 3) interactive communication in real time, providing an active channel for patient reported outcomes, health events and future intervention delivery (Figure 1). Knowledge of these domains in real-world scenarios will help understanding the inter and intra-personal factors that contribute to episodic health events.

Figure 1:

ROAMM framework conceptual diagram. Smartwatches are equipped with a variety of sensors, such as an accelerometer, heart rate monitor, and GPS. It possesses connectivity means and the convenience of developing customizable applications provide a platform that allows a real-time activity monitoring framework to be implemented.

2. Related Works

Monitoring mobility, activity, health events and self-reported health is a well-researched field. Most research has ascertained this information in broad snapshots in time, relying heavily on long-term recall of symptoms and medical events. Additionally, the use of such devices does not allow for real-time access to sensor data and patient reported outcomes (PROs) in free-living participants. Regarding the latter, the ROAMM infrastructure capitalizes on the “Experience Sample Method” of data collection originally developed by Larson and Csikszentmihalyi in 1983 [5]. The method, now often referred to ecological momentary assessment, was originally developed for psychological purposes to assess what activities people do, how they feel, and what they are thinking of during their daily lives. The method asks individuals to provide systematic diaries of their experiences at periodic or system defined occasions. It was brought about by the fact that people are poor at reconstructing their psychological experience after it has occurred. When frequently and randomly sampled, ecological assessments are often regarded as the “truth” because they estimate an unbiased average that has no tendency to either overestimate or underestimate the state. Additionally, questionnaires that rely on recall are noted to suffer from biases due to response sets and cultural/age normativity [6, 7]. It is particularly di cult for individuals to assess or recall complex experiences like pain, mood or fatigue after it has occurred [8, 9]. For example, Stone and colleagues demonstrated that patients with chronic pain overestimate their pain intensity levels by 35% (44 vs. 57 on a 100 point scale) on recalled compared to average of momentary pain experiences. There were also vast differences in pain intensity changes from one week to the next. Recalled changes in pain explained only 15% of the variance in momentary changes in pain [10, 11]. This finding is critically important to the current application because pain intensity is expected to change following an Intervening Health Event (IHE, i.e., episodic falls, injuries, illnesses, hospitalizations). These results are consistent with other reports on pain [10, 11], fatigue [12, 13, 14] and depressive mood [15, 16, 17]. A smartwatch approach to ecological momentary assessments being proposed in this application is a methodological advancement to Larson and Csikszentmihalyi’s paper diaries and current methods using computers, phones or tablet devices. This work presents a convenient approach for responses on a watch face that is synced to sensor data.

Intervening health events are an emerging scientific area in geriatrics and gerontology. Frequent and real-time queries of symptoms and health are a significant advance over traditional means that use infrequent reports of health. Accurately capturing falls is a perfect example of this pervasive issue. Falls occur in about 20–30% of the older adult population and result in 2.8 million injuries treated in the emergency room department [18, 19]. Falls result in a wide spectrum of consequences that span from no injury to death. Additionally, there are millions of fall events that do not result in medical care yet have a significant impact on physical and social independence [20, 21, 22]. Importantly, the severity of a fall dictates the accuracy in which an older person recalls the event prompting the science to move to more frequent ascertainment [23, 24, 25]. Despite decades of research, fall ascertainment remains archaic, costly, and burdensome [23, 24, 25, 26]. Plus, fall detection using wrist worn sensors have largely failed to demonstrate accuracy [27]. For example, the “gold-standard” ascertainment method uses paper and pencil weekly or monthly fall calendars to facilitate recall. Events discovered on calendars trigger questionnaires about the event or telephone interview. There has been some work to convert these methods to the digital age with web-based or smartphone ascertainment, but they tend to be focused on workplace settings or young populations with technology knowhow [28, 29].

There have been two major challenges for EMA studies. The first challenge is the frequency and length of interruptions; user compliance decreases rapidly if they are prompted with long questions and/or too frequently. Furthermore, the burden of responding to interruptions are intensified if the prompting device is not immediately accessible. While smartphones offer some convenience when used to collect EMA data, they are within hands reach for only 50% of the time [30]. Smartwatches are more proper and convenient means for collecting EMA data, because they are worn on the wrist, and have been used in recent works. Intille et. al. [31] have introduced a smartwatch-based framework to interact with participants through short multiple-choice questions that appear on the watch.

Studying physical activity, which requires wearable sensor data, along with EMA has attracted attention in the recent years. Blaauw et. al. [32] have proposed a framework to obtain unified sensor data from smart wearables along with participants’ responses to EMA questionnaire. Smartwatches provide sensor data, the capability to prompt users, and connectivity means that allow for collecting and analyzing sensor and EMA data at the same time and in real-time. The ROAMM infrastructure aims to leverage smartwatches’ capabilities and make an important leap to a wearable interface that offers convenience, real-time connectivity, and periodic prompting about falls or other events that benefit from frequent ascertainment.

Online data monitoring using smart devices has the potential of addressing issues related balancing convenience and research capability. To achieve this balance, several challenges, which were not present for smartphone-based frameworks, must be addressed. These include limiting the computational power used to sensors of importance to maintain battery life. The ROAMM framework described in this paper utilizes a standalone smartwatch application to collect and process raw sensor data. These sensor data are then synced with screen-based information on symptoms, patient-reported outcomes, and recorded incidents. Overall, ROAMM is designed to fill gaps in inadequate measurement tools that are not suited for studies that require continuous data collection for extended periods of time or real-time mobility monitoring.

3. ROAMM Domains and Measures

ROAMM is designed to capture three important domains that include: 1) mobility and activity through sensors, 2) EMA and patient-reported outcomes through the screen, and 3) intervening health events (hospitalizations and falls). The domains described in Table 1 are considered critical pieces of information in Geriatric and Gerontological research as well as clinical care of older adults.

Table 1:

Description of ROAMM domains and measurements

Domain	Description	Collection Method	Frequency*
Mobility and Activity	Physcial activity: the average minutes per day Total activity, light, moderate & vigorous activity, sedentary	Tri-axial accelerometer and derived vector magnitude	Collected at 10 Hz and processed daily
	Walk speed: GPS is triggered during a two-min bout of continuous of activity until the end of the bout (≥1 min of no activity detection). GSP data processed using previously validated Personal Activity and Location Measurement System (PALMS) [29].	Accelerometer trigger and GPS longitude and latitude	Continuous to identify two min duration activity bouts
	Life-Space: Average excursion size maximum distance from the home for each excursion away from home. The maximum distances from multiple excursions are averaged over a day. Average excursion span - average daily maximum distance between all recorded locations away from home. Measures travel clusters away from home, independent of maximal distance traveled from home Smaller values indicate more compact traveling.	GPS longitude and lattitude	Every 15 minutes, 7 AM and 11 PM based on on previous work [33]
Ecological momentary assessments & patient reported outcomes	Pain: A Numerical Pain Rating Scale for rating pain intensity from 0–10 [34, 35]. Poor mood: Adapted from the visual analogue mood scale [36, 37] Fatigue: Adapted from ecological momentary assessment measures of fatigue from references [38, 39]	Likert scale selection using the graphical watch face with rotating bezel and save button (see Figure 3)	Random daily** end-of-day summary rating of maximal pain, highest fatigue and worst mood experienced that day
	Disability: “Do you have severe difficulty with [the task]?” Basic activities (bathing, dressing, walking, and transferring), instrumental activities (shopping, housework, meal preparation, taking medications, or managing finances), and mobility activities (walking a quarter mile, climbing a flight of stairs, or lifting or carrying 10 lb). Positive responses will be following by asking Do you need help from another person [to complete the task]?	Select responses using the graphical watch face with rotating bezel and “save” buttons	Randomly, once per week**. End-of-week summary (over the past week did you have.)
Intervening health event monitoring	Fall: “Did you fall this week?” A yes response will trigger the following: 1) “What day and approximate time did you fall?”; 2) “Did you seek medical care or were hospitalized?”, 3) “Did this result in 3 or more days of restricted activity?”	Select responses using he graphical watch face with rotating bezel and “save buttons	Weekly
	Hospitalization: \Were you hospitalized in the past month?” A hospitalization will trigger collection of medical records for discharge.	Graphical watch face with rotating bezel and “save” buttons	Monthly
	Restricted activity: A_rmative responses to having an IHE will prompt the following questions: 1) “Did [said IHE] result in an inability to leave home for at least one week?”, 2) “Did you cut down on your usual activities because of [said IHE]?”.	Graphical watch face with rotating bezel and “save” buttons	Prompted by affrmative IHE response

^*Note: All values are collected during customized waking hours. During initialization, participants are queried about normal daily routines that are used to program start/stop data collection time points.

^**Random sampling is an excellent method for estimating average participant experience, because the resulting estimate is unbiased, namely, there is no tendency for the average to either overestimate or underestimate the mean of all participant reports [40].

4. ROAMM Architecture

The ROAMM architecture⁶ is an infrastructure for sensor and user-reported data collection, transmission, visualization, and analysis. The infrastructure relies on a smartwatch application for data collection and preprocessing, which is developed for Samsung Gear S2 and S3. The collected data are transmitted to a remote server that maintains them in a database. The server provides means to remotely modify watch application’s configurations such that the framework can be adjusted to any study’s requirements. The ROAMM framework supports the capability of expanding server and application functions to have multiple operating smartwatches in the field. The framework was built with an extensible Application Program Interface (API) that would allow for any mobile device with networking capabilities to communicate with the server. This would allow for newer smartwatches to be integrated into the ROAMM infrastructure effortlessly. We also developed a supplementary web portal frontend to allow for remote interaction with active smartwatches and additional adminstrative functions, such as registering research participants and assigning watches to them. Figure 2 shows an overview of the ROAMM framework. In the following sections, we discuss the smartwatch application (under sections 4.1.1–4.1.4) and the server program (under sections 4.2.1–4.2.3) in further detail.

Figure 2:

The two main components of the ROAMM framework are shown. Smartwatch application collects sensor monitor data, as well as patient-reported (user-reported) outcomes. It processes the collected data into interpretable variables and transmits them to the remote server. The server provides means to register participants, assign watches to them, and configure the application parameters for data collection. It stores the received data into a central database and is equipped with big data framework for enhanced data retrieval, visualization, and analysis.

4.1. Watch Application

The smartwatch application has the following unique features and advan tages:

1. The application collects sensor data (e.g., accelerometer, gyroscope, location, and heart rate) at specified and customizable frequencies and sends them to the server. The data upload is performed using HTTPS communication protocol to ensure the security of data transmission. Furthermore, the address to which data are transmitted are only revealed to registered smartwatches (see 4.2.2).

2. Data are uploaded to the server over a WiFi or 4G network connection. The use of WiFi connection or cellular data for data transmission allows for having the collected data available on the server in real-time and in a minimally obtrusive manner.

3. Beside sending its collected data, the application requests the server to receive configuration parameters and adjusts its sensors utilization accordingly. The server can customize the watch application’s function through the configuration parameters, which include the list of sensors for data collection, their sampling rates, the definition of feature vectors (i.e., calculated variables from sensor data) and PROs.

4. The application is flexible enough to accommodate different types of studies with a variety of target variables and outcomes. Variables can be calculated from the raw data instantly and on the watch. By combining the data collection and variable construction steps, the application reduces the data cleaning time, and thus, expedites the analysis. Furthermore, transmitting variables instead of raw data results in a significant decrease in the size of the data sent to the remote server; hence lowers the transmission costs (e.g., data plan).

5. Smartwatches’ computational power and available sensors are sufficient for detecting non-wear time instantly and accurately. Non-wear times are defined as the times when the device is not worn on the wrist, such as during showering and times when the device needs to be charged. Identifying non-wear periods helps to improve the analysis. In addition, real-time non-wear time detection is used to achieve a power-efficient data collection. The informed decision to collect data only during wear times extends the battery life and make the framework suitable for longer periods of activity monitoring [41].

6. The watch allows for the development of interactive interfaces, such as prompting the user to report symptoms or asking them to charge the watch.

The application is developed and tested for Samsung Gear S2 and S3 smart-watches, although the concept that we discuss applies to similar devices, such as Apple watch⁷. The operating system of these smartwatches is Tizen, which provides APIs for managing and interacting with sensors and system-level procedures. Tizen supports web-based applications written in Javascript and HTML5, which are executed in a webkit-based browser environment operated by Google V8 Javascript engine. Tizen Javascript APIs are the means for all interactions with the watch hardware, such as sensors and memory. There are also HTTP tools in the webkit environment for the network connectivity.

4.1.1. User interface.

The smartwatch application can serve as a reporting tool to obtain participants’ reports of their health. There are two separate ways for users to provide their inputs: 1) for variables where order can be applied (e.g., pain and fatigue levels), rates on a Likert scale such as a 0–10 range as depicted in Figure 3a is used; 2) for categorical variables (e.g., activity types) a single choice user interface (Figure 3b) is displayed. Users can change the numeric values or the categorical choices by rotating the bezel of the watch, and in order to move to the next health status question, they should press the top button (Figure 3c). The application was designed in a modular way to be able to add/remove questions as desired. The application receives a remote configuration with the initial time and intervals during which the user ought to be prompted. Researchers can adjust the configurations remotely, which frees the user from bringing the watch to the research establishment for software updates.

Figure 3:

Smartwatch application user interface. Users are prompted to provide their pain level (a), activity type (b) and more. The rotating bezel is used to select the response to the requested inquiry. A single tap on the screen stores the response locally and on the watch. By selecting “B”, participants can go back to the previous screen (question) to modify their response before finalizing, which is done by the “Save” button.

4.1.2. Data Monitoring.

The ROAMM application collects patient-reported and sensor monitor data. As explained previously, patient-reported data are obtained by prompting the user questions about pre-defined outcomes, such as pain and fatigue. The Data Monitoring module interfaces with the Human Activity Monitor libraries that Tizen provides to collect sensor monitor data. Since accelerometers are widely used for activity monitoring and assessment, a validation of smartwatch’s accelerometer data is warranted and has been the subject of recent works [42]. Similarly, we compared the collected accelerometer data from the Samsung smartwatch with ActiGraph GT9X link to validate its measurements. Due to time drifts and processing shortcuts in smartwatches’ accelerometer data and the fact that time points at which data are sampled are varying [42], we use normalized root mean squared values for each second of data from each axis. Figure 4 shows that the smartwatch’s accelerometers provide similar information as ActiGraph GT9X (correlation > 0.97) for frequencies linked to human movement, i.e., 0.5 to 2.5 Hz. Thus, we consider the smartwatch a valid and accurate replacement of specialized devices for monitoring physical activities.

Figure 4:

Normalized root mean squared (NRMS) values of the acceleration data over one second for each axis. The experiment was conducted for five different frequencies in range of 0.5 Hz to 2.5 Hz, 3 minute each and on a shaker table. Blue line shows data for Actigraph GT9X and red line represents data collected from a Samsung Gear S.

4.1.3. Data Cleaning and Aggregation.

Specialized devices, such as Actigraph accelerometers, collect raw sensor data and researchers can download them after collection. The next step after raw data collection is the data cleaning, where researchers calculate variables according to the the aim of the study. The variables are usually calculated for longer epochs; e.g., as 1-minute activity counts [43] or time- and frequency-domain variables for every 5 to 60 seconds [44, 45]. Maintaining raw data might not be possible due to the large amount of data collected by the sensors.

We tested the size of the collected data by all sensors of Gear S and the results are presented in Figure 5. Figure 5a shows the size of the data collected at 1 Hz sampling rate by each individual sensor for 30 minutes, 1 hour, and 2 hours. At such a low sampling rate, all of the sensors could produce approximately ten megabytes of data after only two hours. Furthermore, we tested each sensor’s data size for different sampling rates (Figure 5b). For 10 Hz sampling rate, which has been shown to be the minimum sampling rate required for physical activity assessment [46], and after two hours accelerometer alone produced approximately 20 megabytes of data. Therefore, preserving raw sensor data for extended periods of time might not be practical for wearables with limited memory (RAM: 1 GB; internal memory: 16 GB). Also, transmitting a large amount of data to a remote server over long periods of time requires purchasing more data plans and thus, increases the cost of research studies.

Figure 5:

Size of generated data by sensor monitors. (a) shows the size of data generated by sensors collecting at 1 Hz after 30 minutes, 1 hour, and 2 hours. (b) depicts the size of collected data by each sensor after two hours anf for different sampling rates.

The smartwatch application performs data cleaning in real time and on the watch. Besides the aforementioned merits, this step has the additional benefit of reducing the size of the data that is transmitted to the server. The processing logic for the current version of this module is implemented in JavaScript which can adapt to any analysis suite of functions that take raw sensor values as input and outputs the required variables. The default implementation calculates time and frequency-domain variables from tri-axial accelerometer data, which have been previously shown to be effective for physical activity assessment [47]. Briefly, the application constructs the following accelerometer variables for every 15-second epoch of collected data at 10 Hz:

1. Time-domain variables: these variables include the mean and standard deviation of vector magnitude (V M). Vector magnitude is calculated as: $\sqrt{x^2+y^2+z^2}$, where x, y, and z represent accelerometer’s axes.

2. Orientation variables: the mean and standard deviation of existing angle between the vector magnitude and vertical axis (x). The angle is calculated as: $\frac{180}{\pi }\times si{n}^{-1}\left(\frac{x}{VM}\right)$.

3. Frequency-domain variables: these variables include the dominant frequency of vector magnitude and its fraction of power, as well as the fraction of power within frequencies related to human movement, i.e., from 0.6 to 2.5 Hz.

The data cleaning step and variable calculations reduces the size of the stored data on the watch by an approximately 150 times. This process has little impact on battery consumption (<1% for two hours) and thus, further justify the existence of this module.

4.1.4. Data Storage and Transmission.

The data storage module is programmed to move the collected data from temporary session storage to a permanent data storage on device’s memory, periodically. We used IndexedDB JavaScript library to implement the permanent data storage. Since the data storage module runs in parallel with the data monitoring module, it asynchronously stores the sensor readings to the permanent data storage without having to stall running processes on memory accesses. The watch does not require to be online for data collection, cleaning, and storing. If not connected, the app runs on default configuration and can store data locally for more than 30 days. When connectivity is established, data transmission module executes batch uploads of the stored data from the permanent data storage via HTTPS requests to the remote server. Data transmission is initiated automatically and when the device is charging or manually and by the user. The uploading subroutine sends fixed-size chunks of the collected data iteratively and stops when all the data are successfully received by the server or when it fails to deliver a chunk for a specified number of attempts. Depending on the configuration, once data are successfully received by the server, they can be deleted from the watch or marked as sent.

As mentioned earlier, one of the biggest challenges in using smartwatches is the limited battery life. Figure 6 shows the battery life consumption by each smartwatch sensor for different durations and sampling rates. Higher sampling rates required only slightly more power; however, it can be seen that even the least battery-consuming sensors (e.g., accelerometer) deplete the battery after a short period of time. It is worth noting that, for this experiment, sensor APIs were called 30 times per second to allocate resources to sensors fully and to prevent Tizen OS from going to “sleep” mode. Furthermore, raw sensor data were stored which uses more memory space. Therefore, Figure 6 depicts the peak of power usage for each sensor. The ROAMM smartwatch application collects sensor readings much less frequently and uses less memory space, and thus, allows for collecting sensor and user-reported data for more than 12 hours.

Figure 6:

Remaining batteries for each sensor. (a) shows the battery life percentage used by sensors collecting at 1 Hz after 30 minutes, 1 hour, and 2 hours. (b) depicts the battery life consumption for different sampling rates after 2 hours.

4.2. Server

The features and advantages of server software are listed, as follows:

1. The server provides a platform to register participants, personalize the application based on their preferences, and configure data collection settings according to study requirements. Personalization includes changing the color theme of the application and modifying the times of day participants prefer to be prompted. Data collection configuration includes identifying active sensors, specifying their sampling rates, and defining the parameters used to aggregate the raw data into study-required variables. All of the configuration steps are done remotely and without the requirement that the watches be collected from and returned to the participants.

2. There is a variety of modules and functionalities embedded in the server, which are accessible via defined roles. New roles with the desired access levels (e.g., Researcher I, Researcher II, Administrator) can be defined and assigned to users to support the minimum required privileges. For better control, the administrative interface for user management and security is housed on the server.

3. The server is capable of interacting with multiple watches and receiving data from them simultaneously. The received data are stored in a centralized fault-tolerant database. Data are encoded and transmitted to the server over HTTPS communications. The server decodes and maintains the collected data securely by granting the access to the database only through its web portal. The web portal itself retrieves the data through a database view with the least privileges.

4. A Map-Reduce framework (Apache Spark [48]), as well as predefined scripts that leverage machine learning methods scaled for big data (SparkML), are included in the server software to be able to retrieve and analyze large amounts of data in real-time. This provides the potential ability to leverage a large number of bigdata toolkits and software that are available today for processing data. These tools are also provided by commercially available clouds, which allows for seamlessly moving the server to a cloud platform (e.g., Amazon Web Services, Google Cloud Platform, or Microsoft Azure) for further scalability purposes.

5. The server provides a web portal that displays information from all actively deployed watches and the collected data for each separate device. It presents summary statistics of activities, the current status of the watches, and detailed visualizations for the activity data. The data can be accessed through the web portal for data exploration and analysis.

The server provides three major functionalities: 1) device administration, which provides the means to register the watches in the framework, assign them to participants, and configure their applications for the customized data collection; 2) data storage, which is responsible for receiving the collected data from the watches, store them in a database, and retrieve them e ciently; and 3) user interface and visualization, which provides graphs and quick summaries of the collected data. We describe each module in detail in the following.

4.2.1. Device administration.

The main functionality of the Device Administration module is to communicate with the smartwatch applications that are active in the field. This is a bidirectional communication where the server sends configuration parameters (e.g., sampling rate, upload size, variables to be calculated, etc.) to smartwatches and receives the sensor and user-reported outcomes from them. The server provides convenient means for researchers to adjust data collection parameters, which are delivered to the smartwatches by the Device Manager module. The smart-watches transmit their collected data to the server, which is received by the Application Manager. The Application Manager can be conveniently modified to accept a new set of variables (e.g., feature vectors) and pass the expected data to the database. The customizability that the Device Administration module provides adds the necessary flexibility to the server so it can adjust to any study requirements with minimum modifications.

The administrative panel on the server’s web portal (Figure 7) has three main parts. Researchers can add a new watch to the system, set the initial configuration parameters of its application, and assign it to a participant for a specific period using the register view (Figure 7a). Researchers can also check the status of all registered smartwatches, their last reported battery levels, and a brief statistic about data they collected, using the status view (Figure 7b). The collected data by smartwatches are accessible and can be retrieved using the download view. Figure 7c shows a snapshot of how a researcher can download data for an individual and a specified time frame.

Figure 7:

The ROAMM server administrative interface. The administrator panel allows researchers to (a) register smartwatches to participate, (b) monitor active watches in the field, and (c) retrieve data transferred by the smartwatches from the database.

4.2.2. Data storage.

The collected data are validated and processed in the previous steps (by the smartwatch application and the Application Manager) before reaching the Data Storage module. This module performs the final checks and modifications and stores data in a high-performance database. The data inserted into the database include an entry for each data item transferred from the watch as well as metadata specifying a timestamp of when the transfer was processed, as well as the size of the received data. Additionally, we maintain anonymized information about which participants were wearing a particular watch during a certain interval.

First, the address to receive data is defined dynamically on the server. Smartwatch applications receive this address upon their startups and once their credentials are verified. This address can be conveniently modified to ensure its anonymity and omitting the possibility of receiving unwanted data from un-trusted parties. Second, the received data are checked to confirm they have been received from a registered participant and smartwatch. Lastly, direct access to the database is blocked by a firewall. The remote access to stored data and information is provided through dedicated web interfaces and database views with the least privileges. All other accesses to data are granted only to the database administrator.

This module also hosts a big data framework (i.e., Apache Spark) to be able to retrieve and explore large amounts of data in real-time. Furthermore, we implement a data model corresponding to the smartwatch application wrist accelerometer model (i.e., a set of variables) to be able to obtain the desired data for any study using the framework. The convenience of adding data models allows multiple research studies with a variety of designs and goals to leverage the framework simultaneously.

4.2.3. Web-based user interface and visualization.

ROAMM framework has a web-based user interface that presents summary statistics of the collected data (e.g., duration of data collection, size of the accumulated data) and the status of the watches (e.g., collecting/stopped, battery level). The web portal is also capable of visualizing the calculated variables from the data, which is an effective way to reduce errors during data collection phase. For example, participants might forget to wear the watch at random times during the requested periods, or the application stops collecting data for an erroneous reason. By constantly monitoring the collected data, researchers can remind participants to wear the smartwatch or contact them to troubleshoot the problems. They can also perform several data validation steps as soon as the data becomes available on the server. Furthermore, due to the flexible and extensible nature of the framework, it is possible to add additional presentation and analysis logic layers on top of the collected data to obtain a more informative summarization.

Figure 8 shows the data flow of the ROAMM framework. Sensor monitor and patient-reported data are collected by the application and transmitted to the server (Figure 8a). Figure 8b shows a simpler visualization of the collected data. Researchers can filter the data by date, time, and participant to obtain time-series graphs of the calculated variables. We use the Google Maps JavaScript API to display an interactive map that provides a visual presentation of the collected location data. Depending on the study goals, the collected data could be further processed to generate more variables for the analysis (Figure 8c).

Figure 8:

ROAMM framework data flow. Sensor monitor and patient-reported data are collected and transmitted to the server by the ROAMM application. (a) shows the ROAMM smartwatch application user interface. (b) shows the visualization of the data on the server. The administrative web portal facilitates filtering and retrieving the collected data. Data are transferred to the server, where they are stored in a high-performance fault-tolerant database. Data can be viewed or run through any custom analysis pipeline for presentation and/or analysis. (c) depicts how data are processed into variables for analysis. Depending on the study, a variety of variables can be constructed from the data, either on the smartwatch or later on the server.

5. Experiments with Participants

As a “proof of concept”, ROAMM was used to collect free-living data from 5 participants who were asked to wear the smartwatch on their left wrist for approximately 2 weeks. From 8 am to 8 pm participants were prompted at four random times with a minimum three-hour gap between two consecutive prompts. Participants were requested to charge the watches every night. Table 2 contains summarized results for each participant over approximately two weeks of wear. Briefly, a total of 777 hours of sensor and patient-reported outcome data were collected. Overall, 50.60 MB of data were collected with most belonging to the sensors (34.90 MB − accelerometer: 24.4 MB; heart rate: 3.5 MB; GPS: 7 MB) and 17% (8.7 MB) for EMA data.

Table 2:

Summary of collected data from participants.

Characteristic	Quantity
Participants	5
Duration	776.61 hours
Sampling Rate	10 Hz
Feature Window Length	15 seconds
Data Size	50.60 MB
Variable Vector Size	18
Sensor Data
Accelerometer (refer to 4.1.3)	7
Heart Rate	1
Location (lattidude & longitude)	2
Patient-Reported Outcome	5
(pain, fatigue, mood, sleep & activity)
Identifiers	3
(ID, tvmeatamp & battery)

There were very few instances where data being collected was not immediately available on the server for visualization. Table 3 contains accelerometer features that were extracted from the raw data. These features have been used in other studies to quantify physical activity type, intensity and energy expenditure [49]. Participants responded 550 times to questions related to their pain, fatigue, and mood levels via the same application. The user-reported data were summarized and related indices required for EMA studies were calculated.

Table 3:

Summary of accelerometer-driven variables for participants.

	#Days	MVM	SDVM	MANGLE	SDANGLE	DF	FPDF	P625
Participant 1	3	9.90 (0.18)	0.29 (0.18)	−4.26 (18.19)	2.41 (6.96)	2.20 (1.53)	0.041 (0.016)	0.390 (0.052)
Participant 2	15	9.77 (0.13)	0.33 (0.57)	2.60 (16.80)	3.87 (8.31)	2.51 (l.55)	0.035 (0.007)	0.395 (0.058)
Participant 3	7	9.83 (0.18)	0.38 (0.65)	−3.89 (22.83)	4.53 (9.39)	2.23 (1.44)	0.035 (0.010)	0.407 (0.062)
Participant 4	15	9.67 (0.96)	0.29 (0.58)	−1.86 (12.16)	2.08 (5.08)	2.23 (1.46)	0.035 (0.011)	0.395 (0.060)
Participant 5	7	9.81 (0.26)	0.46 (0.69)	1.03 (23.05)	5.44 (9.70)	2.02 (1.47)	0.037 (0.013)	0.408 (0.061)
Total	47	976 (0.59)	0.34 (0.61)	−0.39 (17.72)	3.51 (7.82)	2.29 (1.50)	0.036 (0.010)	0.398 (0.060)

Values are reported as Mean (SD)

MVM, mean vector magnitude; SDVM, standard deviation of vector magnitude; MANGLE, mean angle between forearm and the horizontal line; SDANGLE: standard deviation of angle between forearm and the horizontal line; DF: dominant frequency; FPDF: fraction of power covered by dominant frequency; P625: fraction of power covered by frequencies in [0.6, 2.5] Hz.

Table 3 shows details of time, frequency, and orientation-domain features calculated from accelerometer sensor data for each participant. The mean of vector magnitude (MVM) is close to the value of gravity (9.8 m/s²) for all participants and the average of dominant frequencies are > 2 Hz, which indicate that the participants had been engaged in sedentary activity (e.g., sitting) for most of the data collection period.

Figure 9 shows the accelerometer-driven features, i.e., MVM and the average of existing angle of between forearm and the horizontal line (MANGLE) for 350 data points (87.5 minutes) for all of the participants. Every data point is calculated from 15 seconds of accelerometer data. The vector magnitude values are presented as time-series waveforms to show the variability of acceleration in time. The shaded areas for each point show one standard deviation for that particular 15-second epoch. To show MANGLE values, we use polar plots to achieve a better understanding of hand position. For example, 90° represents the times where hand is resting by participant’s side (e.g., natural hand position when standing still). As seen in Figure 9, the angle between P1’s hand and the horizontal line and the low variability in the values of vector magnitude show that he or she has been stationary (i.e., sitting and standing) for most of the times, whereas data from P3 indicates higher activity levels.

Figure 9:

Visualization of 350 data points (87.5 minutes) of accelerometer sensor data for 5 participants. The time-series plots (left) show the average vector magnitude (MVM) over every 15 seconds. For each point, one standard deviation is also shown as the shaded area. The constant gravity force (≈ 9.8 m/s²) is displayed as blue dashed line. On the right, the existing angle between the forearm and the horizontal line (MANGLE) is displayed for the same time periods. Each bar shows the frequency of an angle (±10°). For example, for the first participant, we observe that the hand is placed on a horizontal surface (e.g., when working with a computer) or resting by participant’s side (e.g., natural position of hand when standing) for most of the times.

Table 4 presents details on pain-related indices from the EMA data, such as average pain, peak pain, pain range, and pain variability. Participants could select any value from 0 (no pain) to 10 (worst possible pain). However, the recorded pain values ranged from 0 to 6, where the most frequent selected pain level was 1 (very little pain) overall (n = 332).

Table 4:

Pain-related indices from EMA data.

	#Responses	Peak Pain	Average Pain	Pain Range	Pain Variability
Participant 1	8	3	2.07	[1, 3]	0.83
Participant 2	19	2	1.71	[0, 2]	0.62
Participant 3	50	6	2.58	[0, 6]	1.40
Participant 4	61	5	0.55	[0, 5]	1.06
Participant 5	51	3	0.94	[0, 3]	0.72
Total	189	6	1.37	[0, 6]	1.33

Similar indices were also calculated for fatigue and mood and are presented in Tables 5 and 6. The application allowed selection of fatigue levels from 0 (no fatigue) to 10 (worst possible fatigue) but participants recorded fatigue levels ranged from 0 to 6. The most frequent response for fatigue level was 0 (n = 398). Mood levels could be recorded following a different scale, where −5 represented the poorest, 0 was for neutral, and 5 showed the happiest moods. Except for participant 2, the rest never recorded an unhappy mood (negative value) and the most frequent recorded response was 4 (n = 449).

Table 5:

Fatigue-related indices from EMA data.

	#Responses	Peak Fatigue	Average Fatigue	Fatigue Range	Fatigue Variability
Participant 2	19	3	2.08	[1, 3]	0.77
Participant 3	49	3	0.95	[0, 3]	0.90
Participant 4	60	6	1.95	[0, 6]	1.75
Participant 5	51	3	1.21	[0, 3]	0.91
Total	179	51	1.42	[0, 6]	1.33

Participant 1 did not provide their fatigue data.

Table 6:

Mood-related indices from EMA data.

	#Responses	Peak Mood	Average Mood	Mood Range	Mood Variability
Participant 1	8	4	2.45	[0, 4]	1.69
Participant 2	18	4	1.65	[−3, 4]	2.08
Participant 3	48	4	2.35	[0, 4]	1.52
Participant 4	60	5	1.66	[0, 5]	1.69
Participant 5	48	5	3.42	[0, 5]	1.38
Total	182	5	2.34	[−3, 5]	1.77

6. Discussion & Conclusion

In this paper, we have provided the ROAMM infrastructure for real-time monitoring of personal health. This framework relies on continuous sensor data collection at high frequency for physical activity monitoring and assessment and patient-reported outcomes, queried at random time points throughout a day, for recording EMAs. The infrastructure consists of two main components. The first component is an application implemented for Samsung Gear S2 and Gear S3 smartwatches, which collects sensor and user-reported data, processes them into variables and transmits them to a remote server. The second component is the server, which receives the data from multiple watches and stores them in a database. The workload is distributed among the smartwatch application and server program in a way that the data become available for visualization and analysis with minimum latency.

Systems relying on smart devices for data collection and a remote server for analysis have been used in previous works, where they were shown to be effective approaches for online monitoring of vital signs [50]. Smartwatches are relatively new additions to such frameworks and have been mostly validated for specific applications. Shahmohammadi et al. showed that smartwatches were able to perform accurately for activity recognition and highlighted their advantages over smartphones [51]. Smartwatches were also used in cloud-based frameworks to monitor sensor data and measure risk of asthma continuously [52]. In the presence of the flexibility that the ROAMM framework provides, any desired study model (i.e., set of independent and target variables) can be implemented on the watch and visualized and analyzed on the server in real time.

The main challenge with using smartwatches in research studies, especially for extended periods of time, is the limited battery life. The problem becomes more appranet if all sensors concurrently collect data at high sampling rate. The ROAMM framework leverages detecting wear times to address this issue [41]. This approach is aligned with the recent developments on the smartwatches’ software since they provide proximity APIs, which makes wear-time detection more accurate at no additional computational cost.

The capability of collecting patient-reported outcomes at pre-specified time points, in addition to sensor data, makes it possible to achieve enhanced “gold standard” ascertainment with reduced bias. By time syncing data streams, the ROAMM framework provides efficiently measurement tool to study IHEs, which has been costly and challenging for decades.

The ROAMM framework meets some of the major requirements for the next generation of the Internet of Things for mHealth. ROAMM offers an interactive interface (e.g., prompting for reporting symptoms) and remote application configuration (e.g., modifying data collection rates and types of variables to be calculated), as well as server features for making it flexible for online customization. Additionally, the smartwatch accelerometer hardware provides highly correlated results with a validated, research-grade accelerometer. Given the similar costs with research-grade monitors (~$300), a Gear S3 (or other comparable smartwatches) along with the ROAMM app offers an ideal alternative to access to multi-sensor output, touch screen and physical inputs through a bezel, and broad GSM connectivity for remote device control. These are ideal characteristics for long-term, continuous data collection for capturing episodic intervening health events that is of considerable interest to health researchers.

Highlights:

• Smartwatches possess a collection of sensors required for monitoring mobility and physical activity assessment

• The proposed ROAMM framework leverages smartwatches’ sensor collection and connectivity means to achieve real-time activity recognition.

• The proposed framework collects user-reported data (e.g., pain and fatigue level) which allows for studying ecological momentary assessment, in parallel with physical activity monitoring.