Multi Agent Architecture for Automated Health Coaching

Abstract

For software applications in health coaching domains to be effective, it is vital that they address issues of privacy, modularity, scalability, individualization, data integration, transferability, coordination and flexibility. In this paper, we propose a novel generic multi-agent architecture which serves as a template for health coaching applications involving wearable sensors. Analyzer and communication modules allow different functionalities like goal formation, planning, scheduling, event detection, learning, inter-agent + human communication and long-term data collection, based on the capabilities of the underlying sensor platforms. To show the flexibility of our proposed architecture, we have successfully built two different health coaching systems with the proposed architecture: (1) a static system based on the Fitbit platform where the coaching is done at specific preset times to encourage increased physical activity, and (2) a dynamic system based on the Apple Watch platform where the smart coach adapts and learns when to intervene to encourage physical activity and reduce sedentary behavior.

1. Introduction

With ubiquitous presence and bidirectional communication ability, wearables and smartphones have the potential to extend the reach and impact of health coaching by delivering actionable feedback in real-time [17,20]. With the advancements in computer science (especially artificial intelligence), these devices not only have the capability to replace existing functionalities of human coaches but can also incorporate new functionalities. Kamphorst [10] defines an e-coach or automated coach as “…a set of computerized components that constitutes an artificial entity that can observe, reason about, learn from and predict a user’s behaviors, in context and over time, and that engages proactively in an ongoing collaborative conversation with the user in order to aid planning and promote effective goal striving through the use of persuasive techniques”. Though this definition provides an outline of what an automated coach should do, it is limited to a single user. People often form social networks, and studies [7,12] have demonstrated how social network interventions have been successfully used for a range of health behaviors, including HIV risk practices, smoking, exercise, dieting, family planning, bullying, and mental health. So, in reality automated health coaches should not only posses the functionalities described by Kamphorst but should also be able to accommodate users to form social networks and allow communication and coordination over intervention strategies to achieve behavioral changes.

The above definition of automated health coaches only provides a theoretical foundation but to build software applications that satisfy the above definition, underlying architectures should posses the following features: Privacy: Privacy of health data is of utmost importance and automated coaches should provide a secure platform. Scalability: Automated coaching should be able to accommodate vast numbers of users and should be able to scale well, as opposed to systems that only augment human coaching. Individualization: Just-in-time adaptive intervention (JITAI) design monitors changes to a person’s psychological (e.g. mood), behavioral, and physical states and contexts over time for the purpose of delivering support at the moment it is needed, and when the person is most likely to be receptive [21], requiring both individualization and customization of interventions [19]. Transferability: When adapting interventions, data is limited because probing the intervention recipient (i.e. human) too many times may annoy or discourage them from achieving their target goals. In order to address this data limitation problem, automated health coaches should be built in a way to allow transferability of selective data from other, similar individuals (while preserving privacy). Integration: Both adaptation and goal support require extensive recognition of the user’s current context (state), which in turn requires the integration of information from many sources, both local and external to the coaching system. Flexibility and Modularity: Automated health coaching has a wide variety of important applications, such as increasing physical activity, reducing sedentary behaviour, smoking cessation, sleep coaching, or other similar applications. Modular design of systems help to loosely couple system components allowing systems to adapt to different requirements. Coordination: In behavior change, especially in PA, people who worked in pairs or a group had higher success than those who did not [7]. This requires coaches to coordinate over user actions.

Considering the above features we propose a novel generic multi-agent architecture, SMART-ALEC (System to Monitor, Adapt, and Refine Target goals by Agent-based Learning in Everyday Contexts) that can serve as an effective platform for multiple kinds of health coaching applications. SMART-ALEC is a multi-agent platform which provides necessary architectural services to extract and send data in real time to multiple sources, communicate and transfer data securely among health coaches, analyze, learn, plan and schedule interventions in real time.

2. Related Work

Kitsiou et al [11] developed iCardia, a mobile health platform designed to support remote monitoring and health coaching of cardiac rehabilitation (CR) patients, through Fitbit wearable sensor devices, smartphones, and personalized SMS text-messages. Li et al [14] introduced MobiCardio, an architecture which integrates multi-sensor wearable technologies, machine-learning based sleep analytics and electronic health records information to manage cardiovascular disease (CVD) patients for clinical scenarios. Though these architectures provide effective platforms for CVD applications, they lack flexibility to serve as platforms for other health coaching problems. Blok et al [3] and Ochoa et al [18] both try to address this problem by developing flexible and loosely coupled architectures for personalized virtual coaches. Even though these architectures are built such that system structure is decoupled from its behavior they lack some of the vital features of health coaching problems such as scalability, privacy, transferability and coordination. To address the issue of scalability, Ibrahim et al [8] developed a multiagent platform to automate the process of collecting user-provided clinical outcome measures without clinician intervention though the primary aim of this work is not to serve as an automated health coach. Azim et al [1] introduced a hierarchical fog-assisted computing architecture, HiCH, for remote IoT-based patient monitoring systems featuring autonomous data management and processing at the edge. HiCH is both flexible and scalable but lacks on transferability and coordination. Current health coaching architectures posses capabilities to address only subsets of features of automated health coaching problem, and current literature lacks a holistic approach required to address the automated health coaching problem. In this paper we attempt to solve this problem by proposing a new architecture.

3. SMART-ALEC Implementation

Figure 1 represents the high level structure of the SMART-ALEC architecture from a subjective view of a single agent. SMART-ALEC riffs on some ideas found in both RETSINA [5] and DECAF [9], albeit updated for this setting with different levels of hardware and sensor access. Each agent has two major modules, analyser and communicator, running concurrently on different threads. A central agent (not shown in the figure), acts as a dashboard/presentation layer for researchers/human coaches, providing an interface to view and interact with user data.

Fig. 1.

SMART-ALEC Architecture

Planner.

SMART-ALEC divides top-level goal achievement into three levels: planning, scheduling and learning. The division of these levels is motivated by the varied needs of applications, some requiring a deterministic set of plans along with schedules others requiring non-deterministic policies to be learned. The planner is involved whenever a new goal or a goal change for the user should be accomplished. Generated plans are structured as trees, top level nodes replicating the overall goal to be achieved and bottom level as atomic actions, replicating HTN style plans [13]. Being abstract in nature, SMART-ALEC can accommodate any type of classical planning algorithm which suits the needs of the applications.

Scheduler.

The purpose of scheduler is to determine which actions can be executed now, which should be executed now, and in what order. The scheduler is tightly intertwined with the planning and executor modules, because of the dynamic nature of health coaching applications. As failures of actions are quite common in the health coaching domain, it is important that the scheduler has information about the outcome of user actions in real time which allows for the scheduler to either dynamically reschedule or invoke the planner to replan the entirety or subgoals of existing goals.

Learner.

The purpose of learner is to construct policies in domains where how to achieve goals is not explicitly given beforehand. To do this, the learner module interacts closely with the context analyser to capture user states (location, weather, time, etc) along with user feedback. User feedback is obtained by providing interventions to user under varying contexts and collecting user responses directly or indirectly through observing the user’s subsequent actions. The learner module can accommodate any learning algorithm like a reinforcement learner or decision trees depending on the needs of the application.

Context Analyzer.

Context analyzer is used for constructing user state in real time. This is done by collecting continuous user data from varied sources (wearable data, GPS, weather, etc) and consolidating it. The context analyzer is tightly coupled with the executor and learner modules serving as a means of input to both these modules.

Executor/Monitor.

Once a compiled policy is constructed either through the planning/scheduling modules or the learning module the policy is deployed to the executor. The executor module not only implements the current complied policy but also acts as a feedback provider for the learner module, by capturing user responses for a given policy. The executor in SMART-ALEC can be run on either the cloud, the user’s smartphone or the wearable itself if supported (this is one reason why this part is separated out). Allowing executors to be run on wearables (i.e. Apple Watch) allows the construction of true JITAIs, dynamically reacting to the current context in realtime. The executor can be thought of as independent agent, continuously communicating and cooperating with cloud/phone agents (with no analysis component).

Communicator.

The communicator module is responsible for transferability and coordination features of SMART-ALEC architecture. The communicator can transfer learned policies of users to other similar users. Coordination can be achieved through communicating commitments [13] or personal preferences over actions, enabling agent schedulers to synchronize activities among users.

4. Case studies

4.1. BeSmart

The effect that moderate-to-vigrous physical activity (MVPA) and improved sleep quality have on cardiovascular disease (CVD) risk is irrefutable [2]. The aim of this project, was to build BeSmart (Bio-behavioral Systems to Motivate and Reinforce Heart Health Trail) which leverages short message service (SMS) and Fitbit technology within a comprehensive cloud-based, bi-directional feedback system that provides individually tailored behavior-change messages based on the collection of objective behavioral and biometric data to increase MVPA and improve sleep metrics among midlife adults. The impact of changes in MVPA and sleep metrics on blood pressure were measured.

Before the beginning of the trail each participant consulted with a human health coach where a customized SMART PA (Physical Activity) and sleep goals were assigned. Once these goals were assigned the job of an automated coach (BeSmart) was two fold: (1) To help achieve participants reach their goals (2): Modify goals for the upcoming week, making goals harder/easier. To achieve the above goals the BeSmart system interacted with participants three times a week at preferred preset times, all interactions were done via SMS/MMS (Multimedia Message Service). At the beginning of the week (Mondays) each participant was sent a reminder about the goals they were suppose to achieve that week. At mid week, BeSmart system held a small dialogue with participants, informing about their goal progression and sending customized reinforcing messages for participants who were/weren’t on track to achieve their goals. At the end of week, BeSmart held a full dialogue with participants, informing whether they achieved their goals, modifying goals if necessary and recommending strategies to overcome barriers. The goal progression for the week was depicted graphically, pie charts for PA goals and bar charts for sleep goals.

Figure 2 shows the architecture of the BeSmart system. The context analyzer block is responsible for extracting data from Fitbit (raw PA and sleep data) and Withings (blood pressure data) and sending messages to and from agents to central agent. As the raw data was unstructured so we opted for a NoSQL mongoBD database. Once data was pushed into database, analyser was responsible to process raw minute level heart rate data into continuous bouts for PA goals and calculate sleep efficiency for sleep goals. The other functionality of analyzer (scheduler block) was to generate dialogues with participants. Finite State Machines (FSM) were to used to conduct dialogues, each state represented dialogue turn and responses from participants were mapped to actions in FSM. Participants answers in dialogues were constrained by prefixed options, this enabled BeSmart to map FSMs to dialogues. The Analyzer and communicator blocks were tightly coupled to achieve smooth generation of dialogues, analyzer maintaining the state of the dialogue and communicator sending and receiving messages from participants via central agent.

Fig. 2.

BeSmart architecture

A central agent was used for two purposes, (1) Maintain a consolidated view of participant’s data (2) Send and receive messages via twillio to participants. BeSmart architecture is written in python and flask was used as web framework. In our beta testing, architecture was evaluated on two parameters (1) percentage of messages successfully sent and received and (2) Time delay of messages when scaled. 98% of the messages were successfully sent and received, 2% were lost when central agent interacted with twillio server, the cause of loss was unknown as message loss was erratic, to mitigate this, central agent resent messages whenever it detected a message loss. To test time delay, we scripted up to 250 dummy agents to send and receive messages, no significant delay was detected.

4.2. Walking with JITAIs

Just-in-time adaptive intervention (JITAI) use mobile and wearable sensor technology to monitor health, behaviors and environmental factors, designed to respond to predefined events in real-time to facilitate behavior change [16,15,4,6]. Compared to our previous approach with BeSmart that allows for interventions at fixed times with participants, this project allows for real-time interventions. This is because the Fitbit platform used in BeSmart is not capable of providing sensor data in real time. In Walking with JITAIs we built an innovative, remote-delivered and personally tailored JITAI system using Apple watches and IPhones to improve MVPA and reduce sedentary behavior, both of which are key CVD risk factors.

In this pilot study, the objective of the system is to learn whether/when to intervene to increase physical activity and reduce sedentary behavior. To do this, real-time contextual information of participants is collected. Time, location, weather, events from calendar, heart rate, step activity are combined to form the current context of participant. Participants were under 2-week initial observation period, during this time participant’s data regarding to physical activity, sedentary times under different contexts were collected to feed decision trees to learn whether/when to intervene. To effectively use location, participants were asked to tag in their most frequented locations (home, work, gym and other) before the 2-week observation trial. After the observation period, participants received notification messages for 2-weeks on their Apple watches when the system detects periods of walking and prolonged sitting. Walking messages prompted subjects to walk faster or to increase the duration of their current walk. Sedentary messages prompted subjects to replace a proportion of their sitting time with light activities such as standing or taking a short walking break. Evaluation included not only nudge success rate under all contexts, but also the duration and effort put forth (e.g. how much increase in speed or duration).

Figure 3 shows the architecture of Walking with JITAIs. Walking with JITAIs has two components to it, a server and client side. The client side of the application runs on participants’ Apple watches and iPhones, watches are responsible to deliver JITAI in the form of notifications. Watches run a customized decision trees to decide if and when to intervene. Because of the energy constrains of Apple watch, training of decision trees is done on the server side, watches collect data points (context along with response of participants) and sends them to server. Context analyzer module is responsible for receiving data from iPhones and sending trained decision trees back to watches via phones. Sedentary data points consist of the following: time, location, activity, notification text (“It looks like you have been sitting for a while”), participant’s response to notification (“Yes, I am”, “No, I am not”, no response), participant’s follow up (did they stand up or not). Watch collects two types of data points relating to physical activity. 1. Whenever a change in activity is detected, data point consists of the following: start time, activity, location, end time. 2. Data point collection relating to PA interventions are triggered whenever the watch detects walking beyond a preset threshold (example: 250m from starting location), having a preset threshold helps not to provide interventions unnecessarily. Data collection regarding PA interventions starts with an initial data point: time, location, activity, followed by a more comprehensive data point when participant reaches their PA threshold consisting of: start time, start location, distance from start location, current location, weather, current activity, current time. This data point helps decision trees to decide whether to provide an intervention, if the watch provides an intervention then for every 30 seconds a new data point is collected consisting of the following: start time, start location, distance from start location, current location, weather, current activity, current speed, current time, heart rate, notification text (“It looks like you are walking, you should walk faster” or “It looks like you are walking, you should walk longer”), response to notification (“Yes”, “No”, no response).

Fig. 3.

Walking with JITAIs

The learner module is responsible for running decision trees which customize interventions for participants, learning if and when to intervene. “If” part of the intervention relates to the outcome of decision tree (Yes or No) and when part relates to the threshold (250m or 150m customized to participants) in walking. Response of participants to interventions along with follow up activities help to form a supervised training data set. Time, location, current activity, weather, threshold, response to previous intervention act as attributes to training data. Decision trees are updated with every new training data point, agent sends an updated model to the watch via phone every night. A python flask web framework is built on the central agent allowing web page based dashboard to navigate and analyze participants data. Evaluation of architecture was done on 2 parameters 1. Data loss 2. Scalability. No data loss was detected whatsoever in our beta testing when scaled up to 250 agents.

5. Conclusion and Future Work

In this paper, we proposed a novel generic multi-agent architecture for automated health coaching. To show flexibility of our system we have successfully built two health coaching systems, a static goal-oriented system based on Fitbit which helps participants achieve goal oriented behavior change by providing motivational dialogue at preset times, and a dynamic real-time intervention system based on Apple Watch which adapts and learns to provide true JITAI messages. In the future, we plan to combine goal setting/modification, motivational dialogue along with JITAI with added functionalities of transfer learning and natural language processing to provide a Just-In-Time adaptive automated coach. We also plan to build coaches which will have the ability to schedule action plans and coping plans according to user preferences. We plan to build these systems using the architecture described here.