Towards Visualization of Time–Series Ecological Momentary Assessment (EMA) Data on Standalone Voice–First Virtual Assistants

Abstract

Population aging is an increasingly important consideration for health care in the 21th century, and continuing to have access and interact with digital health information is a key challenge for aging populations. Voice-based Intelligent Virtual Assistants (IVAs) are promising to improve the Quality of Life (QoL) of older adults, and coupled with Ecological Momentary Assessments (EMA) they can be effective to collect important health information from older adults, especially when it comes to repeated time-based events. However, this same EMA data is hard to access for the older adult: although the newest IVAs are equipped with a display, the effectiveness of visualizing time–series based EMA data on standalone IVAs has not been explored. To investigate the potential opportunities for visualizing time–series based EMA data on standalone IVAs, we designed a prototype system, where older adults are able to query and examine the time–series EMA data on Amazon Echo Show — a widely used commercially available standalone screen–based IVA. We conducted a preliminary semi–structured interview with a geriatrician and an older adult, and identified three findings that should be carefully considered when designing such visualizations.

1. INTRODUCTION

Population aging is a global health consideration in the 21th century [8], especially when it comes to keeping track of older adults’ daily health data. To more effectively support older adults and their clinicians to collect daily health check-ins, studies have shown the effectiveness of using Ecological Momentary Assessments (EMA), a well–known method to more conveniently assess specific recurring behaviors and track specific health states. An effective EMA strategy could allow clinicians to constantly monitor older adults and ensure their physical and mental well-being, hopefully leading to an increase in their Quality of Life (QoL) [9, 18].

However, such strategies often necessitates tremendous amounts of time and effort; and further issues, such as communication challenges, could arise [20]. While a wide variety of technologies have been used to keep track of the older adults’ conditions [16, 19], many older adults are still facing obstacles while using smart devices, most notably those with complicated Graphical User Interfaces (GUI) [13, 14].

Voice based conversational user interfaces offers an promising alternative interaction channel for older adults to interact with their digital health information. Such modality allows older adults to naturally and easily use voice based conversation–like commands to delegate tasks to, or query information from their healthcare providers. While Trajkova et al. [21] demonstrates the limitations of today’s voice assistants among aging users, others (e.g., [3–5, 15, 17]) show the feasibility and potential usefulness of using standalone voice–based Intelligent Virtual Assistants (IVAs) for conducting EMA among older adults patients.

Despite the potential of deploying EMAs through IVAs, due to the nature of ambiguity of conversational voice user interfaces, Chen et al. [5, 6] pointed out the necessity of also integrating simple visual components for facilitating better interaction with the EMAs deployed on the device. The same group of researchers [6] also investigated how to visualize simple interaction elements, such as the texts of EMA questions and input buttons on Amazon Echo Show [10], a widely used commercially available standalone IVAs. However, while data collection through EMA is important, previous work did not yet explore how to designs and visualize the data collected through EMAs on the IVA device itself, especially when it comes to display time–series visualizations.

In this paper we present the first preliminary exploratory effort to design time–series data visualizations on screen based voice–first IVAs. We chose Amazon Echo Show [10] as the standalone voice–based IVA device with a built–in touchscreen due to its dominant market share [1], however, many of our findings could be transferred to other screen–based voice–first IVAs. We prototyped a system that enables older adults to examine their past EMA data on the built–in touchscreen of Echo Show. By observing interaction with the IVA visualization, and especially older adults’ past data and corresponding trends, we found that older adults might be able to make in-situ decisions to address possible unhealthy lifestyle.

We conducted a remote co–design workshop with two representative stakeholders – an older adult and a geriatrician. After thematic analysis of the recordings, we present findings and potential future development.

2. PRELIMINARY DESIGN

Our preliminary designs consist of two example scenarios demonstrating how time–series EMA data can be visualized and are beneficial to older adults (see Figs. 2a and b). In both cases, we assume that the EMA data has already been collected and stored in a secure Electronic Health Record (EHR). We considered scenarios of one–measure visualization, where we only visualize one measure (e.g., the hour of sleep as shown in Fig. 2a) over the time domain, and two–measure visualization, that places two one–measure graphs with the same dimension vertically, which might indicate possible correlations between two dependent variables. Visualizations with more than two measures are out of our scope due to the consideration of the screen size and participants’ cognitive load.

Figure 2:

Two sample demonstrations on visualizing time-series data on Amazon Echo Show with voice input (a–b) and a semi-structured interview exploring their utility (c). Faces are pixelized to protect participants’ personally identifable information.

Scenario 1 – One–Measure Visualization:

Fig. 2a demonstrates a one–measure visualization for hours of sleep, one important time-series EMA data used in geriatrics to help tracking older adults’ QoL [7]. A voice skill could be initiated by a simple query: “Hey Alexa, what’s my average hours of sleep everyday in the week?” The mean hours of sleep by days in a week will then be computed, yielding a line plot shown on the IVA’s screen. The trend of the graph could help older adults recall the common activities on particular days in a week that possibly led to an irregularity of sleeping hours.

Scenario 2 – Two–Measure Visualization:

Older adults should also be able to visualize two–measure plots with the same time span to examine any potential correlations. As an example, Fig. 2b demonstrates how older adults could query about their sleep hours and alcohol consumption in the past 10 days. This plot can visually guide the user to find the correlation between the daily alcohol consumption and sleep, so that the user can properly in–situ adjust to a healthier lifestyle.

Prototype:

We prototyped the system based on Amazon Alexa and Amazon Web Services (AWS). The overall system architecture is shown in Fig. 3, and consists of three major components: (1) a front–end converting users’ utterances to queries, (2) a middle–tier that handles the queries and requests the graphs, and (3) a back–end to generate the graphs. In the front–end, speech recognition technology in the IVAs, such as Alexa Voice Service (AVS) in an Echo Show, are used to recognize the raw audio input. An AWS Lambda Function is then triggered and passes through the transcribed text to the back-end server via RESTful API which will then generate a time-series graph. Through Amazon Presentation Language (APL) [12], the plot is then rendered on the touchscreen as a background image. Through this process, older adults are able to view their time–series healthcare information as a graph on their standalone devices.

Figure 3:

When older adults request a graph from voice assistant Alexa. AWS Lambda handles this request by asking EC2 to generate a graph and send it back to Echo show, paired with Alexa.

3. DESIGN AND USER EVALUATION

To understand how data visualization can help the aging population, we conducted a remote 30–minute co–design workshop through Zoom [22], and interviewed an older adult (P1) from the retirement community “Vi” in La Jolla, California and a geriatric provider (P2) from the University of California San Diego Health System. Our study has been approved by the Institutional Review Board (IRB). We showed the interviewees our two initial visualization prototypes (see Fig. 2a and b) and solicited feedback, encouraging them to propose alternatives. Specifically, the participants were asked to express their opinions regarding (a) the potential functionality and effects of the proposed EMA visualizations, as well as (b) how useful these visualizations would be to their daily routine. Fig. 2c shows a screenshot of our Zoom session with the two stakeholders.

4. PRELIMINARY FINDINGS

We recorded the complete Zoom session and analyzed the discussion using analysis [11]. In this section we summarize our findings in terms of three aspects.

(1). Selections of Measures to be Visualized Should be Carefully Considered.

Participants agreed that mean values, as visualized in Fig. 2a, could provide clear insights while retrospecting the older adults’ activities over the previous weeks. However, participants also expressed a preference for seeing additional measures to help evaluating the spread of the measured data. For example, P2 mentioned that if “error bars” existed in Fig. 2a, “the preciseness of the number of sleeping hours” could be shown. This revision would help visualize the consistency of sleeping time among aging populations.

(2). The Span of Horizontal and Vertical Axis Should be Carefully Designed.

Participants suggested to have more flexibility in terms of the range for the vertical axis. As shown in Fig. 2a, the range of the vertical axis is a four–hour difference. Changing the range of the vertical axis to be more flexible would allow users to display a more accurate variability of the number of sleeping hours. P2 mentioned that having a set vertical span has the limitation of being “bounded by the responses they [the users] have” and emphasized that having a flexible vertical range could be implemented “so people don’t potentially freak out”.

Participants also emphasized the importance of carefully designing the time span (i.e., visualized as horizontal axis). In Figs. 2a and b, the graphs show data within only a short time span (e.g., the time span of Fig. 2a is a week and the time span of Fig. 2b is 10 days). However, healthcare providers could request data from a longer time span, such as from the past several months or years. P2 in fact questioned the effectiveness of graphing only tens days of data in Fig. 2b and suggested to “make the axis longer so you[the visualization] show more” health data. Future designs should adapt the time span according to the user’s demand.

(3). The Types of Data to be Visualized Should be Carefully Chosen.

Currently, the prototypes only display quantitative data, but there are numerous essential qualitative data points and types that could be included in the graphs, as suggested by the participants. The quality of sleep, an example of qualitative data, can provide a clearer picture on how alcohol is affecting the user’s sleep. This data type, however, is difficult to be implemented in the visualizations. One method is to “build it in as a color … to overlay the number of [sleeping] hours with quality”, as suggested by P2. The addition of “setting goals” perhaps overlaid on top of the quantitative data would also make the visualizations more personal to the user, since meeting their goal encourages the user to maintain a better lifestyle. P1 mentioned that “you might set your goal fairly low… so if you’re consistently meeting your lower goal, then that gives you a chance to consider whether you want to increase the goal or not”. In the graphs showing the correlation between the amount of consumed alcohol and the sleeping time, the time difference between when the user last consumed alcohol and the time they fell asleep the same day is also a factor that affects sleep. Therefore, this variable could also be considered to add into the visualizations.

5. CONCLUSION AND FUTURE WORK

To help older adults visualize their time-series EMA data through voice input, we designed and implemented a prototype that can generate one–measure and two–measure graphs on standalone voice–first virtual assistants. We explored opportunities of the system and summarized needs on 3 major components of this visualization, which are measures, types, and time span of data. Although some of these opportunities can be easily implemented, others like providing a better picture of the user’s conditions and goals, can be difficult to integrate into the visualizations and interpreted correctly by the user and the healthcare providers.

This work is only scratching the surface in terms of how to design graphical visualization of EMAs on IVAs, and outline a number of challenges that still need to be resolved. In future work, we plan to conduct user studies with older participants to fully explore opportunities and challenges faced by this a system like the one we have prototyped. Meanwhile, integrating some straight-forward suggested refinements as part of our existing prototype will already make visualizations more adaptive to context and representation of qualitative data.

Figure 1:

With touchscreen based standalone voice–first virtual assistant, older adults are able to query and visualize the time–series based Ecological Momentary Assessment (EMA) data (e.g., the quality and time of the sleep).

CCS CONCEPTS.

• Human-centered computing → Empirical studies in accessibility.