Usability Evaluation for the Amulet Wearable Device in Rural Older Adults with Obesity

Abstract

Mobile health (mHealth) interventions hold the promise of augmenting existing health promotion interventions. Older adults present unique challenges in advancing new models of health promotion using technology including sensory limitations and less experience with mHealth, underscoring the need for specialized usability testing. We use an open-source mHealth device as a case example for its integration in a newly designed health services intervention. We performed a convergent, parallel mixed-methods study including semi-structured interviews, focus groups, and questionnaires, using purposive sampling of 29 older adults, 4 community leaders and 7 clinicians in a rural setting We transcribed the data, developed codes informed by thematic analysis using inductive and deductive methods, and assessed the quantitative data using descriptive statistics. Our results suggest the importance of end-users in user-centered design of mHealth devices and that aesthetics are critically important. The prototype could potentially be feasibly integrated within health behavior interventions. Centralized dashboards were desired by all participants and ecological momentary assessment could be an important part of monitoring. Concerns of mHealth, including the prototype device, include the device’s accuracy, its intrusiveness in daily life and privacy. Formative evaluations are critically important prior to deploying large-scale interventions.

INTRODUCTION

The growing epidemic of geriatric obesity¹ is associated with significant health risks². Current programs focus on weight-loss rather than function-based outcomes and are marginally able to achieve clinically significant weight loss³. Initiatives are time-intensive and impractical to implement in busy practices, particularly in rural areas that lack specialized services and trained staff^4–6. Mobile health (mHealth) technologies may overcome many of the barriers faced by existing health promotion interventions. Devices have the potential to monitor and provide real-time motivating patient-oriented feedback to adults wishing to change behavior. Devices provide automation to improve adherence and overcome reliance on direct interactions with busy clinicians who focus on health issues other than obesity^7–9.

A significant social stigma exists in this population, where perceptions suggest that older adults may not agree to wear or use novel technologies¹⁰. For instance, they may not have the latest version of a cellular phone, or have older prototypes that are inconsistent with the current social norms. Ensuring product designers provide a device that is inclusively designed for all age groups while being matched to specific groups’ requirements is essential. User-centered design processes are a cornerstone in the development and marketing of any device to be used for commercial, everyday patient use. Unfashionable devices, even when designed for older adults, may propagate the stigma of limitations observed in older age¹¹. The emergence of commercially available devices for older adults is a major predictor of acceptance^{12, 13}.

Older adults often have sensory and medical limitations that can affect the appropriate and effective use of devices^{14, 15}. This is of particular importance in a rural population, where disability rates are higher, transportation issues and broadband issues are abound and disparities have been well established in health. While older adults are the fastest-growing demographic using technology¹⁶, the need to use a team-science approach by including the end-user as a member of the research and design team is important¹⁷. What is unknown is whether using user-centered design processes can be successfully applied in this demographic, specifically in the formative stages of intervention development. While obesity-related wearables for health promotion interventions have been tested in younger populations^{18, 19}, previous studies have only focused on fall detection techniques^{20, 21}, safety ^22–24, and activity monitoring in older populations^{25, 26}. Other than an internet-based studies ^{27, 28}, we are unaware of studies focusing on this particular study population, with a primary intent to be considered in older adult obesity trials.

Without involving the appropriate stakeholders in the design of products and services based upon emerging technology, their market adoption may fail, thus reducing the chances of intervention impact and benefit. Our study used rigorous qualitative methods by using a multi-stakeholder co-design (community leaders, patient participants, and clinicians). Our goal was to evaluate the potential for [Device Name]²⁹, an open-source mHealth platform that has the ability to load custom-made applications for specific populations to support a health-behavior intervention.

METHODS

Study Setting/Design

We undertook a qualitative study including both semi-structured interviews and focus groups between October 2016 and April 2017. The purpose of this analysis was to gain insight and information among three different stakeholder groups (patient participants, clinicians and community leaders) related to the design and usability of the [Device Name] device (see below). The Committee for the Protection of Human Subjects at [Institution] approved the study.

We recruited participants from a rural, primary care academic practice at [Institution] in [Location], which is located on the [State] border with [State] and is the state’s sole academic center. The center serves over 4,000 patients aged over 65 in primary care. Posts on institututional, academic (through [name of research center] and community based (@vitalcommunities.org) listservs (disseminated this study information. All participants gave informed consent prior to the interviews.

Recruitment & Sampling

Purposive sampling of the three stakeholder groups facilitated insight into different viewpoints and perspectives^30–33. Our target recruitment goals were 8 semi-structured individual interviews, 4 focus groups of 6–8 participants each, 4 community-leader interviews, 6 clinician and 8 patient participant semi-structured interviews. We conducted homogeneous sampling for our patient participants (focusing on older adults and obesity), and typical case sampling for clinicians and community leaders. We achieved theoretical saturation following our 21^st patient participant within the focus group setting, at which time we concluded conducting additional interviews as no new data was appearing and all concepts and themes were clear to the research team.

All patient participants were English speaking, with a body mass index ≥30kg/m² or a waist circumference ≥88 in females or ≥102cm in males. Screening was assessed by medical record review and the investigators approached those by telephone who met criteria. We sought community leaders from local networking and email lists whose roles were to lead aging initiatives within their respective communities; two of the six community leaders identified did not respond or did not feel they could contribute to the purpose of the study. Clinicians were all primary care providers, three who had additional training in geriatrics. All approached clinicians participated. Data were collected by self-reported questionnaire for clinicians and community leaders and medical records were not reviewed.

Interviews & Focus Groups

These sessions asked a broad range of aging, wellness and technology questions (Appendix 1). One component specifically focused on [Device Name], the focus of this paper. [Device Name] will eventually be integrated within a multicomponent obesity wellness intervention for rural older adults, as a tool to augment in-person and remote behavioral change. All participants held, felt, and wore an [Device Name] during the session as part of initial usability testing. We also presented images of applications and asked participants to comment on the illustrated designs. Our methods focused on heuristic evaluation and cognitive walk-through of applications as to the capabilities of the proposed applications^34–37. We also asked participants to “think aloud” when holding the [Device Name], asking them to speak exactly what was coming into their minds.

We conducted open-ended semi-structured interviews (lasting 45–60 minutes) and focus groups (90–120 minutes) for patients, consisting of clarifying probes in which we encouraged participants to elaborate upon open-ended questions. Members of the study team ran focus groups. Investigators running the interviews and focus groups had formal training in interviewing, self-reflection, and conduct.

[Device Name] Prototype

[Device Name] (Figure 1) is [institutional affiliations] project designed with the goal of a long battery life and the ability to operate independently of other devices (e.g. tablet/smartphone). The device was meant to be employed within a research setting with minimal input from specific design engineers. [Device Name]’s dimensions are 4cm², with two buttons, and a slider with a black and white screen. The device can also evaluate motion through built-in sensors that measure acceleration, rotation, sound, light, and temperature. In addition, it has the capability of ecological momentary assessment, a technique that investigates and gathers user data on affective, behavioral and contextual experiences real case in real time³⁸. The platform is capable of running applications specifically designed for the purposes of monitoring and tracking health, including steps, activity, stress and strength^39–41. Data can be stored on a micro-SD card, or transmitted over Bluetooth to a nearby device such as a tablet or smartphone, when these devices are available.

Figure 1:

Image of the Amulet wearable device

As part of the proposed intervention, the investigators deliberately chose a prototype as opposed to a commercial device. In the project’s conception, the open-source platform of this device would provide unique opportunities for custom-made development of applications that far outstretch the capabilities to modify the application programming interface of devices such as Fitbit or Garmin. Importantly, a concurrent goal was to evaluate basic signal data to allow creation of custom-made algorithms for an older adult population.

Data Analysis

Using a digital voice recorder, we recorded each encounter and transcribed the text. De-identified data was uploaded into Dedoose (www.dedoose.com), a qualitative software application. Each transcript was reviewed by [Author Name] and 10% of the audio-recordings were reviewed by [Author Name]. We developed a codebook using both inductive and deductive categories related to [Device Name]’s usability. For the present analysis, we focused on the following [Device Name] domains and codes: features (good, poor, missing); potential usage problems (screen, communication); and aesthetics (appearance, comfort). All coding involved multiple researchers to enhance reproducibility and trustworthiness. Consensus resolved all discrepancies. A third researcher provided independent input when consensus could not be reached. Major themes were reviewed by a qualitative methodologist [Author Name] allowing the construction of the main themes for this usability study.

RESULTS

Demographics

We interviewed 29 patient participants, 7 clinicians and 4 community leaders. Mean ages were 72.9±4.6, 46.7±12.1, and 64.3±8.7 years, respectively, with the majority female gender, and white race, in all groups. Body mass index was higher in patient participants than in clinicians or community leaders (32.9±2.5, 23.8±2.1, and 26.4 ±4.6 kg/m²). Clinicians and community leaders had been involved for 14±9.5 and 13.5±5.5 years in their respective roles.

Usability

Our usability assessment focused on three separate domains: desired features of the existing [Device Name] prototype; potential problems that this demographic could face in using the device; and the aesthetics of the prototype (Table 1). Generally, all three groups had a favorable impression of the device design and features of tracking, monitoring and representing the data. For some participants, the device’s durability was a key factor to its potential use. Clinicians in particular were more skeptical about the use of the side buttons and the slider in those with limited dexterity (e.g., osteoarthritis); however, this concern was not mentioned by patient participants. An additional improvement suggested was to reconsider the wrist strap, as older adults with reduced fine motor capabilities would be unable to strap the watch on. The prototype used an off-the-shelf plastic strap with a metal buckle, as is common in traditional wristwatches; it would be easy to swap this strap for a Velcro or other style of easily donned strap.

Table #1:

Usability of the [Device Name]

Participants’ Views of the Device’s Features	Illustrative Quote
Good Features of the Device	Community 1: It’s very lightweight. It’s not threatening at all. Focus Group 2: No, it’s good [the lack of glare] Focus Group 5: It seems sturdy and I like the material
Poor Features of the Device	Clinician 6: if somebody had a lot of osteoarthritis and other stuff, this seems like it [the device] might be a little more difficult to use Community 1: for folks that may have some visual problems, this [information] might be tough to see Focus Group 4: Well I couldn’t put it on cause I have trouble putting a pin on that’s why I have this kind of band
Missing Features Needed in the Device	Focus Group 3: Are there ones for lefties Patient 5: I would want to connect with my smartphone Focus Group 2: They should put holes on this side too. For sweating. Just like holes here Community 1: For people that have Lifeline maybe being able to connect to it? Patient 3: So like a visual representation of [the activity] you’ve done
Potential Problems in the Current Prototype Device
Concerns with the Screen Size	Patient 5: I like bigger screens. See my phone’s big so I can read it. I had a smaller one, but you can’t do anything with it. Patient 7: I think the screen size would be ideal if it’s going to show steps, things like that. I would say we need a pretty big screen to me.
Alerts and Communication Effects that Older Adults were concerned about	Patient 8: I guess the first thing is how intrusive would I feel it to be. And now today I might consider it to be intrusive. Clinician 2: Some get agitated with too many alarms and voice. Focus Group 2: I think you’d have to be careful not to give too much [information or alerts]
Participant Views of the Device’s Aesthetics
Current Appearance of the Device	Patient 3: [It is] not particularly appealing, physically, in terms of something. And I’m not a big fashion person, but this does not have fashion associated with it Focus Group 2: Patient N:You gotta get people to wear it. And they won’t wear it if it’s … weird. Clinician 4: The band feels very comfortable, but the block nature of it seems bulky and if people want to wear it as an aesthetic device, that would feel like a barrier to me. Feels like they’re wearing more of a computerized device
Comfort Level of Wearing the Device	Community 4: Don’t love [the way it] feels when I’m moving my hand this way. Patient 8: Some articles of clothing might be hard to pull it over. Patient 4. The material of that band you wouldn’t like, necessarily? I’m pretty sure I wouldn’t like to wear that. At all.

Individuals were quick to criticize the aesthetics and comfort of the device as it differed significantly from current commercial devices like the Apple Watch. A general theme was the need to ensure the device communicated with their phone or tablet. Patient participants felt that a larger screen would be helpful to account for their sensory challenges (e.g., eyesight). However, all individuals did not feel that this would significantly preclude the user from wearing the device.

Desired Functions

The evaluation also sought to learn what other key features and capabilities would be important to incorporate, and to probe some of the participant concerns. We asked all three stakeholder groups to ‘dream big’ and imagine what type of monitoring could be implemented on [Device Name] to improve their health. We classified the monitoring capabilities in groups as illustrated in Table 2 (vital signs, physical activity, nutrition, somatic, and other).

Table #2:

Desired Functions of the [Device Name]

Objective Measures Desired by Participants in a Wearable Device	Specific Capabilities that could prompt behavioral change and/or data visualization	Overall Participant Concerns for each of these Features
Vital Signs ▪  Blood pressure ▪  Heart rate ▪  Temperature	Dashboard: I think the less that they have to input the better.	Accuracy: Clinician 7: I think if it’s accurate, the calorie burning is helpful.
Physical Activity ▪  Steps ▪  Type of Exercises	Ecological Momentary Assessment: Community 1: The concept of messaging? Absolutely. Especially if they had some sense that it was a human they knew. I’m seeing this measuring things but really what it should, I would say, is that it would go “bing” six times a day. And it would say during the last two hours, did you have any of these things to eat? “Bing” during the last two hours, what did you do? When you give the answer, you just push a button.	Need person (life coach) I would think, that you would want to have a life coach
Nutrition Monitoring ▪  Day-to-day weights ▪  Calorie burning    Meal planning ▪  Muscle mass	Constant diagnosing: Ideally some small device like that, which would have to be portable to be able to monitor you all the time. People go to medical school and spend a heck of a lot of time and many, many sleepless nights learning everything you have to learn to be a successful. Why not go to the doctor in a little portable device? It would at least be able to do diagnosis on a constant basis, and, diagnosis is the key to successful medicine. You got to know what the heck you’re dealing with first.	Intrusiveness Community 2: generally, nothing intrusive, but… things like heart rates and blood pressure and just general wellness things,” and see if there’s any connection to things relating to things… and, “Oh, by the way, you’ll get feedback on the, the very things that we’re collecting data.”
Somatic ▪  Pain ▪  Mood ▪  Stress	Communication with healthcare provider: Send my heart rate and blood pressure to my doctor. I’m not feeling well, and maybe an email would get generated that all of a sudden they’d say, “No, you better come in.”	Privacy: Focus Group 2: It’s like a big daddy.
Other ▪  Sleep monitoring ▪  Prescription monitoring ▪  Seizure Disorder	Transfer information: But it, but if it were something that would transfer information to somewhere else that at the end of the day I could go look at. How did I do today?

Participants felt that a dashboard would be helpful to centralize all data inputs. All participants believed that this would be an important mechanism to feed information to the participant and research team and could enhance an individual’s own personal knowledge and health. Patient participants, in particular, felt it would be important to have all this information relayed to their healthcare provider. Ecological momentary assessment was an important feature raised as part of monitoring and tracking.”Buzzer fatigue” was a concern expressed by most groups.

Finally, participants raised several concerns. First, application accuracy of any measurement or application was important to all enrollees. Second, while the technology could be a useful adjunct to current behavioral strategies, it may not in and of itself solve the problem at hand (i.e.obesity) and that coupling technology with human interaction would provide critical guidance to individuals in their health efforts. Third, the intrusiveness of constant monitoring and privacy concerns were raised as secondary issues.

DISCUSSION

Our results are an example of how involving the end-user in co-design and usability assessments can be used to improve the user interface and user experience to ensure the preliminary viability of prototypes mHealth devices. This qualitative work provided a number of suggestions to improve [Device Name]’s capabilities in the older adult population that can be extended to other devices. In this study, we demonstrated that [Device Name] potentially could be a usable mHealth device for older adults with obesity, with capabilities to understand health behaviors in this population.

The use of mHealth applications in older adults with obesity is an emerging intervention strategy. Batsis and colleagues evaluated the FitBit ‘Zip’ device on eight adults aged 65 years and older from a rural practice, and demonstrated that this device was feasible and acceptable for use among older rural adults with obesity⁴². In a separate study, older adults engaged in more activity in a web-based weight-loss protocol as compared to younger adults, with a greater number of logins and higher degree of weight loss²⁸. Findings of this current pilot support the acceptability and promise of mHealth interventions for older adults in rural settings where transportation and other potential barriers limit access and effectiveness of office-based interventions. Related research includes a study by O’Brien demonstrating in 24 older adults (mean age 69±8 years) that using the Internet-delivered “Lose-It” app was feasible, usable and led to increased self-efficacy²⁷. Our group previously evaluated [Device Name]’s ‘ActivityAware’ application, which monitors activity intensity in older adults, and demonstrated its accuracy³⁹. Unique studies focusing on older adults are critically lacking, an important gap as weight loss affects older adults differently. The special challenges of safe and effective weight loss in obese older adults are often underappreciated and overlooked³.

While our study cohort was generally accepting of [Device Name], there was a broad desire for integration of this intervention with the medical point of care system. A major challenge of mHealth interventions is the volume of data and material generated by continuous-monitoring apps and devices. Cloud-based systems need to process data into a usable form for the consumer and provider. Our participants all asked for integration with the electronic health record, and that their clinical providers have access to such information. The current healthcare environment requires connectivity between existing devices and electronic medical records in the United States^{43, 44}, however. Providers are overwhelmed with existing infrastructures and additional data potentially can lead to medicolegal and information-technology challenges, a subject needing further evaluation.

We obtained a comprehensive understanding of [Device Name] from three distinct groups. Triangulation enhances the ability to obtain different perspectives on a given problem. Our interviews and cognitive techniques add to the internal validity of our results. The coding process reduced bias and increased the rigor of the findings. Cognitive techniques (i.e.,Think Aloud) are affordable, robust, easy, and can capture a participant’s preference and performance simultaneously. We acknowledge our study’s limitations including the non-random sample. We also caution against generalizing our results to other settings (e.g., non-rural, more ethnically/racially diverse, lower education). Our participants had previous experience in using smart phones and computers; those who are technology-naïve may have different responses. Findings were obtained within an interview setting, and reactions may differ if the [Device Name] were provided to participants in another space (such as home or community).

Lessons learned & Implications

Further studies should evaluate [Device Name]’s functionality, its use in different populations, and robustness outside of a research setting. Ascertaining input of participants who may not have access to or familiarity with mHealth technologies may advance our understanding of engagement factors in different socioeconomic and reduced health literacy populations. Evaluating [Device Name] in urban settings may help confirm, modify, or refute our findings. Adapting [Device Name] for visual and dexterity impairments is important, as is an ability for audible (rather than haptic buzzer) alerts. We recognize that academia generally lacks the commercial resources to develop modern and unique devices that could be deployed commercially. Industrial design is considerably important in ensuring adequate usability and academic/industry partnerships in conjunction with stakeholders are critical to success. Future prototype development should pay particular attention to usability and aesthetics, given the importance of these factors to participants. We emphasize that end-users are critical in conducting usability testing. This point is particularly important with older adults, a group that often has limitations that make it difficult or impossible for it to adopt emerging technologies.

Our results suggest potential adaptations to the device to address the normative changes of aging. Older participants raised important factors related to changes in perceptual alteration, including larger fonts and buttons to accommodate the visual changes with aging, and reducing high-pitched alerts. Motor system changes may also impose difficulties in using the device (e.g., putting the strap on, scrolling). While these particular issues were not an issue in this study’s cohort, the normal and abnormal cognitive changes in aging can lead to inappropriate or slowed processing and attention⁴⁵. Such elements not only lead to hesitation in the use and adoption of technology among older adults, but potentially prevent an aging patient from using this technology effectively. Additionally, future research will evaluate [Device Name]’s use in the field and whether its aesthetics are a significant limitation to its broad acceptance and use, as this was a significant concern raised by the study participants.

Our study has considerable implications for clinicians and researchers alike. First, we highlight using qualitative methods in informing future interventions. This is a vital step in any systems-engineering approach for both designers and interventionists using the [device name] as a case and can provide robust information in device development. Stakeholder engagement allows harmonization of opinions – congruent vs. discongruent – in ensuring the design will be acceptable within the intervention. Our findings and observations could easily be applied to any process focusing on optimizing user-centered experiences in older adults with obesity.

Conclusions:

Our qualitative study provides considerable formative data that can be helpful in future interventional studies. First, end-users are critical in the development of any device. Second, aesthetics are important. Third, prototypes may be acceptable to older adults over commercial devices. However, future work should focus on demonstrating the efficacy and capability of the device to augment weight change in an intervention in this population.

Appendix #1 –. Questions

[Device Name] [show [Device Name] and demonstrate its functions]

This is a prototype [Device Name] device. An idea we have is to track your level of physical activity using this device. It was developed by the Department of Computer Science at Dartmouth College and can connect to the Internet through a cell phone or tablet.

• 1What are your first impressions of this device?

  • Probe: What looks good? What does not look good? Do you see any problems at the outset that you might have?

• 2In an ideal ‘dream big’ world, what types of information would you want the [Device Name] to measure?

  • Probes: Steps, activity, strength

Feedback through messaging

One goal of the [Device Name] is to provide feedback through messaging. Throughout the day, you will receive short messages on the display. Some feedback messages will be automated and some will be delivered by an actual person. In addition to measuring and monitoring your steps, activity, and strength, it will have the ability to relay some of this information to you.

• 3What do you think of the [Device Name] now?
• 4How do you think receiving updates or messages would be helpful in tracking your progress through the [Device Name]?

[show slides of [Device Name] screenshots]

• 5What do you think of the screen now that you can see some of the information that can be relayed?