Examining engagement and usability in an online discussion platform for older adults: Findings from pilot studies

Abstract

Social media may facilitate older adults’ ability to engage socially and explore health information, but it can present difficulties for older adults. Therefore, it is important to explore older adults’ experience of usability and user engagement. We conducted two rounds of pilot studies where we used Facebook to engage older adults. We performed a mixed-methods evaluation of user engagement and usability. A directed content analysis of qualitative data from the pilot studies was used to explore engagement and perceived usability, and the Mann-Whitney test was used to examine differences in feature usage and engagement. We analyzed qualitative data from 13 participants. Qualitative data analysis yielded themes pertaining to three main domains: user engagement, usability, and usability related to aging-related changes. In terms of user engagement and usability, participants in both pilot studies reported positive feedback on felt involvement and endurability, and the second pilot group reported more positive comments regarding perceived usefulness compared to the first pilot group. There was no statistically significant difference in usage over the two studies. The findings of this study suggest opportunities to improve older adults’ experience of online discussion platforms. Considering changes that improve perceived aesthetic appeal and focused attention will be helpful.

INTRODUCTION

Older adults’ interest in technology has been growing,^1,2 and in recent years, there has been increased research on social media and its potential to facilitate health management.^3–5 However, the uptake of social media platforms such as Facebook among people aged 65 and older has also raised concerns about privacy, absence of non-verbal expressions such as gestures and tone of voice, preference for more familiar communication methods, and time commitment.⁶

The importance of understanding older adults’ experiences regarding technology use when developing future technologies or modifying existing technologies has been well documented.^7,8 Models of usability and technology adoption can help us to characterize older adults’ intentions with technology. For example, MOLD-US identifies four main categories of aging-related barriers to mHealth usability: physical abilities (e.g., hand-eye coordination), perception (e.g., visual acuity), cognition (e.g., working memory, dynamic/selective attention, reasoning), and motivation (e.g., trust in own ability, computer literacy).^4,9 Technology Acceptance Model seeks to explain how users’ attitudes and beliefs influence their acceptance or rejection of technology.^10–12

Aside from usability, it is also important to consider user experience and user engagement in digital health interventions.^13–15 Broadly defined, user experience can be defined as the experience a user has with technology,¹⁶ whereas user engagement is the quality of that experience.¹⁷ There are many ways that engagement can be assessed including self-report questionnaires, qualitative analysis of self-report data, system usage data, sensor data, and psychophysiological measures.¹⁵ Despite the diversity of data produced during the course of digital health interventions, multi-faceted evaluations that aim to leverage the different types of data that are produced, are scarce.¹³

In this study, we take a two-fold approach to evaluate a social media intervention to assist older adults in health management. Our analysis includes: 1) a qualitative analysis of older adults’ experience based on the concepts of user experience, user engagement, and usability and 2) a quantitative comparison of usage and self-reported user engagement.

MATERIALS AND METHODS

Study Design and Sample

We employed an iterative approach involving multiple pilot studies to improve the online discussion platform. We used a mixed-methods approach to examine user experience, engagement, and perceived usability based on data from the first two pilot studies (Figure 1). Our target sample for each pilot was ten older adults aged 65 years or older who met at least one of the frailty criteria, such as low physical function, exhaustion, low physical activity, and weight loss, from the short Women’s Health Initiative (sWHI) frailty screening measure.¹⁸ Full details of our inclusion criteria and sampling methods can be found elsewhere.^19,20

Figure 1.

Virtual Online Communities for Aging Life Experiences (VOCALE) design process

The first pilot social media intervention study was conducted over ten weeks from August to October 2018, consisting of weekly discussion topics regarding frailty and aging-related experiences, such as pain, sleep, and fatigue, and coping strategies. Based on feedback from the first study, the second pilot study, which was conducted from April to June 2019, introduced problem solving therapy and reduced the length of the intervention to eight weeks. The first three weeks were similar to the first pilot study in that participants were presented weekly health-related topics to discuss; in the remaining five weeks, participants were presented with a fictional persona developed based on health concerns shared by the first pilot participants, and they were asked to apply problem solving skills to help this fictional persona address her health issues.

Data Collection

This study analyzed multiple types of data, including questionnaires, participants’ online discussion posts, and semi-structured exit interviews that asked about participants’ experiences and perceptions of engaging in the intervention. The administered questionnaires included a baseline questionnaire about demographics, health, and technology use/comfort, and a shortened version of the User Engagement Scale (UES). The version of the UES that we used consisted of 27 items categorized into six dimensions: perceived usability, endurability, novelty, felt involvement, aesthetics, and focused attention, and the items were 7-pt. Likert-type scale items in which users were asked to rate the extent of agreement.^17,21,22

Qualitative Data Analysis

We analyzed the data collected in the exit interviews from the two pilot studies using directed content analysis.²³ We synthesized a conceptual framework based on extant literature on user experience, user engagement, and perceived usability (Figure 2).^4,10,13,22 This framework included three categories:

Figure 2.

Conceptual framework of user experience for older adults (adapt from Chen et al,¹³ Davis,¹⁰ O’Brien & Toms,²² Wildenbos et al,⁴ respectively for each domain)

1. User engagement (endurability, novelty, felt involvement, aesthetics, and focused attention)

2. Perceived usability (perceived ease of use and usefulness)

3. Usability aspects among older adults (focusing on the themes of physical abilities, cognition, perception, and motivation)

The codebook was developed based on the first three transcripts and then applied to the rest of the transcripts. We also coded the polarity of each theme (positive, neutral, and negative). Two authors (SH and AT) analyzed the data independently, and another author (AC) provided guidance upon resolving discrepancies between the two coders to reach a consensus. After discussion, a consensus about all discrepancies was reached. We employed Atlas.ti for qualitative coding analysis.

Quantitative Data Analysis

We compared participants’ usage and self-reported ratings of engagement over the two pilot studies. The Mann-Whitney test, a non-parametric test, was applied to compare the two pilot studies, each with small, unequal sample sizes of less than ten individuals.^24–26 For usage, we employed the Mann-Whitney test to examine differences in participants’ average responses per week. For self-reported engagement, we used the Mann-Whitney test to compare the participants’ scores on the UES. A higher score of each element of the UES indicates more positive feedback. The scores were ranked from 1 to 6, using 1 for the highest score. All quantitative analyses were conducted in R programming language (version 4.1.1). A p-value of < 0.05 was taken as statistically significant. In this study, we used qualitative data to contextualize and enrich our understanding of the quantitative ratings of user experience and usability as assessed by the UES.

Ethical Considerations

All participants voluntarily provided written informed consent and took part in the study. The study was confidential. Participants were informed that the data would not be shared except for a research purpose and that they can withdraw from the study at any time. This study was approved by the university Institutional Review Board.

RESULTS

Sample

Our sample comprised of all participants who enrolled, did not drop-out, and completed the exit interview in the pilot studies (N=13). Sample characteristics over the two pilot studies were similar with participants’ mean age being 82.3 (SD=3.36) years in the first study and 82.6 (SD=6.60) years in the second study. Most of the sample were female (Table 1). Almost all participants felt at least somewhat comfortable using a computer, and all but one participant had prior experience using Facebook.

Table 1.

Sample (Total N=13, Round 1 n=6 and Round 2 n=7)

Pilot Study No.	ID	Age	Gender	Education	Comfortableness of Computer Use	Facebook Experience	Gait Speed (m/sec)	Frailty Symptoms
1	P1	81	Female	Doctoral	Somewhat comfortable	Yes	0.53	Low physical function, exhaustion, low physical activity
1	P2	84	Female	Baccalaureate	Somewhat uncomfortable	Yes	0.79	Low physical function, low physical activity
1	P3	79	Male	Master’s	Somewhat comfortable	Yes	0.84	Low physical function, low physical activity
1	P4	80	Female	Vocational or Associate	Somewhat comfortable	Yes	0.72	Low physical function
1	P5	89	Female	Baccalaureate	Somewhat comfortable	Yes	1.67	Exhaustion
1	P6	81	Female	Doctoral	Very comfortable	Yes	0.88	Low physical function, low physical activity
2	P7	82	Female	Master’s	Very comfortable	Yes	1.06	Weight loss
2	P8	92	Female	High school diploma/GEDa	Very comfortable	Yes	1.17	Low physical function, low physical activity
2	P9	86	Female	Vocational or Associate	Very comfortable	Yes	0.83	Low physical activity
2	P10	69	Female	Master’s	Very comfortable	Yes	0.83	Low physical function, weight loss
2	P11	83	Female	Baccalaureate	Somewhat comfortable	Yes	1.05	Low physical function, low physical activity
2	P12	86	Female	Doctoral	Very comfortable	No	1.09	Low physical function, low physical activity
2	P13	80	Male	Master’s	Somewhat comfortable	Yes	1.27	Weight loss

Note. Doctoral degrees include Ph.D, MD, and JD.

Low physical function: The response ‘limited a lot’ or ‘limited a little’ to the item is considered as low physical function.

Exhaustion: The response ‘all the time’ or ‘most of the time’ to the item is considered as exhaustion.

Low physical activity: The response indicating walking outside less than twice a week is considered as low physical activity.

Weight loss: The response ‘yes’ to the item is considered as weight loss.

^aGED, general educational development.

Qualitative Analysis of User Experience and Usability

We performed content analysis of the exit interviews from both pilot studies. Supplemental Table 1 presents the final codebook, consisting of coding categories, definitions, and exemplars, and Table 2 presents the descriptive statistics relating to the codes. In the subsequent section, we describe themes related to three domains: user engagement, perceived usability, and aging-related changes to usability.

Table 2.

Description of three domains and specific coding themes in each domain in Rounds 1 and 2

Pilot Study No.	Coding Category	Coding Theme	Number and proportion of exemplars based on each coding theme
			Positive	Neutral or Mixed with positive and negative	Negative	Total
1	Domain 1: User engagement	Felt involvement	12 (44.4%)	6 (22.2%)	9 (33.3%)	27
		Endurability	6 (50.5%)	2 (16.7%)	4 (33.3%)	12
		Aesthetics	0	1 (16.7%)	5 (83.3%)	6
		Novelty	2 (50.0%)	0	2 (50.0%)	4
		Focused attention	0	1 (100.0%)	0	1
	Domain 2: Perceived usability	Perceived ease of use	1 (4.0%)	12 (48.0%)	12 (48.0%)	25
		Perceived usefulness	5 (50.0%)	1 (10.0%)	4 (40.0%)	10
	Domain 3: Usability aspects related to four domains regarding aging-related changes	Motivation	5 (45.5%)	5 (45.5%)	1 (9.1%)	11
		Cognition	0	0	5 (100.0%)	5
		Physical abilities	1 (50.0%)	0	1 (50.0%)	2
		Perception	0	0	0	0
2	Domain 1: User engagement element	Endurability	23 (56.1%)	8 (19.5%)	10 (24.4%)	41
		Felt involvement	13 (59.1%)	3 (13.6%)	6 (27.3%)	22
		Aesthetics	3 (37.5%)	0	5 (62.5%)	8
		Novelty	6 (85.7%)	0	1 (14.3%)	7
		Focused attention	2 (28.6%)	0	5 (71.4%)	7
	Domain 2: Perceived usability	Perceived usefulness	19 (61.3%)	6 (19.4%)	6 (19.4%)	31
		Perceived ease of use	4 (28.6%)	3 (21.4%)	7 (50.0%)	14
	Domain 3: Usability aspects related to four domains regarding aging-related changes	Physical abilities	1 (100.0%)	0	0	1
		Motivation	0	0	0	0
		Cognition	0	0	0	0
		Perception	0	0	0	0

Domain 1: User Engagement and Experience

Participants expressed diverse opinions about their experience and engagement. We focus on the three most common aspects emerged from the qualitative data: 1) felt involvement, 2) endurability, and 3) aesthetics.

1). Felt involvement

In this study, we conceptualized felt involvement as the perception that one is involved in the online environment. Some participants appreciated the opportunity to provide social support and have connections with others. “Somebody else has the same problem. I’m not here alone, right, and it’s okay and they’re doing fine. I felt support from them when I see them in the hall. I felt I made friends in a way. And so, I felt closer to those people” (P5, Pilot 1). However, other participants did not feel similarly: “Because I didn’t hear any discussion. … I never discussed anything” (P2, Pilot 1), and “I sort of felt that it wasn’t really communication. … you’re not sure if anybody’s gonna read it…” (P3, Pilot 1).

There were more positive comments about felt involvement in the second pilot study. As some participants said, “I especially liked having other, to read other’s impressions and suggestions” (P7, Pilot 2) and “I especially was triggered by comments and opinions of the other people involved in the study” (P8, Pilot 2).

2). Endurability

O’Brien and Toms characterized “endurability” as the extent to which an individual is likely to remember an experience and return to it.²² In this study, the term was conceptualized as an experience that was successful, rewarding, and worthwhile. Some participants appreciated the value of the online discussion. One participant said, “I enjoyed the concept, and I enjoyed it, the relationships, the conversations with the other, some of the other people” (P5, Pilot 1). However, others experienced stress or challenges of online discussion use, and one participant felt that the study should have had a stronger educational component: “I don’t think I gained any knowledge” (P6, Pilot 1).

In the second pilot, which incorporated problem solving therapy, more participants had positive comments about the value of the online discussion, with one participant saying, “It just gave me an insight into what other people’s problems are. And what other people think, which I find very helpful” (P12, Pilot 2). Some participants liked the problem solving approach. “Well, the problem solving I think is the area where I need the most focus. And so that was my favorite discussion, and I enjoyed seeing how others responded” (P10, Pilot 2).

3). Aesthetics

In the first pilot study, participants noted various challenges. Some participants shared difficulties in using the online platform due to confusing layout and advertising, and a few participants from the first pilot study expressed frustration and difficulty with the layout (Table 2). The first pilot study employed the default Facebook newsfeed layout, which ordered posts by recency and thus often resulted in content not being in order by week, as some participants responded to previous discussions at later times. We found that some participants had trouble finding specific posts and discussion topics. One participant said, “my biggest complaint is that I had just a little bit of trouble with the layout” (P4, Pilot 1).

We took various steps to improve usability. In the second pilot study, a chronological layout was introduced using the Facebook group ‘Units’ feature. By counting one week as a single unit, this fixed the order of the discussion topics and allowed for easier wayfinding within the platform. We also revised the training materials for clarity. In the second pilot study, there were more positive comments on the platform. As one participant said, “I thought it was very user-friendly. It really was. And you would highlight what week it was” (P11, Pilot 2). Another participant remarked, “I like the Facebook format as compared to just the general Facebook format” (P10, Pilot 2).

Domain 2: Perceived Usability

Perceived usability was comprised of two constructs, perceived ease of use and perceived usefulness. In the first pilot study, positive comments regarding perceived ease of use included navigable topics listed by weeks, a shortcut to the Facebook group, and a well-designed outline. Negative comments regarding perceived ease of use included confusing layout, disappeared comments, trouble in posting, difficulty in locating comments, and information overload during training sessions. As one participant said, “Maybe too much information and confusing” (P4, Pilot 1). Regarding perceived usefulness, positive comments included connection with people, obtaining a better understanding of health issues and management or strategies, and support from others and negative comments included repetitive information with no new information.

In the second pilot study, positive comments regarding perceived ease of use included easy to find topics listed by week numbers on Facebook and convenient shortcuts that can directly navigate to the Facebook group. Negative comments regarding perceived ease of use included difficulty in using it on a cell phone because of a vision problem, and confusing arrangement. Positive comments regarding perceived usefulness included interesting, informative suggestions and ideas, finding oneself good at problem solving, having a relaxed environment by using a persona’s story as part of the problem solving approach, and peer motivation for physical activities. One participant said, “I know as people age, it looks like we’re going to live longer, longer, and longer. … I now have a better understanding and appreciation of how important exercise is. … I never had anybody tell me exercise is necessary to keep your energy up” (P8, Pilot 2). Negative comments regarding perceived usefulness included the lack of perceived relevance and reluctance or unwillingness to share personal thoughts because of a preference to be more private.

Domain 3: Usability Related to Aging-Related Changes

The identified themes about aging-related changes included cognition (e.g., confusion due to memory or cognitive issues), physical abilities (e.g., decreased vision or hand dexterity), and motivation (e.g., tendency to enjoy communicating in old age to some older adults). With regard to physical abilities and cognition, one participant said, “I’m losing the ability now to move my fingers, so I type ‘k’ almost every word… I tried to think of easy to type things to say. … I don’t think I’m able to type all of that” (P1, Pilot 1). This participant also added, “The instruction I had was good… one mistake I made was not doing it right after you left… as a senior citizen my mentality declining, I can’t keep track of things” (P1, Pilot 1).

We made modifications to assist with navigation and to reduce cognitive overload. The modifications included reducing text length, enlarging font size, and directional arrows to aid navigation. With the modified features based on feedback from the first pilot group, negative feedback related to aging-related changes was not reported in the second pilot group (Table 2). One participant alluded to her vision problems and mentioned that it was easier to use a bigger screen: “Sometimes it was not as easy to use on my cell phone as it was on my iPad. You see, my vision is going a bit” (P8, Pilot 2).

Comparing Usage and User Engagement between the Two Pilot Studies

As the design of the online discussion platform evolved for the second pilot study based on the experiences and suggestions of participants from the first pilot study, we compared usage and user engagement in the two studies. Figure 3 presents the average number of responses per week per participant, and Figure 4 presents the UES ratings for each participant. Although the average responses per week are generally higher in the second pilot study (M=3.43/SD=1.26) compared to the first pilot study (M=2.22, SD=0.39), there is no statistically significant difference (Mann-Whitney: W = 10, p = 0.1331) (Figure 3). In addition, Figure 3 shows that there was substantial variability in the mean number of responses per participant.

Figure 3.

Average number of responses per week

Figure 4.

Rank of the elements of User Engagement Scale in each participant

We compared the rank of the elements of the UES over the two pilot studies. There is no significant difference (Mann-Whitney: W = 758, p = 0.9878). In general, participants reported higher ratings on novelty, followed by felt involvement, and lower ratings for focused attention and perceived usability, in both pilot studies (Figure 4).

DISCUSSION

In this study, we examined user experience, engagement, and usability of a social media intervention to improve health management among older adults over two rounds of pilot testing. Although there were no significant differences in participants’ usage and self-reported engagement, we observed qualitative differences in user experience, usability, and aging-related usability over the two pilot studies. We made various design modifications over the course of the pilot rounds, and we observed improvements in felt involvement, endurability, aesthetics, novelty, focused attention, and perceived usability in the second pilot study. Considering the small sample size, the lack of quantitative differences between the two pilot studies is not surprising. However, the additional insight afforded through the qualitative analysis illustrates the importance of mixed-methods evaluations in formative stages of research.

This study enhances extant knowledge relating to the design and usability of technology for older adults in multiple ways. At the outset, we synthesized existing literature from multiple disciplines to develop a conceptual framework for evaluating user experience, engagement, and usability in technological interventions for older adults. Though we retain the important focus on aging-related issues in usability among older adults, the inclusion of user experience and engagement facilitates a more holistic perspective of older adults’ experiences, to include not only usability problems but also contextual factors of the user experience.

Extant literature has argued that ease of use and usefulness are factors that affect technology acceptance among older adults.²⁷ In this study as well, participants expressed confusion when we employed the default recency sort on Facebook, resulting in negative comments. However, we were able to address this in the subsequent pilot study by leveraging extant Facebook features.

Extant literature also suggests that the use of social networking sites may enhance cognitive functions such as complex working memory function among older adults,²⁸ and with adequate support, older adults can engage with platforms intended to provide both informational and peer-to-peer social support.²⁹ In this study, some participants experienced some difficulties due to manual dexterity and memory. We implemented various measures to address these issues, including adding additional support to facilitate recall of how to use the features of the platform as well as making revisions to the interface to more clearly indicate how to perform actions. Though the research team included those clinical backgrounds, in the future we hope to involve nurses and other healthcare professionals to provide additional feedback on the design of online interventions for older adults.

This study has several limitations. Because this study employed a mixed-methods approach to evaluation, there could be issues of subjectivity, disagreement, and limited reproducibility.³⁰ Additionally, another limitation is that our sample could have been more diverse, and there is a need to consider how other participants might engage with this intervention. We encountered a challenge in recruitment due to increased public concern about privacy and Facebook around the time of study recruitment, and some potential participants expressed hesitation to participate. As a result, it led us to consider moving our study to a different discussion platform in the future.

CONCLUSIONS

Online discussion platforms can serve as a venue in which older adults with similar health problems exchange and help one another improve their health management. However, older adults can experience challenges in the use of these platforms; this study demonstrated how quantitative and qualitative data could be combined to perform a holistic assessment of usability and user experience on multiple dimensions. In addition, the study illustrated how the user-centered design approach, employed over multiple pilot rounds, could be used to address usability issues. This highlights the value and importance of continuous exploration and deliberation of making the most out of online discussion platforms among older adults in order to improve their health.

Supplementary Material

Supplemental Data File (.doc, .tif, pdf, etc.)

Supplemental Table 1. Coding categories, definitions, and exemplars for older adults’ user engagement