Older Adult Engagement with Facebook Interventions: A Challenge for Nursing Research

Background

Facebook is a popular platform for hosting online support groups. The readily available, socially acceptable, and free platform holds many advantages for researchers designing and implementing interventions with social media. Statista reports that in the second quarter of 2020 there were 2.7 billion monthly users of Facebook (Statista Research Department, 2020). According to the Pew Research Center, six in ten U.S. adults use Facebook, and seniors are the fastest growing segment, doubling since 2012 (Perrin & Anderson, 2021). The private hidden groups on Facebook allow researchers to control who is in a group and prevent the group’s activity from being accessible to non-members or being searched, making private groups an effective way to explore sensitive topics and test related interventions.

The Facebook platform is not without challenges for nurse scientists interested in using the platform for intervention research. Despite the popularity of Facebook online support groups, health outcomes such as anxiety or depression which may be impacted by participation in those groups have not yet been extensively assessed. For example, it is unknown what components of Facebook participation lead to a change in an outcome, such as anxiety. It is also unknown if the change in outcome is the result of what is learned or found meaningful in online content or the result of social support from discussions with persons in similar circumstances. Answering questions regarding the individual effect of Facebook support group interventions requires researchers to be able to identify individual data from social media content. For example, researchers must understand how an individual participant in an online group engages with the group to evaluate the impact of the encounter on an outcome like anxiety. Does an individual need to create a post or comment on a post to reduce their anxiety or is anxiety reduced simply by reading another’s posts and comments? How engaged in the group does an individual have to be in order to benefit? Similar to face-to-face support groups, the question of how engaged or how much participation is required to benefit is an important issue. In the case of an online group, the ability to measure an individual’s participation and engagement with the group and its content requires identifying which posts and comments belong to a specific individual, as well as understanding how to measure the value of that content. Currently, Facebook does not allow downloading of individual identities with specific posts, comments, or reactions when data are extracted from any Facebook group. This has not always been the case; this changed in 2018.

In 2018, the Cambridge Analytica scandal called into question the privacy and security of the Facebook platform (Wong, 2019). As a result, Facebook made numerous changes, including the elimination of links to individual names from their analytics data as part of a “clear history” initiative (Egan, 2018). After the changes, the ability for researchers to collect individual-level data and measure individual engagement was suddenly taken away. This paper illustrates one research team’s attempt to measure individual engagement since the Cambridge Analytica scandal. The experience is offered as a way to help in the development of research methodologies for social media research.

Facebook Challenges in One Randomized Controlled Trial

The ACCESS (Access for Cancer Caregivers to Education and Social Support) study, funded by the National Cancer Institute in 2017, is a five-year, pragmatic, cluster-crossover randomized controlled trial; details are reported elsewhere (Parker Oliver et al., 2020). The ACCESS intervention seeks to reduce anxiety and depression among family caregivers of hospice cancer patients through increased social support and education provided via a Facebook group. The Facebook group was created and facilitated by our research team, and caregivers of hospice cancer patients were recruited and randomized by hospice agency into the group. The group was created as a hidden private Facebook group, which protected individual privacy by making the group unsearchable and allowed our team control over its membership.

During the early testing of the ACCESS intervention, we noticed substantial variability in the engagement of participants in the Facebook group, as some individuals posted or commented several times per day and others did not post or comment at all. In our work, engagement is defined as the ways individuals interact as they use and respond to the content provided in the Facebook group. Given the variability of participants’ involvement in the intervention, it became clear that we needed to measure if and how engagement was associated with our hypothesized study outcomes (anxiety and depression). At the beginning of the project, we could access reports that provided data on the activities of individuals in the group. An engagement statistic that took into account how much an individual contributed to the group activity for a specified time period was readily available. Unexpectedly and without warning, Facebook changed its privacy policy, and we lost our ability to automatically obtain participant-level engagement statistics. Individual posts and comments were no longer identified when we extracted data.

The research question addressed in this paper is: How can researchers measure individual engagement in Facebook online support groups? The goal of this paper is to share our process and experience in answering this question. We describe in detail how we captured data and developed an individual engagement statistic for our participants, contributing to the literature options for nurse researchers who are developing and analyzing Facebook interventions.

Methods

Our team conducted an informal literature review to understand how others have measured individual engagement in Facebook since the Cambridge Analytica change. We searched for studies published in English from 2019 to 2021 (i.e., post-Cambridge Analytica). We included papers focused on measuring engagement in a Facebook group for a health-related intervention. Campaigns involving paid/sponsored content, awareness-only campaigns, interventions or campaigns that did not incorporate quantifying engagement, and papers without results were excluded. The search terms “Facebook”, “engagement”, and “measure” were used. We searched PubMed and CINAHL.

Based on the results of the literature review, we used a four-step process to quantify individual engagement. First, a third-party program was used weekly to download all posts and comments from August 2017-April 2021 to an Excel spreadsheet. Only comments and posts could be extracted, as they are narrative in nature. “Likes” and other reactions that involve the simple task of clicking a button were not automatically extractable. Posts and comments were then manually identified by research staff who reviewed the original text on the Facebook group page itself and added the name of the participant who wrote the narrative in a new column on the spreadsheet. In some cases, we also had to match the Facebook name on the page with the participant’s research name, as some people used nicknames on Facebook while using a legal name in the research study.

The second step involved organizing each participant’s individual engagement data using a separate Excel spreadsheet. Members of the research team manually counted and documented posts, comments, likes, and reactions for each participant. If a participant changed their privacy settings after leaving the Facebook group, their data disappeared from the Facebook page and history. Therefore, if we had not downloaded data before a participant left the group and changed their settings, we lost all access to the individual participant’s data.

In the third step, data from the two sets of spreadsheets were merged. A new variable was computed to represent the number of days individuals had the opportunity to engage in the private Facebook group. We counted days from Facebook group enrollment to the date we conducted an exit interview with the participant (the study protocol required individuals to be removed from the Facebook group after they completed the exit interview). In those few cases where the interview was never completed because a participant could not be reached by phone, we used the last date we had contact with them. The fourth step involved creating the engagement score.

Assumptions Underlying the Engagement Score

Based on the same assumptions as Wu et al. (2020), we noted that Facebook posts and comments were more time-consuming for participants than likes or other reactions. We estimated that it took half the time to record a like or other reaction as it did a simple post or comment. Therefore, like Wu et al. (2020), we developed a weighted scoring system in which we scored posts and comments as 1 and everything else (i.e., likes and other reactions) as 0.5 to account for the differences between the different types of interactions. We decided against assessing the quality of every individual post or comment, instead assigning the allotted value to each response of the same type. Posts and comments made by the facilitator were not included in the analysis.

Creating an Engagement Score

Once we had complete data, we computed an engagement score. First, after counting all the documented interactions of all participants, we computed engagement per day for each group member. The individual weighted counts of posts, comments, and reactions were summed for a total score. This total score was then divided by the total number of days a participant had the opportunity to engage in the group. The resulting score indicated the average per day engagement score for each participant. For instance, a score of 2 is interpreted as an individual averaging a combination of 2 written interactions (posts, comments, reactions, likes) per day.

Next, we used a subsample of our study exit interviews for a face validity check. We selected participants who were in both the interview dataset and our engagement dataset. We compared participants’ self-assessment of their Facebook activity from interview data with their daily engagement score. The subsample interview participants are not meant to be representative of the entire sample; they simply represent the only sample we had to provide any face validity. In the interview, participants were asked to describe their level of participation in the group. The participant self-reports in a subsample of interviews were consistent with the level of engagement reflected in the daily engagement value. For example, when individuals stated they did not post anything, their engagement value was near or close to 0. Likewise, if the participants reported they were active nearly every day, their daily engagement score was close to 1 or greater than 1, representing an average of one interaction per day.

Statistical Analysis

The distribution of the raw data was examined, including measures of central tendency and normality (skewness and kurtosis), to determine the best approach to develop the engagement score. First, we grouped scores of 0 and scores greater than 1 into their distinct groups: no engagement and very high engagement. The remaining data, which were the scores between 0 and 1, were grouped into tertiles, with each tertile representing “low”, “moderate”, and “high” engagement groups. After establishing our score categories, we then examined descriptive statistics of our sample per engagement group.

Results

Our literature review found 149 unduplicated articles and, after reviewing abstracts, we analyzed 38 full-text articles. After applying inclusion and exclusion criteria, only four articles remained. Of these four articles, two described the measurement of group engagement rather than individual engagement (Healy & Marchand, 2020; Kashian & Jacobson, 2020). Although the two remaining articles described measurement of individual engagement, the methods used were inadequate for our research question, as noted in Table 1. Table 1 summarizes the findings in the four eligible studies included in this review. For example, the study by Ng et al. (2019) used the total number of posts and divided by the number of individuals in the group for an average number per member rather than an actual number of posts per individual. This method would not have allowed us to connect individual engagement to individual outcomes. While our review did not provide us with a usable engagement measure method, it provided context to help develop our own and guided decisions as noted above in that process.

Table 1.

Summary of the Literature Reporting Measurement of Engagement Statistic Since Data Use Changes in 2019 (i.e., Cambridge Analytica)

Author, Year	Aim(s)	Individual or Group Score	Method for Measuring Engagement	Sample Size	Applicability to Our Study
Healy & Marchand, 2020	To examine the feasibility of CHASE, a parent-mediated physical activity intervention for children with Autism Spectrum Disorder using a private Facebook group.	Group	General group Facebook engagement data were collected via Facebook analytics.	16	Did not apply, as we wanted an individual measure.
Kashian & Jacobson, 2020	To examine factors of online engagement and the relationship between online engagement and health expectations in a private Facebook group for individuals with stage IV breast cancer.	Group	Engagement was assessed using an aggregate measure of members’ comments and reactions (e.g., likes, hearts, and smiley faces) to the Facebook group.	74	Did not apply, as we wanted an individual measure.
Ng et al., 2019	To evaluate usability and acceptability of a patient-informed mHealth support program for young adults with Type 1 diabetes using Facebook groups.	Individual	Engagement was calculated as mass engagement divided by number of group members.	34	This did not accurately report the activity of individuals. Instead, it reported the mean activity by all individuals.
Nour et al., 2019	To explore user engagement with a 4-week smartphone program for improving vegetable intake with additional Facebook content.	Individual	Engagement with Facebook content was measured by tracking views, likes, and comments or posts within an app.	97	Did not apply, as no additional app other than Facebook was used in our study.

The final sample contained 170 caregivers in our Facebook group who had data to compute an engagement score. We estimate that it took three research team members a total of more than 300 hours of work to compute these 170 scores.

Measures of normality revealed that the engagement scores were extremely right-skewed (skewness=5.85) and leptokurtic (long and skinny-tailed; kurtosis=47.8). The average number of days enrolled in the Facebook group was 76 (standard deviation [SD]=37.6). There were 112 posts, 999 comments, and 1383 reactions from all participants. The total counts of Facebook interactions (1 for posts/comments and 0.5 for reactions) ranged from 0 to 107 interactions per participant, while the per day engagement score ranged from 0 to 4.78. The overwhelming majority of caregivers had engagement scores <1 per day, while seven caregivers had very high engagement, with average engagement per day ranging from 1.0-4.78. Box 1 provides two case examples of computed scores.

Box 1. Case Examples of Engagement Scoring, Classification, and Face Validity Assessment.

Case 1: Study Number 3-1066-01

# of Posts: 27 (number of these interactions) x 1.0 (value of interaction) = 27

# of Comments: 310 (number of these interactions) x.1.0 (value of interaction) = 310

# of Likes or Other Reactions: 463 (number of these interactions) x 0.5 (value of interaction) = 231.5

Total for posts, comments, likes, other reactions: 568.5

Total number of days in the group: 119

Average daily engagement score: 4.78

This was in the top group (average daily engagement score < 1), so they were classified as very high engagement.

Response to interview question for face validity:

“How often do you participate in the Facebook group?”

Probably about every day or every other day. I may not write anything, but a lot of times it’s just reading or re-reading an article, or re-watching a video. Sometimes where I do write something, or I interact with somebody, it is still helpful. It just helps me to know that these are real people.

Case 2: Study Number 5-2731-02

# of Posts: 0 (number of these interactions) x 1.0 (value of interaction) = 0

# of Comments: 0 (number of these interactions) x 1.0 (value of interaction) = 0

# of Likes or Other Reactions: 1 (number of these interactions) x. 05 (value of interaction) = 0.5

Total number of days in the group: 76

Average daily engagement score = .007

This was in the bottom tertile (average daily engagement score = 0.007 to 0.059), so they were classified as low engagement.

Response to interview question for face validity:

“How often do you participate in the Facebook group?”

I’m really not a social media kind of guy. I think I only visited a couple of times. I basically just introduced myself and read some material. That’s probably about it.

There were 24 (14.1%) participants with “no engagement,” 48 (28.2%) classified as “low engagement,” 45 (26.5%) as “moderate engagement,” 46 (27.0%) as “high engagement”, and 7 (4.1%) as “very high engagement.” Figure 1 shows the full distribution of engagement scores and illustrates the distribution by group as well. Based on dividing the groups into tertiles, low engagement scores ranged from 0.007 to 0.059, moderate engagement from 0.059 to 0.174, high engagement from 0.174 to 1.00. Scores equal to 0 and greater than 1 were placed in their own distinct groups. Box 1 provides two contrasting examples of computed engagement scores, category assignment, and face validity check based upon the participant interviews.

Figure 1. Caregiver Engagement per Day.

Note. Engagement is defined as the degree to which individuals interact in the group and use and respond to the content provided in the group.

Table 2 summarizes the participants in total and by engagement. Men were a greater percentage of the total caregivers in the “no” and “low” engagement groups at 30% and 22%, respectively, when compared to their presence in the “moderate” and “high” engagement groups (10% and 12%, respectively). Age did not seem to vary across the groups, with mean age ranging from 53 years in the no engagement group to 56 years in the low engagement group. While their numbers in the sample were relatively small, Black or African American caregivers appeared in greater percentages in the “moderate” and “high” engagement groups, at 17% and 18%, respectively, compared to 8% each for “no” and “low” engagement. Marital status did not seem to make a difference with roughly equal frequency distribution in each group from married, non-married or other, and separated/divorced participants. Education also had relatively similar frequency distributions from all education groups across all levels of engagement. Employment was interestingly different between the no engagement and very high engagement groups. Seventy one percent of those in the no engagement group were employed and 29% were unemployed, and 71% of those in the very high engagement group were unemployed and 29% were employed. Adult children had the highest proportion of each engagement group, but this share decreased as engagement got higher, from 58% in the no engagement group to 29% in the very high engagement group. Those caregivers who lived with the patient also had higher percentages in the “moderate” and “high” engagement groups at 59% and 68% compared to 46% and 45% for “no” and “low” engagement groups.

Table 2.

Summary of Participant Demographics and Engagement

	Overall (N=170)	No engagement (N=24)	Low engagement (N=48)	Moderate engagement (N=45)	High engagement (N=46)	Very high engagement (N=7)
Gender
Female	140 (83%)	16 (70%)	37 (77%)	40 (89%)	41 (89%)	6 (86%)
Male	29 (17%)	7 (30%)	11 (23%)	5 (11%)	5 (11%)	1 (14%)
Age (years)
Mean (SD)	54.6 (12.5)	53.0 (12.6)	56.0 (12.2)	54.5 (12.0)	54.1 (13.3)	54.3 (14.0)
Median [Min, Max]	56.0 [22.0, 82.0]	58.0 [22.0, 68.0]	56.0 [26.0, 78.0]	56.0 [26.0, 74.0]	55.0 [31.0, 82.0]	62.0 [39.0, 70.0]
Race
Black/AA	23 (14%)	2 (8%)	4 (8%)	8 (18%)	8 (18%)	1 (14%)
Other	4 (2%)	1 (4%)	0 (0%)	1 (2%)	1 (2%)	1 (14%)
White/Caucasian	142 (84%)	21 (88%)	44 (92%)	36 (80%)	36 (80%)	5 (71%)
Marital Status
Married	122 (72%)	17 (71%)	33 (69%)	31 (70%)	35 (76%)	6 (86%)
Never Married/Widowed/Other	36 (21%)	6 (25%)	10 (21%)	12 (27%)	7 (15%)	1 (14%)
Separated/Divorced	11 (7%)	1 (4%)	5 (10%)	1 (2%)	4 (9%)	0 (0%)
Education
Graduate Degree	26 (15%)	6 (26%)	6 (12%)	8 (18%)	5 (11%)	1 (14%)
High School/GED	33 (20%)	4 (17%)	13 (27%)	9 (20%)	6 (13%)	1 (14%)
Some College/trade school	71 (42%)	8 (35%)	17 (35%)	16 (36%)	27 (59%)	3 (43%)
Undergraduate degree	33 (20%)	5 (22%)	11 (23%)	9 (20%)	7 (15%)	1 (14%)
< High School	5 (3%)	0 (0%)	1 (2%)	2 (5%)	1 (2%)	1 (14%)
Employment Status
Employed	100 (60%)	17 (71%)	26 (57%)	26 (58%)	29 (63%)	2 (29%)
Unemployed	68 (40%)	7 (29%)	20 (43%)	19 (42%)	17 (37%)	5 (71%)
Relationship to Patient
Adult child	75 (44%)	14 (58%)	23 (48%)	17 (39%)	19 (41%)	2 (29%)
Other	50 (30%)	4 (17%)	18 (38%)	13 (30%)	12 (26%)	3 (43%)
Spouse/partner	44 (26%)	6 (25%)	7 (15%)	14 (32%)	15 (33%)	2 (29%)
Lives with Patient
No	74 (45%)	13 (54%)	27 (56%)	18 (42%)	12 (27%)	4 (57%)
Yes	92 (55%)	11 (46%)	21 (44%)	25 (58%)	32 (73%)	3 (43%)
Days in the Study
Mean (SD)	76.6 (37.5)	76.9 (51.8)	82.2 (31.4)	78.8 (35.2)	68.9 (37.2)	73.1 (36.8)
Median [Min, Max]	72.0 [0, 188]	69.0 [0, 188]	80.0 [32.0, 142]	75.0 [32.0, 167]	67.5 [1.00, 151]	66.0 [35.0, 128]

Discussion

Having a way to formally assess and quantify engagement for interventions that use social media platforms is critical in the evaluation of the intervention process and outcomes. When thinking of the efficacy of an online support group intervention we need to understand not only the effect of the group on the outcome but also the effect of the involvement in the group, or engagement. Our literature review found no other online support groups that had a method, other than subject self-report, of calculating individual Facebook engagement scores. In our own clinical trial, the engagement score we generated will be a useful statistic as we move toward our final analysis and consider the outcomes of the intervention; however, its computation was very time consuming.

The Facebook platform allows posts and comments to be identified in private groups if participants provide permission within their Facebook settings. Agreeing to allow researchers to track identifiable posts and comments, however, requires participants to also agree to give researchers access to entire personal Facebook accounts. For ethical reasons, our team did not ask participants to give our research team this level of access. This decision limited our access to participant data but further protected participant privacy.

Creating a valid and reliable process for developing an engagement score was subject to many limitations. It is important to note that the changes made to the Facebook privacy policy came during the first year of our five-year study. Thus, we were not prepared for the change. As there was no way to anticipate this problem, we had to adapt as best we could.

The initial challenge was developing a database with which we could calculate an engagement score. The process was time-intensive and tedious, and the potential for miscounting was great. We did not have a method to conduct complete reliability checks because once a participant left the group, depending on their individual privacy settings, their data would disappear. While we downloaded data weekly, it was often significantly longer before staff could add identifiers to the downloaded data. While some data were entered by two staff members working together rather than one individual, this was not consistent, as these activities were add-ons to our originally proposed work. Additionally, “read and view” data are not able to be downloaded, thus we are unable to quantify the number of times data was read and/or seen by individuals unless they posted, commented, reacted to or liked a post. This is a limitation to the data and we had no way to control for it. We share this issue in hopes that other Facebook researchers can plan for unexpected data losses during their design phase.

It is not the purpose of this paper to analyze the impact engagement may have on study outcomes but rather to emphasize the challenges inherent in creating an engagement score and to share our process to facilitate future research. Engagement’s impact on outcomes is an important question for future research. We would anticipate that the counted engagement may not be related to the outcome, as we had individuals note they had informational benefit from the group but did not personally make a post or comment.

Another critical question about engagement is the role that age and digital literacy may play in engagement of older adults in Facebook interventions. A recent study found that older adults use Facebook more often, but report lower digital literacy toward social media in general and Facebook in particular (Parker Oliver et al., in press). The ability to measure engagement is necessary before we can explore its relationship to either outcomes, digital literacy, or older age.

The differences between “no engagement” and any engagement are important characteristics to measure among users of social media. In the first case, participants with no engagement are just watching, or “lurking,” as they read posts made by others. While there is a noted “read” on individual posts, it means that the users scrolled past the post; there is no guarantee that they actually read the material. Furthermore, this “read” designation cannot be downloaded for easy calculation. It can only be collected by actually reviewing the Facebook page, and even then it will be missed if a participant reads (or scrolls past) the post after the researcher collects the data. Therefore, these “reads” could not be counted, and participants who may have read something and not posted anything are counted as not engaging with the material. We have defined engagement as active interaction including an effort to respond in some way to another individual, of acknowledging that someone is heard, which is more than just reading the words of other participants. Another important question for future research that we could not answer is, “What are the benefits of online support groups to those who do not interact with the group but do read the interactions of others?”

It is important that we continue to refine our process and search for ways to automate the computation of engagement in a more efficient way. We employed only one strategy, and it was a time-intensive one. Additionally, it is important that we come to understand the relative value of the different components of engagement (posts, comments, likes, and other reactions), not only to the individual doing the posting, but also to the recipient of the action. Communication within a social media platform is unique and requires new tools for data collection, measurement, and analysis.

Finally, the ethical issues involved in using social media data for research continue, and the solutions are slow to follow. The Cambridge Analytica case demonstrated the importance of individual privacy, yet the swift answer to make privacy settings all-or-none has limited individuals’ control over how private they can make their data. We suggest that individuals be able to adjust their privacy setting dependent upon the group they are involved with. Allowing access to a specific group, however, should not allow access to a person’s entire Facebook profile. As social media continues to grow and online nursing interventions become more common, these measurement and privacy issues will continue to emerge and cause pause for discussion.