Feasibility, Acceptability, and Potential Utility of Peer-supported Ecological Momentary Assessment Among People with Serious Mental Illness: a Pilot Study

Abstract

To examine the feasibility, acceptability, and initial validity of using smartphone-based peer-supported ecological momentary assessment (EMA) as a tool to assess loneliness and functioning among adults with a serious mental illness diagnosis. Twenty-one adults with a diagnosis of a serious mental illness (i.e., schizophrenia, schizoaffective disorder, bipolar disorder, or treatment-refractory major depressive disorder) and at least one medical comorbidity (i.e., cardiovascular disease, obesity, diabetes, chronic obstructive pulmonary disease, hypertension, and/or high cholesterol) aged 18 years and older completed EMA surveys via smartphones once per day for 12-weeks. Nine peer support specialists prompted patients with SMI to complete the EMA surveys. Data were collected at baseline and 12-weeks. EMA acceptability (15.9%) was reported, and participants rated their experience with EMA methods positively. EMA responses were correlated with higher social support at 3 months. Higher levels of EMA-measured loneliness were significantly correlated with levels of social support, less hope, and less empowerment at 3 months. Lastly, those who contacted their peer specialist reported higher levels of loneliness and lower levels of functioning on that day suggesting that participants were able to use their peers for social support. Peer-supported EMA via smartphones is a feasible and acceptable data collection method among adults with SMI and appears to be a promising mobile tool to assess loneliness and functioning. These preliminary findings indicate EMA-measured loneliness and functioning are significantly predicted by baseline variables and such variables may impact engagement in EMA. EMA may contribute to future research examining the clinical utility of peer support specialists to alleviate feelings of loneliness and improve functioning.

People with a diagnosis of a serious mental illness (SMI; defined as individuals diagnosed with schizophrenia spectrum disorder, bipolar disorder, or treatment-refractory major depressive disorder) utilize peer support services to maintain their health and recovery. Peer support is defined as shared knowledge, experience, emotional, social, and/or practical assistance to support others with similar lived experiences [1]. Mental health peer support is classified by the World Health Organization as an essential element of recovery [1] and can augment traditional mental health treatment through providing support services to maintain recovery between clinical encounters [2]. Despite these benefits, a consequence of the COVID-19 restrictions has meant that people with SMI have less access to in-person peer support services [3]. Beyond online surveys, it is unclear how adults with SMI are impacted by COVID-19, who may need the most support, and which modifiable factors might be targeted by peer support specialists to enhance overall health and recovery and reduce costly hospitalizations and early nursing home placement.

Researchers have employed digital phenotyping (i.e., defined as the in situ data collection of people’s phenotypes using digital devices) to address these challenges [4]. While promising evidence exists for digital phenotyping with people with SMI, by contrast, evidence indicates people with SMI have reported concern engaging with passive monitoring technologies and prefer active monitoring [5]. Active monitoring such as ecological momentary assessment (EMA) may be a more acceptable (thus, higher likelihood of uptake) approach for the measurement of living in real-world settings, and potentially also allow for early intervention. EMA is a daily diary method that involves asking people to record experiences, thoughts, and behaviors in real-time and real-world settings. EMA is easily collected on a smartphone device and individuals with SMI report high levels of satisfaction and engagement with this methodology to measure and record their mental health [6]. EMA may assist with identifying a novel mechanism of action associated with hospitalizations and early nursing home placement that could greatly enhance the clinical utility of peer support specialists to offer just-in-time interventions.

Prior studies have found strong support for the feasibility and application of EMA in people with SMI [7–9] yet some studies are impacted by low rates of engagement (e.g., 28–31%, [10]), limiting the potential utility of this tool. To date, most telehealth and other mobile technologies for people with mental health conditions are impacted with the lowest levels of patient engagement [11]. In fact, patients with SMIs have the lowest levels of engagement with health technology and the highest healthcare costs of any population [12] and are, therefore, especially vulnerable to falling ill due to COVID-19. Reciprocal accountability (i.e., mutually learning and helping each other) [13], supportive accountability (i.e., adherence to expectations and goals due to the support of a trustworthy coach) [14], and object-relations (i.e., smartphone as a transitional object to feel comforted and connected to others) [15] all suggest the integration of human connection in digital health technologies is necessary to develop a therapeutic alliance and promote engagement.

Digital peer support interventions for people with SMI use a variety of technology modalities (i.e., smartphones, social media, videogames, video conferencing) and have shown to be a promising human factor to increase engagement in digital mental health services [16–18]. Digital peer support is defined as live or automated peer support services delivered through any technology medium (e.g., videogames, videoconferences, smartphone applications, cell phones, virtual reality) [19]. A recent systematic review examined features of apps and their impact on engagement [20] and found that live (not automatic or artificial) peer support had the highest engagement (17%) compared to other features (i.e., trackers = 6.3%; mindfulness/ mediation = 4.1%; breathing exercises = 1.6%; and psychoeducation = 3%) [20]. Thus, the integration of peer-supported EMA may have potential clinical utility.

The current study is part of an iterative design process to develop and test “PeerTECH,” a digital peer support self-management intervention that included peer-supported EMA functionality.

PeerTECH Intervention

“PeerTECH” is an adaptation of a 12-month clinician-delivered evidence-based intervention–Integrated Illness Management and Recovery (I-IMR) [16]. A randomized trial of I-IMR among adults with SMI found increased medical and psychiatric self-management skills and decreased hospitalizations compared to usual care [16, 21]. In developing PeerTECH, we employed the Peer and Academic partnership framework [22] and partnered with peer support specialists through intervention development. This partnership conducted a series of clinic-based usability tests to iteratively refine the intervention [22, 23] and align PeerTECH with evidence-based design principles for people with SMI [23, 24]. As described below, PeerTECH is a mobile technology platform designed to facilitate the delivery of evidence-based principles that have been shown to promote self-management in people with SMI (i.e., coping skills training, psychoeducation, medical management, relapse prevention planning, healthy behaviors, and peer support) [25, 26]. Previous pilot studies found that PeerTECH (N = 8) was delivered with high fidelity by peer support specialists [27], and associated with empirical improvements in medical and psychiatric self-management and self-efficacy to manage chronic disease among both peer support specialists and patients [28]. The first PeerTECH study led to the following intervention modifications: enhancement of peer support specialists PeerTECH training; non-interventionists’ fidelity rating scale; inclusion of participants aged 18 + years (instead of aged 50 years +); intervention content on social health (i.e., developing relationships and addressing feelings of loneliness); an electronic library guided PeerTECH sessions were available on the smartphone app (not on a separate website only accessible through a tablet).

PeerTECH Mobile Technology Platform

The PeerTECH mobile technology platform includes a smartphone application and a peer support specialists’ care management dashboard. The smartphone application is designed for patients to reinforce skills learned from in-person sessions with a peer support specialist. The smartphone application includes: (a) access to personalized self-management support; (b) intervention components that correspond to patients’ needs and goals (see Fig. 1); (c) a HIPAA-compliant chat feature for use between peer support specialists’ care management dashboard and patients’ smartphone application; (d) an on-demand library of peer-led self-management narrative videos (see Fig. 1); (e) EMA surveys that can be set to any time/date. The PeerTECH library includes classes designed to be reviewed together by a peer support specialist and patient on a smartphone during one-hour, weekly, in-person, or telephone-based classes (see Table 1). Each library class includes peer-led videos and guiding materials to discuss the interconnection between mental health, physical health, and social health, the role of stress in the development of or the worsening of mental health and physical health conditions, coping skills training, and unscripted lived experiences of self-management challenges and successes. Library features can be accessed offline and can also be accessed multiple times throughout the 12-week PeerTECH intervention.

Fig. 1.

Screenshots from the Library and Goals pages of the PeerTECH application

Table 1.

PeerTECH Library

Class #1: Introductions, Smartphone Orientation, and Recovery and Health

Class #2: Good Mental Health Starts with Good Physical Health and Social Health (vice versus)

Class #3: Recovery is a Daily Process

Class #4: How Stress Impacts Our Health

Class #5: Smoking and Living a Healthy Lifestyle

Class #6: Healthy Sleep

Class #7: Developing and Maintaining Relationships

Class #8: Dental Health

Class #9: Exercise

Class #10: Getting the Help You Want from Communities and the Physical   Healthcare and Mental Health System

Peer Care Management Dashboard

The peer care management dashboard is stored on a secure website that monitors patients’ personalized recovery goals, their personalized wellness plan, the chat between peer support specialists and patients, and survey responses. High levels of loneliness as determined as a score of 3 incited a push notification to a peer support specialist to check in on patients. Peer support specialists sign in securely to the dashboard on a desktop or laptop to send secure, HIPAA-compliant mobile messaging to the smartphone application. Dashboard data is managed by a peer support specialist and monitored by the PI (blinded for review) (see Fig. 2).

Fig. 2.

The PeerTECH EMA System

Methods

A single-arm pre-/post- pilot study was conducted in collaboration with a community mental health center that provides care management, coordination of services, and referrals for adults with SMI, aged 18 years and older. Peer support specialists provided PeerTECH within a community setting (e.g., outdoor park), the participant’s home, and or virtually (via telephone) four times per month (over a 12-week period) and text messaged participants a minimum of three times per week throughout the study. First, “paper-and-pencil” (delivered during in-person sessions rather than through the PeerTECH platform) baseline assessments were conducted by a trained rater at the participant’s home or in the community mental health center. These assessments covered typical peer support intervention targets including hope, loneliness, empowerment, social support, psychiatric and medical self-management, and self-efficacy. Over the course of the study, participants were asked to complete EMA surveys (to capture loneliness and general functioning) that were sent daily at 10am EST to their smartphones. Finally, instruments measuring the same variables as the baseline assessment were delivered at the 12-week mark (conclusion of the intervention) and were conducted over the telephone due to COVID-19. This study was approved by [blinded for review] Institutional Review Board.

Participants

The pilot study included N = 30 adults aged 18 years and older with SMI and one medical comorbidity. Eligibility for patient study participation included the following: (1) community-dwelling adult; (2) aged 18 + ; (3) diagnosis of schizophrenia, schizoaffective disorder, bipolar disorder, or major depressive disorder; (4) at least one medical condition defined as cardiovascular disease, obesity (defined as a body mass index of 30 and over), diabetes, chronic obstructive pulmonary disease, chronic pain, hypertension, high cholesterol, or current tobacco use identified through chart review; (5) ability to speak and read English; (6) provide voluntary informed consent; and (7) qualify for Medicaid reimbursements. Study participants were excluded based on the following criteria: (1) major visual or motor impairment as evidenced by turning a smartphone on and reporting they cannot clearly see the screen; (2) a chart diagnosis of dementia, or evidence of significant cognitive impairment as indicated by a Mini Mental Status Examination [29] score of less than 24. Peer support specialists were recruited from a single community mental health site; eligibility included: (1) trained and accredited certified peer support specialist in the state of [blinded for review]; (2) aged 18 + ; (3) ability to speak and read English; (4) willingness to provide voluntary informed consent; (5) currently in recovery as self-reported by peer support specialist; and (6) employed at community mental health center research site.

Sociodemographic Characteristics of the Study Sample

Descriptive statistics were conducted to describe the sociodemographic characteristics of the study sample. Descriptive statistics and analyses were computed using SPSS version 24. A total of 30 patient participants completed baseline and 21 patient participants completed the follow-up assessment. The demographics and clinical characteristics for those 21 participants who completed 1 EMA response are presented in Table 2. Participants were white (100%), had been married (n = 7, 33%), completed high school/GED (n = 10, 47.6%), lived independently (n = 9, 42.9%) and were unemployed (n = 16, 76.2%). Participants had a primary mental health diagnosis of: schizophrenia spectrum disorder (n = 6, 29%), bipolar disorder (n = 9, 43%), major depressive disorder (n = 3, 14%), and “other” including posttraumatic stress disorder, obsessive–compulsive disorder, attention-deficit/hyperactivity disorder, anxiety, anorexia nervosa, and autism spectrum disorder (n = 16,76%). Multiple participants had co-morbid conditions; hence the number of diagnoses exceeds the number of participants. Fourteen participants reported they were smartphone owners and had used a smartphone before participating in this study. Among the 21 participants assessed at the follow-up visit, 66.6% were female, with a mean age of 37.3 (SD = 9.16).

Table 2.

Demographic variables

Demographic
	Mean	SD
Age	37.3	9.16
	N	%
Gender: Male	7	33%
Race: White	21	100%
Ethnicity: Hispanic	3	13%
Marital status: Ever married	7	33%
Highest education
Some elementary school	1	4.8%
Some high school	4	19%
Completed high school	10	47.6%
Complete some college	5	23.8%
Completed associates degree	1	4.8%
Housing status
Homeless	2	9.5%
Supervised facility	7	33.3%
Supervised non-facility	2	9.5%
Supported	1	4.8%
Independent	9	42.9%
Work status
Not working	16	76.2%
Volunteer	1	4.8%
Part-time	3	14.3%
Full-time	1	4.8%
Mental health diagnosis
Schizophrenia	2	10%
Schizoaffective disorder	4	19%
Bipolar disorder	9	43%
Major depressive disorder	3	14%
Other (PTSD (N = 6), OCD (N = 2), ADHD    (N = 2), anxiety (N = 4), anorexia nervosa    (N = 1), autism spectrum disorder (N = 3))	16	76%
Physical health diagnosis
Diabetes	8	38%
Heart disease	4	19%
Obesity	9	43%
High blood pressure	10	48%
High cholesterol	9	43%
Osteoporosis	2	10%
Fibromyalgia	2	10%
GERD	7	33%
Osteoarthritis	5	24%
COPD	3	14%
Congestive heart failure	2	10%
Coronary artery disease	3	14%
Chronic pain	13	62%
Other (asthma, lupus, seizures, migraines,    osteopenia, superior canal dehiscence    disease)	9	43%
Smartphone ownership (% yes)	14	67%

Nine patient participants were lost to follow-up. Of these nine, seven participants did not respond to repeated telephone calls to schedule the 12-week interview. One individual decided after hearing about the study and completing the informed consent that they were not interested in participating in the study; the other met with a peer one time and decided they did not want to work with their assigned peer support specialist and no longer wanted to be involved with the study. The remaining twenty-one participants completed the PeerTECH intervention. The main intervention results can be found in Fortuna et al. [30], which reported a significant increase in empowerment and IMRS scores and a trend-level increase in self-efficacy from baseline to post-intervention.

Fifteen participants owned a smartphone and used their own smartphone and data plan for the study. Of the six participants using loaned smartphones, the smartphone was configured only to deliver the intervention and not for personal use. One smartphone broke during the study and was returned to the PI. Eighteen people completed at least one EMA response. Out of all the EMA reports, 19% (N = 69) were in December, 31% (N = 112) were in January, 24% (N = 87) in February, and 26% (N = 92) in March. Completing a class on that day or receiving a text from the peer was not associated with an increased likelihood of completing EMA surveys, respectively (P = 0.14, P = 0.95). The intervention was over 12 weeks and participants completed up to 113 days of EMA data collection; actual responses ranged from 1–88 days. Seven participants completed at least 20% of possible responses in the 12-week intervention.

Procedures

Patient Recruitment

The PI met with the clinical team leader to discuss the purpose of the study and the recruitment process. The clinical team leader was a licensed clinical social worker who reviewed current patient caseloads along with other peer support specialists. Together, they identified potential participants that met inclusion criteria and telephoned the potential participants to speak with the individual about the study. The clinical team leader read a scripted one-page summary of the study over the telephone to the potential participant. If they were interested in the study, they verbally agreed to meet with a trained rater and the peer support specialist in a location of their choosing.

Patient Informed Consent

During the scheduled in-person baseline data collection meeting, potential participants were provided a description of the study, shown the PeerTECH smartphone application, and informed their information was confidential and that their participation in the study was voluntary. Potential participants were evaluated for study criteria. If the participant met the criteria and provided informed consent to participate in the study, the trained rater completed the baseline assessments independently with the participant using REDCap in a private room within the community mental health center or the participant’s home. Thereafter, the peer support specialist scheduled with the participant. Participants were loaned a ZTE Blade Vantage 2 Prepaid Android phone and 12-week data plan (at no cost to the participant).

Peer Support Recruitment

The peer support specialists’ supervisor at the research site identified peer support specialists to be trained to deliver PeerTECH. The peer support specialists’ supervisor assessed employees’ interests in this study and made recommendations to the PI. All interested peer support specialists were trained in PeerTECH. All peers previously completed the [blinded for review] certified peer support specialists training in order to work as a certified peer support specialist at the community mental health center. Peer specialists were certified by the state of [blinded for review]. This state-specific certification training takes place over 10 weeks and includes six day-long trainings and a three-day retreat [31]. Participants included nine peer support specialists between the ages of 25 and 54 years (mean 39 years) who were employed as peer support specialists for 1 to 11 years (mean 4.25 years) and had access to a work-funded smartphone device and data plan. Peer support specialists’ workload did increase during the PeerTECH study from 30 to 40 h. As peer support specialists are employees, a record of their mental health diagnosis was not requested.

Peer Support Specialists Informed Consent

During peer support specialist training, potential peer participants were provided a description of the study, shown the PeerTECH smartphone application and care management dashboard, and informed their information was confidential and that their participation in the study was voluntary. If they met the criteria they were invited to complete an informed consent to participate in the study.

PeerTECH Training

Once peer support specialists indicated their interest in the study, they completed the PeerTECH training. The PeerTECH training included 16 h of in-person training over two consecutive days. PeerTECH training included: (1) the importance of addressing both physical health, mental health, and social health; (2) integration of recovery within medical challenges; (3) techniques used in PeerTECH (i.e., psychoeducation, coping skills training, peer support); (4) defining personally meaningful, achievable goals and actions steps with the participant; (5) delivering PeerTECH sessions using the smartphone; (6) the structure of the weekly session and between session text messaging between peers and participants; (7) teaching others how to use technology; (8) maintaining engagement; (9) sharing lived experience intentionally to teach self-management concepts; and (10) experiential training using the smartphone application and the peer care management dashboard. All peers also completed the Digital Peer Support Certification [32] throughout the course of the study. The Digital Peer Support Certification is a 12-week training led by the PI that includes two education and simulation training sessions and ongoing synchronous and asynchronous support services and audit and feedback. Training focuses on digital communication skills; technology literacy; technology usage skills with the PeerTECH system (e.g., downloading apps, sending SMS text messages, entering goals, saving information, increasing the volume on a smartphone, watching videos in the library, and offering digital peer support services); available digital peer support technologies; organizational policies and compliance issues; separating work and personal life; digital crisis intervention; and privacy and confidentiality. Peer support specialist participants were given an informational manual on how to use the smartphone and complete the surveys.

Peer Supervision

Peer support specialists individually met in-person or over the telephone with a peer supervisor (also a peer support specialist and a trained supervisor) once a week for one hour. Discussions centered on concerns working with participants as part of PeerTECH and problems with PeerTECH technology. Peer supervision revealed if the peer needed additional technical support regarding PeerTECH or if participants needed extra services or technical assistance with the PeerTECH smartphone application.

Instruments

EMA Surveys

The EMA survey captured data about (1) loneliness using the UCLA-3-item [33] measure and (2) functioning using the PROMIS–Global Health Scale [34] (see Table 3). Both of which are associated with health behaviors correlated with hospitalizations and early nursing home placement. Measures classified as EMA methodology have been found to be reliable and valid [35].

Table 3.

EMA Instruments

UCLA 3-Item Loneliness*

1. First, how often do you feel that you lack companionship? 2. How often do you feel left out? 3. How often do you feel isolated from others?

PROMIS-Global Health Scale**

1. In general, would you say your health is: 2. In general, would you say your quality of life is: 3. In general, how would you rate your physical health? 4. In general, how would you rate your mental health, including your mood and your ability to think? 5. In general, how would you rate your satisfaction with your social activities and relationships? 6. To what extent are you able to carry out your everyday physical activities such as walking, climbing stairs, carrying groceries, or moving a chair? 7. How would you rate your pain on average? 8. In the past 7 days, how would you rate your fatigue on average? 9. In general, please rate how well you carry out your usual social activities and roles. (This includes activities at home, at work and in your community, and responsibilities as a parent, child, spouse, employee, friend, etc.) 10. In the past 7 days, how often have you been bothered by emotional problems such as feeling anxious, depressed or irritable?

*Response options included 1 = Hardly ever, 2 = Some of the time, 3 = Often; **Response options included 5 = Excellent, 4 = Very Good, 3 = Good, 2 = Fair, 1 = Poor (for questions 1–5,9); 5 = Completely, 4 = Mostly, 3 = Moderately, 2 = A Little, 1 = Not At All (for question 6); 5 = Never, 4 = Rarely, 3 = Sometimes, 2 = Often, 1 = Always (for questions 8,10); and a scale of 0 (No Pain) to 10 (Worst Imaginable Pain) for question 7

EMA Protocol/Procedures

Participants were loaned a ZTE Blade Vantage 2 Prepaid Android phone and a 12-week limited data plan (at no cost to the participant). Fifteen participants (50%) reported they were smartphone owners and had used a smartphone before participating in this study. Peer support specialists were trained to teach service users how to complete EMA data (see peer training below).

Participants were asked to complete EMA-based surveys 1-time per day for 12 weeks, providing 118 data points per person (participants continued to provide data even after the 12 weeks). This frequency of assessment was determined by feedback from peer support specialists and service users. The timing of the surveys was set to 10am EST as determined by peer partners. Time-stamped, de-identified and encrypted responses were automatically transferred to a password-protected server. After the completion of the EMA assessment period, participants returned the smartphones and completed a follow-up interview regarding their experience carrying and operating the smartphone and completed the following assessments with a trained interviewer.

Study instruments were administered in-person at baseline and over the telephone at 12-week time intervals by a trained rater. Self-report data were entered into REDCap by the trained rater. Instruments were selected to reflect peer support intervention targets described in the research literature (i.e., hope, empowerment, social support [36–38]), and mechanisms to promote engagement in medical and psychiatric self-management behaviors (i.e., self-efficacy [39], loneliness [40]).

To measure hope, the trained rater administered the 12-item Herth Hope Index (HHI) [41], which has shown reliability and validity in medically complex nursing home patients [42] and individuals with cognitive impairments [43]. Sample questions include the following, “I feel all alone” and “I have short and/or long-range goals”. Response choices include the following, “strongly disagree”, “disagree”, “agree”, and “strongly agree”. Scores on the HHI range from 12–48 (note: higher scores indicate a higher self-reported level of hope).

To measure feelings of loneliness, the trained rater administered the 20-item UCLA Loneliness Scale [33]. Each of the 20 items was self-rated on a four-point scale, with higher values indicating greater feelings of loneliness (i.e., 1 = never; 2 = rarely; 3 = sometimes; 4 = always). Sample items include “how often individuals felt left out” or “isolated from others”, and how often they “felt that there are people that really understand them or that they can talk to”. Consistent with scoring, items are summed, giving a total score ranging from 20 to 80 [33]. The original UCLA Loneliness Scale demonstrates good validity and reliability [44, 45].

To measure empowerment, the trained rater administered the Empowerment Scale [46], which is a widely used valid, reliable 28-item instrument that measures personal empowerment [47, 48]. Sample questions include the following, “I can pretty much determine what will happen in my life” and “people are only limited by what they think is possible.” Response options include the following, “strongly agree”, “agree”, “disagree”, and “strongly disagree”. Consistent with scoring, scores were aggregated and averaged, in which lower scores indicated higher levels of empowerment. Total scores range from one to four.

The Medical Outcomes Study (MOS) Social Support Survey instrument [49] is a valid, reliable 19-item instrument [50] that examines different domains of social support (i.e., emotional/informational, tangible, affectionate, and social interaction). The trained rater asked participants how often each type of social support was available to them. Response options include the following, “none of the time”, “a little of the time”, “some of the time”, “most of the time”, and “all of the time”. Each domain’s score was averaged across and aggregated. Scores range from 0–100, in which higher scores indicate higher levels of social support.

Psychiatric self-management skill development was assessed by administering the Illness Management and Recovery Scale (IMRS) [51]. The IMRS is a valid, reliable 15-item instrument that examines domains of illness self-management [52, 53]. An example item reads, “how much do you know about symptoms, treatment, coping strategies (coping methods), and medication”. Response options include the following, “not very much”, “a little”, “some”, “quite a bit” and “a great deal”. Scores range from 15–75. Higher scores indicated higher levels of psychiatric self-management skills.

Medical self-management skill development was assessed using the Self-Rated Abilities for Health Practices Scale (SRAHPS) [54]. SRAHPS is a 28-item instrument that examines confidence to implement health practices [54]. SRAHPS has demonstrated reliability and validity with adults with disabilities [54]. SRAHPS includes four subscales with seven items each. Subscales include ye following health practices: (1) exercise, (2) nutrition, (3) responsible health practice, and (4) psychological well-being. The trained rater asked participants to rate the extent to which they are able to execute health practices in each of the domains. An example item reads, “I am able to get help from others when I need it.” Each item is rated on a four-point scale (i.e., zero [not at all] to four [completely]). Subscale ratings were summed to produce subscale scores, and then, totaled to obtain an overall score. Participants could score between zero-112 points. Higher scores indicated higher levels of medical self-management skills.

A trained rater implemented the Self-Efficacy for Managing Chronic Disease Scale (SEMCD) [39] to measure self-efficacy. SEMCD is a six-item scale that examines the following domains: symptom control, role function, emotional functioning, and communicating with physicians. SEMCD has established reliability and validity in people with chronic physical health conditions [39, 55]. Participants answer each item on a one-10 point scale (i.e., one = not confident at all to 10 totally confident), and the final SEMCD score is the average of the six items. Scores can range from one to ten. Higher scores indicated higher self-efficacy.

Statistical Analyses

Mann–Whitney U Tests were used to compare those who adhered to the EMA assessments (e.g., completed at least 20% of responses) to those who did not adhere to EMA on the seven outcome measures. Spearman correlations were used to examine the relationship between mean EMA loneliness or mean EMA functioning with outcome variables at baseline and post-intervention using the HHI, empowerment, MOS social support, IMRS, SRAHPS, and SEMCD scales. Finally, independent t-tests were conducted to examine whether text messages to peers from the participant, or vice versa, on that day were associated with EMA reported loneliness and functioning, including each survey as a new timepoint.

Results

Feasibility and Acceptability

In total, participants sent 72 text messages while peers sent 134 messages. Results demonstrated an EMA average acceptability per person rate of 15.9% (i.e., completing approximately 18 out of 113 surveys; SD = 24.70 surveys; range = 1–88 surveys). Results also demonstrated an EMA average acceptability per person rate of 15.9% (i.e., completing approximately 18 out of 113 surveys; SD = 24.70 surveys; range = 1–88 surveys). Overall, 11 participants completed 0 surveys, 10 participants completed 1–5% of the surveys, 1 participant completed 5–10% of the surveys, no participants completed 10–15% or 15–20%, and 7 participants completed 20% or more of the surveys. No participants reported difficulty responding to EMA surveys or difficulty understanding the questions. No participants reported difficulty responding to EMA surveys or difficulty understanding the questions.

Predictors of EMA adherence

Those who completed > 20% of EMA responses (N = 7) were compared to those who did not (N = 22) on the seven baseline or 3-month assessments (HHI, UCLA, empowerment scale, MOS social support, IMRS, SRAHPS, and SEMCD). The relationship between EMA assessments and higher social support at 3 months (M = 79.22, SD = 17.55) was trending towards significant as compared to those who did not adhere to assessments (M = 59.60, SD = 24.80), U(N _<20% = 15, N_>20% = 6), 23.00, Z = -1.72, P = 095. All other measures were non-significant (P > .05).

Clinical Correlates of EMA Loneliness

For all participants who provided any EMA data, higher mean EMA-measured loneliness was significantly associated with lower levels of social support at 3 months (rs = -.80, P = .002) and baseline IMRS (rs = .66, P = .026). In participants who provided > 20% of EMA data (N = 7), there was a significant negative correlation between mean EMA-measured loneliness and hope at 3 months (rs = -0.84, P = .04, N = 6), empowerment at 3 months (rs = .83, P = .04, N = 6), social support at 3 months (rs = -0.81, P = .05, N = 6).

We examined the relationships between paper-and-pencil and EMA-measured loneliness. In all participants who provided any EMA data, there was a significant negative correlation between mean EMA-measured loneliness and baseline paper-and-pencil loneliness (rs = -0.62, P = .04) and significant positive correlation with 3-month paper-and-pencil loneliness (rs = .79, P = .002). In participants who provided > 20% of EMA data, there was no significant correlation between mean EMA-measured loneliness and baseline paper-and-pencil loneliness (rs = -0.80, P = 0.2, N = 5), but there was a significant negative correlation with 3-month paper-and-pencil loneliness (rs = -0.83, P = 0.04, N = 6).

Participants who were contacted by their peers that day had significantly lower EMA-measured UCLA scores (M = 4.28, SD = 1.99), than those who were not contacted by peers (M = 5.82, SD = 2.27), t(358) = -2.84, P = .005. In contrast, participants who texted their peer on a given day had significantly higher EMA-measured UCLA score (M = 5.08, SD = 2.06), than those who did not contact their peer (M = 3.00, SD = 0.00), t(12.00) = 3.64, P = .003.

Clinical Correlates of EMA Functioning

In all participants who provided any EMA data, higher mean EMA-measured functioning was significantly associated with a higher hope score at 3 months (rs = .53, P = .07), higher social support score at baseline (rs = 0.83, P = 0.01), higher social support score at 3 months (rs = .87, P < .001), higher loneliness (UCLA) at 3 months (rs = .79, P = .002), and higher self-efficacy at 3 months (rs = 0.68, P = .02). In participants who provided > 20% of EMA data, there was a significant correlation between mean EMA-measured functioning and empowerment at 3 months (rs = -.83, P = .04, N = 6). As empowerment was negatively scored, this suggests higher mean EMA-measured functioning was associated with higher levels of empowerment.

Participants who were texted by their peers that day had significantly lower EMA-measured global functioning scores (M = 32.11, SD = 1.99) than those who were not contacted by peers (M = 34.77, SD = 6.63), t(21.84) = -2.51, P = .02. Similarly, those who texted their peer on a particular day had significantly lower EMA-measured global functioning scores (M = 30.46, SD = 4.67) compared to those who did not contact their peer that day (M = 34.20, SD = 2.10), t(17.52) = -2.57, P = .02.

Discussion

The pilot study demonstrated early, promising evidence of the feasibility, acceptability, and initial validity of using smartphone-based peer-supported EMA to assess loneliness and functioning among adults with a diagnosis of a SMI. Participants rated their experience with EMA methods positively, although EMA acceptability (15.9%) was lower than expected. EMA responses were correlated with higher levels of hope, social support, and self-efficacy. Lastly, results indicated significant correlations between mean EMA-measured loneliness with baseline paper-and-pencil loneliness and 3-month paper-and-pencil loneliness. Completing a class on that day or receiving a text from the peer was not associated with an increased likelihood of completing EMA surveys.

Application and implementation of smartphone-based peer-supported EMA can have important implications in taking a more holistic approach in measuring the needs of participants and better understanding their needs. In our study, among participants who provided any EMA data, higher mean EMA scores of loneliness scores were significantly correlated with lower levels of social support at 3 months. Participants that adhered to EMA assessments, at three months, reported trend levels of higher social support than those who did not adhere to the assessment. This indicates that having an EMA assessment tool integrated into a peer support-based intervention not only has the potential to be an effective way of measuring and adapting service delivery but may also serve as a contact point that may enhance the user’s perceived sense of social support from peers and their engagement in an intervention. Furthermore, those participants who contacted their peer specialist reported higher levels of loneliness and lower levels of functioning on that day suggesting that participants were able to use their peers for social support.

The implementation of peer-supported EMA was also correlated with participants’ sense of loneliness. Across participants who provided any EMA data, higher mean EMA scores of loneliness were significantly correlated with a lower sense of social support at 3 months. Furthermore, EMA loneliness scores across participants who provided > 20% of EMA data indicate a negative correlation with participants’ sense of hope, empowerment, and social support at 3 months. This finding is particularly important as this suggests state loneliness may constitute, or lead to, a sense of social disconnectedness, low levels of hope, and isolation. Across participants who provided any EMA data, EMA loneliness scores were significantly correlated at both baseline (negative correlation) and 3 months (positive correlation) with the paper-and-pencil loneliness scores [45], indicating that measuring loneliness via EMA could be an effective and reliable way of gauging and actively monitoring participants’ sense of future loneliness. Active measurement tools, such as the EMA, can provide user autonomy and enable peer support specialists to deliver customizable support, meet participants’ needs, monitor mental health, and prompt user autonomy.

With regards to the texting data, participants who reached out to their peers and those who were contacted by their peers that day had lower EMA-measured functioning scores. It might be that those with low scores reached out to/were contacted by their peers because they were the ones who required assistance the most on that particular day. In addition, participants who were contacted by a peer on a given day had lower EMA-measured loneliness scores while those who reached out had higher scores. It is possible that talking to a peer might have a therapeutic benefit by a) reducing subjective levels of loneliness when participants were contacted by their peers first and b) acting as a source of comfort when they were especially lonely.

Study findings should be interpreted with caution. Here we present several important limitations. First, this pilot study included a small sample size consistent with our principal objective of examining feasibility and acceptability. Yet, this study was not powered to detect examination of the validity of our method. For instance, considering that only 7 participants completed > 20% of the EMA assessments, the study can only comment on its emerging significance and results cannot be generalized. While higher levels of EMA adherence may be predictive of higher levels of social support, the cutoff of 20% is too little to be considered significant. Thus, more research is required to investigate whether recruitment based on levels of social support can facilitate adherence or be a potential intervention target. Moreover, the correlation between EMA loneliness and paper-and-pencil assessments was negative for baseline while positive for the 3-month follow-up. While one can argue that this might be because participants gained a better sense of their loneliness after completing the EMAs over time, such conclusions need to be interpreted with caution due to the small sample size. Second, we recruited certified peer specialists from one state. As peer certification training varies by state, we do not know if these findings generalize beyond the state of [blinded for review]. Future studies with a fully powered sample will allow us to explore nested designs to account for variation in intervention delivery. Third, the sample purposefully included a heterogeneous grouping of psychotic disorders and mood disorders to mimic real-world conditions in community mental health centers. As such, we do not expect that SMI diagnostic heterogeneity impacted results as prior EMA studies that include people with diverse SMI diagnoses found no differences by diagnosis with respect to feasibility and acceptability via EMA [56, 57]. However, the sample was racially and ethnically homogenous, limiting generalizability to racially and ethnically diverse individuals. Future studies could address this concern by including a more racially and ethnically heterogeneous sample and by implementing other languages on the PeerTECH platform (for EMA assessment and text messaging) to increase its accessibility and feasibility, and generalizability across populations.

Conclusion

The pilot study demonstrated promising evidence of the feasibility, acceptability, and initial validity of using smartphone-based peer-supported EMA to assess loneliness and functioning among adults with a diagnosis of a SMI. Our results indicated that peer-supported EMA via smartphones can be a feasible and acceptable data collection method among adults with SMI. EMA may be a promising tool to assess loneliness as well as functioning in people with SMI. These preliminary findings indicate that, for people with SMI, higher EMA-measured loneliness was associated with lower levels of social support, lower illness self-management, less hope, and less empowerment. Higher EMA-measured functioning was associated with higher hope, social support, loneliness, and empowerment. EMA may contribute to future research examining the clinical utility of peer support specialists in impacting feelings of loneliness and functioning.

Biographies

Karen L. Fortuna, PhD, LICSW

Dr. Karen L. Fortuna is an Assistant Professor of Psychiatry at Dartmouth College. Dr. Fortuna’s work focuses on increasing the lifespan of people with serious mental illness–in hope, patients can achieve older adulthood. She works in equal partnership with patient partners in co-producing and empirically testing digital peer support technologies to extend the lifespan of people with serious mental illness. Dr. Fortuna has over 85 peer-reviewed publications co-produced with patient partners. She has an NIMH K01 award and was recently awarded a PCORI convening meeting award to bring together individuals across the globe to develop the first-ever patient-centered research agenda to address early mortality in people with serious mental illness.

Abigail C. Wright, PhD

Dr. Abigail Wright is a Senior Research Lead at the Institute for Clinical and Economic Review in Boston, MA.

George Mois, LISW

George Mois is a Licensed Social Worker from the School of Social Work at the University of Georgia.

Amanda L. Myers, MPH

Amanda Myers is a Research Associate at The Heller School for Social Policy and Management at Brandeis University.

Arya Kadakia

Arya Kadakia is an undergraduate student at Dartmouth College and a research assistant at the Geisel School of Medicine.

Caroline Collins-Pisano

Caroline Collins-Pisano is an undergraduate student at Dartmouth College and a research assistant at the Geisel School of Medicine.

Author Contributions

All authors contributed equally in the production of this manuscript.

Funding

This study was supported by Dr. Fortuna’s NARSAD Young Investigator Grant from the Brain and Behavior Foundation (#26800), Alvin R. Tarlov & John E. Ware Jr. Doctoral Dissertation and Post- Doctoral Awards in Patient Reported Outcomes, and K01 award from the National Institute of Mental Health (K01MH117496).

Declarations

Ethics Approval

Approval was obtained from the IRB at Dartmouth College. The procedures used in this study adhere to the tenets of the Declaration of Helsinki.

Consent to Participate

Informed consent was obtained from all individual participants included in the study.

Consent to Publish

Participants signed informed consent regarding publishing their data.

Competing Interests

Dr. Fortuna offers consulting services through Social Wellness.