Feasibility and Acceptability of Smartphone Assessment in Older Adults with Cognitive and Emotional Difficulties

Abstract

Objectives

Ecological momentary assessment (EMA) has several advantages in clinical research yet little is known about the feasibility of collecting EMA data with mobile technologies in older adults, particularly those with emotional or cognitive difficulties. The aim of this feasibility study was to assess perceived acceptability, adherence rates, and reasons for non-adherence to smartphone-based EMA.

Method

At two sites, participants (n=103) aged 65 years or older with a DSM-IV-defined anxiety or depressive disorder and cognitive concerns responded three times daily to smartphone-based EMA questions assessing clinical outcomes for two 10-day periods. Quantitative and qualitative measures assessed acceptability, adherence, and reasons for non-adherence following both 10-day EMA periods.

Results

Participants were moderately satisfied with and comfortable using smartphone-based EMA. Overall, 76% of participants completed surveys on ≥10 of the 20 assessment days, and 70% of participants completed at least 30% of the total surveys. Reasons for non-adherence included technical (malfunction), logistical (competing demands), physiological (hearing difficulties), and cognitive (forgetting) issues.

Discussion

Smartphone-based EMA is feasible in older adults with cognitive and emotional difficulties. EMA tools should be responsive to the needs and preferences of participants to ensure adequate acceptability and adherence in this population. Our findings can inform the design, development, and implementation of mobile technologies in older adults in research and clinical contexts.

The use of mobile technology for the assessment, treatment, and management of medical and psychiatric conditions is an issue receiving increased attention in clinical and health services research (Mohr, Burns, Schueller, Clarke, & Klinkman, 2013; Ben-Zeev et al., 2014; Marsch, Carroll, & Kiluk, 2014). Clinical researchers have begun to harness the advantages of smartphones and wearable sensors to capture behavioral and physiological data repeatedly over time in the naturalistic setting (Dobkin, 2013; Gaggioli et al., 2013). With the release of platforms such as Apple Research Kit (www.apple.com/researchkit) and Northwestern University's Purple application suite (Schueller, Begale, Penedo, & Mohr, 2014), it is anticipated that many clinical research studies will incorporate mobile technology for multi-modal assessment (Shen, 2015).

A common function of mobile technology in patient-oriented research is to enable ecological momentary assessment (EMA; Jean, Swendsen, Sibon, Fehér, & Husky, 2013). EMA is a method of repeated collection of “real-time” patient-reported data on individuals’ symptoms, affect, and behavior in natural environments (McVay, Kane, & Kwapil, 2009; Shiffman, Stone, & Hufford, 2008). EMA provides several advantages over traditional survey methods that rely on retrospective global reports. In a typical EMA paradigm, individuals are prompted multiple times throughout the day by a mobile device to answer a short set of questions about their experiences in the moment, rather than asking for summative judgments of their experiences over the past 7 (or more) days. Compared to retrospective reports, EMA is less subject to recall and other biases (Shiffman et al., 2008, Kahneman, 2011), enables assessment of temporal associations between variables, and helps inform contextual and individual variation related to fluctuations in disorders such as anxiety and depression (Ebner-Priemer & Trull, 2009; Moskowitz & Young, 2006).

Given the diminished reliance on retrospective memory, EMA could be especially useful for enhancing the accuracy of assessment in older adults, who are more likely than younger adults to experience cognitive impairments (Rullier et al., 2014). Although older adults may be less likely to own and be proficient with mobile devices (Smith, 2014), this “digital divide” is rapidly shrinking, and evidence suggests that older adults may actually prefer electronic data collection platforms over more traditional paper-based diaries (Shiffman et al., 2008). Previous studies have examined the acceptability and feasibility of EMA (Wenze & Miller, 2010), but few studies have addressed these issues in older adults, particularly those with clinical disorders or cognitive difficulties (Cain, Depp, & Jeste, 2009; Rullier et al., 2014). The use of EMA in this unique population is novel, and additional research is needed to better understand and maximize EMA acceptability and adherence in older adults with cognitive and emotional difficulties.

Ultimately, EMA might even directly improve human health by providing a platform for intervention in older adults with cognitive and mental health problems (van der Wardt, Bandelow, & Hogervorst, 2012). As a first step in this line of work, however, the patient-level implementation issues associated with EMA in older adults with clinical disorders and cognitive difficulties must be better understood. Thus, additional research addressing the unique needs regarding technology use in this population is needed to inform the future design, development, and training of new mobile technologies and their use both in research and clinical contexts.

Successful implementation of technology-based tools can be problematic with respect to consumer acceptance and maintenance of patient-facing technologies that require consumers to interact with electronic tools directly and with limited support or assistance (Ramsey, Lord, Torrey, Marsch, & Lardiere, 2016). Of particular concern in technology-based assessment implementation is cognitive dysfunction, as age-related declines in memory, executive function, and reaction time occur in the vast majority of older adults (Salthouse, 2010). This cognitive decline interferes with learning new technology (Charness & Boot, 2009). In addition, common mental health problems such as depression or anxiety disorders may diminish motivation and reduce confidence in using technology. However, to our knowledge, the feasibility of electronic data capture has not been examined in older adults diagnosed with mental disorders. Understanding of the specific barriers to feasibility and acceptability of EMA is thus particularly needed in older adults with cognitive and emotional difficulties.

Present Study

The data presented here were collected as part of a randomized controlled trial (RCT). The large-scale trial compared the effectiveness of mindfulness-based stress reduction (MBSR) and a health education intervention in older adults with subjective cognitive dysfunction plus anxiety and/or depressive disorders. More information about the parent study can be found at NIH Reporter, https://projectreporter.nih.gov/project_info_description.cfm?aid=8532828&icde=27141587&ddparam=&ddvalue=&ddsub=&cr=3&csb=default&cs=ASC. Although feasibility studies are typically conducted to inform future clinical trials, no previous feasibility data of this nature existed. Therefore, the current study provided a unique opportunity to contribute to our understanding of EMA in a vulnerable population and to inform future trials in this area. In addition to the limited research on EMA methods in those with cognitive and mental health problems, the nature of these presenting concerns—particularly cognitive dysfunction—motivated the decision to perform the present study as a component of the RCT, using EMA in which repeated measures are used to capture recent experiences and perceptions that may have otherwise been forgotten or misreported. Patient reports of anxiety, mindfulness, and depression were collected via 10-day smartphone-based EMA at baseline and post-treatment. Feasibility of EMA in this population was assessed in line with recommendations by Thabane et al. (2010) using the following metrics: (1) acceptability, as rated by satisfaction and comfort with using EMA, (2) objective and self-reported rates of EMA adherence, and (3) reasons for non-adherence to smartphone-based EMA.

Methods

Participants

Participants were 103 adults aged 65 years or older participating in a two-site research study (n=51 at Site 1 and n=52 at Site 2). Site 1 was located in the greater San Diego area, and Site 2 was located in the greater St. Louis area. Inclusion criteria included clinically significant anxiety or depressive symptoms, as defined by PROMIS (Gershon et al., 2010) scores at screening (Anxiety score ≥ 14 or a PROMIS Depression score ≥ 16), and a current diagnosis of a depressive and/or anxiety disorder (i.e., Major Depressive Disorder, Dysthymia, Depressive Disorder Not Otherwise Specified [NOS], Generalized Anxiety Disorder, Panic Disorder With or Without Agoraphobia, or Anxiety Disorder NOS) using the Structured Diagnostic Interview for DSM-IV (First, Spitzer, Gibbon, & Williams, 1997). All participants also endorsed current subjective aging-related neurocognitive problems on the question “Have you noticed that you have any trouble with your memory or concentration?”, which was selected due to its high face validity. Exclusion criteria included diagnosis of dementia as defined by scoring ≥10 errors on the Short Blessed Test (Katzman, Brown, & Fuld, 1983) or a chart diagnosis of dementia, prescription of cognitive enhancing medication (e.g., donepezil), alcohol or substance use disorder within the past six months, current or lifetime diagnosis of psychotic or bipolar disorder, current participation in psychotherapy or regular engagement in mindfulness practice or yoga, corticosteroid use, and serious medical illness (e.g., congestive heart failure) that would prevent study participation or accurate data collection. Table 1 provides comprehensive baseline demographic data on the full sample and by site.

Table 1.

Means and Standard Deviations for Baseline Variables by Site

	All Patients n=103 M (SD) or n (%)	Site 1 (San Diego) n=51 M (SD) or n (%)	Site 2 (St. Louis) n=52 M (SD) or n (%)
Age	71.9 (5.4)	72.5 (5.3)	71.4 (5.6)
Gender
Male	n=28 (27%)	n=19 (37%)	n=9 (17%)
Female	n=75 (73%)	n=32 (63%)	n=43 (83%)
Ethnicity
Caucasian	n=83 (81%)	n=38 (75%)	n=45 (87%)
African American	n=9 (9%)	n=3 (6%)	n=6 (12%)
Asian / Pacific Islander	n=5 (5%)	n=4 (8%)	n=1 (2%)
Hispanic	n=4 (4%)	n=4 (8%)	n=0 (0%)
Other or Unknown	n=2 (2%)	n=2 (4%)	n=0 (0%)
Education, years	15.6 (2.6)	16.1 (2.6)	15.0 (2.6)
PROMIS Depression	20.0 (7.2)	17.9 (7.5)	22.1 (6.2)
PROMIS Anxiety	20.4 (5.8)	18.8 (5.8)	21.9 (5.4)
Cognitive and Affective Mindfulness	31.5 (6.0)	32.5 (7.0)	30.4 (4.8)
PROMIS Cognitive Concerns	23.7 (7.3)	23.8 (6.6)	23.7 (8.0)
Premorbid Neurocognitive Function (WTAR 4)	39.3 (8.5)	37.8 (8.4)	40.8 (8.3)
Executive Function Composite	0.01 (0.61)	0.04 (0.64)	−0.02 (0.57)
Memory Composite	−0.16 (0.78)	−0.38 (0.70)	0.07 (0.79)

Note. PROMIS Depression (8 items; 1=Never, 5=Always); PROMIS Anxiety (7 items; 1=Never, 5=Always); Mindfulness (12 items; 1=Rarely/Not at all, 4=Almost always); PROMIS Cognitive Concerns (8 items; 1=Never, 5=Very Often); Premorbid Neurocognitive Function (50 items; 0=Incorrect, 1=Correct); Executive Function Composite [averaged z-score of the Delis-Kaplan Executive Function System (DKEFS) Verbal Fluency test and the DKEFS Stroop test]; Memory Composite (averaged z-score of the immediate and delayed paragraph and list recall tests).

Procedure

Participants were recruited into the parent RCT via newspaper advertising, university registries, and word of mouth. Upon enrollment into the parent study, participants were made aware that they could refuse any study procedures, including the EMA, without jeopardizing their participation in the RCT. Although participants reported subjective complaints about their cognitive abilities, individuals with dementia were explicitly excluded from participation. As is standard practice in geriatric research (Morone, Greco, & Weiner, 2008; Wells et al., 2013), older adults without dementia were treated as competent subjects for purposes of informed consent to participate in this research study. All participants gave informed consent to participate, and the study was reviewed and approved by the Institutional Review Boards at both sites.

Participation in the present study involved use of a smartphone-based EMA tool to respond to questions assessing anxiety, depression, and mindfulness for 10 days at baseline and 10 days immediately following the intervention (MBSR or health education). Depression and anxiety questions were included because participants were selected for this study based on meeting criteria for depression or an anxiety disorder. Depression and anxiety were measured with National Institutes of Health (NIH) Patient Reported Outcomes Measurement Information System (PROMIS) short form instruments which have been validated in prior research (Cella et al., 2007, 2010; Reeve et al., 2007) and found to have excellent internal consistency when administered via EMA (Moore, Depp, Wetherell, & Lenze, 2016). Mindfulness questions were included because we believed that mindfulness—a construct that emphasizes present moment experience—was particularly well suited for EMA data collection and relevant to the proposed mechanism underlying the RCT. Mindfulness was measured using items from the Cognitive Affective Mindfulness Scale-Revised (CAMS-R) which has been validated in prior research (Feldman, Hayes, Kumar, Greeson, & Laurenceau, 2007) and found to have adequate internal consistency equivalent to that of paper-and-pencil administration (Moore et al., 2016).

EMA survey prompts occurred three times per day at random times with a total of 12 questions included in each EMA survey. Research staff provided subjects with a device (Samsung Galaxy S), trained them to respond to questions and charge the device, and explained that the purpose of the EMA surveys was to assess daily experiences. Research staff encouraged the completion of all measures and explained that they would be able to monitor adherence to the surveys remotely. EMA adherence was defined at the level of the individual EMA survey. Upon survey completion, EMA data were transferred immediately to a customized database using the JavaScript Object Notation (JSON) data-interchange format so that study personnel could monitor ongoing EMA adherence. Research staff at both sites trained all participants to operate the smartphone and respond to questions when prompted by the device. A common set of procedures was used across both sites to train staff and participants in EMA use; a research assistant trained staff members at both sites to use the EMA and supervised each staff member teaching the EMA to at least one participant.

Research staff conducted in-person and telephone training, instructing participants how to use the smartphone and EMA tool. Every day that a participant missed all three attempts to complete the survey, study staff received notification by email and attempted to troubleshoot the issue by phone within 24 hours. If the problem could not be solved over the phone, staff either attempted to correct the issue at the next scheduled in-person appointment (for either an intervention class or assessment) or offered to provide new equipment for another two week period. Minor differences in the approach between sites included actual practice with EMA using a “practice phone” at baseline with research staff supervision and more rapid technical assistance to troubleshoot user response problems and motivate EMA adherence in Site 2. These had a small albeit statistically significant effect on adherence, F(1,75) = 4.30, p = .042, indicating somewhat higher EMA adherence at Site 2 than at Site 1.

Measures

Time 1 (after baseline EMA but prior to the intervention) and Time 2 assessments of the acceptability of the EMA protocol to participants, as rated by self-reported satisfaction (Likert-type scale: 0=very dissatisfied; 4=very satisfied) and comfort levels (0=very uncomfortable; 4=very comfortable), were examined via paper-based surveys. These Time 1 and Time 2 surveys also included self-reported EMA adherence (i.e., “Do you think you missed any study questionnaires on the cell phone during the study so far?”) and self-reported frequency of non-adherence (i.e., If Yes [to self-reported non-adherence], how many times? [A few times, Half of the time, Most of the time, All of the time]), as well as an open-ended question on self-reported reasons for missing surveys. These one-item measures were developed for the purposes of this study as there was a lack of existing measures of these variables that were appropriate for this population. Individual difference characteristics were also measured at baseline, including memory, executive function, premorbid neurocognitive function, anxiety, depression, cognitive concerns, mindfulness, diagnosis, medication, medical burden, and years of education (see Table 1 for baseline descriptive results on these measures).

Analytic Strategy

Using a mix of quantitative and qualitative data, we assessed acceptability and feasibility of using the EMA tool. For the qualitative responses informing EMA non-adherence (i.e., missed surveys), we used a directed content analysis approach and assigned thematic codings to each response using Atlas.ti Qualitative Data Analysis and Research software. We also tracked EMA adherence objectively through device-generated data reports, and used the standard set by Rullier et al. (2014) of 30% of total surveys completed as an indication of acceptable EMA adherence. For comparisons of EMA adherence rates between Time 1 and Time 2, p-value was set to 0.05 to remain consistent with the standard level of statistical significance in scientific research (Bross, 1971).

Results

Acceptability of Smartphone-Based EMA

Quantitative reports of satisfaction and comfort

Regarding participant acceptability, participants at Time 1 were moderately satisfied (M = 2.63; SD = 1.36) and this increased significantly at Time 2 (M = 2.82; SD = 1.25), t(82) = 5.10, p < .001. Similarly, participants were moderately comfortable with the EMA protocol at Time 1 (M = 2.95; SD = 1.43) and at Time 2 (M = 3.00; SD = 1.36), although this increase only approached statistical significance, t(82) = 1.93, p = .057. The majority of participants (60% at Time 1; 69% at Time 2) were either fairly or very satisfied with the EMA tool. Approximately 27% of participants at Time 1 and 16% at Time 2 reported being fairly or very dissatisfied with the EMA tool. Additionally, most participants (71% at Time 1; 73% at Time 2) were either fairly or very comfortable with using EMA. Approximately 21% of participants at Time 1 and 18% at Time 2 reported being fairly or very uncomfortable with the EMA tool.

Qualitative reports of satisfaction and comfort

Based on open-ended feedback, participants expressed perceptions of acceptability and usefulness regarding the EMA protocol through representative quotes, including “helped identify mood at time of questions,” “it's a thing to bring me back to the present,” and “gave words to how I was feeling and allowed me to ponder that.” There were also a range of negative perceptions, reflecting dissatisfaction or discomfort with the EMA protocol. Representative quotes for this perception included “the redundancy was irritating,” “it was like having a newborn baby, always on alert,” and “sometimes I was too upset and nervous to have any feelings or to be thinking about feelings.”

EMA Adherence Rates

Across sites and time points, 76% of participants completed one or more surveys on at least 10 different days, and 70% of participants demonstrated “acceptable” EMA adherence by completing at least 30% of the total surveys. Across sites, total EMA response rate was 46% (M = .46; SD = .31; Range = 0.00-1.00; Median = .44) at Time 1 (prior to the intervention) and 48% (M = .48; SD = .30; Range = 0.00-0.98; Median = .45) at Time 2 (following intervention); that is, 46-48% of the three-times-daily assessments were completed and not missed. There were no differences in response rate between Time 1 and Time 2, t(76) = 0.55, p = .587. Time 1 adherence was positively correlated with Time 2 adherence, r(75) =.47, p < .001.

Additionally, all participants missed at least one assessment, although 18% of participants at Time 1 and 16% of participants at Time 2 self-reported perfect adherence. Pearson correlation analyses indicated that actual non-adherence and self-reported non-adherence were significantly but modestly related at Time 1, r(83) = .31, p = .004, and at Time 2, r(71) = .32, p = .005. Similarly, actual non-adherence and self-reported frequency of non-adherence were significantly associated at Time 1, r(68) = .39, p < .001 and at Time 2, r(59) = .53, p < .001.

Reasons for EMA Non-Adherence

Table 2 displays the primary self-reported reasons for non-adherence, key examples of each type of reason, representative quotes for each type of reason, and the number (and proportion) of participants reporting each type of reason. Seventy-seven participants provided valid open-ended responses regarding reasons for EMA non-adherence (responses such as “N/A” or “not sure” were deemed invalid), and participants were allowed to report multiple reasons. The most common self-reported reasons for EMA non-adherence were grouped into the following themes: 1) being busy during the alarm or bad timing in general (e.g., at doctor's appointment, survey prompt was too late at night); 2) not having the cell phone on person during the alarm (e.g., temporarily misplaced phone, left phone at home); 3) not hearing the alarm (e.g., at loud event, phone buried deep in purse); 4) technical difficulties with the questionnaires or phone itself (e.g., phone not working, survey-related glitch); and 5) user error (e.g., accidentally left phone in airplane mode, accidentally pressed wrong button). Given the overlap between cognitive impairment and problems with memory, clear thinking, and visual perception, many of the reasons grouped in the user error theme (e.g., forgetting to turn the phone back on) could be considered related to cognitive issues. While less commonly reported, participants also cited the burden of responding, poor reception or connectivity, and the survey timing out too soon as reasons for EMA non-adherence.

Table 2.

Rank-Ordered Reasons for EMA Non-Adherence

Reasons for Non-Adherence	Key Examples of Reason Themes	Representative Quotes	# of Participants Reporting Reason
Being busy during the alarm or bad timing in general	Work meeting Sleeping / too early / too late Inconvenient time	“I would hear it but something would be happening, like a long distance phone call and I would get distracted and forget” “I was in a meeting for work or in the middle of a conference call”	44 (57%)
Not having the cell phone on person during the alarm	Left at home In other room Misplaced phone	“Left home without it. In the evening, the afternoon survey was not available to me.” “Forgot to take it with me: a) left house, b) phone was upstairs”	41 (53%)
Not hearing the alarm	Prompt too quiet Too much noise Phone in purse	“Didn't hear ringtone while watching basketball [or] at an evening concert” “Sometimes I didn't hear it ring three times a day so I surmise that I missed a call even though I had the phone near me all day”	29 (38%)
Technical difficulties with the questionnaires or phone itself	Phone out of range Phone not working Survey not presented	“The phone and/or its program was not cooperative—I would log in and “the page not available” would flash on the screen” “The cell phone didn't work when I answered (about 1/3 of the time) and I was out of town 3 days (range?)”	23 (30%)
User error	Airplane mode Incorrect button Phone still off	“Phone was on airplane mode while we were traveling and I didn't realize it” “Perhaps I did not press the button right and it did not show the check mark sign”	14 (18%)

Discussion

This is one of the first studies to assess the feasibility of mobile technology assessment in older adults with emotional disturbance and self-reported cognitive dysfunction. Several potentially important findings emerged: 1) adherence rates were comparable to rates previously reported in younger samples and in older adults without emotional or cognitive concerns; 2) ratings of satisfaction and comfort with smartphone-based EMA were moderately high yet somewhat variable among participants; and 3) a variety of reasons for non-adherence were identified. Adherence rates were influenced by technical (e.g., malfunction), logistical (e.g., competing demands), physiological (e.g., hearing difficulties), and cognitive (e.g., memory, user error) issues, suggesting a need for comprehensive strategies to enhance adherence and acceptability in this population.

Considering prior research on EMA adherence rates in the general population (ranging between 55-87%; Cain et al., 2009), results of this study suggest that using mobile phones for EMA appears feasible in older adult populations with emotional and/or cognitive concerns. Although participants only approached 50% adherence for all surveys, 76% of participants completed surveys on at least 10 different days, and 70% completed at least 30% of the total surveys—an EMA adherence rate deemed acceptable in prior research (Rullier et al., 2014). Thus, the majority of participants contributed sufficient observations to allow for adequate analysis and improve on retrospective self-report data collection methods.

These findings are highly relevant to a number of information system theories and are perhaps best explained by the Technology Acceptance Model (TAM; Davis, 1989). Broadly, the TAM hypothesizes that a user's perceived ease of use and perceived usefulness of a technology-based system (e.g., EMA) determine their intention to use that system, which then mediates actual use of the system. The technical (e.g., system malfunction), physiological (e.g., hearing/visual difficulties), and cognitive (e.g., risk of user error and forgetting) factors cited are highly aligned with users’ perceived ease of use of EMA and may have contributed to suboptimal adherence for some users. Additionally, some of the logistical barriers to adherence (e.g., competing demands, leaving the phone in the other room) may reflect a relatively low perceived usefulness of the EMA system for some users. Although many users cited benefits of using the system (e.g., helped identify mood, facilitated present moment awareness), it may have been perceived as only a research tool for some users. In these cases, the lack of clear personal usefulness of the system may have led to limited EMA adherence. Efforts to enhance mobile technology use, including EMA adherence, in future trials must consider ways to maximize the perceived usability and usefulness of the technology-based system.

Several reasons for non-adherence (e.g., barriers) were identified that may help further inform strategies to address issues and improve adherence. In light of the reported reasons for non-adherence, additional strategies to improve adherence may include: 1) working with clients to plan acceptable timeframes for EMA prompts; 2) gaining buy-in to ensure that clients carry mobile devices with them during the day; 3) turning the prompt notification (e.g., buzzer, ringer) up higher for older adults with hearing impairments; and 4) enhancing the training session(s), including sufficient user practice, with older adults to ensure an adequate level of EMA user ability. Based on open-ended responses involving cognitive (i.e., user error, forgetting) issues and in line with the TAM framework, EMA systems should simplify the design and process of responding to surveys and incorporate reminders shortly after the initial prompt for this population. This may facilitate adherence when users are busy during the initial prompt and then forget about the prompt when eventually available to take the survey. These strategies build on prior sensory-based recommendations to avoid EMA systems with small screens or poor sound quality in elderly populations with visual or hearing impairments (Shiffman et al., 2008).

Relatedly, incorporation of a community-based participatory research (CBPR) approach and inclusion of user feedback on issues related to system usefulness and ease of use early in the process likely would have mitigated non-adherence (Giger et al., 2015). As part of this process, baseline assessment of anticipated problems and concerns could help target training and buy-in needs and reduce experienced stress and anxiety when using EMA. Similarly, early-stage user reports of poor system usability or usefulness could prompt system redevelopment or adaptations necessary for adequate system use. Given the emphasis of CBPR on building trusting relationships, involvement of key stakeholders (e.g., older adult EMA users) throughout the process, and ongoing assessment and cyclical improvements to the system, using CBPR principles in future work may increase comfort and satisfaction, reduce barriers, and ultimately, improve adherence to the EMA protocol (Tapp et al., 2014).

From a user perspective, the EMA protocol of three times daily for 20 days is consistent with other EMA research (Cain et al., 2009; Shiffman et al., 2008; Wilson et al., 2015). While it is possible that a less intensive EMA schedule would improve adherence in older adults with cognitive problems, the qualitative findings do not suggest that the relatively labor intensive nature of EMA was a predominant factor in non-adherence. Instead, more usable technology (e.g., fewer “bugs”, customized to user needs) coupled with proper training and buy-in are key areas for improvement in future research. Additionally, supervised use of a practice phone and more rapid technical assistance—as conducted in Site 2—appeared to boost EMA adherence rates. This may constitute an implementation strategy worth examining more systematically in future research.

From a researcher or provider standpoint, while the adherence monitoring and technical assistance in the current study required several hours of labor from one research staff member, this process likely resulted in significantly higher EMA adherence and, thus, richer data. These supportive monitoring processes could be automated (e.g., system-generated “nudge” reminders), and future research should examine whether the benefits on EMA adherence are comparable to the benefits from the human monitoring approach described here. The feasibility of adherence monitoring and technical support may depend on organizational size, however. For instance, Ramsey et al. (2016) found that larger organizations reported fewer barriers to using technology-based behavioral health tools, likely due to relatively greater resources. Large human service organizations serving large numbers of clients may benefit from more automation to keep up with monitoring and technical support needs. However, smaller organizations less able to invest in such automated strategies may instead find that their human service providers can manage the smaller client caseload with human monitoring and in-person or telephone-based technical assistance.

Limitations

Limitations of the current study include measuring acceptability through single-item, forced choice methods which are limited in their measurement of this construct and, alone, do not allow for actionable data to inform services or future research. Therefore, we added a qualitative assessment of acceptability, and we believe it is important to incorporate mixed methods approaches whenever possible (Creswell, 2013). We find that the forced choice allowed for efficient quantification of acceptability and provided a starting point from which qualitative data expanded on and brought additional meaning and interpretation to the quantitative results.

Although this study yields important results regarding EMA adherence rates, EMA acceptability levels, and reasons for poor EMA adherence in older adults with mental health and cognitive issues, future studies should use comparative designs to examine whether EMA adherence can be improved with enhanced training, technical assistance, or other implementation strategies. Additionally, a previous review has identified research on EMA methods in primarily healthy older adults (Cain et al., 2009); therefore, our focus in this feasibility study was to examine EMA methods in a sample with cognitive and mental health problems. Still, future research may benefit from directly comparing older adult samples with and without cognitive and mental health problems in the same study. It is also worth noting that the unique characteristics and needs of this older adult sample with subjective cognitive and mental health problems may limit the generalization of these results to other populations of EMA users. Finally, the study involved research subjects responding to questions designed to measure outcomes in the context of a research trial. Future studies should examine rates of adherence to EMA in real-world clinical care contexts.

Conclusion

The current study found that the acceptability and feasibility of smartphone-based EMA was adequate yet somewhat variable in older adults with cognitive and emotional difficulties. There is some evidence that satisfaction and comfort with using EMA improved over time (i.e., with increased experience) in this population. Qualitative results on reasons for non-adherence of EMA suggest that researchers and service providers should aim to gain initial input and buy-in from EMA users, attend to technical details to ensure usability and appropriateness with the population, and facilitate sufficient training and practice to maximize EMA user ability. These findings are timely and significant considering aging trends in the United States and beyond. This research contributes a greater understanding of the feasibility and barriers to the reliable use of smartphone-based EMA in a vulnerable population, which may inform design and implementation strategies to enhance user adherence and acceptability in this population.