Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Nov 1.
Published in final edited form as: Pers Med Psychiatry. 2021 Jul 16;29-30:100085. doi: 10.1016/j.pmip.2021.100085

A cross-diagnostic study of Adherence to Ecological Momentary Assessment: Comparisons across study length and daily survey frequency find that early adherence is a potent predictor of study-long adherence

Sara E Jones 1, Raeanne C Moore 2, Amy E Pinkham 3,4, Colin A Depp 2,5, Eric Granholm 2,5, Philip D Harvey 1,6
PMCID: PMC8442600  NIHMSID: NIHMS1729924  PMID: 34541425

Abstract

Background:

Ecological momentary assessment (EMA) offers a highly valid strategy to assess everyday functioning in people with severe mental illness. Adherence is generally good, but several questions regarding the impact of study length, daily density of sampling, and symptom severity on adherence remain.

Methods:

EMA adherence in two separate studies was examined. One sampled participants with schizophrenia (n=106) and healthy controls (n=76) 7 times per day for 7 days and the other sampled participants with schizophrenia (n=104) and participants with bipolar illness (n=76) 3 times per day for 30 days. Participants were asked where they were, who they were with, what they were doing and how they were feeling in both studies. The impact of rates of very early adherence on eventual adherence was investigated across the samples, and adherence rates were examined for associations with mood state and most common location when answering surveys

Results:

Median levels of adherence were over 80% across the samples, and the 10th percentile for adherence was approximately 45% of surveys answered. Early adherence predicted study-long adherence quite substantially in every sample. Mood states did not correlate with adherence in the patient samples and being home correlated with adherence in only the bipolar sample.

Implications:

Adherence was quite high and was not correlated with the length of the study or the density of sampling per study day. There was a tendency for bipolar participants who were more commonly away from home to answer fewer surveys but overall adherence for the bipolar patients was quite high. These data suggest that early nonadherence is a potential predictor of eventual nonadherence and study noncompletion.

Keywords: ecological momentary assessment, survey adherence, EMA completion rates, bipolar disorder, schizophrenia

1. Introduction

Ecological momentary assessment (EMA), also known as the Experience Sampling Method (ESM), is a broad term used to describe a range of research methodologies focused on assessing participants on a repeated basis in their natural environments [1]. In its simplest and earliest form, EMA consists of asking participants to keep a paper-and-pencil daily diary of their experiences and/or symptoms of interest. With technological developments, EMA is now commonly employed as a methodology involving timed smartphone surveys to which participants are asked to respond multiple times per day.

EMA offers three main benefits over typical clinical assessments: reduced recall bias, increased ecological validity, and increased surveillance accuracy of dynamic processes [2,3]. In a typical clinical assessment, a study participant may be asked to provide a summary of their symptomatology and daily activities retrospectively over a specific time period which can be as long as their “lifetime to date”. As Smyth and Stone note, recall bias can be generated by a participant’s current mood state, the valence or outcome of the event that the participant is asked to recall, and the participant’s general beliefs about themself and the world at large [2]. EMA mitigates this bias by instead asking a participant questions that focus on present experiences, feelings, and activities only, which has the additional benefit of reduction of recall failures in the absence of biases. Additionally, assessing a participant only in a clinical environment, as in traditional methods, is inherently unnatural, as it exposes the participants to stressors and experiences incongruent with their daily life. Finally, whether mental, physical, or both, a participant’s symptoms may fluctuate on a daily, hourly, or momentary basis, and standard clinical assessments are unable to fully capture these dynamic changes. Accordingly, by assessing the participant at many different timepoints and in their natural environment, EMA data can be used to create an aggregate picture of a participant’s symptomatology and activities, without relying on the participant themself to accurately generate a retrospective summary, which is likely prone to unconscious biases.

EMA has been utilized as a research method in a variety of fields. While a large portion of research involving EMA has centered on psychiatric illnesses, it has also been used heavily in the areas of nutrition, intentional weight loss, smoking cessation, and chronic illness management [4–7]. Inventive protocols have also combined active data collection through EMA with passive data collection via an accelerometer to monitor activity levels in post-stroke patients, pediatric chronic abdominal pain patients, and participants with severe mental illness [8,9,10]. In psychiatry, EMA has sparked particular interest in its ability to aid in the creation of personalized treatment plans for serious mental illness (SMI) and its ability to enhance understanding of medication adherence and nonadherence [3,11,12].

However, the success of EMA as a clinical tool relies on consistent data collection and many of the active assessments in EMA require a response on the part of the participants. Jones et al. have suggested an adherence goal for EMA studies of 80% [13]. A 2017 study based in Spain piloted an EMA tool for psychiatric outpatients to self-monitor symptoms and share data with their psychiatry team; it offered no reminders to participate or incentives to complete surveys and found that only 20% of patients ever logged into the system [14]. However, numerous other EMA studies involving patients in both psychiatric care and medical care have achieved far higher completion and adherence rates by establishing response schedules, incentivizing responses, and sending reminders to enter responses. EMA studies of adults with chronic illnesses, such as human immunodeficiency virus (HIV) and history of acute coronary syndrome (ACS), have reported strong adherence rates, ranging from 79% survey completion to 91% survey completion [15–17]. EMA studies of adolescents with chronic illnesses such as type 1 diabetes mellitus and asthma have reported lower adherence rates, ranging from 59% to 67% [18,19]. A 2019 systematic review of EMA studies involving patients with major depressive disorder (MDD) found that reported survey completion rates ranged from 65%−85%, although some studies did not report adherence rates [20]. An initial study with patients with bipolar disorder [21] reported 43% adherence to smartphone surveys and 71% adherence (after excluding participants who provided no data) for paper and pencil dairies. Despite the lower smartphone-based adherence, the smartphone responses were significantly correlated with end of study clinical ratings of both depression and mania, while the paper diary data were not. This study did not compensate participants on a survey by survey basis which may account for the lower adherence despite excellent validity. Later EMA studies conducted in participants with bipolar disorder (BPD) have reported 80% adherence in both adolescents and adults [22,23], and studies in schizophrenia (SCZ) have reported greater than 85% adherence in adults [24–25]. Therefore, excellent adherence rates are achievable in participants, including those with psychiatric illnesses, through careful study design.

Nonadherence in EMA studies poses several issues for researchers: it increases costs, it may introduce systematic bias, and it lowers statistical power [13]. Several intrinsic and extrinsic factors may contribute to reduced responding. For example, the frequency of survey requests and duration of the sampling period could theoretically affect adherence. It is also possible that being more symptomatic could lead to reduced adherence in mental health studies or conversely that more severe symptomatology could incentivize greater participation. Additionally, while the result may seem paradoxical, studies with fewer surveys per day often find lower adherence, perhaps because frequent surveys become routine to participants [26]. Equivalent amounts of reimbursement for each weekly sampling burst are associated with greater adherence if the reimbursement is attached to each completed survey rather than delivered in aggregate at the end of the week [3].

In the work presented here, we evaluated several parameters of active EMA sampling across psychiatric diagnoses and in healthy controls. We compare results from studies that lasted 30 and 7 days, with sampling frequency at 3 and 7 times per day, respectively. The studies did not differ in compensation protocol, with participants being paid $1.00 for each completed survey with the payment delivered contemporaneously. We also aimed to determine if participants with lower levels of adherence can be identified early in EMA studies, by correlating early to total adherence. As our two studies had different designs, we defined early as the first day of a 7-day sampling period in the first study and as the first week in the second. This one-week period could correspond to a run-in for a clinical trial.

2. Methods

2.1. Participants:

There are two separate samples of participants with various diagnoses included in these analyses. One sample answered EMA queries 7 times per day for 7 days (sample A) and included participants with schizophrenia and healthy controls. The full study protocol was previously described and can be referenced elsewhere [26]. This study was fully conducted in San Diego, California, and was completed before the second study was initiated. The second study was conducted in San Diego, Dallas, TX, and Miami, FL, and the previously described full study protocol can be referenced elsewhere ([27]; Sample B). Sample B included participants with schizophrenia and bipolar illness, and these participants were queried 3 times per day for 30 days. Although there is overlap in the study teams across the studies, there is no overlap of participants between samples. All participants provided written informed consent and the studies were approved by Institutional Review Boards at the University of California San Diego, the University of Texas at Dallas, and the University of Miami as appropriate.

In both studies, the diagnosis of schizophrenia and bipolar illness was made by trained interviewers using validated methodology, which included the Mini International Neuropsychiatric Interview (MINI; [28]) and the psychosis module of the Structured Clinical Interview for DSM Disorders-5 (SCID-5; [29]), followed by a local consensus procedure to reify final diagnoses. For participants with schizophrenia, we have grouped schizophrenia and schizoaffective disorder, and for bipolar illness, we included participants with bipolar I disorder, with and without psychotic features, and bipolar II disorder.

2.1.1. Inclusion and Exclusion Criteria for All Participants in Both Studies:

Inclusion criteria for clinical participants included having a diagnosis of schizophrenia, any subtype, or schizoaffective disorder, based on DSM-5 criteria or bipolar disorder type I, with or without psychotic features, or bipolar II disorder. Clinical participants were included if they were clinically stable for at least four weeks on psychotropic medication with no anticipated medication adjustments, were age 18 to 65, were fluent in English, and were able to give valid informed consent. Stability was defined as no hospitalizations or emergency room visits in the prior 4-week period.

Inclusion criteria for healthy control participants included no DSM-5 diagnoses of past or current mood, anxiety, or psychotic disorders (based on the SCID-Nonpatient Version). Participants were age 18 to 65 and able to give valid informed consent.

Exclusion criteria for all participants included the presence of medical or neurological comorbidities that would interfere with adequate participation and subsequent assessments, including substance dependence other than tobacco not in remission for at least 6 months, cerebrovascular disease, CNS tumors, epilepsy, intellectual disability or developmental delay as defined by DSM-5 criteria, significant hearing or visual impairment, or inability to communicate in English. Potential participants were not enrolled if they could not demonstrate adequate reading skills as measured via a Wide Range Achievement Test-3rd edition (WRAT-3; [30]) grade equivalent score of at least 8th grade.

2.2. EMA Procedures

Participants were given a Samsung Android OS smartphone and were surveyed several times a day. Sample A was administered surveys 7 times per day over 7 days. Sample B was administered surveys 3 times per day over 30 days. Surveys asked participants to numerically rate, on a scale of one to seven, intensity of their current experience of four mood states: happiness, sadness, relaxation, and anxiety. The higher the number, the greater the subjective experience of that mood. Surveys also asked participants to report whether they were home or away from home and alone or with another person, as well as identify which activities they were engaged in from a list that was tailored to where they were and who they were with. Thus, there was there was a home-alone survey, a home with someone survey, and an away from home survey.

Participants in sample B were also asked questions regarding the presence and severity of psychotic symptoms. See Harvey et al. for the results of those surveys [31]. The activity questions were slightly different from the questions asked of sample A as sample B’s survey was refined based on the findings from the first study. However, the mood, location, and social context rating questions were identical and are examined here as correlates of adherence.

After the participant was surveyed in sample A, they were asked what they had been doing in “the last hour “; in sample B the query was “today”, for the first survey, and “since the last survey” for surveys 2 and 3. Surveys were closed and designated as unanswered after 60 minutes.

In order to increase adherence and aid participation, investigators formally instructed participants on how to use the smartphone and respond to surveys, paid participants a micro-reimbursement for each survey completed ($1.00 USD per survey), and formatted surveys as user-friendly check-box questions.

2.3. Data Analyses

Our primary interest was in the percentage of surveys validly answered within the allocated time frame. This was defined simply as surveys where information about social context and mood state, as well as activities were completed. We calculated the median and mode for percentage of surveys answered overall. We also examined the mean and standard deviation (SD) overall as well as identified the 10th and 90th percentiles for adherence for early surveys and total surveys. “Early” in sample A was defined as the first day (out of 7) and for sample B as the first 21 surveys (the first week) (out of 90 surveys collected over 30 days). We also used Pearson Correlations to calculate the correlation between early versus later adherence and the average of all of the samples for the two negative mood states (sadness and anxiety) averaged across the surveys (it is not possible to measure how sad or anxious one is on a survey that is not answered) and the proportion of surveys answered at home. We planned to use tests of the significance of the difference between proportions; however, adherence outcomes were so similar across all 5 samples and all variables that we only compared the least and most adherent group with a t-test. We calculated the Pearson correlations between total adherence, number of surveys answered while home, number of surveys answered while alone, and the two negative moods: sadness and anxiety. We included only negative moods because of the redundancy that we found when examined the between correlation between happy and sad moods: in the entire sample B, the MMRM association (adjusting for random intercept, day, and survey) between the [up to] 90 happy ratings and time-linked linked sadness ratings was: X2(6) =3771, p=3.2*10−6, B=−.48 (Durand et al. [27]). We computed the Pearson Correlations separately in each sample to see the consistency of effects across the different samples.

3. Results

Participant demographic characteristics are presented in Supplementary Table 1, as these data have previously published. Table 1 presents the adherence to EMA surveys. We defined early adherence as the first day (out of 7) in sample A and the first week (out of 30 days) in sample B. As can be seen in the table, mean overall adherence was quite high across all four subgroups, ranging from 75% to 83%. A t-test on the percentage total adherence between the two most divergent groups, the sample A schizophrenia patients and bipolar patients was statistically significant, t (180) =2.55, p<.05, but no other comparisons would have been significant. Early adherence was slightly higher than total adherence, ranging from 81% to 83%. However, across the groups adherence only declined over the rest of the protocol by 6% in two of the groups and less in the other two groups. All modal adherence values were over 93 %. The lowest 10% of adherence ranged from 44% to 51% adherence; participants in the lowest 10% included 5 participants in sample A who were excluded from the analyses in that previous publication for lower than 33% adherence.

Table 1.

Adherence Characteristics Across Four Samples of EMA Participants

Sample A Schizophrenia Sample B Schizophrenia Bipolar Healthy Control
N 106 104 76 76
% % % %
Mean Overall (SD) Adherence 83.00 (20.00) 75.00 (23.13) 74.7 (22.16) 80.1 (22.46)
Mean First Period (SD) Adherence 82.2 (24.05) 81.11 (22.90) 81.2 (21.08) 82.5 (25.1)
Modal Adherence Overall 98 97 93 95
Median Adherence Overall 91 81 81 88
90% percentile Total Study Adherence 98 97 94 98
10% percentile Overall Adherence 51 47 44 45
90% percentile First Period Adherence 98 100 95 99
10% percentile First Period Adherence 43 48 57 43
Open in a new tab

Note: First period is day 1 (7 total Surveys) for Sample A and HC. First period is week 1 (21 total surveys) for Sample B and Bipolar.

Table 2 presents the proportion of surveys answered at home and the means and standard deviations for the two mood variables. The sample A patients answered more surveys from home than the other three groups, X2 (3) = 9.63, p<.02. There were no differences in the likelihood of being alone across the four samples, X2 (3) = 0.59, p=.90. The healthy participants from sample A were less sad, t(180) =6.22, p<.001, and less anxious, t(180) =6.06, p<.001, than the schizophrenia patients in that sample.

Table 2.

Scores on Correlational Variables

Sample A Schizophrenia Sample B Schizophrenia Bipolar Healthy
% Surveys at Home 80% 66% 59% 61%
% Surveys Alone 53% 54% 49% 53%
M SD M SD M SD M SD
EMA Sadness Scores 2.6 1.3 2.6 1.3 2.9 1.4 1.6 0.7
EMA Anxiety Scores 2.6 1.3 3.0 1.5 3.4 1.4 1.7 0.7
Open in a new tab

Note. Sadness and Anxiety Survey scores can range from 1 (no mood experience) to 7 (Extreme mood Experience). Every survey that was answered had a valid mood rating.

Table 3 presents correlations between early and later adherence, as well as the proportion of surveys answered at home and alone and the average scores on the two mood state variables. For earlier and later adherence, the correlations were all statistically significant at p<.001. Lower total adherence was only correlated with answering fewer surveys at home in the bipolar patients; higher scores on the anxiety and depression variables were only correlated with lower adherence in the healthy controls. Schizophrenia patients in sample A answered more surveys when they were alone, at a minimally significant level. Correlations between early adherence and the other variables did not differ significantly from correlations with total adherence.

Table 3.

Correlations between First Period and Total Adherence and other Clinical Variables

Schizophrenia Sample A
Overall Adherence Surveys at Home Surveys Alone Sadness Anxiety
Period 1 Adherence .72*** −.10 .18 0.01 −.04
Overall Adherence - - 0.03 .23* −.10 −.08
Surveys at Home - - - - .00 0.03 −.04
Surveys Alone - - - - - - −.01 .08
Mean Sadness - - - - - - 0.86***
Schizophrenia Sample B
Overall Adherence Surveys at Home Surveys Alone Sadness Anxiety
Period 1 Adherence .84*** −.02 0.00 −.01 −.06
Overall Adherence - - −.09 −.01 −.03 −.11
Surveys at Home - - - - −.12 −.08 −.01
Surveys Alone - - - - - - .04 .10
Mean Sadness - - - - - - .73***
Bipolar Patients
Overall Adherence Surveys at Home Surveys Alone Sadness Anxiety
Period 1 Adherence .82*** −.28* −.13 −.02 −.14
Overall Adherence - - −.29* .03 .05 −.13
Surveys at Home - - - - −.08 .09 .08
Surveys Alone - - - - - - .10 .12
Mean Sadness - - - - - - .83***
Healthy Controls
Overall Adherence Surveys at Home Surveys Alone Sadness Anxiety
Period 1 Adherence .70*** .11 .08 .−.28* −.35**
Overall Adherence - - .21 .14 −.16 −.26*
Surveys at Home - - - - .06 −.03 −.05
Surveys Alone - - - - - - .22 .08
Mean Sadness - - - - - - .87***
Open in a new tab

Note:

*:

p<.05

**:

p<.01

***

p<.001.

4. Discussion

Despite differing EMA parameters and psychiatric diagnoses, we found relatively high adherence rates across these two protocols, ranging from 75%−83%. Of the four participant groups (schizophrenia Sample A, schizophrenia Sample B, bipolar disorder, and healthy controls), the only groups whose adherence significantly differed were schizophrenia Sample A and the bipolar disorder sample, who had the highest and lowest adherence, respectively. While this may suggest higher adherence by sampling 7 times per day for 7 days versus sampling 3 days per day for 30 days, the two schizophrenia samples did not differ significantly in adherence, which weakens this idea.

As discussed previously, an 80% adherence goal for EMA projects has been suggested in the literature [13]. While only two of the four samples presented here reached mean adherence levels of 80% or greater (schizophrenia Sample A and healthy controls), both the median and modal adherences for all 4 samples exceed 80%. Accordingly, we realize that most participants in our EMA trials are extremely adherent and complete the vast majority of their surveys, but mean adherence is greatly decreased by nonadherent outliers. Understanding that large-scale EMA projects can be resource intensive, and that low adherence lessens statistical power, the question then becomes: how early can we recognize these outliers? Our data reveals that, for all four samples, early adherence predicted late adherence at the p<.001 level. If a participant is poorly adherent in the first day (in the case of 7-day studies) or in the first week (in the case of 4-week studies), the correlational results suggest that they will be poorly adherent for the duration of the study. Accordingly, we suggest that researchers bear this correlation in mind and consider incorporating an early phase and late phase into EMA trials in order to adjust their strategies according to early adherence results in order to optimize study data outcomes.

A very recent study examined adherence to passive and active digital phenotyping in people with schizophrenia and healthy controls [32]. In that study, 55 schizophrenia patients were assessed over a 6-day EMA period with active surveys responses and passive measurement and compared for adherence to sample of healthy controls. Event-related EMA sampling occurred 3 times per day and there 8 more EMA surveys throughout the day on a quasi -random basis. Thus, there were a total of 66 possible surveys to be answered. Adherence to the planned surveys was greater than for the randomly delivered ones, with adherence at 56% for the 8 daily random surveys for schizophrenia participants and 67% for the HC, while the pre-planned morning surveys had adherence rates of 75% for patients and 85% for controls. Thus, this study suggests that there may be an upper limit on the number of EMA surveys per day.

The current results also demonstrate a relatively limited impact of negative mood states on EMA adherence, particularly within the clinical populations included here. In healthy individuals, increased sadness and anxiety were associated with reduced adherence; however, these effect sizes were small and intensity of the mood experiences was minimal on average. Additionally, being home vs. away and alone or with someone also appeared to have a limited effect, more surveys answered by the bipolar patients while at home and, in one sample of the schizophrenia patients, when alone. We view these results as encouraging in that they suggest EMA surveys are widely accessible across clinical states and real-world contexts and that the outcomes of interest for the study of negative symptoms, being home and alone, do not affect the likelihood of surveys being answered. Replication and expansion of these results across different populations, sampling densities, and compensation strategies seems important as well.

As with any research study, the results presented here have some limitations. Due to the differing study designs, we are unable to directly compare parametric adherence differences in participants with bipolar disorder and healthy controls. We also averaged our mood ratings because missing surveys lead to missing mood data; our previous results suggested no significant effects of day for the sad moods in the schizophrenia participants [31]. Additionally, while the intent of the work presented here is intended to help develop strategies to maximize adherence, previous work has suggested that EMA is less feasible for individuals experiencing significant social barriers, such as living in transitional housing or having a history of incarceration [33]. However, we suggest that researchers make inclusion-focused efforts including providing all necessary equipment and technological support for study participation. Finally, while adherence in our samples decreased minimally across the study duration, this is consistent with trends in the existing body of EMA research [6,34].

5. Conclusions

In line with previous suggestions, median adherence in the current analyses was over 80% across diagnoses and study designs; the lowest 10th percentile in terms of adherence neared 40% response rates. Adherence did not vary systematically as a function of the length of the study or the density of the daily sampling although other studies with even more dense sampling have found somewhat lower adherence to surveys. Overall levels of sadness and anxiety did not predict nonadherence in the patient participants but did predict nonadherence in healthy controls. Early adherence was a significant predictor of study-wide adherence, suggesting that early adherence rates could be used to help determine the best allocation of study resources (e.g., reminder phone calls to participants after missed surveys, etc.). On the whole, these results support the utility of EMA for collecting consistent data across a wide variety of samples, clinical states, and environmental contexts.

Supplementary Material

Supplemental Material

Acknowledgments

This research was supported by NIMH grant RO1MH112620 to Dr. Pinkham and VA Merit Review Grant # 1I01CX000810 to Dr. Granholm.

References

  • [1].Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu. Rev. Clin. Psychol. 2008;4(1):1–32. doi: 10.1146/annurev.clinpsy.3.022806.091415. [DOI] [PubMed] [Google Scholar]
  • [2].Smyth JM, Stone AA. Ecological Momentary Assessment Research in Behavioral Medicine. J. Happiness Stud 2003;4(1):35–52. doi: 10.1023/A:1023657221954. [DOI] [Google Scholar]
  • [3].Depp CA, Moore RC, Perivoliotis D, Granholm E. Technology to assess and support self-management in serious mental illness. Dialogues Clin. Neurosci. 2016;18(2):171–83. doi: 10.31887/DCNS.2016.18.2/cdepp. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [4].Shiffman S How Many Cigarettes Did You Smoke? Assessing Cigarette Consumption by Global Report, Time-Line Follow-Back, and Ecological Momentary Assessment. Health Psychol. 2009;28(5):519–26. doi: 10.1037/a0015197. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [5].Burke LE, Shiffman S, Music E, Styn MA, Kriska A, Smailagic A, et al. Ecological Momentary Assessment in Behavioral Research: Addressing Technological and Human Participant Challenges. J. Med. Internet Res. 2017;19(3):e77. doi: 10.2196/jmir.7138. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [6].Forman EM, Schumacher LM, Crosby R, Manasse SM, Goldstein SP, Butryn ML, et al. Ecological Momentary Assessment of Dietary Lapses Across Behavioral Weight Loss Treatment: Characteristics, Predictors, and Relationships with Weight Change. Ann. Behav. Med. 2017;51(5):741–53. doi: 10.1007/s12160-017-9897-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [7].Crochiere RJ, Mangubat CJ, Manasse SM, Forman EM. Does executive function moderate the relation between momentary affective and physical states and subsequent dietary lapse? An EMA investigation. J. Behav. Med. 2019;42(6):1148–52. doi: 10.1007/s10865-019-00039-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [8].Cushing CC, Kichline T, Blossom JB, Friesen CA, Schurman JV. Tailoring Individualized Evaluation of Pediatric Abdominal Pain Using Ecological Momentary Assessment (EMA): A Pilot Study Testing Feasibility and Acceptability. Clin. J. Pain 2019;35(11):859–68. doi: 10.1097/ajp.0000000000000750. [DOI] [PubMed] [Google Scholar]
  • [9].Chen YA, Demers M, Lewthwaite R, Schweighofer N, Monterosso JR, Fisher BE, et al. A Novel Combination of Accelerometry and Ecological Momentary Assessment for Post-Stroke Paretic Arm/Hand Use: Feasibility and Validity. J Clin Med 2021;10(6) doi: 10.3390/jcm10061328. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [10].Raugh IM, James SH, Gonzalez CM, Chapman HC, Cohen AS, Kirkpatrick B, Strauss GP. Geolocation as a Digital Phenotyping Measure of Negative Symptoms and Functional Outcome. Schizophr Bull. 2020;46(6):1596–1607. doi: 10.1093/schbul/sbaa121.PMID: 32851401; PMCID: PMC7751192. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [11].Gaudiano BA, Moitra E, Ellenberg S, Armey MF. The Promises and Challenges of Ecological Momentary Assessment in Schizophrenia: Development of an Initial Experimental Protocol. Healthcare (Basel) 2015;3(3):556–73. doi: 10.3390/healthcare3030556. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [12].Bell IH, Lim MH, Rossell SL, Thomas N. Ecological Momentary Assessment and Intervention in the Treatment of Psychotic Disorders: A Systematic Review. Psychiatr. Serv. 2017;68(11):1172–81. doi: 10.1176/appi.ps.201600523. [DOI] [PubMed] [Google Scholar]
  • [13].Jones A, Remmerswaal D, Verveer I, Robinson E, Franken IHA, Wen CKF, et al. Compliance with ecological momentary assessment protocols in substance users: a meta-analysis. Addiction 2019;114(4):609–19. doi: 10.1111/add.14503. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [14].Barrigón ML, Berrouiguet S, Carballo JJ, Bonal-Giménez C, Fernández-Navarro P, Pfang B, et al. User profiles of an electronic mental health tool for ecological momentary assessment: MEmind. Int. J. Methods Psychiatr. Res. 2017;26(1) doi: 10.1002/mpr.1554. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [15].Moore RC, Kaufmann CN, Rooney AS, Moore DJ, Eyler LT, Granholm E, et al. Feasibility and Acceptability of Ecological Momentary Assessment of Daily Functioning Among Older Adults with HIV. Am. J. Geriatr. Psychiatry 2017;25(8):829–40. doi: 10.1016/j.jagp.2016.11.019. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [16].Campbell LM, Paolillo EW, Heaton A, Tang B, Depp CA, Granholm E, et al. Daily Activities Related to Mobile Cognitive Performance in Middle-Aged and Older Adults: An Ecological Momentary Cognitive Assessment Study. JMIR Mhealth Uhealth 2020;8(9):e19579. doi: 10.2196/19579. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [17].Jean FAM, Sibon I, Husky M, Couffinhal T, Swendsen J. Feasibility and validity of Ecological Momentary Assessment in patients with acute coronary syndrome. BMC Cardiovasc. Disord. 2020;20(1):499. doi: 10.1186/s12872-020-01774-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [18].Mulvaney SA, Ho YX, Cala CM, Chen Q, Nian H, Patterson BL, et al. Assessing adolescent asthma symptoms and adherence using mobile phones. J. Med. Internet Res. 2013;15(7):e141. doi: 10.2196/jmir.2413. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [19].Mulvaney SA, Rothman RL, Dietrich MS, Wallston KA, Grove E, Elasy TA, et al. Using mobile phones to measure adolescent diabetes adherence. Health Psychol. 2012;31(1):43–50. doi: 10.1037/a0025543. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [20].Colombo D, Fernández-Álvarez J, Patané A, Semonella M, Kwiatkowska M, García-Palacios A, et al. Current State and Future Directions of Technology-Based Ecological Momentary Assessment and Intervention for Major Depressive Disorder: A Systematic Review. J Clin Med 2019;8(4) doi: 10.3390/jcm8040465. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [21].Depp CA, Kim DH, Vergel de Dios L, Wang V, Ceglowski J. A Pilot Study of Mood Ratings Captured by Mobile Phone Versus Paper-and-Pencil Mood Charts in Bipolar Disorder. J. Dual Diagn. 2012;8(4):326–32. doi: 10.1080/15504263.2012.723318 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [22].Husky MM, Gindre C, Mazure CM, Brebant C, Nolen-Hoeksema S, Sanacora G, et al. Computerized ambulatory monitoring in mood disorders: Feasibility, compliance, and reactivity. Psychiatry Res. 2010;178(2):440–2. doi: 10.1016/j.psychres.2010.04.045. [DOI] [PubMed] [Google Scholar]
  • [23].Gershon A, Kaufmann CN, Torous J, Depp C, Ketter TA. Electronic Ecological Momentary Assessment (EMA) in youth with bipolar disorder: Demographic and clinical predictors of electronic EMA adherence. J. Psychiatr. Res. 2019;116:14–8. doi: 10.1016/j.jpsychires.2019.05.026. [DOI] [PubMed] [Google Scholar]
  • [24].Johnson EI, Grondin O, Barrault M, Faytout M, Helbig S, Husky M, et al. Computerized ambulatory monitoring in psychiatry: a multi-site collaborative study of acceptability, compliance, and reactivity. Int. J. Methods Psychiatr. Res. 2009;18(1):48–57. doi: 10.1002/mpr.276. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [25].Swendsen J, Ben-Zeev D, Granholm E. Real-Time Electronic Ambulatory Monitoring of Substance Use and Symptom Expression in Schizophrenia. Am. J. Psychiatry 2011;168(2):202–9. doi: 10.1176/appi.ajp.2010.10030463. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [26].Granholm E, Holden JL, Mikhael T, Link PC, Swendsen J, Depp C, et al. What Do People With Schizophrenia Do All Day? Ecological Momentary Assessment of Real-World Functioning in Schizophrenia. Schizophr. Bull. 2020;46(2):242–51. doi: 10.1093/schbul/sbz070. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [27].Durand D, Strassnig MT, Moore RC, Depp CA, Ackerman RA, Pinkham AE, et al. Self-reported social functioning and social cognition in schizophrenia and bipolar disorder: Using ecological momentary assessment to identify the origin of bias. Schizophr. Res. 2021;230:17–23. doi: 10.1016/j.schres.2021.02.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [28].Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, et al. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J. Clin. Psychiatry 1998;59Suppl 20:22–33. [PubMed] [Google Scholar]
  • [29].First M, Williams J, Karg R, Spitzer R. Structured Clinical Interview for DSM-5 Disorders, Clinical Trials Version (SCID-5-CT). Arlington, VA: American Psychiatric Association, 2015. [Google Scholar]
  • [30].Jastak S, Wilkinson G. Wide Range Achievement Test-Revised. Wilmington, DE: Jastak Associates, 1984. [Google Scholar]
  • [31].Harvey PD, Miller ML, Moore RC, Depp CA, Parrish EM, Pinkham AE. Capturing Clinical Symptoms with Ecological Momentary Assessment: Convergence of Momentary Reports of Psychotic and Mood Symptoms with Diagnoses and Standard Clinical Assessments. Innov. Clin. Neurosci. 2021;18:E12. [PMC free article] [PubMed] [Google Scholar]
  • [32].Raugh IM, James SH, Gonzalez CM, Chapman HC, Cohen AS, Kirkpatrick B, Strauss GP. Digital phenotyping adherence, feasibility, and tolerability in outpatients with schizophrenia. J Psychiatr Res. 2021;138:436–443. doi: 10.1016/j.jpsychires.2021.04.022.Epub ahead of print.PMID: 33964681. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [33].Turner CM, Arayasirikul S, Trujillo D, Lê V, Wilson EC. Social Inequity and Structural Barriers to Completion of Ecological Momentary Assessments for Young Men Who Have Sex With Men and Trans Women Living With HIV in San Francisco. JMIR Mhealth Uhealth 2019;7(5):e13241. doi: 10.2196/13241. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [34].Glenn CR, Kleiman EM, Kearns JC, Santee AC, Esposito EC, Conwell Y, et al. Feasibility and Acceptability of Ecological Momentary Assessment with High-Risk Suicidal Adolescents Following Acute Psychiatric Care. J. Clin. Child Adolesc. Psychol. 2020:1–17. doi: 10.1080/15374416.2020.1741377. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Material
NIHMS1729924-supplement-Supplemental_Material.docx (13.3KB, docx)

RESOURCES