Feasibility and Adherence Paradigm to Ecological Momentary Assessments in Urban Minority Youth

Abstract

All adolescents in general, including ethnic and racial minorities, report high levels of cell phone use, making mobile technology a useful tool for assessment and intervention. Known health and education disparities based on minority status motivated us to conduct an in-depth investigation regarding feasibility of and adherence to the ecological momentary assessment (EMA) research protocol, studying daily life of urban minority youth. Additionally, this paper presents a methodological approach to conceptualizing and reporting adherence in EMA studies. The sample was comprised of 126 youth (41.3% boys; 40.5% seventh and 59.5% eight graders; 75.4% African-American, and 20.6% Hispanic) who carried a mobile phone for 10 days, including two weekends, reporting on activities, moods, and attitudes. Mean level of adherence was 81% for momentary and 93.8% for daily assessments; it decreased over time, and was higher during the week compared to weekends. Adherence was lower on days when participants reported high levels of negative affect and on days when they were engaged in physical activities. Our findings underscore the importance of differentiating between human and technology factors when computing adherence rates and portray adherence as a complex and dynamic construct that also varies across individuals. Specific study recommendations and methodological discussion are useful for designing future studies.

INTRODUCTION

Adolescence is a pivotal developmental period, often characterized by increased independence, restructuring of social relationships, self-expression, and psychological as well as physiological changes (Erikson, 1994). Four important contexts of developmental growth in the lives of adolescents are school, leisure, family, and work (Silbereisen & Lerner, 2007). To better understand development as well as emotional and behavioral functioning within these contexts, researchers have increasingly employed self-report methods that collect data in real time and in real environments (see Hektner, 2012, for a review). These methods are known as experience sampling (Csikszentmihalyi, Larson, & Prescott, 1977) or ecological momentary assessment (EMA; Stone, Shiffman, Atienza, & Nebeling, 2007; Stone & Shiffman, 1994). In EMA studies, adolescents respond to momentary prompts (delivered by an electronic device such as a phone) and report on their current activities, psychological as well as physical states, and contextual surroundings. The immediacy of reported experiences helps to reduce the recall bias substantially. Further, repeated and frequent assessments help to capture within-person dynamics across contexts and time. Finally, ecological integrity of data is improved by assessments gathered in natural environments, over the course of adolescents’ day-to-day lives (Csikszentmihalyi, Larson, & Prescott, 1977; Hektner, Schmidt, & Csikszentmihalyi, 2007; Hufford, 2007).

While EMAs have been used predominantly in studies with adults, adolescents are increasingly dexterous with technology and accept and adopt it easily. Based on the latest statistics, 78% of teens own a cell phone, and 37% own a smart phone (Madden et al., 2013). For minorities, rates of cell phone ownership are slightly lower: 72% for Black non-Hispanic and 64% for Hispanic youth. Contrary, rates for smartphone ownership are slightly higher: 40% for Black non-Hispanic and 43% for Hispanic youth. Proliferation of technology is a reality, and governmental agencies (e.g., National Institute of Health) emphasize the use of mobile phones as a medium of reach and intervention (Krishna, Boren, & Balas, 2013; Riley et al., 2011; Tate, 2008).

For minority groups, in particular, technology can be a gateway for educational and prevention outreach, given its low cost and accessibility. Ethnic and racial disparities in rates of literacy, healthcare access, and chronic health conditions are well-documented and present a serious public health concern (CDC, 2015; Russell, 2010; U.S. Department of Health and Human Services, 2011). To capitalize on the potential of portable technology, the in-depth understanding of feasibility, acceptability, and opportunities of using mobiles phones with minority youth is the first step.

Due to high prevalence of mobile devices among youth, one would expect a relative ease of implementing EMA studies in this population. However, an imposition of frequent and repeated assessments presents unique challenges related to self-discipline in the developmental context characterized by elevated levels of impulsivity and a general propensity towards immediate gratification (Romer, Duckworth, Sznitman, & Park, 2010; Van den Bos, Rodriguez, Schweitzer, & McClure, 2015; Romer et al., 2010). Prior research has also demonstrated that African-American youth exhibit higher levels of impulsivity compared to European peers (Pedersen, Molina, Belendiuk, & Donovan, 2012). Thus, feasibility and EMA adherence should be examined directly in the population of interest.

Beyond relevance to public health and education, issues of adherence have several important implications. First, specific experiences of adolescents may be misrepresented if they are systematically absent from assessments (e.g., not responding to assessments when involved in physical activity), resulting in biased generalizations about the types of activities engaged in and the amount of time spent on various activities. Second, aggregate scores (e.g., daily or overall averages) that are computed from EMA data are under or overestimated when certain assessments are missed. Third, a number of techniques for analysis of EMA data (e.g., random-effects modeling) employ the likelihood-based estimation procedure, which is based on the assumption that data are missing at random (such that the value of missing scores does not depend on whether that score is missing; Black, Harel, & Matthews, 2012, Schafer & Graham, 2002); therefore, the nature of data missingness in EMA studies should be closely evaluated to substantiate use of such statistical tools. Finally, a practical reason for understanding rates and patterns of adherence is directly related to determining the feasibility of EMA studies and informing about phenomena suitable to EMA research within minority youth.

Past Studies on EMA Adherence with Adolescents

Previous reports of adherence rates in EMA studies among adolescents vary greatly, ranging from 46% (Shernoff & Schmidt, 2008) to 94% (Shernoff & Vandell, 2007). This high degree of variability may be attributed to the lack of standards regarding how adherence is conceptualized and computed. Stone and Shiffman (2002) proposed assessing adherence rates as a percentage (%) of completed assessments against the originally planned schedule. Yet, it has been noted that assessments are often not delivered due to technical difficulties arising from software and hardware malfunctioning (DeVito Dabbs et al., 2011). To account for that, we proposed several indices that highlight different aspects of adherence (The Authors, under review). Generally, if device fidelity is not accounted for, resulting adherence rates are underestimated and impact conclusions around feasibility.

Current knowledge of real-time data collection adherence rates among adolescents comes mostly from studies that have employed paper-and-pencil methods, in which electronic prompts (e.g., delivered by programmable watches) were followed by entries in a paper journal. One drawback of the paper-and-pencil medium is that adherence cannot be tracked in real time. Moreover, a cautionary statistic from an adherence study with adults (Stone, Shiffman, Schwartz, Broderick, & Hufford, 2003) suggests that up to 90% of all paper-and-pencil entries were back entered at later time points (e.g., end-of-the-day or end-of-the-study), raising concerns about data validity. A study with children (Mean age = 12.3 years) demonstrated that electronic assessments yielded more completed days (Mean = 6.6 days out of 7 possible) compared to paper entries (Mean = 3.8 days out of 7; Palermo, Valenzuela, & Stork, 2004). The same study showed that electronic records contained fewer errors and question omissions. Thus, a detailed investigation of adherence that is tracked in real time could expand the current knowledge of the feasibility of EMA studies among adolescents.

Mobile Technology and Minority Youth

Little knowledge around adherence to EMA with minority youth parallels the general paucity of research in this area. A few existing investigations are promising. For instance, a feasibility pilot study by Garcia et al. (2004) examined a text-based messaging system for Latina girls in the context of assessing physical, mental, emotional, and social well-being. Response rates to text-delivered assessments varied from 53 to 99%, with an average response rate of 80%. While no formal statistical analyses were conducted, rates on weekends appeared to be higher. Another study utilized wearable watches to signal time to record momentary perceptions of social support in a paper diary in a sample of African-American sixth graders (Hammack et al., 2004). Completion-based inclusion cut-off point was set at 30% (an additional complication was that diary entries could not be time verified), and the median response rate was recorded as 82%. One other study utilized paper-and-pencil diaries with inner-city African-American youth (Bohnert, Richards, Kohl, & Randall, 2009; Larson, Richards, Sims, & Dworkin, 2001), but rates of adherence could not be objectively verified given no automated time stamps.

None of the aforementioned studies examined inter-individual differences or the impact of contextual factors on adherence, which could play an important role. In summary, the paucity of studies performed specifically with minority youth highlight a general gap in knowledge around utilizing EMA methodology with this population.

Inter-Individual Differences and Contextual Factors Related to EMA Adherence

Inter-individual differences in adherence rates have been noted (e.g., Garcia et al., 2004); however, very few studies have examined systematic variability due to personal and contextual factors. Some personal characteristics stand out, however. For instance gender has been noted to impact adherence in some studies. As an example, Palermo and colleagues (2004) found higher rates of adherence among 8 −16 year old boys filling out pain diaries. In contrast, Sokolovsky and colleagues (2014) found lower rates for boys enrolled in a study around smoking behaviors. Finally, Heyer and Rose (2015) reported no gender differences in youth with migraine. Age can be another contributing factor, yet the effect of this variable is equally inconclusive (Bohnert et al., 2009; Sokolovky et al., 2014).

Beyond individual differences, frequency of assessments and contextual factors can also play a role. For instance, demands associated with a more frequent schedule of self-report may impose an additional burden and result in lower adherence (e.g., Courvoisier, Eid, & Lischetzke, 2012). Structural environmental factors around days of a week (e.g., weekends vs. weekdays) can impact adherence, given variability in routine and schedule (Garcia et al., 2004; Heyer & Rose, 2015). The emotional context can also play a role. Sokolovsky and colleagues (2014) found positive affect to be associated with a higher likelihood of responding to momentary EMA prompts, while general mean negative affect associated with the lower overall rate of adherence in adolescents. Finally, the same study found that being at home is associated with lower likelihood of responding to EMAs. Given the relatively new area of research, the impact of most contextual and personal factors is still largely unexplored.

The Present Study

The present study aims to fill several gaps in the literature. First, we aim to examine feasibility and acceptability of using EMA methodology with urban minority youth – a generally understudied population. Second, we aim to contribute methodologically by conducting an in-depth investigation of adherence to an EMA protocol in a context of examining out-of-school physical activity and inactivity among urban minority youth.

The following research questions (RQ) guided our investigation: 1) What quantitative indices of adherence can be used to parse out device fidelity from individual engagement to promote accurate and clear reporting? 2) How does the frequency of assessments (one, three, or six times a day) influence adherence? 3) What is the impact of situational factors on adherence (e.g., study days, times of the day, engagement in physical activity)? 4) What are the effects of personal characteristics on adherence (e.g., gender, mood)?

We hypothesized that adherence rates would be generally comparable to those found with non-minority youth. We did not have directional hypotheses around gender and age. We hypothesized that higher frequency of assessments would be associated with lower adherence. Further, we expected to see a progressive decrease in adherence over the course of the study and, generally, on weekends. Finally, we hypothesized that negative affect and engagement in activities that might interfere with device use (e.g., physical activity) would be associated with lower adherence. We did not have a directional hypothesis for feeling socially connected with others.

In the following sections, we provide details of the study design, explore different computational techniques for assessing adherence rates, and summarize details of the findings. The focus of the discussion is on the implications of the current findings for future EMA studies and directions of future lines of research.

METHODS

Participants and Procedure

In the current study, EMA data were collected to determine the impact of contextual factors, attitudes, and feelings on the level of out-of-school physical activity and inactivity among urban minority youth. One hundred and twenty-six seventh- and eighth- grade students from two public middle schools (75 and 51 from each school) in Central Pennsylvania participated in the study, and everyone completed the study (100% completion rate). The study protocol was approved by a university Institutional Review Board and the school district; written informed consent forms were collected from students’ parents and students provided written assent. Of the total sample, 52 were boys (41.3%) and 51 (40.5%) were seventh graders. The racial/ethnic distribution was as follows: 95 (75.4%) African-American, 26 (20.6%) Hispanic, and 5 (4%) other.

Students’ daily experiences were assessed via electronic surveys delivered with preprogrammed mobile phones. The study lasted for 10 days and included 4 weekend days and 6 week/school days. Surveys were comprised of momentary and end-of-the-day assessments. For momentary assessments during weekdays, a total of three survey prompts occurred after school hours, one each within the following two-hour intervals: 3–5pm, 5–7pm, and 7–9pm. During weekends, six prompts were scheduled during the following intervals: 10am–12pm, 12–2pm, 2–4pm, 4–6pm, 6–8pm, and 8–10pm. Thus, assessments followed a pseudo-random schedule, such that they occurred within each time block and were spaced at least 30 minutes apart. Over the course of 10 days, each student was prompted to complete 42 assessments. Momentary assessments were designed to capture immediate experiences and psychological states (e.g., “Where are you right now?”, “Are you feeling sad right now?”).

In addition, at the end of each day, students were asked to respond to an end-of-the-day assessment, which contained global questions about experiences during an entire day (described further). A total of 10 end-of-the-day assessments were expected. These assessments were used in this study to investigate adherence and provide information about context and personality traits. All prompts and responses were time and date stamped, and data were transmitted to a secure server in real time. At the end of the study, participants who returned devices with at least 80% of the prompts answered were compensated $40 for their participation and returning the device.

Measures

Device fidelity

When reporting on EMA data, most studies do not differentiate between human and technology factors that contribute towards the overall volume of gathered data. Considering and parsing out device fidelity, however, is necessary for the overall clear and accurate picture (see DeVito Dabbs et al., 2011, for some discussion). In this study, device fidelity was established by comparing times and days when prompts were actually delivered to the predetermined assessment schedule. Rates of device fidelity were computed for momentary (out of 42 expected) and daily (out of 10 expected) assessments.

Individual adherence was expressed by three indicators. First, for each momentary delivered prompt, we established whether or not it was answered (1/0). Second, daily adherence rates to momentary assessments were computed as a ratio of the answered to delivered assessments. For example, a day with all delivered prompts being answered yielded a 100% index, whereas a day with 5 answered prompts out of 6 delivered yielded a 5/6 or 83.3% index. Of note, denominator scores varied to reflect the number of assessments actually delivered to each participant on a given day. Third, for each end-of-the-day assessment, binary indicators were constructed to reflect whether a particular prompt was answered (1/0). Through adjustment of denominators, all above adherence indices take into account information on device fidelity and do not penalize participants for not responding to assessments that were never delivered in the first place.

Day and time characteristics were recorded for each assessment prompt and included binary weekend/weekday indicators, assessment order (1 through 3 on weekdays and 1 through 6 on weekends), and a study day (1 through 10). Based on preliminary evidence, day of the week and, by extension, time of the day could influence adherence rates (e.g., Courvoisier, Eid & Lischetzke, 2012; Garcia et al., 2014).

Situational and personal characteristics were constructed in order to differentiate between days with higher and lower momentary and daily adherence rates and between individuals with different adherence behaviors. Situational characteristics were comprised of responses to end-of-the-day assessments and included summaries of time spent on different activities (relaxing, hours of sleep, physical-, outdoor-, and work-related activities), quality of social interactions rated on the 0–100 slider scale (“Overall, did you get along with people today”) as well as emotional well-being (feeling bored, energetic, in control, relaxed, anxious, calm, sad, happy, lonely, angry, safe, good about oneself, having fun, all rated on the 0 – 100 slider scale with a time reference of “Overall, how … were you today”). The design of the study combined momentary with end-of-the-day assessments, which presented a unique opportunity to investigate the impact of activities and emotional states on adherence rates. Typically, situational factors are not available for missing data. Personal characteristics were constructed as mean summaries of end-of-the-day responses to represent more stable person-specific tendencies towards daily activities and social and emotional well-being. To derive global scores of emotional experiences, we carried out exploratory factor analysis on mean summaries of daily ratings of mood and attitudes. Factor extraction was based on the Kaiser rule of eigenvalues > 1 and interpretability. A three-class solution was chosen and included positive affect (comprised of responses to items around having fun, getting along, feeling good, safe, in control, happy, energetic, and bored), negative affect (feeling sad, lonely and angry), and anxiety (anxious, relaxed, calm). The Varimax orthogonal rotation was applied to compute individual factor scores.

Analytical Approach

To determine rates of adherence and the impact of assessment frequency (RQ1 and RQ2), descriptive statistics were computed for all indicators of device fidelity and individual adherence. Changes in adherence rates to momentary assessments across time (RQ3) were examined using the two-part random-effects model for semicontinuous longitudinal data to account for the non-normal and highly skewed outcome distribution (Olsen & Schafer, 2001) and nested data structure (observations within individuals). In accordance with the model, the outcome Y_ij (non-adherence) for a person i measured on a day j was converted to binary, u_ij, and continuous, v_ij, indicators such that

$$u_{\mathit{\text{ij}}}=\{\begin{array}{l}1\mathit{\text{if}}{Y}_{\mathit{\text{ij}}}\ne 0\hfill \\ 0\mathit{\text{if}}{Y}_{\mathit{\text{ij}}}=0\hfill \end{array}\text{and}{v}_{\mathit{\text{ij}}}=\{\begin{array}{l}\mathit{\text{log}}(Y_{\mathit{\text{ij}}})\mathit{\text{if}}{Y}_{\mathit{\text{ij}}}\ne 0\hfill \\ \mathit{\text{missing if}}{Y}_{\mathit{\text{ij}}}=0.\hfill \end{array}$$

The variable u_ij indicates non-adherence of a person i on a particular day j. Only for a nonadherent person, the magnitude of non-adherence is expressed by v_ij; otherwise, the value is missing. The binary outcome, u_ij, is modeled as a logistic mixed-effects model, where the logarithm of the odds (logit) of being non-adherent is modeled as a linear function of days in the study, centered at day 1. A similar linear effect of days is examined for the continuous outcome, v_ij, via a linear mixed-effects model. To examine inter individual differences in the initial adherence (intercept) and changes in adherence over time (slope) (RQ4), both models included personal characteristics as covariates. The random coefficients from both models are assumed to be jointly normal and possibly correlated. This two-part modeling approach takes care of the complex outcome that violates the assumption of normality, is measured on a continuous scale (adherence rate), and has an overrepresentation of floor values corresponding to perfect adherence.

In a separate two-part random-effects model, we examined the impact of situational predictors and weekday/weekend effects (RQ3) on the probability of non-adherence and its magnitude, controlling for interpersonal differences. Due to the large number of measures of context, we preliminarily selected those that correlated with rates of daily adherence at the magnitude of ≥ |.05| (while, likely, correlations were attenuated due to clustering of data, they were used for a preliminary investigation, and the cut-off point was chosen to consider only variables with the strongest potential for impact). In both two-part random-effects models described above, a backward elimination technique was used to arrive at the final solution.

Finally, two binary logistic mixed-effects models were fitted to examine whether the assessment order (RQ3) played a role in adhering to individual momentary assessments, while controlling for personal factors. Dummy variables were constructed with the first assessment of the day as the reference point. For all the models, variables were retained at the p-value of ≤ .1.

RESULTS

Device Fidelity (RQ1)

Fifty-five participants (43.7%) received all 42 scheduled momentary prompts across 10 days of the study, and an additional 22 (17.5%) received 41 prompts. One hundred and ten participants (87.3%) received at least half (≥ 21) of all scheduled prompts. The median number of delivered momentary assessments was 41 (Mean = 35.2, SD = 9.8). In regards to daily assessments, 21 participants (17.2%) received all 10 end-of-the-day assessments. The median number of delivered daily assessments was 7 out of 10 (Mean = 6.6, SD = 2.7). Technical difficulties accounted for the missed prompts and were triggered by battery removal, hard or soft reset of phones, and phone misuse. The study personnel were available on a daily basis to handle technical difficulties and provided students with remediated phones as soon as possible.

Individual Adherence on Weekdays and Weekends (RQ2)

Out of 4,431 delivered momentary prompts, participants responded to a total of 3,612 (81.5%). The response rate was 83.3% on weekdays and 80.2% on weekends. Based on the individual adherence scores, the median daily adherence rate to momentary assessments was 86.7% (Mean = 81%, SD = 18.4). On weekdays, the median rate was 88.9% (Mean = 82.2%, SD = 19.8) and, on weekends, the median rate was 87.5% (Mean = 78.9%, SD = 21.9). Adherence to end-of-the-day assessments was, on average, 93.8% (SD = 12.6), with 88 study participants responding to all delivered prompts (72.1%).

As a point of comparison, if device fidelity is not accounted for and adherence rates are computed solely based on the number of expected assessments (42 overall, 18 on weekdays and 24 on weekends), the overall adherence would equal to 68.3% (SD = 25.1%). Weekday adherence would be 70.2% (SD = 26.9%) and weekend 66.8% (SD = 26.9%). In this case, technical malfunctioning is counted against participant adherence.

Descriptive Statistics for Personal Characteristics

Table 1 summarizes descriptive statistics for personal characteristics, including demographic variables, indicators of time spent on different daily activities, and ratings of emotions and attitudes. Overall, ratings of positive emotions tended to cluster in the upper part of the response scale. Negative emotions were less widely represented.

Table 1.

Descriptive Statistics of Personal Characteristics (N = 126)

Variable Name	Frequency (%) / Mean	Standard Deviation
Female	75 (59.5%)
Grade 7	51 (40.5%)
Sleep Hr/Day	7.26	1.58
Physical Activity Hr/Day	1.78	.72
Working Hr/Day	1.07	.56
Relaxing Hr/Day	1.86	.71
Getting Along	80.88	18.52
Lonely	23.28	19.99
In Control	83.10	19.08
Having Fun	79.22	19.43
Energetic	75.01	22.63
Bored	63.46	25.63
Relaxed	76.42	20.34
Anxious	45.65	26.70
Calm	73.98	22.15
Sad	25.41	21.65
Happy	77.10	20.83
Angry	27.96	20.44
Feeling Good	83.61	19.10
Feeling Safe	83.89	17.98

Change in Adherence Rates to Momentary Assessments over Time and Effects of Personal Characteristics (RQ3 & RQ4)

Based on the results of the two-part random-effects model for semicontinuous longitudinal data, we examined temporal changes in the probability of full adherence to momentary prompts (the first part of the outcome in the two-part model) and the degree of non-adherence for those without full adherence (the second part of the model). Results (presented in Table 2) suggest that the initial probability of perfect adherence was estimated at .74 and was computed by the following formula: 1 − (exp^Parameter/(1 + exp^Parameter)), controlling for covariates. Over time, the probability decreased in a linear fashion (solid line in Figure 1). Adolescents with a higher level of overall positive affect demonstrated a higher chance for perfect adherence (dotted line in Figure 1); conversely, those who spent more time than average engaged in physical activity had a lower chance for perfect adherence (dashed line in Figure 1). For adolescents who did not demonstrate perfect adherence, the rate of adherence was stable over time (second part of Table 2) and estimated to be .76. It was higher for those with higher ratings of overall positive affect and lower for those who spent more hours engaged in physical activity.

Table 2.

Results of the Two-Part Random-Effects Model Examining Changes in Adherence Rates to Momentary Assessments Over Time Controlling for Personal Characteristics

Parameter Name (Centered at Value)	Parameter Estimates (Standard Error)	p-value	Exponent of Parameter
1. Logistic mixed-effects model
Intercept 1	−1.044 (.157)	< .001	.352
Day (0)	.137 (.026)	< .001	1.147
Physical Activity (1.75)	.336 (.152)	.027	1.399
Positive Affect (0)	−.160 (.092)	.081	.852
2. Linear mixed-effects model
Intercept 2	−1.163 (.056)	< .001	.313
Day (0)	.015 (.010)	.119	1.015
Day*Physical Activity (1.75)	.026 (.009)	.004	1.026
Day*Positive Affect (0)	−.015 (.006)	.016	.985
Residual Variances
Var (Intercept 1)	.713 (.218)	.001
Var (Intercept 2)	.031 (.017)	.065
Var (1. Day)	.001 (<.001)	.002
Var (2. Residual)	.234 (.018)	< .001

Figure 1.

Estimated Changes in the Probability of Perfect (100%) Adherence over the Study Period (10 Days) Based on Results from the Two-Part Random-Effects Model

Situational Predictors of Adherence Rates to Momentary Assessments (RQ3 & RQ4)

In addition to temporal changes, we examined situational characteristics differentiating between levels of adherence. On days when individuals reported higher than average levels of boredom, rates of perfect adherence were lower (first model in Table 3). On days when adherence was not perfect, non-adherence was higher when adolescents reported higher levels of boredom and anger and lower when they reported higher levels of anxiety, with general levels of positive affect and physical activity controlled for (second model in Table 3).

Table 3.

Results of the Two-Part Random-Effects Model Examining Situational and Personal Characteristics Affecting Adherence Rates to Momentary Assessments

Parameter Name (Centered at Value)	Parameter Estimates (Standard Error)	p-value	Exponent of Parameter
1. Logistic mixed-effects model
Intercept 1	− .991 (.136)	<.001	.371
Weekend	1.084 (.178)	<.001	2.956
Bored.D (63)	.006 (.003)	.046	1.006
Physical Activity (1.75)	.381 (.181)	.035	1.464
Positive Affect (0)	−.226 (.120)	.060	.798
2. Linear mixed-effects model
Intercept 2	−.923 (.034)	<.001	.397
Weekend	−.477 (.048)	<.001	.621
Bored.D (63)	.002 (.001)	.047	1.002
Anxious.D (45)	−.002 (.001)	.020	.998
Angry.D (28)	.003 (.001)	.002	1.003
Residual Variances
Var (Intercept 1)	.603 (.250)	.016
Var (Intercept 2)	.039 (.013)	.003
Var (2. Residual)	.144 (.017)	< .001

Effect of Assessment Order on Adherence to Momentary Assessments on Weekdays and Weekends (RQ3 & RQ4)

On weekdays, the last assessment of the day had the highest probability of being responded to (left panel of Figure 2). Individuals with higher overall positive affect were more likely to respond to momentary prompts during the week (p < .05; dotted line in Figure 2), and individuals who reported higher than average levels of physical activity had a decreased likelihood of responding to momentary prompts during the week (p < .05; dashed line in Figure 2). On weekends, the middle of the day assessments (between the hours of 12 and 4 pm) were less likely to be responded to (right panel of Figure 2). This finding was robust for all study participants.

Figure 2.

Estimated Effects of the Assessment Order on Adherence to Momentary Assessments Based on the Results from the Logistic Random-Effects Model. The First Assessment of the Day Was Used as a Reference Points. Significant Differences in the Probabilities (p < .05) are Marked with Asterisks (*).

DISCUSSION

With 93% of adolescents between the ages of 13 and 17 reporting cell-phone use (Lenhart, Purcell, Smith, & Zickuhr, 2010), and a general high acceptance of survey technologies among this population (Palermo, Valenzuela, & Stork, 2004), EMA methods are increasingly being used in studies with adolescents. Recent applications of EMA range from studies on academic engagement (Shernoff & Schmidt, 2008) and after-school activities (Shernoff & Vandell, 2007) to mental (Primack et al., 2011; Silk et al., 2011) and physical health (Helgeson, Lopez, & Kamarck, 2009; Mulvaney et al., 2012). The ubiquitousness of smart phones makes collecting EMA data highly feasible, presenting new opportunities for understanding adolescent development in a more fluid manner. The current study provides a careful and detailed exploration of adherence among urban minority youth in order to assist future researchers in conceptualizing adherence measures and conducting EMA studies with these populations.

The rate of adherence in the current study was relatively high, especially given the fact that all study participants completed the study, returned devices and met requirements for compensation. The overall mean adherence rate for all momentary assessments was 81%, and the adherence rate for end-of-the-day assessments was 93.8%, with both indicators computed based on the adjusted number of received prompts. These rates are comparable to or even on the higher end of those in adults, which have been shown to range from 50 to 90% (Stone & Shiffman, 2002), and seemingly challenge the previous notion of suboptimal adherence among adolescents (Hektner, Schmidt, & Csikszentmihalyi, 2007). A direct comparison between the two populations, however, should be made with caution, since formulae for computing adherence vary across studies.

Computed rates are also comparable to or even somewhat higher than those found in studies with other minority samples (Bohnert, Richards, Kohl, & Randall, 2009; Garcia et al., 2014, Hammack et al., 2004; Larson, Richards, Sims, & Dworkin, 2001). Differences can be attributed to the sample composition (e.g., Latina girls, Garcia et al., 2004) or the method of EMA administration (e.g., paper and pencil; Palermo et al., 2004). Further, no investigations considered failure in signal delivery as a contributing factor to lower adherence rates. The previous studies have not explicitly focused on adherence and lack a detailed conceptualization of this phenomenon, which limits a more direct comparison.

An important implication of our analyses is that adherence can be, and, we believe, should be, represented as a complex, dynamic, and pluralistic construct. Isolating device fidelity from computations of adherence is of great importance because it diminishes the risk of underestimating a participant’s level of engagement with the device. In line with our earlier work (Authors, under review), in this study, adolescents were considered accountable only for those assessments that were successfully delivered by phone. We observed multiple assessments that were not delivered either at the momentary or daily levels, related to technical difficulties. Such difficulties are not uncommon (Shiffman, Stone, & Hufford, 2008) but rarely explicitly reported. Given that our team was proactive and experienced in monitoring real-time data collection, studies that utilize technology should explicitly budget for data loss due to technical difficulties. In addition, extensive pilot work would help to ensure proper device functionality. New generations of phones and other wearable devices also make the task of collecting data in real time and real life more user- and research- friendly.

We found that a number of factors had an impact on adherence rates. As in previous studies with adults (Broderick, Schwartz, Shiffman, Hufford, & Stone, 2003; Courvoisier, Eid, & Lischetzke, 2012; Litcher-Kelly, Kellerman, Stone, & Hanauer, 2007), adherence in our study decreased over time, with the probability of perfect adherence decreasing from about 75% on day one to 50% on day ten. Given the high demand of some intensive EMA schedules, initial training should emphasize the importance of consistent engagement with the device. In addition, continuous real-time monitoring of adherence can be conducted, and motivational interventions (e.g., text messages, phone calls) as well as incentives can be provided in an effort to sustain high levels of adherence over time (Hufford, 2007). These factors warrant further exploration.

The design of the current study offered an interesting opportunity to examine the effect of assessment frequency on adherence rates. Adolescents were very successful with end-of-the-day assessments, with an average completion rate of 93.8%. By design, school hours were exempt from assessment prompting, and weekends offered an opportunity for a higher sampling rate (six vs. three per day). The rate of adherence during the week was higher (83.3%) compared to that on weekends (80.2%). One should be cautious, however, in attributing this effect solely to the increased demands of sampling, given the potential vast differences in activities of engagement. Optimistically speaking, the difference between weekday and weekend adherence was rather small, which, from a methodological standpoint, supports the feasibility of a relatively high sampling rate. When deciding on frequency of sampling, factors such as the nature of the construct under investigation, the rate at which it is expected to change, and feasibility should all be considered (for a more detailed discussion of sampling in EMA studies, see Collins, 2006; Shiyko & Ram, 2011) .

Both the time of the day and the order of assessments may be possible modifiers of adherence (Courvoisier, Eid, & Lischetzke, 2012; Csikszentmihalyi & Hunter, 2003). Contrary to previous research demonstrating increased challenges with collecting the first and the last assessments of the day, our sample had the most challenge with middle-of-the-day assessments. The types of activities that adolescents are involved in during such times may account for the lower adherence rates observed during the mid-day periods. Since missed assessments provide no indication of the reason for missingness, only indirect explanations can be inferred. End-of-the-day assessments may provide some insight into situational characteristics that may possibly account for decreased rates of adherence. On days when participants reported higher levels of boredom and anger, in particular, adherence suffered. More research is needed in this area to replicate and extend findings regarding situational factors that have an impact on adherence. Asking directly about reasons for missing assessments, at later times, and using collateral sources of information (e.g. pedometers, global positioning systems) may also shed light onto contextual factors.

Examination of personal characteristics helped to understand inter-individual differences further rather than treating the sample as a homogenous unit. Encouragingly, our study showed no differences in adherence between boys and girls and between the two grade levels (grades 7 and 8). However, adolescents involved in higher levels of physical activity consistently demonstrated lower levels of adherence. This finding is concerning, given the fact that EMA methodology is frequently used to study extent of involvement in and attitudes towards physical activity (e.g., Dunton, Whalen, Jamner, & Floro, 2007; Dunton, Kawabata, Intille, Wolch, & Pentz, 2011). It appears that real-time self-report might not be the best mechanism to accurately measure physical activity, since the activity itself may interfere with carrying the device, hearing audible prompts, and responding to questions in real time. Alternative, more unobtrusive ways may be preferable (e.g., accelerometers; Martinez-Gomez, Welk, Calle, Marcos, & Veiga, 2009; Robertson, Stewart-Brown, Wilcock, Oldfield, & Thorogood, 2011; Van Coevering et al., 2005), with EMA offering complementary information. Finally, emotional health was positively associated with adherence, such that adolescents reporting higher overall levels of positive affect also demonstrated higher adherence rates. This may be an artifact of the social desirability bias, with youth over-reporting on positive experiences and underreporting on negative (Crowne & Marlowe, 1960). Thus, for studies in which affect is the primary outcome of interest, the issue of missed assessments when negative emotions are heightened may pose important implications and require additional training efforts, real-time monitoring of adherence, and auxiliary non-intrusive methods of measurement.

In the interest of promoting accurate interpretations of the results discussed herein, it is important that we note several limitations of the current study. While overall adherence rates were notably high, the sample was comprised of volunteers, and personal interest in the study could have played a role. Given the study design, we had limited access to important personal information on study participants, such as academic performance, which has been shown to affect adherence in past research (Salamon, Johnson, Grondin, & Swendsen, 2009). Finally, the study design was correlational, and effects of situational and personal characteristics should be interpreted with this knowledge in mind.

CONCLUSIONS

To our knowledge, this was the first study to investigate, in detail, adherence rates to mobile-phone based surveys among a sample of urban, minority youth. The technology allowed for the tracking of adherence in real time and our analyses demonstrated the complexity of this phenomenon. Differences in reported rates of adherence in the literature may be largely an artifact of different studies using different formulae for computing adherence. Based on our findings, we recommend that personal and situational factors that can either enhance or compromise adherence be carefully considered when developing EMA study protocols. In conclusion, this study underscored the importance of examining the phenomenon of adherence in EMA studies employing adolescent populations, demonstrated the feasibility of conducting EMA research with minority adolescents, and contributed to the current body of literature on EMA study design.