Compliance to cell phone-based EMA among Latino youth in outpatient treatment

Abstract

Outpatient treatment practices for adolescent substance users utilize retrospective self-report to monitor drug use. Cell phone-based ecological momentary assessment (CEMA) overcomes retrospective self-report biases and can enhance outpatient treatment, particularly among Latino adolescents, who have been understudied with regard to CEMA. This study explores compliance to text message-based CEMA with youth (n=28; 93% Latino) in outpatient treatment. Participants were rotated through daily, random, and event-based CEMA strategies for one-month periods. Overall compliance was high (> 80%). Compliance decreased slightly over the study period and was less during random versus daily strategies and on days when alcohol use was retrospectively reported. Findings suggest that CEMA is a viable monitoring tool for Latino youth in outpatient treatment, but further study is needed to determine optimal CEMA strategies, monitoring time periods, and the appropriateness of CEMA for differing levels of substance use.

INTRODUCTION

Current outpatient treatment practices for adolescent substance abusers utilize cognitive-behavioral (CB) based therapies for relapse prevention (Dennis et al., 2002, 2004; Kaminer, 2001). During group therapy, youth are taught to self-monitor and identify affect, cognitions, and situational factors that make them vulnerable to relapse. However, their retrospective reports of antecedents of alcohol and other drug (AOD) use are subject to recall biases and limitations in autobiographical memory (Bradburn, Rips, & Shevell, 1987; Stone & Shiffman, 1994).

To avoid these sources of error, ecological momentary assessment (EMA) has been developed to monitor affect, cognitions, and behavior in real time in a person’s natural environment (Stone & Shiffman, 1994; Piasecki, Hufford, Solhan, & Trull, 2007). EMA has been used to assess smoking and drinking (Carney, Tennen, Affleck, del Boca, & Kranzler, 1998; Collins, Morsheimer, Shiffman, Paty, Gnys, & Papandonatos, 1998; Shiffman et al., 2002), antecedents of relapse (Shiffman et al., 2002; Armeli, Todd, & Mohr, 2005), and reactivity of AOD use to EMA (Hufford, 2007; Hufford, Shields, Shiffman, Paty, & Balabanis, 2002; Kirchner & Shiffman, 2013; Litt, Cooney, & Morse, 1998; Shiffman & Stone, 1998; Simpson, Kivlahan, Bush, & McFall, 2005) among adult populations.

The use of electronic devices (e.g., laptop) for EMA addresses many of the compliance challenges found with traditional paper diaries, including time stamping to safeguard against faked compliance (Litt, Cooney, & Morse, 1998; Collins, Kashdan, & Gollnisch, 2003; Stone, Shiffman, Schwartz, Broderick, & Hufford, 2002, 2003) and the early identification of data entry errors (Collins, Kashdan, & Gollnisch, 2003). However, electronic devices may still be cumbersome to use for EMA multiple times a day, which necessitates participants staying in close proximity to the recording device throughout the day.

EMA collected via cell phone (CEMA) has the benefits of electronic devices and the added benefit of using a device that is already embedded in participants’ daily routines. This is especially true for youth; 78% of 12–17 year olds in the U.S. own cell phones (Pew Internet & American Life Project, 2013). Furthermore, EMA of affect, activities, social environment, dietary intake, and AOD use via handheld computer has been shown to be acceptable and feasible in adolescent populations (Henker, Whalen, Jamner, & Delfino, 2002; Whalen, Jamner, Henker, & Delfino, 2001; Whalen, Jamner, Henker, Delfino, & Lozano, 2002; Whalen et al., 2006).

The potential for CEMA to enhance current AOD treatment programs for youth is an important consideration in light of the impact that AOD has on this population. AOD use remains as a significant contributor to poor outcomes in adolescents, including unsafe sexual practices (Riggs et al., 2013; Staton et al., 1999; Zimmer-Gembeck & Helfand, 2008), poor academic performance (Diego, Field, & Sanders, 2003; Johnston, O’Malley, & Bachman, 2002), and delinquency (Brown, Gleghorn, Schuckit, Myers, & Mott, 1996; Brown & Ramo, 2006). Latino youth are disproportionately affected by AOD. In 2011, Latino youth in the 8^th and 10^th grades reported higher consumption of alcohol and drug use across multiple categories compared to African American and White youth (Johnston, O’Malley, Bachman, & Schulenberg, 2012). In addition, Latino youth suffering from substance use disorders (SUD) are less likely to receive SUD treatment than White adolescents (Cummings, Wen, & Druss, 2011), partly due to cultural differences that are not addressed in many treatment programs (Burrow-Sanchez, Martinez Jr, Hops, & Wrona, 2011; Goldbach, Thompson, & Holleran Steiker, 2011). In turn, there is little data on effective AOD treatment strategies for Latino youth, especially in regards to CEMA.

To date, the exploration of CEMA and other EMA schemes in treatment settings has taken place in adult and non-Latino youth populations. Successful demonstrations include CEMA on cocaine-addicted homeless patients (Freedman, Lester, McNamara, Milby, & Schumacher, 2006), an interactive telephone response system for self-monitoring following an alcohol treatment program (Rose, Skelly, Badger, Naylor, & Helzer, 2012), a CEMA application to facilitate treatment of youth mental health symptoms with general practitioners (Kauer et al., 2012; Reid et al., 2013), and CEMA of alcohol use in a youth health clinic setting (Kauer, Reid, Sanci, & Patton, 2009).

First steps to address this gap in knowledge call for an examination of the feasibility and acceptability of CEMA in Latino youth during outpatient drug treatment. A tangible and key metric of CEMA acceptability is compliance to filling out assigned assessments. Based on the intensity of repeated assessment that is inherent to EMA and CEMA, participant fatigue is an important consideration. Similar to retrospective report, (C)EMA compliance rates impact the quality of the data and the utility of (C)EMA as a relapse monitoring tool.

In this article, we examine CEMA compliance data from a feasibility study that administered CEMA as part of an adolescent outpatient drug treatment program in Los Angeles, California; most adolescents were Latino. Data was collected through text-message based CEMA of AOD and risky sexual behavior. We hypothesize two predictors of compliance based on findings from prior EMA studies in different populations. First, we hypothesize that youth will be less compliant over time (Broderick, Schwartz, Shiffman, Hufford, & Stone, 2003; Courvoisier, Eid, & Lischetzke, 2012). Second, we hypothesize that AOD use will be associated with lower compliance. The second hypothesis can be tested because we administered a timeline follow-back assessment that allowed us to examine reported AOD use on days when CEMA was not filled out. This hypothesis is based on a commonly-held notion among substance abuse researchers that non-compliance results from AOD use (McPherson, Barbosa-Leiker, Burns, Howell, & Roll, 2012; Shiffman, 2009), especially in filling out EMA diaries that may be difficult to complete while intoxicated (Shiffman, 2009).

METHOD

PARTICIPANTS

Youth between the ages of 12 to 18 were recruited from an adolescent outpatient treatment setting in Los Angeles, from November 2010 to June 2011. Approximately 35 to 40 youth were enrolled in treatment at any given time. Youth were informed about the study during group sessions or individually approached by research staff. Eligible youth were: 1) enrolled for treatment, with current enrollment expected to last at least a month; 2) able to use a cell phone; and 3) English-speaking in order to respond to a text message-based CEMA. A high proportion of the youth in treatment were Latino, though eligibility criterion 3) was not a recruitment barrier. All youth we encountered in the treatment setting spoke English; many youth were bilingual, speaking both English and Spanish. For youth younger than 18 years, we obtained parental consent for their participation and youth assent. Youth who were 18 years and older signed consent forms. Youth participating in the study received a $15 gift certificate for completing a baseline assessment. All participants received $25 per week and 500 free cell phone minutes every month for the entire duration of the study. In addition, youth who were compliant in responding to CEMA received additional cell phone minutes on a weekly basis.

The use of a cell that was provided by the study during the study period and incentivized cell phone minutes generated a high degree of interest in study participation among youth at the treatment setting. There were only two reasons that interested youth were not enrolled in the study. Some parent(s) did not want their children to participate in the study. Given a limited number of study cell phones and resources, we could only accommodate approximately 15 study participants at any one time. Study procedures were approved by the Institutional Review Board of the University of California, Los Angeles.

PROCEDURES

Eligible youth were administered a baseline assessment immediately following a screening interview that was conducted at the outpatient treatment setting in a private area. Afterwards, youth were assigned a mobile phone they would use to respond to a text message-based CEMA. During assignment, youth were trained and practiced filling out the CEMA with research staff. Wording of the CEMA questions and response choices were informed by focus groups and key informant interviews conducted with youth currently and formerly in treatment.

The study field tested CEMA in the framework of three CEMA sampling strategies that would likely be implemented in a treatment program: daily, event-based, and random assessments. Daily reports have been commonly utilized in prior EMA studies, due in part to reliance on paper diaries that were most tenable for once-a-day reporting. Electronic data capture facilitates event-based and random assessment of events that vary in intensity throughout the day, e.g. AOD use (Shiffman, 2007). Further details on the sampling strategies in this study follow:

• Daily assessment (D): Youth received one assessment questionnaire from an automated text messaging system once a day around 9:00 PM at night. Messages asked whether youth engaged in AOD use within the past 24 hours and if they engaged in sexual behavior after using alcohol or drugs.

• Event-based assessment (E): Youth self-initiated an assessment by typing a six digit code whenever they engaged in AOD.

• Random assessment (R): Youth received one assessment questionnaire from an automated text messaging system once a day at a random time after school hours, between the hours of 3:00 PM and 9:00PM. Questions in the first part of the assessment were similar to the daily assessment questions, except that the queried time period is since the last survey. Participants were also queried on any AOD and sexual behavior that they were engaging in at the time of the random assessment.

• Combined assessment (C): Youth received a daily assessment (D) and they were instructed to initiate an event-based assessment (E) if they engaged in AOD use.

Over the course of the study, participants were rotated through the four CEMA strategies (D, E, R, or C), with a month of participation in each rotation, followed by a month-long rest period in between rotations.

During the first two rotations (Rotations 1 and 2), participants were assigned to one of three basic CEMA strategies (D, E, or R), to establish our CEMA procedures and the acceptability of CEMA with participants. During Rotations 3 and 4, we added the C strategy as a test case for another viable CEMA strategy in an adolescent treatment setting. CEMA strategy assignment was not random and was done at the discretion of the research staff. For example, the E strategy was more likely to be assigned to youth who were newer to the treatment program, more likely to be using AOD, and in turn, more likely to have AOD-usage events to report on throughout the day. After Rotation 1, assignment was carried out to minimize repeating the same CEMA strategy in subsequent rotations. The flow of study participants through the different EMA strategies is summarized in Figure 1. We allowed participants to enroll during any of the rotations. Eleven participants were enrolled in Rotation 1. Four of 11 participants who started in Rotation 1 completed all four possible rotations.

FIGURE 1.

Flow of study participants through four rotations by assigned assessment modality: Daily (D), Event-based (E), Random (R), or Combination assessment (C; D and E).

An overview of the assessment is shown in Figure 2, including the order of the questions and conditional branching of questions based on participant responses. There was a preprogrammed algorithm for each assessment and a participant’s response determined the number of follow-up questions he/she needed to answer. The R assessment questions covered time periods since the last assessment and events that were currently happening. Therefore, conditional branches in the R assessment tended to contain more questions (range = 10 to 40 questions) than branches in the E (range = 10 to 19 questions) and D assessments (range = 6 to 21 questions). The majority of the CEMA questions were multiple choice. For example, the final stem question in the D assessment was displayed in text-message format as follows: “Have you used any drugs today? a. Yes, b. No”. A participant could reply with an “A” or “B” response. Some questions allowed for unique responses (e.g., what went through your mind before you used drugs?).

FIGURE 2.

Outline of CEMA structure showing the order of the questions, conditional branching of questions based on study participant responses, and the number of questions in each branch (n). * Rotation 1 did not query on sexual behavior since last survey in random assessment if participant did not report alcohol or drug use since last survey.

Question prompts were repeated if a text message response was not sent within three minutes of the prompt. Non-response to the second question prompt resulted in an incomplete assessment for the day if a response was not within 24 hours. Overall, response times were reasonable. Median completion times were 8 minutes for D assessment, 16 minutes for R assessment, and 192 minutes for E assessment. As shown in Figure 2, the E assessment was designed to be longer. Initial questions to ascertain current AOD use were followed by a three-hour recovery period. Afterwards, more detailed questions were prompted. Participants also took more than three hours to complete nine percent of the D and R assessments, which was an unanticipated result of the flexible response times that were permitted.

MEASURES

Baseline assessment

Participant characteristics included age, gender, ethnicity, current school attendance (yes / no), if currently on probation (yes / no), and the number of days in drug treatment at recruitment site.

Substance use severity was measured with the 16 item Substance Problems Scale (SPS) of the QAIN-Q Instrument (Titus & Dennis, 2004) and is used to identify adolescents in need of a referral for a more detailed assessment. The first item queried participants on AOD use during the past 12 months. If AOD was indicated, participants were queried on yes/no items from two subscales: the Substance Use and Abuse Scale (8 items, alpha = .66) and the Substance Dependence Scale (7 items, alpha = .65). Based on the percentage of items endorsed in each subscale, participants were classified into minimal, moderate, or high urgency levels for needing a referral / treatment.

Participants were administered the extended version of the Timeline Follow-Back (Ext-TLFB; Crosby, Stall, Paul, Barrett, & Midanik, 1996; Midanik, Hines, Barrett, Paul, Crosby, & Stall, 1998) to assess for AOD use over the past month, including counts of the number of days using alcohol, marijuana, and other drugs. TLFB was administered both at baseline and at the end of each rotation during the study period.

Cellular ecological momentary assessment (CEMA)

The assessment questions focused on 1) whether or not alcohol and/or drug use took place; 2) the quantity used; 3) the antecedents to relapse (e.g, the affective, cognitive, and situational factors that trigger a person to use); 4) intensity of craving; 5) any engagement in risky behaviors; and 6) the activities and location of the research participant. Depending on the assessment strategy, the questions differed on how participants were instructed to recall last drug use, e.g. “today” or “since the last time you completed a survey”.

Alcohol consumption

Participants were asked if they used alcohol “today” on the daily assessment, “now” on the event-based assessment, and “since the last time you completed a survey” on the random assessment. Participants indicating alcohol consumption were further queried on the type of alcohol (“beer”, “hard liquor”, or “beer and hard liquor”) and the quantity. Beer drinkers were asked two additional questions for the number of “regular cans” and the number of “40’s” that they drank. Participants who indicated drinking hard liquor were asked an additional question for the number of “shots or mixed drinks” that they drank.

Drug use

Participants were asked if they used drugs in parallel time frames to alcohol consumption. Participants indicating drug use were asked to indicate which drugs they used, including “Marijuana”, “Ecstacy”, “Cocaine / Crack”, “Inhalants”, “Hallucinogens”, “Painkillers”, and “Meth”. Participants indicating marijuana use were also asked two additional questions: “How much did you use?” and “How many people shared with you?”.

Risky sexual behaviors

Participants were asked if they engaged in “sex (anal, oral, or vaginal)”. Participants indicating sexual activity were then asked three additional questions: “with whom”, if they or their partner used a “condom”, and if they or their partner used “birth control pills”.

Alcohol and drug-related contextual questions

Participants were asked when they used (e.g., “In the morning”), where they used (e.g., “Friend’s house”), with whom they used (e.g., “with girlfriend/boyfriend), and their activities (e.g., “hanging out”), thoughts (e.g., “had a bad day”), and feelings (e.g., “stressed”) prior to their use of alcohol and drugs.

CEMA compliance

Compliance to filling out CEMA on a given day during D and R assessment strategies was categorized as “yes”, for a completed assessment (1); or “no”, for a partially completed or missed CEMA for that day (0). We did not differentiate between partially completed (2% or 14 of 602 D assessments and 5% or 28 of 614 R assessments) and missed CEMA (10% of both D (n = 59 of 602) and R assessments (n = 71 of 614)), due to the small number of observations for partially completed CEMA. Analyses on a three-category indicator of compliance would have been impractical. For the E assessment strategy, non-compliance (0) could only result from partially completed CEMA, since E assessments were initiated by study participants. Given the different definition of compliance, E assessments were not included in statistical analyses on compliance.

For the C assessment strategy, analyses grouped daily assessments with assessments that originated from the D assessment strategy; E assessments were grouped similarly. This seemed reasonable, since levels of compliance did not significantly differ for daily assessments that were collected during the D and C strategies.

STATISTICAL ANALSES

A key study design feature was the rotation of study participants through different sampling strategies. This within-subjects study design potentially reduced variability on comparisons between sampling strategies relative to a study design that would have compared sampling strategies between separate groups of individuals. In turn, within-subjects comparisons offer greater statistical power relative to comparisons on independent groups. All regression models included random-effects (RE) at the participant-level to properly model correlations between repeated compliance data points on the same participant over time. RE act as intercepts that allow each adolescent’s mean outcome level to be lower or higher than the overall mean outcome level for a given set of covariate values.

RE multinomial logistic regression was used to estimate the probability of assignment to CEMA assessment strategy in order to check for imbalances across background characteristics that may have resulted from non-random assessment strategy assignment. F-statistics, degrees of freedom, and p-values are presented for omnibus tests of differences between at least two assessment strategies. Main analyses used RE logistic regression to examine predictors of compliance to CEMA (yes / no). We present t-tests, degrees of freedom, and p-values for regression coefficients. The predictive margin is estimated as the average difference in the probability of compliance between different response values for a given covariate in x, e.g. between a participant using and not using AOD. All analyses were conducted in SAS software Version 9.3 (SAS Institute Inc., Cary, NC). Logistic regression models were fit in the PROC GLIMMIX procedure; models incorporated random effects through the “random” statement.

RESULTS

Sample characteristics

On average, study participants (n = 28) were 16 years old (range = 13 to 18) and had been in the current drug treatment episode for three months (95.6 days; range = 9 to 254 days). About half of the participants were male gender (57%; n = 16) and were on probation (46%; n = 13). Most participants were attending school (82%; n = 23) and identified as being Latino (93%; n = 26); one participant reported being Asian or Pacific Islander and another participant indicated belonging to an ethnic group other than Latino, White, Black, Asian or Pacific Islander.

At baseline assessment, over half of the participants reported consuming alcohol (79%; n = 22) and marijuana (61%; n = 17) within the past 30 days; a little less than half reported using other drugs (43%; n = 12), including stimulants (n = 3), inhalants (n = 5), party drugs (n = 6), hallucinogens (n = 1), cocaine or crack (n = 2) and opiates (n = 3). No one reported using barbiturates, sedatives, heroin, or injecting drugs. Half of the 12 participants reporting the use of other drugs also reported using multiple drugs. On average, participants endorsed 5.8 out of a possible 8 items and 4.1 out of a possible 7 items on the SPS use-abuse and dependence subscales, respectively, indicating a moderate level of urgency for further AOD assessment and treatment.

Table 1 summarizes baseline characteristics across CEMA assessment strategies as a visual check for imbalances that may have resulted from non-random assignment. No obvious differences were observed. Likewise, CEMA strategy was not found to differ significantly based on separate regressions on each baseline characteristic (F(3, 49) = 1.29 to 0.09, p = .29 to .96).

TABLE 1.

Baseline characteristics of sample by assessment strategies within rotations.

	Daily (n = 8 youth)		Event-based (n = 13 youth)		Random (n = 21 youth)		Combination (n = 13 youth)
	n	%	n	%	n	%	n	%
Age, mean (SD)	15.6	(0.9)	16.2	(1.1)	15.5	(1.1)	15.9	(1.2)
Male gender	5	62.5%	8	61.5%	10	48%	7	54%
Attending school	8	100%	11	85%	20	95%	9	69%
On probation	3	37.5%	7	54%	8	38%	5	38.5%
Days in treatment, mean (SD)	96.8	(70.6)	87.5	(52.6)	96.2	(62.3)	88.8	(55.8)
AOD use over the past 30 days
Alcohol use	7	87.5%	10	77%	16	76%	10	77%
Marijuana use	4	50%	7	54%	13	62%	8	61.5%
Other drug usea	4	50%	6	46%	8	38%	7	54%
Substance use severity, mean (SD)
Substance use / Abuse scale	5.6	1.8	5.4	1.9	5.8	1.7	6.0	2.3
Dependency scale	4.8	1.8	4.1	1.8	4.2	1.9	4.8	1.5

^aIncludes use of cocaine, crack, hallucinogens, inhalants, opiates, party drugs, or stimulants.

CEMA reporting characteristics

There were a total of 1307 CEMA prompts that were received during daily (D) and random (R) assessment strategies or initiated during event-based assessment strategies (E) by the 28 participants over the study. The number of prompts closely matched the total number of days that study participants were in the study (roughly 30 days for each rotation in Figure 1), except for days where multiple event-based assessments were initiated or a daily and event-based assessment were filled out by participants assigned to the combination (C) assessment strategy. The analysis data contained 1282 of the 1307 total number of CEMA prompts (98%) across 1248 days of reporting (mean = 44.6 days, range = 2 to 118 days). Prompts excluded from the analysis data resulted from glitches in the preprogrammed algorithm for administering the text-message CEMA and nonsensical response patterns that made compliance difficult to calculate, e.g. event-based assessments that were initiated but then ended because participants did not indicate any alcohol or drug use.

Alcohol and other drug (AOD) use during the study

Table 2 shows the median and maximum number of days of AOD use (and percentage out of the total number of reporting days) that was reported by each participant. Both CEMA and TLFB reports are shown for comparison. Regardless of the reporting mechanism, reports of use were fairly low. Half of the participants reporting using alcohol or marijuana for three or fewer days and did not report any use of other drugs during the study period. We also calculated differences between numbers of CEMA and TLFB reports of AOD use across participants. As indicated in Table 2, reports of use were quite low across most substances, making meaningful comparisons difficult. However, we did find that CEMA during D, R, and C assessment strategies captured a higher number of days of alcohol use relative to TLFB; fewer days of alcohol use were reported by E assessment (Wilcoxon signed rank sum test, both p = .02).

TABLE 2.

Median and maximum counts of daily AOD use reported during daily (D), random (R), combination (C), and event-based (E) CEMA assessment strategies. Reports of AOD use reported by time line follow-back (TLFB) over the same time periods are shown for comparison.

	D, R, and C strategies (n = 25 youth)								E strategy (n = 13 youth)
	CEMA				TLFB				CEMA				TLFB
	Median		Max		Median		Max		Median		Max		Median		Max
	n	%	n	%	n	%	n	%	n	%	n	%	n	%	n	%
Alcohol	3	6.7%	18	62.1%	2	3.4%	16	55.2%	1	3.4%	3	10.7%	1	3.6%	10	34.5%
Marijuana	2	5%	22	58.6%	1	1.1%	26	45.6%	0	0%	10	34.5%	0	0	25	86.2%
Ecstasy	0	0%	2	5%	0	0%	2	5%	0	0%	2	7.1%	0	0%	2	7.1%
Cocaine	0	0%	2	3.5%	0	0%	1	3.3%	0	0%	2	6.9%	0	0%	0	0%
Inhalants	0	0%	13	22.8%	0	0%	15	26.3%	0	0%	2	7.1%	0	0%	3	10.7%
LSD	0	0%	1	1.8%	0	0%	0	0%	0	0%	0	0%	0	0%	0	0%
Painkillers	0	0%	3	5%	0	0%	2	3.3%	0	0%	4	7.1%	0	0%	2	14.3%
Meth	0	0%	2	6.7%	0	0%	6	10.5%	0	0%	2	7.1%	0	0%	5	17.9%

Compliance to CEMA

Table 3 summarizes the observed compliance data and shows high compliance across assessment days by CEMA strategy (D, E, and R) and rotation. Over 80% of assessments were completed across strategies. Table 3 also shows that compliance to D across rotations (88%) was 5% and 4% higher than compliance to E and R, respectively.

TABLE 3.

Percent (n / N) of completed assessments by rotation and CEMA strategy out of 1282 assessments among 28 study participants

Rotation
	1	2	3	4	Total
Daily	87% (97/111)	89% (99/111)	78% (110/141)	93% (223/239)	88% (529/602)
Event-based	90% (19/21)	88% (15/17)	82% (9/11)	71% (12/17)	83% (55/66)
Random	85% (100/118)	84% (98/116)	87% (125/144)	81% (129/236)	84% (515/614)
Total	86% (216/250)	87% (212/244)	82% (244/296)	87% (427/492)

Compliance was regressed on the following covariates in separate models: CEMA strategy (D or R), time from the baseline assessment (study days), time from the start of a rotation (rotation days) and demographic / baseline characteristics, including age, gender, current school attendance, probation status, the number of days in the AOD treatment program at the start of the study, indicator variables from the baseline assessment for the use of alcohol, marijuana, or other drugs during the past 30 days. Time-varying covariates were also included for TLFB-reported alcohol and marijuana use (yes / no). Other drugs were not included, due to low rates of reporting (Table 2).

Differences in compliance that were observed between D and R assessment strategies in Table 3 (88% versus 84%) were significant (t (1196) = 2.47, p = .01). Increases in both study days and rotation days were associated with lower compliance (t (1196) = −2.83 and −5.67 respectively, both p < .01). Compliance was significantly lower on days when alcohol use was reported by TLFB (t (1042) = −5.09, p < .001); the association between compliance and TLFB-reported marijuana use was in the same direction but not significant (t (1036) = −1.54, p = .12). None of the demographic and baseline characteristics were significantly associated with compliance.

The four significant covariates were entered into a multiple-predictor logistic regression model; study days was no longer significant (t (1039) = −1.62, p = .11) and excluded from the final model. Two-way interactions between remaining covariates were also tested in the model; none were significant. Results of the final model are shown in Table 4. The ratio between the generalized Chi-square statistic and its degrees of freedom for the final model is 0.51. A value greater than one would indicate over-dispersion in the data or model misspecification. Regression coefficient effects were in the same direction as they were in single-predictor models. Therefore, interpretation is the same. In terms of predicted probabilities, significant differences were relatively small. Compliance was estimated to be 4.0% less for R versus D assessment, holding other model covariates constant. Similarly, compliance was estimated to be 5.7% less at the end of a typical rotation (i.e., day 30) compared to the beginning of a rotation and 10.1% less on days when alcohol use was reported versus days when alcohol use was not reported.

TABLE 4.

Parameter estimates (B) and standard errors (SE) from random-effect logistic regression analysis of CEMA compliance across 1068 days of reporting (n = 25 participants)

Covariate	B	SE	t (df = 1040)	p
Daily vs. random assessment	1.61	0.51	3.17	.002
Days in rotation	−0.062	0.023	−2.73	.007
TLFB report of alcohol use	−1.99	0.44	−4.50	< .001

DISCUSSION

Two aspects of our study highlight the potential of CEMA as a tool for use in outpatient treatment with adolescents. First, overall compliance to CEMA was fairly high, above 80% for almost all EMA prompt types and rounds of participation. Compliance rates in our study are in line with a CEMA compliance rate of 80% that was found in an outpatient treatment program for mostly African American cocaine-addicted homeless patients (Freedman, Lester, McNamara, Milby, & Schumacher, 2006). Compliance rates in our study are notably higher than compliance rates to CEMA of AOD use in non-patient populations, including compliance rates of 50% in HIV-positive adults from various ethnic backgrounds (Swendeman et al., 2014), 69% and 30% in school and health clinic-based samples of Australian youth, respectively (Kauer, Reid, Sanci, & Patton, 2009). Higher CEMA compliance rates in treatment settings may benefit from the self-selection of motivated individuals who participate in treatment programs. Given the paucity of CEMA studies that have been conducted in treatment settings, further study is needed before conclusions can be drawn. Higher CEMA compliance rates we found in our study relative to non-CEMA assessment strategies in other non-patient Latino populations may also partly be due to the self-selection of motivated individuals in our sample. For example, 75% of Latino college students were retained for at least 80 days of daily automated telephone assessment of sexual behavior and AOD use before sex over three months (Schroder, Johnson, & Wiebe, 2007) and 60% of Latino students responded to a one-time mail-in survey that assessed adjustment to college (Hurtado, Carter, & Spuler, 1996).

Second, text message-based assessment that was utilized in our studyis easily scaled to treatment programs where patients use their own cell phones; smartphone features were not required. Costs are currently roughly estimated at about $500 per 500 messages sent per month, including hosting, and decreasing at scale (see dimagi.com for example). The ability to tailor text messages and CEMA, in general, is another important feature of cell phone delivery that opens the door for CEMA across treatment populations of different cultural and ethnic origins. Text messages in our study were written in English in line with the English-speaking population in the study clinic. Text messages can easily accommodate Spanish and other languages, as well as cultural nuances in the wording of questions. Text message-based assessment can handle fairly complex skip patterns as we demonstrated in our study to address differing levels of engagement in AOD use and sexual encounters. Similarly, Rodgers et al. (2005) tailored a text-message based intervention for smoking cessation based on a number of individual characteristics, including smoking history and barriers to cessation. Freedman, Lester, McNamara, Milby, and Schumacher (2006) recommended that researchers consider randomizing call times based on individuals’ reported sleep patterns and other lifestyle factors.

As hypothesized, we found decreases in compliance over the study period and on days when alcohol use was reported. A ten percent decrease in the probability of compliance was predicted for days when alcohol use was reported. Unfortunately, use of other substances was too low to adequately test for similar associations between use and non-compliance. Lower levels of compliance in the random (R) strategy compared to the daily (D) strategy was unexpected. Non-random assignment to the assessment strategies makes the impact of strategy on compliance difficult to interpret. As a potentially related issue, the median assessment completion time was twice as long for the R versus the D assessment strategy. We do not have an explanation for this. The median R assessment completion time does not decrease if we exclude participants who indicated using AOD now, which could have partly explained the longer median completion time. R assessment generally resulted in a larger number of questions than D assessment for comparable AOD and sexual behavioral patterns, but not enough to reasonably explain the difference in median completion times between R and D assessment. For example, a participant in the R strategy who indicating drinking beer, using marijuana, and engaging in sex since their last survey, but not using alcohol or drugs now was prompted to answer 26 questions. A comparable participant in the D strategy was prompted to answer 20 questions.

It is also important to acknowledge that the R strategy only prompted participants once a day, done in part because participants were likely to be in school at earlier times of the day. We cannot tell how a multiple-times-per-day R strategy would perform against a D strategy in terms of compliance. More intensive assessment offers more opportunities to respond but may be offset by introducing additional burden. Of course, a once-a-day assessment may not give a reliable picture of behaviors throughout the day. Clinicians are left with a delicate balancing act between the ability to minimize participant burden and potentially obtain better information. This points to the importance of considering synergistic effects between the timing of CEMA prompts throughout the day and the time frame over which the CEMA is conducted when designing a plan to support a treatment program. Questions around information utility for treatment providers and patient self-management support are high priorities for future research.

A number of study limitations need to be acknowledged. Our study sample was small and use of AOD was relatively low. Examining potential biases that were introduced by non-random assignment to CEMA assessment strategies was difficult as statistical comparisons of baseline characteristics and AOD use between assessment strategies was underpowered. As noted by Shiffman (2009), EMA needs to be tested on the spectrum of drug users. This is certainly needed in a larger sample of Latino youth in outpatient treatment before conclusions can really be drawn on the impact AOD use and other factors on compliance. Despite these limitations, our study provided an important proof-of-concept for the administration of text message-based CEMA in adolescent outpatient settings.