Examining the Feasibility of Ecological Momentary Assessment Using Short Message Service Surveying with Homeless Youth: Lessons Learned

Abstract

We assessed the feasibility of ecological momentary assessment using short message service (SMS) surveying with 150 homeless youth. We found that on average, participants completed 18.8 days of texts, but 30% of youth (N = 44) had texting data on 28 or more days. The average number of texts answered per day was 8.49 (of a possible 11). Forty-three percent of days had answers to all 11 texts sent that day, and 69.4% of days had answers to eight or more texts. We found significant differences in response rates by phone type. Seventy-three percent of youth reported that responding to our texts was very easy or somewhat easy and 69% endorsed the response option “just enough,” regarding the number of texts sent per day. The 30-day texting period was deemed “about right” by 74% of youth. Details of using SMS with homeless youth are provided, and results indicate that this data collection technique is feasible with this population.

Introduction

Each year in the United States, approximately 1.6 million adolescents experience homelessness (The National Center on Family Homelessness 2011). This group suffers numerous negative outcomes as a result of their unstable living situation including victimization (Tyler et al. 2004), substance misuse (Hadland et al. 2011), and poor mental health (Brown et al. 2015). Moreover, homeless youth experience multiple transitions (Tyler and Schmitz 2013) and have a transient lifestyle (Tyler and Whitbeck 2004), which makes data collection with this group especially challenging.

One of the main innovations in technology is ecological momentary assessment (EMA), which developed from experience sampling (Cziks-zentmihalyi and Larson 1987) and allows researchers to capture data in the moment about an individual’s current behavior in their natural environment (Shiffman et al. 2008). EMA via short message service (SMS) surveying verifies the timing of one behavior relative to another, allowing for temporal sequencing (Cohn et al. 2011), and minimizes recall biases.

We used SMS to implement EMA with a transient population, which is innovative, and to the best of our knowledge, the first study to date to use SMS to collect daily data from homeless youth. Specifically, the purpose of this study was to empirically assess the feasibility of SMS surveying via EMA with homeless youth. SMS permitted us to gather information on youths’ current feelings (e.g., worried), the type of support (e.g., emotional) and services utilized (e.g., shelter), and the social context of substance use (e.g., drinking with friends) during their day-to-day lives. Our work is an improvement over prior retrospective studies of homeless youth as our study used cell phones via EMA to gather real-time daily data from homeless youth and thus contributes to the broader literature. Moreover, cell phones have the potential to be used as an intervention tool (Tyler and Schmitz 2017) to improve the lives of homeless youth and other highly mobile populations.

Cell Phones and Technology for Maintaining Contact with Homeless Young People

Bender and colleagues (2014) examined the feasibility of various technologies to follow homeless youth over time. Results showed that persistence and frequent contact were still necessary even when utilizing phone calls, text messages, e-mail, and Facebook. Cell phone calls and texting were used more consistently by youth. Although youth were provided with a prepaid study phone for three months at the first interview, the researchers caution that a cell phone alone is not enough to prevent attrition and recommend frequent contact and rapport (Bender et al. 2014).

Cell Phones for Data Collection on Drug Use among Various Populations

Cell phones have increasingly been used to survey similarly at-risk populations such as those formerly in prison, addicted homeless individuals, and minority youth receiving treatment for substance abuse. Sugie (2016) used smartphones with 135 men recently released from prison and followed them for three months. Overall, 68% of men completed more than 75% of received surveys on their phone but 10% completed less than half of received surveys. Few phones were reported to be stolen, and Sugie concluded that the smartphone as a survey instrument worked well with this population.

Freedman and colleagues (2006) tested the feasibility of cell phones and automated telephone interviewing to collect EMA data from 30 homeless crack cocaine–addicted adults in treatment. Eighty percent of adults completed the full two weeks of data collection, though half of the sample thought the two-week period was too long while others complained about the repetitive questions or poor cell phone service. In a feasibility study among 28 outpatient treatment youth, Comulada et al. (2015) found that lower compliance was linked to both an increase in the number of study days and retrospective reporting of alcohol use. Overall compliance rate was 80%.

Method

Sample Eligibility Criteria and Recruitment Sites

Data are from the Homeless Youth Texting Project, a pilot study designed to examine risk and protective factors for substance use and to field test EMA via SMS to ascertain its utility and feasibility with homeless youth. From August 2014 through October 2015, 150 homeless youth were interviewed in two Midwestern cities. Of the 150 respondents interviewed at baseline, 112 youth or 75% completed a follow-up interview. The university institutional review board approved our study.

Eligibility required youth to be between 16 and 22 years of age and homeless or runaway on the night prior to screening. Homeless includes those who lack permanent housing such as spending the previous night in a shelter, public place, or on the street, staying with friends or in a transitional facility, or other places not intended as a domicile (National Center for Homeless Education and The National Association for the Education of Homeless Children and Youth 2010). Runaway includes those under age 18 who spent the previous night away from home without parental permission (Ennett et al. 1999). Participants were recruited through three local agencies that offer emergency shelter, food programs, transitional living services, and street outreach.

Four trained and experienced interviewers conducted the interviews. Interviewers approached youth at shelters, food programs, and during street outreach. Informed consent was obtained from youth who were informed that the study had three parts, and if they agreed to participate, they would need to complete a baseline, structured interview, the SMS portion, and a follow-up, structured interview. The two interviews, which were conducted in shelter interview rooms, local library, or outside (weather permitting), lasted 45 minutes and 15 minutes, respectively. Participants received a US$20 and US$10 gift card to a local store for completing the baseline and follow-up interview, respectively. Less than 3% of youth (n = 5) refused to participate or were ineligible.

Cell Phone Distribution

On completing the baseline interview, participants were given a disposable cell phone and told they would receive 11 texts per day (sent in blocks of two–four questions three times per day) over the next 28–30 days and then would be recontacted in approximately 30 days for a follow-up interview. The block of texts came at 10:00 a.m., 4:00 p.m., and 9:30 p.m. Because the first day of the texting limited the opportunity for youth to answer all 11 texts (due to signups throughout the day), we examine only days after the first study day. Text questions were sent from an automated system, set up to send out text questions in the same order and at the same time each day. Responding to each text question required participants to enter a number or numbers. Participants were shown how to operate the phone and instructed to contact their interviewer if they encountered problems.

Youth who failed to answer the support question in the 10:00 a.m. block did not receive the remaining two questions in this block but did receive the items in the 4:00 p.m. block. If the youth later answered questions from the 10:00 a.m. block, the SMS system would send the remaining questions. We used the paradata files (Olson and Parkhurst 2013) containing time and date stamps for both the sent and returned text messages to link answers with appropriate days.

Typically, three to four days prior to the end of their texting period, youth were sent a text informing them how many texting days were left and to set up a follow-up interview. Participants were texted a reminder the day before and the day of their appointment to minimize attrition. Those who responded to every text question (11 texts per day) were paid US$50 cash (prorated at US$0.14 per response), and those who responded to at least 85% of texts also received a bonus US$10 gift card.

Measures

Text questions (response options, in parentheses below) were all multiple choice. Four text questions were asked at 10:00 a.m.: (1) “Where did you sleep last night” (outside or car, youth shelter, adult shelter, with friend/ partner, stranger or acquaintance, and transitional living); (2) “What type of support did you receive yesterday” (emotional, help with money, safety, shelter, and none); (3) “Who did you see yesterday” (caseworker, pastor, mentor, teacher, friends from home, family, and no one); and (4) “Which services did you use yesterday” (shelter, meals, counseling, street outreach, health, and none). At 4:00 p.m., these four questions were asked: (5) “Today I felt depressed or lonely”; (6) “Today I felt worried or concerned” (both yes/no); (7) “Today were you” (beat up, robbed, threatened with weapon, touched sexually, sexually assaulted, and no); and (8) “Today I had trouble finding” (shelter, food, clothes, money, street outreach, and nothing). At 9:30 p.m., these three questions were asked: (9) “How many drinks tonight” (range: no drinks to seven or more drinks); (10) “Used any of these drugs tonight” (weed, crank, meth, coke, inhalant, heroin, ecstasy, other, and none); and (11) “Drank or used drugs with friends tonight” (drank with friends, used drugs with friends, drank and did drugs with friends, and did neither).

Feasibility Measures

Feasibility refers to the likelihood that homeless youth are able to respond to daily texts. We measure feasibility in five ways: (1) Youth were asked seven questions (e.g., how easy/difficult was it for you to respond to our texts throughout the day?) to assess their impression of using SMS to provide daily data (see Online Appendix for Supplemental Table 2 for a full list of questions); (2) the average number of texts youth answered per day; (3) the average number of days youth texted; (4) question-level response rates; and (5) the return rate for the phones. Each measure is described in detail below.

Cell Phones

When we queried agencies serving homeless youth about how many youth own their own cell phone, agencies estimated about 25%, although some youth did not have current phone plans. Because of this and budget constraints, we purchased inexpensive cell phones for all participants. We used two phones: TRACFONE and AT&T GoPhone (see Figure 1).

Figure 1.

Cell phones provided to homeless youth with sample text questions.

A TRACFONE cost US$10. A phone card, for example, cost US$40 for 90 days of activation time with 600 minutes. Phones were not charged for incoming texts. Outgoing texts (i.e., text responses) cost one-third of a minute per response (three responses = one minute of airtime). Youth typically received a TRACFONE with 800 minutes and needed approximately 110 minutes to complete 100% of the text responses. The AT&T GoPhone cost US$30. The phone card cost US$25 and included unlimited texting and Internet access and 250 free calling minutes for 30 days. The AT&T GoPhone required activation (i.e., minutes added to the phone) within two weeks of phone registration and obtaining a cell phone number. Once the phone was activated, the 30-day period began and could only be extended by adding another phone card. While the AT&T GoPhone had unlimited texting and eventually, unlimited calling, the drawback was the activation time of 30 days versus the TRACFONE, which remained activated for 90 days. The overall cost of the AT&T GoPhone was lower, and we achieved higher response rates with this phone compared to the TRACFONE.

We asked youth to return phones for two reasons. First, we believed this would increase the likelihood youth would return for their follow-up interview. Second, all local stores and online retailers limited purchases to two phones per transaction; thus, we could not order 150 phones at one time. Instead, we purchased phones on a “needs basis.” Returned phones were cleaned, returned to factory specifications, assigned a new phone number, and reused. Phones returned in poor condition were not reused.

Texting Logs

The first author monitored incoming text logs daily. When a respondent did not send a single text response on a given day, the first author or a second interviewer would send the participant a message: “We noticed you didn’t text yesterday so we are checking to see if everything is ok.” Conversely, we also sent a text stating: “You are doing a great job texting; keep up the good work!” to youth who consistently completed texts. This procedure assisted with keeping in contact with youth and likely increased cooperation rates.

Statistical Analysis

Feasibility analyses were based on follow-up surveys with youth and the texting data. Youth were asked their impressions of the SMS study portion; frequency distributions are reported for seven items. For the texting data analyses, we first examined a count of the total number of days with any texting data. Then, we compared the percent of youth who had data for the entire study period (28 days) versus texting data for less than 28 days and whether this varied by phone type. Next, we examined question-level response rates and mean number of texts per day overall and by phone type to evaluate whether particular questions or sets of questions were less likely to be answered by youth. If youth did not answer at all that day, they were not included in the analyses for that day. We examined whether youth completed a text message for each item for each day, using a dichotomous measure of 1 = there is an answer for that day for that question and 0 = there is no answer for that day for that question. We tested whether there were differences in response rates by phone type using hierarchical random effects logistic regression models accounting for the nesting of days (j) within youth (i) (logit(P(answered_ij = 1)) = β₀ + β₁PhoneType_i + u_i), assuming that u_i is normally distributed with mean zero and variance τ₀ (Raudenbush and Bryk 2002).

We also calculated the total number of texts answered for that day by each youth. We tested differences in the mean number of texts using hierarchical random effects linear regression models, again accounting for the clustering of days within youth: nwmberoftexts_ij = β₀ + β₁PhoneType_i+ u_i + e_ij. Because of the nonnormal distribution of number of texts, we then dichotomized the number of texts answered per day into completing all 11 texts (=1) versus between one and 10 texts (=0), testing for differences across phone type using a hierarchical random effects logistic regression model. Results did not differ when we accounted for the city in which we interviewed the youth. Finally, we summarized the number of youth who returned the phones.

Results

Sample Characteristics

Our sample included 150 homeless youth from two Midwestern cities. Ages ranged from 16 to 22 years (M = 19.4 years). One-half (51%) were female, 22% identified as lesbian, gay, bisexual, and 59% were nonwhite. The average age at which youth first left home was 14.8 years. Youth reported running away from home between one and 35 times (M = 4.9 times). The most frequently reported reason for leaving home was family conflict, reported by 34.4% of youth. While 27.4% of youth have been away from home for six months or less, 21.9% have been away for more than four years (M = 31.5 months; ~2½ years).

Feasibility of Approach

We asked youth their impressions of the SMS study portion in the follow-up interview. Seventy-three percent of youth said that responding to texts throughout the day was “very easy or somewhat easy”; almost 69% said that the 11 texts per day was “just enough”; and 74% reported that the 30-day period was “about right.” If they repeated this study in the future, 81% of youth said they would prefer doing SMS versus an online survey, and almost 56% preferred to use their own phone versus the study phone. Youth favored the SMS study portion, and results indicated that this method is feasible with homeless youth (see Supplemental Table 2 in Online Appendix).

Days of Texting and Phone Type

We evaluated field outcomes measuring the feasibility of using texting to gather data from homeless youth. We found that on average, participants completed 18.8 days (SD = 9.49) of texts, but 44 youth (29.9%) had texting data on 28 or more days. Drop-off was relatively steady throughout the field period, with a daily attrition rate of 1–3%. We have complete data (all 11 texts) on 1,128 days (43.0%). There were modest differences in the number of days with any texting data across the type of phone assigned to youth, although they were not significantly different with this sample size; 34.3% of the AT&T GoPhone respondents had 28 or more days of data compared to only 25.7% of the TRACFONE respondents (χ² = 1.287, p = .257). Three youth who completed the initial interview did not complete any texts.

Question-level Response Rates Overall and Phone Type

There were 2,621 opportunities (youth-days) for the 147 youth on which we have texting data to answer any given question after the first day. Question-level response rates ranged from 84.3% (question 2) to 69.1% (question 4; see Table 1). Overall, item response rates were higher for youth who received the AT&T GoPhone compared to youth who received the TRAC-FONE. For the 10:00 a.m. questions, only 76.5% of youth who received the TRACFONE answered question 1, compared to 83.5% of youth who received the AT&T GoPhone (z = 2.54,p = .011). Similarly, item response rates were significantly higher for questions 4–6 and questions 9–11 for youth who received the AT&T GoPhone. Thus, respondents who received the AT&T GoPhone tended to provide more complete answers on individual questions.

Table 1.

Number and Percentage of Youth-days with Answers to QI –Q II by Phone Type.

	Days after Day 1
	Overall		TRACFONE		AT&T GoPhone
Text Questions	#	%	#	%	#	%	z	P Value
Asked at 10:00 a.m.
Q1 (slept last night)	2,106	80.35	908	76.50	1,198	83.54	2.54**	.011
Q2 (support)	2,210	84.32	1,000	84.25	1,210	84.38	0.84	.401
Q3 (saw who)	2,129	81.23	964	81.21	1,165	81.24	0.83	.405
Q4 (used services)	1,811	69.10	752	63.35	1,059	73.85	2.76***	.006
Asked at 4:00 p.m.
Q5 (depressed)	2,156	82.26	955	80.45	1,201	83.75	1.93*	.054
Q6 (worried)	2,006	76.54	872	73.46	1,134	79.08	2.08**	.038
Q7 (victimization)	2,001	76.34	893	75.23	1,108	77.27	1.17	.243
Q8 (trouble finding)	1,966	75.01	881	74.22	1,085	75.66	1.20	.230
Asked at 9:30 p.m.
Q9 (# drinks)	1,923	73.37	815	68.66	1,108	77.27	2 80***	.005
Q10 (drugs)	2,111	80.54	932	78.52	1,179	82.22	1.79*	.073
Q11 (drink w/ friends)	1,821	69.48	770	64.87	1,051	73.29	2.63***	.009

Note: Test comparing TRACFONE to AT&T GoPhone.

^*p < .10.

^**p < .05.

^***p < .01.

Texts Answered Per Day

The average number of texts answered per day was 8.49 (SD = 2.94). Forty-three percent of days had answers to all 11 texts sent that day, with 69.4% of days having answers to eight or more texts. The number of texts answered per day varied by phone type. On average, respondents with the TRACFONE provided 8.21 (SD = 2.93) answers to texts per day, compared to 8.72 (SD = 2.93) for those with the AT&T GoPhone; this difference was statistically significant (z = 2.18, p = .029). Only 35.1% of days had answers to all 11 texts on the TRACFONE, compared to 49.6% of days for the AT&T GoPhone (z = 3.06,p = .002).

Return of Phones

Of the 50 phones given out to youth in city #1, 37 phones were returned (74%), 2 were reported lost, and 11 youth could not be relocated. Overall, 76% of TRACFONEs were returned compared to 72% of AT&T GoPhone (z = –0.317, p = .751). Of the 100 phones given out to youth in city #2, 61 phones were returned (61%), 12 were reported lost, and 27 youth could not be relocated. Overall, 51% of TRACFONEs were returned compared to 73% of AT&T GoPhone (z = 2.243, p = .025).

Discussion

The purpose of this article was to assess the feasibility of SMS via EMA with 150 homeless youth over 30 days. Using SMS surveying to implement EMA in this population is innovative and to the best of our knowledge, the first study to date to use SMS to collect daily data from homeless young people. In terms of feasibility, our results overall reveal that collecting data via SMS is feasible with homeless youth.

Feasibility was measured in five ways in the current study. In terms of youths’ impression of using SMS to provide daily data, three-fourths thought that responding to texts throughout the day was very easy or some-what easy, while a similar percentage reported that the 30 days of texting was “about right.” The majority of youth (almost 69%) also indicated that the 11 texts sent per day was “just enough.” We also found that on average, participants completed approximately 19 days of texts, whereas almost 30% of youth had texting data on 28 or more days. The number of days with any texting data across the type of phone assigned to youth was not significantly different. The average number of texts answered per day was 8.49. Forty-three percent of days had answers to all 11 texts sent that day, and 69% of days had answers to eight or more texts, though this varied by phone type. Finally, we found that in city #2, significantly more AT&T GoPhones were returned compared to TRACFONEs, but no differences were found for city #1 in terms of phone returns.

Our results reveal an upper limit for number of days to which homeless youth will respond to text questions, which, on average, is approximately 19 days. Our response rates are generally consistent with prior studies of at-risk populations using cell phones to collect data including men recently released from prison (Sugie 2016), crack cocaine–addicted adults in treatment (Freedman et al. 2006), and youth in outpatient treatment (Comulada et al. 2015).

Given the numerous negative outcomes that homeless young people experience such as victimization (Tyler et al. 2004), substance misuse (Hadland et al. 2011), and poor mental health (Brown et al. 2015), using SMS to gather daily data increases opportunities to make contact with this group and subsequently provides a chance for service providers to offer and deliver interventions to improve homeless youths’ health outcomes. Additionally, using SMS to collect daily data eliminates some of the barriers that are associated with data collection with highly mobile populations. That is, even though homeless youth experience multiple transitions (Tyler and Schmitz 2013) and a transient lifestyle (Tyler and Whitbeck 2004), youth do not need to be physically present for researchers to collect data from them.

In addition to these benefits, research on homeless youth using retrospective reporting does not allow researchers to link a specific victimization incident with a particular drug use episode. With SMS, we can examine whether a victimization experience earlier in the day is associated with a specific drinking or drug use episode later that day. Moreover, this level of specificity provides a better understanding of homeless youths’ daily experiences and thus allows service providers to more effectively aid this population. Relatedly, cell phones could be used as an intervention tool to connect youth with agencies and services and their social networks and to help youth schedule their daily lives, all of which can improve youths’ well-being (Tyler and Schmitz 2017).

Limitations

Our study has limitations. Given the difficulties of recruiting hard-to-reach populations, we used a convenience sample of homeless youth; thus, our findings cannot be generalized to all homeless youth. For example, homeless youth in larger metropolitan areas appear to have greater cell phone ownership (Rice et al. 2011) than youth in our Midwestern sample. Additionally, it is unknown how many more youth who were assigned the TRACFONE would have completed the study if they had unlimited minutes. Finally, our analyses of predictors of item-level completion look at only one predictor—phone type. Future analyses will examine correlates of nonresponse to the text questions, including demographic and other predictors.

Conclusion

Researchers seeking to collect data via SMS with hard-to-reach or highly mobile populations can do several things to increase compliance rates. First, researchers should select cell phones that have unlimited texting and calling minutes and an extended activation period as different phones and phone plans may result in different response rates. Second, sending periodic texts to respondents can increase cooperation rates. Third, the text questions should be easy to respond to and researchers will want to send a reasonable number of daily texts (e.g., 20% of our respondents thought 11 texts a day was too many). Fourth, an upper limit for our population is approximately 20 days of texting. Fifth, respondents may prefer to use their own cell phone; however, it is important to check whether they have unlimited texting otherwise, they may have insufficient minutes to complete the study. Finally, our respondents preferred SMS over online surveys. In sum, collecting data via SMS is feasible with hard-to-reach or highly mobile populations.