Response patterns to weekly short message service health surveys among diverse youth at high risk for acquiring HIV

Abstract

HIV researchers use short messaging service (SMS)-based surveys to monitor health behaviors more closely than what would be possible with in-person assessment. Benefits are tempered by nonresponse to completing surveys. Understanding response patterns and their associated study participant characteristics would guide more tailored use of SMS-based surveys for HIV studies. We examined response to weekly 7-item SMS surveys administered as part of an HIV prevention trial. Using Mixture hidden Markov models (MHMM), we identified the underlying response patterns shared by subgroups of participants over time and quantified the association between these response patterns and participant characteristics. Three underlying response patterns were identified; responders, responders with phone-related errors, and non-responders. Non-responders versus responders were more likely to be younger, male, cis-gender, Black and Latinx participants with histories of homelessness, incarceration, and social support service utilization. Responders with phone-related errors compared to non-responders were more likely to be Black, Latinx, female, students, and have a history of incarceration and social support service utilization. More nuanced results from MHMM analyses better inform what strategies to use for increasing SMS response rates, including assisting in securing phone ownership/service for responders with phone-related errors and identifying alternative strategies for non-responders. Actively collecting and monitoring non-delivery notification data available from SMS gateway service companies offers another opportunity to identify and connect with participants when they are willing but unable to respond during follow-up.

INTRODUCTION

Innovative social and behavioral research, including HIV prevention and adherence research, increasingly capitalizes on the widescale uptake of mobile devices to reach marginalized populations who might not be engaged in clinic settings. Researchers use short messaging service (SMS) text messages and smart phone applications to deliver ecological momentary assessment (EMA) and periodic diaries in HIV care and other research settings [1–3], especially among youth [4–8]. SMS and/or smart-phone based interventions [9–13], sometimes referred to as ecological momentary interventions [14] (EMI) or just-in-time adaptive interventions [15] (JITAI), have also been successfully used. There is strong evidence indicating that EMI and JITAI positively impact disparate health-related behaviors such as adherence to antiretroviral therapy (ART), weight-loss, and diabetes self-management [9–12, 14, 16]. In practice, it is often hard to distinguish the use of EMA from EMI when the assessment components include self-monitoring behavioral questions, such as the amount of alcohol drinking, adherence to HIV medication, or consistent condom use.

Despite its promise, the success of SMS-delivered assessments hinges on participants’ willingness to respond as requested. Assessment nonresponse disrupts the feedback loop needed for intervention and limits outcome evaluation. In addition, missing EMA data due to nonresponse often leads to biased statistical inferences as missing data often depend on mechanisms that are not easily controlled for through observed variables [17, 18]. While systematic review suggests that EMA response rate is not linked to prompt frequency or length of assessment period [19], individual studies almost inevitably see more nonresponse from participants as follow-up continues [20]. Few studies, however, have tracked response rates to EMA for more than 12 months [21].

Tracking and analyzing willingness to engage in EMA over a sustained period of time can provide important information for the development of effective engagement and retention strategies in HIV research and services, especially if they are to be implemented at a large scale. Furthermore, understanding overall response patterns and factors associated with varying levels of responsiveness to EMAs can be used to tailor technology-based interventions to those who will engage with them, identify factors that may increase engagement, and understand potential biases when data are missing. For example, different incentive structures may be needed if nonresponse is largely due to attrition versus intermittent missingness when participants remain in the study. However, past studies yielded insufficient data to inform these questions for the following reasons. First, none of the past studies collected or reported the reason(s) for participants’ nonresponse [16, 18]. For instance, researchers could not determine whether a nonresponse was due to an outright refusal or to phone-related barriers such as service suspension that preclude the participant from responding. In principle, inability to respond should be treated differently from an unwillingness to respond. Second, among studies that reported response rates, many collapsed responses over time into an aggregate measure (e.g., percentage responded) without looking at the overall longitudinal response pattern, making it impossible to distinguish study participants who respond to EMA only intermittently from those who become nonresponsive after a period of consistent responses. This issue is further complicated by different definitions for acceptable levels of response that are used to define analysis samples (e.g., excluding participants who respond to EMA less than 60% of the time) [16, 17]. Third, several studies investigated participant characteristics associated with response rates but conducted analyses under a regression framework, either treating response as a repeated outcome measure [22–24], or summarizing response over time into an individual response rate [25]. Standard statistical approaches do not provide a comprehensive understanding of the complex response patterns and their associated participant characteristics [16, 19, 20]. To better understand response patterns and advance assessment and intervention development efforts, studies that apply novel statistical approaches to handle nuanced, longitudinal response data are sorely needed. The current study addresses this gap.

For this paper, we conducted secondary analysis of weekly brief self-monitoring surveys administered via SMS over twelve months to youth at high risk (YHR) for acquiring HIV participating in a multi-site HIV prevention trial [26] where we collected text message delivery status data via server logs [27]. We used mixture hidden Markov modeling (MHMM) to summarize response patterns over time and to evaluate associations between response patterns and individual-level characteristics assessed at baseline (e.g., sociodemographic characteristics, risk behaviors). Our analytic approach allows us to tease out nuanced temporal response patterns in contrast to simpler response patterns that would be estimated by regression models, such as mean percentage response at baseline and the slope over time. Details of MHMM are provided in statistical methods section.

In the absence of prior MHMM analyses on this topic, we derived our hypothesized associations between individual characteristics and response patterns from prior longitudinal studies that evaluated predictors of EMA response rates as well as study attrition, preferably among youths or other populations at high risk of HIV infection. We hypothesize that different response patterns are associated with participant age [25, 28], birth sex, gender [22, 23, 29], race, ethnicity [25, 30], mental health (e.g., anxiety and depression) [22, 31], structural barriers such as homelessness history and lower education [7, 25, 28], drug and alcohol use [20, 28, 29, 32], and greater number of lifetime sexual partners [33]. We are also interested in understanding whether sexual orientation, additional behavioral indicators (e.g., condom use and smoking history), and past trauma experience predict response patterns as these factors could also be used to uniquely identify participants with specific response patterns.

METHOD

Sample

We analyzed data from a randomized controlled trial conducted through the Adolescent Medicine Trials Network (ATN). The protocol is described in detail elsewhere [26]. Eligible participants were between 12 and 24 years of age and were recruited mostly from community agencies and clinics serving homeless, sexual and gender minorities, and other vulnerable youth in Los Angeles, California and New Orleans, Louisiana between May, 2017 and August, 2019. Participants received $50 for each 4-month study visit and the interviewers provided referrals to services as indicated in the study visit assessments. HIV negative youth met a minimum eligibility score that was a weighted sum of the following risk factors: race/ethnicity, gender identity, sexual orientation, and history of homelessness, incarceration, mental health hospitalization, substance abuse treatment, HIV pre-exposure or post-exposure prophylaxis (PrEP/PEP) use, positive test for a sexually transmitted infection, needle sharing, and sex exchange. All participants provided signed informed consent. Behavioral and biomarker data were recorded on a secure, HIPAA compliant data capture system and identified via unique participant ID numbers. Ethical approval for all study procedures was obtained from the Institutional Review Board at the University of California, Los Angeles (IRB #16–001674-AM-00006). See protocol papers [27] and [34] for additional details.

Weekly Monitoring Surveys

We began sending automated weekly survey assessments on Tuesdays to all participants upon enrollment and offered them a $1 incentive for each completed survey. We disclosed potential confidentiality issues to participants by informing them of the content of the SMS survey, which could be considered as sensitive, prior to registration. Furthermore, we informed participants that they could opt out of the survey at any time point during follow-up and choose the time of day (but not what day) to receive the survey. Table 1 shows the seven questions included in the survey. In cases of nonresponse, we offered participants an email option with a link to a web-based assessment. Few participants chose the email option making it impractical to analyze them as a subgroup, thus we excluded them from analyses in line with our focus on SMS survey delivery. Once participants received the initial request, they had 48 hours to complete the survey. If participants did not respond within the first 24 hours, they would receive another prompt at the 25th hour. In total, each participant received a maximum of 53 surveys during their first twelve months in the study.

Table 1.

List of EMA assessment questions and range of responses allowed

Item #	Question	Single choice response categories
1	Have you had a fever, severe headache, sore throat, swollen glands, rash, extreme fatigue, vomiting/diarrhea, or no appetite in last week?	Yes/No
2	How many days did you feel sad or depressed in the past 7 days?	0–7
3	Did you have any genital itching/pain/discharge, burning during urination, lower stomach pain, or discomfort during sex?	Yes/No
4	How many times did you have sex without using condoms in the past 7 days?	0–7
5	How many days did you use alcohol and/or drugs in the past 7 days?	0–7
6	How many days did you not have a place to sleep and/or not enough to eat in the past 7 days?	0–7
7	How many days did you miss taking medications in the past 7 days?	0–7 or NA if not taking medications

Possible Observed States for Survey Response

If a participant responded to at least 1 of the 7 questions in Table 1, the participant was labeled as having responded to the survey at that time point. Survey nonresponse occurred when participants did not respond or did not receive the survey for one of following reasons: (1) they had previously opted-out of receiving the survey by replying “STOP,” which automatically stops messages from being sent by the SMS gateway server, (2) they had dropped out of the study, (3) the survey was not delivered due to mobile phone service suspension (e.g., non-payment), (4) service station overload or being outside the service region, or (5) participants did not possess valid cell phone numbers (e.g., the number was a landline). For analysis purposes, we treated opt-out (1) and study drop-out (2) the same as nonresponse for participants who received survey. Nonresponse due to cell phone service error (3), server error (4), or invalid phone numbers (5) were grouped as experiencing phone-related (and server) error. Thus, we ended up with 3 observed states for assessment response: response, nonresponse, and phone-related error.

Measures of Participants’ Characteristics

The following variables measured at baseline were considered a priori as candidate predictors of participants’ EMA response patterns.

Demographic characteristics.

We included study location, categorized as Los Angeles or New Orleans, age, race/ethnicity, sex at birth, gender identity, sexual orientation, education level, and employment status. Employment status was categorized into three categories: employed (full or part time); student, not employed; not employed and not a student.

Mental health.

We assessed the presence of moderate to severe anxiety and depression symptoms over the past two weeks prior to baseline using a 7-item anxiety scale (GAD-7; [35]; Cronbach’s alpha (α) = 0.88) and the 9-item Patient Health Questionnaire (PHQ-9; [36]; α = 0.85), respectively. We created binary indictors for moderate to severe anxiety if GAD-7 scores were greater than 7 and for moderate to severe depression if PHQ-9 scores were equal to or greater than 9. We did not adopt the originally proposed cutoff of 10 for generalized anxiety disorder because a cutoff of 8 was found to improve sensitivity while maintaining the specificity [37]. Participants also reported if they had ever attempted suicide and whether they had been ever hospitalized for mental health issues.

Traumatic events.

Participants reported if any of the following traumatic events happened to them in their lifetime: forced or frightened to participate in a sexual (oral, vaginal, or anal) act, had sex with someone who was five or more years older than them before they turned 16 years old, attacked or robbed (including threats), seen someone seriously injured or killed or had a close friend or family member murdered. They also reported if they had ever been the victim of intimate partner violence (IPV; e.g., having a current or former partner slap or throw something at them). We included these traumatic event indicators as such experiences in childhood or adolescence could result in loss of core capacities for self-regulation and interpersonal relatedness, which might affect response rates [38].

Other structural factors.

In each of the following domains, we evaluated on a binary scale if participants had each of the following or not: health insurance at baseline; an income below the federal poverty line; had been homeless; participated in a diversion program or time in a juvenile detention center, jail or prison (i.e., incarcerated) in their lifetime; and whether they had received any of the following support or services: (1) HIV prevention program, (2) substance abuse treatment program, or (3) housing, food, clothing, toiletries, hygiene products, transportation, employment services, etc.

Substance use.

Participants indicated whether they had ever smoked cigarettes or e-cigarettes for longer than four months. The 3-item AUDIT-C was used to identify participants engaged in heavy or hazardous drinking during the past four months [39]. Participants also indicated if they used marijuana and other substances over their lifetime, including synthetic marijuana, cocaine or crack, heroin, Ecstasy, methamphetamine, prescription stimulants, inhalants, hallucinogens, and prescription painkillers that were not prescribed.

Sexual behaviors.

Participants reported if they ever had vaginal or anal sex without a condom, where a ‘No’ response indicated 100% condom use, and if they had ever used pre-exposure prophylaxis (PrEP) and post-exposure prophylaxis (PEP) to prevent HIV. Participants were asked about the number of men, women, and transgender men and women with whom they had sexual intercourse over their lifetime. Last, we also created an indicator of having ever had an STI if participants either self-reported ever having syphilis, gonorrhea, trichomonas, chlamydia, HSV-2, HPV, hepatitis A/B/C or scabies, or if participants tested positive at baseline for syphilis, gonorrhea, or chlamydia.

Randomization arms.

Participants randomly assigned to Arm 1 received only daily SMS health promotion messages as intervention (SMS only). Arm 2 participants received both health promotion messages as well as peer support through an interactive online forum (SMS + Peer support). Arm 3 participants received telehealth coaching in addition to health promotion SMS messages (SMS + Coaching). Participants in Arm 4 received all three interventions: SMS, peer support, and coaching.

Statistical Methods

We utilized first-order MHMM to identify hidden clusters of survey response patterns and their association with participants’ baseline characteristics. Given the novelty of MHMM relative to standard statistical methods, modeling details are provided, starting with first-order hidden Markov models. At each time point in this study, we assume that a participant could be in one of two hidden states: “intend to respond” or “did not intend to respond”. Figure 1 illustrates a sequence of hidden states for one participant over the 52-week period. The hidden states directly control the observed states (i.e., response, nonresponse, and phone-related error) via a probabilistic relationship. The hidden Markov model is a first-order model because the probability of a participant transitioning to a hidden state at the next time point only depends on their current hidden state.

Figure 1.

Hidden and observed state sequence for one participant starting from week 0 to week 52. The circles refer to the hidden states and the squares refer to the observed states.

When many participants are modeled together, there may be temporal response patterns such that we can categorize participants based on similarities in their response patterns over time (e.g., a cluster of participants who initially respond and stop after the first 10 weeks of follow-up). We do not know how these participants will cluster beforehand; thus, we use a MHMM to identify clusters and model each participant’s response pattern through separate hidden Markov models.

There are three types of probabilities associated with a MHMM: initial probabilities, transition probabilities, and emission probabilities. Each cluster has a unique set of initial probabilities, which are the probabilities of starting in the hidden states. For instance, cluster 1 may have initial probabilities of .80 and .20 for the “intend to respond” and “did not intend to respond” hidden states, respectively. A transition probability is the probability of moving from one hidden state to another at the next time point versus the probability of remaining in the same state. Finally, an emission probability is the probability of a hidden state emitting an observed state. For example, if a participant is in an “intend to respond” hidden state at a given time point, what are the probabilities of responding to the survey, experiencing phone related error, and not responding given that hidden state? These three probability types (i.e., initial, transition, and emission) vary by cluster and help us interpret the general pattern of each cluster [40].

We fit MHMM using the Expectation-Maximization (EM) algorithm. We allowed EM algorithms to restart 10 times with different initial values where a total of 5000 iterations was allowed. As a first step, we specified the number of hidden states to be two for intention to respond or not. We then selected the number of clusters based on models unadjusted for covariates. We considered models with two to six clusters. To avoid the mixture algorithm being trapped in a local minimum, we fit each model with four different uniformly generated initial, transition, and emission probabilities [40]. We then selected the unadjusted models with the minimum BIC for a given cluster number. A balance was struck between model parsimony and retention of clusters to aid interpretability of the observed response patterns. Among candidate models, if the BIC value of a model with a larger number of clusters decreased more than 2% compared with the BIC value of a model with a smaller number of clusters, we favored the model with a greater number of clusters. The final model was the first model whose BIC decreased less than 2%.

After selecting the number of hidden states and clusters, bivariate analyses were conducted by fitting separate MHMM to covariates (i.e., participant characteristics discussed in the measures section). Covariates yielding 95% confidence intervals (CI) that excluded the null for predicting at least one cluster assignment were considered for multivariable MHMM. Education level and having health insurance at baseline were excluded from multivariable MHMM a priori due to their strong collinearity with age and employment among youth.

Due to difficulties in handling missing covariate data through MHMM, the analysis sample retained participants without missing baseline covariate values (86%; n= 1248/1486). Alternatively, we could have imputed these covariate values but techniques such as multiple imputation would make our current research question computationally intractable and is not the focus of our exercise. As mentioned before, participants who chose web-based survey were excluded from analyses as the subsample (N = 79) is insufficient for analysis. The analysis sample was comprised of 1169 participants.

To check whether the (hidden) clusters we identified reflect the observed response patterns, for each cluster we calculated the overall prevalence of each observed state (i.e., survey response, nonresponse, and phone-related error rates) across participants and across each weekly time point.

We used R 4.0 and R studio to conduct all statistical analyses [41]. We used seqHMM package to fit MHMM [40].

RESULTS

Sample Characteristics

Table 2 shows baseline characteristics of the analysis sample. Most youth (57.9%) were from Los Angeles and the rest from New Orleans. Most participants were between 19–24 years of age (85.8%), born male (81.3%), identify as homosexual or bisexual (72.6%), and reported being Black (45.3%) or Latinx (29.4%). About half of the participants reported at least some education beyond high school (52.6%).

Table 2.

Distribution of candidate predictors among analysis sample (n=1169)

Characteristic	n	%	Characteristic	n	%	Characteristic	n	%
Demographics			Mental health			Behavioral risk factors
Study location			Baseline GAD-7 anxiety score >=8	444	38.0	Smoking, 4 months or longer	458	39.2
Los Angeles	677	57.9	Baseline PHQ-9 depression score >=10 (moderate depression)	358	30.6	Problematic/binge drinking, past 4 months	468	40.0
New Orleans	492	42.1	Ever physically attempted suicide	376	32.2	Lifetime marijuana use	1037	88.7
Age group			Ever hospitalized for mental health issues	333	28.5	Lifetime use of any drug, non-marijuana	724	61.9
14–18 years	166	14.2				Lifetime polydrug use, non-marijuana	536	45.9
19–21 years	495	42.3	Traumatic events, lifetime			100% condom use	216	18.5
22–24 years	508	43.5	Close friend or family member murdered	488	41.7	Ever PEP use	58	5.0
Race/ethnicity			Seen someone seriously injured/killed	565	48.3	Ever PrEP use	164	14.0
Black or African American	530	45.3	Force or threats to attack/rob you	347	29.7	Number of partners, lifetime+
Latinx	344	29.4	Experienced intimate partner violence	428	36.6	0–2	213	18.2
White	223	19.1	Forced do something sexually as a child	352	30.1	3–10	491	42.0
Other race/ethnicity	72	6.2	Forced have sex with someone 5+ older	342	29.3	More than 10	465	39.8
Sex at birth						Any STI up to baseline+	526	45.0
Male	950	81.3	Structural Factors
Female	219	18.7	Has health insurance at baseline*	876	74.9
Gender identity			Income below federal poverty line	343	29.3	Randomization groups
Cisgender	1018	87.1	Ever homeless	557	47.6	Arm 1, SMS only	497	42.5
Transgender/gender diverse	151	12.9	Ever incarcerated	292	25.0	Arm 2, SMS + peer support	225	19.2
Sexual orientation			Service Utilization			Arm 3, SMS + coach navigation	232	19.8
Homosexual/bisexual	849	72.6	HIV prevention program, lifetime	248	21.2	Arm 4, SMS + peer support + coach navigation	215	18.4
Other sexual orientation (straight, asexual, pansexual etc.)	320	27.4	Substance abuse program, lifetime	234	20.0
Employment status at baseline			Food/housing support, past 4 months	593	50.7
Employed	530	45.3
Student, not employed	310	26.5
Not employed, not a student	329	28.1
Education level*
Below high-school degree	255	21.8
High-school degree	299	25.6
Any higher education	615	52.6

^*Not included in the multivariate analysis model due to concern of strong collinearity with other predictors.

⁺Not included in multivariate analysis due to insignificance in predicting any cluster in the bivariate analysis.

While most participants reported having health insurance at baseline (74.9%) and income above the federal poverty line (70.7%), almost half had a history of homelessness (47.6%) and recently received food/housing support (50.7%). A quarter of the participants had been incarcerated (25.0%). Close to a third of the participants indicated at least mild anxiety (38.0%), indicated moderate or more severe depression (30.6%), and had attempted suicide during their lifetime (32.2%). More than a third of the participants had experienced one or multiple traumatic events.

Survey Delivery and Overall Response Statistics

The overall response rate to weekly SMS survey was 45.5%. Over two-thirds of participants responded at least once (68.9%; N = 805). Almost all had a working cell phone at some point during the study period (91.2%; N = 1066) but 1070 (91.5%) had experienced issues with their cell phone (e.g., possessing no working cell phone, no cell phone service, out of service area or service provider breakdown) at least once over the follow-up period.

Underlying Survey Response Patterns

We selected three underlying clusters of survey response patterns based on model selection criteria described in the methods section (Appendix A.2). Figure 2 shows model-based simulations of observed and hidden state sequences for the three clusters. Based on simulated observed sequences in Figure 2, we labeled the three clusters as (1) non-responders (n=407), (2) responders (n=480), and (3) responders with phone-related errors (n=282). Responders tend to consistently respond to surveys but also experience reduced response over time. Responders with phone-related errors tend to be participants who have more intermittent missingness in responses largely in the form of phone-related errors. Overall, the cluster assignment is supported by observed response patterns (Appendix A3). For example, the responder and non-responder clusters differ mainly in terms of the prevalence of observed response states (34.17% versus 29.84%) and server error states (25.85% versus 32.76%) even if nonresponse percentage seems to be higher for responder cluster. Also as expected, the cluster of responders with phone-related error has the highest observed phone-related error prevalence (36.60%) followed by non-responders and then responders.

Figure 2.

Model-based Response Sequence by Cluster

Figure 3 shows graphs for Markov state transitions between hidden states for each of the three survey response clusters. Each pie chart represents a hidden state for each cluster, with emission probabilities for each type of observed response represented as slices. Initial probabilities of being in each hidden state are shown underneath each pie chart and transition probabilities between hidden states are shown between pie charts. Referring to cluster 1 for non-responders, they had a 0.51 probability of being in the second hidden state representing “intend to respond” where nonresponse was largely due to server error. In other words, they might have responded if not for phone-related issues. We also note that participants across all three clusters were much more likely to continue in the same hidden state than to transition between states as indicated by small transition probabilities (range = 0.009 – 0.062). Using cluster 1 of non-responders as an example again, even though participants in this cluster likely could not respond due to phone-related errors, they had a high chance of becoming those who had no intention to respond. And once the transition from “intend to respond” to “not intend to respond” happened, they would not respond even if their phone-related issues were resolved.

Figure 3.

Markov States Transition Graph (Hidden and Observed States)

MHMM Analyses

In bivariate analyses, number of lifetime sexual partners and any positive STI history were not found to be significant predictors for either responder or phone-related responder compared with non-responder and thus excluded from the multivariable model (Appendix A4). Odds ratios and 95% confidence intervals for covariates in the multivariable model are visualized in Figure 4. Numeric results are presented in Appendix A5.

Figure 4.

Adjusted Odds Ratio and 95% Confidence Interval Associating Participants’ Baseline Covariates (including Intervention Arms) and Clusters (Multivariate HMHH).

Multivariable results were mostly consistent with bivariate analysis results. Responders and responders with phone-related error were more likely to be older (19–24), student (vs unemployed), and participants of an HIV prevention program compared with non-responders (odds ratio (OR) = 1.12–3.45). Responder or responder with phone-related error versus non-responder were also more likely to have had suicidal attempts (OR = 1.14 and 1.47, respectively). Conversely, responders with phone-related error are less likely to be cisgender, male, or long-term smoker compared with non-responders (OR = 0.24–0.72).

Participants reporting Black, Latinx, or other race/ethnicities (compared with white) are less likely to be a responder (OR = 0.24–0.80) but more likely to be responders with phone-related errors versus non-responders (OR = 1.49–1.60). Participants who identify as gay, lesbian or bisexual are more likely to be responders (OR = 1.26, 95%CI[1.18,1.34]) and less likely to be responders with phone-related errors (OR = 0.91, 95%CI[0.84,0.98]) versus non-responders. Overall, most factors indicating higher mental health burdens (at least moderate depression, mental health-related hospitalizations) and traumatic events experience predicted lower likelihood of being responders versus a non-responders (OR = 0.48–0.84) except for having mild anxiety (OR = 0.95, 95%CI[0.86,1.04]), attempting suicide (OR = 1.47, 95%CI[1.39,1.55]), being forced to do something sexually as a child (OR = 1.57, 95%CI[1.49,1.65]), or having sex with someone 5+ years older (OR = 1.07, 95%CI[1.00,1.14]).

Homelessness history, incarceration history, and recent use of food/housing support services strongly predicted lower likelihood of being responders vs non-responders (OR = 0.34–0.74), while being below the poverty line or participating in a substance abuse program did not. Incarceration history and food/housing support service utilization are also associated with greater likelihood of being responders with phone-related errors versus non-responders (OR = 1.74–1.92).

Results for behavioral risk factors are mixed. Compared to non-responders, responders are less likely to be consistent condom users, long-term smokers, marijuana users and non-marijuana polydrug users (OR = 0.50–0.78). Participants who had reported recent problematic/binge drinking, contrary to expectation, are more likely to be responders (OR = 1.44, 95%CI[1.37,1.51]) but less likely to be responders with phone-related errors (OR = 0.90, 95%CI[0.83, 0.97]) compared to non-responders.

Being randomized to receive the peer support or coaching interventions did not increase the odds of being a responder versus non-responder (OR = 0.90–1.09) compared with being randomized to receive SMS daily message only (Arm 1). However, randomization to receive SMS daily messages and coaching (Arm 3) versus SMS only (Arm 1) significantly increased the odds of being responders versus responders with phone-related errors (OR = 2.02, 95%CI[1.50,2.83], this result is not shown in Figure 4).

DISCUSSION

As far as the team is aware, this study is the first to adopt MHMM as an analytical approach to summarize and predict long-term EMA response patterns and to demonstrate that vulnerable youth at-risk for HIV can be successfully engaged to complete weekly SMS monitoring surveys over 12 months. It sheds additional light on how SMS-based EMA can be scaled to address HIV and other health conditions [42]. First, unlike previous studies, our results uniquely show that in addition to responders and non-responders, there is a distinct group of a substantial number of participants who might have responded had they experienced fewer issues with their phone such as service suspension, loss of phone ownership, or shared mobile phone ownership. This serves as a major opportunity for future studies to increase response rates, for example, by assisting in securing phone ownership and stable mobile service to convert these participants into responders. Our analyses suggest that structural barriers may be the underlying driver to one’s inability to respond since the main predictors of being a responder with phone-related errors (versus responder) included unemployment, history of incarceration, and homelessness. Importantly, participants who are unable to respond to SMS-based EMA may also fail to respond to other platforms such as social media messaging and smartphone applications. Thus, investigators should take into account the phone-related challenges faced by the participants when designing future interventions that rely on mobile phone engagement.

Second, our study confirms a salient fact that a subgroup of participants simply have low willingness to respond to any SMS-based EMAs. This suggests that researchers must ensure that the methods they plan to use are compatible with and appropriate for their target population, or that the population enrolled is appropriate for the study aims. For example, our analysis suggests that if non-missing EMA data from SMS non-responders (who are likely to be young, unemployed males) are desired, other assessment and engagement strategies should be simultaneously considered in addition to SMS-based surveys. In the case of this study, weekly SMS surveying was a secondary assessment method and not critical to the primary aims of the study.

Third, this is also the first study to examine EMA response patterns over an extended study period among a diverse community sample of youth at high risk for HIV. Upon study completion, our study participants would have provided EMA responses for a total of 104 weeks, of which the results of the first 52 weeks are presented in this paper, compared with the 4 to 42 day duration reported by past EMA study protocols targeting youth [16]. As HIV prevention and treatment efforts among youth often involves long-term follow-up, sufficient length of follow-up time is crucial in understanding youths’ behavior, which can often be episodic and thus cannot be accurately captured over short periods of time [43]

It is important to note that our collection of detailed response data in the form of text message delivery logs drove our ability to elucidate different types of response patterns using MHMM analysis. Therefore, future studies should consider expanding the scope of data collection to include text message delivery service logs, which contain rich information with respect to when and why text messages are not delivered. For example, study teams receive specific error codes in the service logs whenever a text message survey fails to be delivered, with different error codes corresponding to various reasons of non-delivery including service not available (indicative of service suspension), invalid phone number (indicative of landline or deactivated phone number), out of service area, etc. In fact, throughout the study, we utilized such error messages to prompt further action items from the research team whenever resources allowed, such as following up with participants through other media when their phone service was suspended in order to improve overall study retention [27].

In this study, overall response to weekly SMS surveys hovered around 45%, similar to what was reported in prior studies with youth living with HIV [11, 44]. Higher response rates to EMA and similar assessment schemes have been found in non-clinical samples of youth and some clinical samples. However, these higher response rates are often only observed in studies with limited follow-up (e.g., on average, 30.29 days among substance use program participants and 13.27 days among children and adolescents) and sample sizes (e.g., on average, 154.21 substance abuse program participants and 98.81 children and adolescent), or studies where participants were more motivated to respond to EMA to start with (e.g., substance use treatment program participants, or weight management program participating youth) [16, 19, 21]. Moreover, as we have demonstrated, response rates are a less informative way for understanding response patterns and identifying strategies to increase response rates. Simple one-size-fits-all study design modifications may be less effective to increase response rates. For example, we changed the weekly survey delivery day from Tuesday to Sunday over a 3-month window in 2017 based on some participants’ feedback to increase (weekly) response rates but saw no such improvement, thus we reverted back to Tuesday delivery.

Consistent with previous reports on predictors of nonresponse and loss to follow-up, behavioral risk factors including smoking, use of marijuana and multiple non-marijuana drugs also predicted greater likelihood of being a non-responder versus a responder [20, 32]. However, our study did not identify any important sexual risk factors in predicting response patterns. Further studies are needed to investigate whether there are additional participant characteristics that help uniquely identify subgroups of participants or additional modifiable factors that can be implemented to improve response rates.

As a technical reminder for interpreting results from associational MHMM analysis, a strong predictor of nonresponse or response with phone related errors (versus response) is not necessarily useful in profiling these two groups of participants. For example, homelessness history is a strong predictor for non-responder versus responder, but since it does not distinguish non-responder from responders with phone-related errors, as indicated by the inclusion of null value 1 by the 95% CI, it is not a good factor for uniquely identifying non-responders. A good identifying factor of responders with phone-related errors, for example, should be a variable that is predictive of responders with phone-related error group when compared against both non-responder and responder groups. In our output we only provided ORs comparing responders or responders with phone-related error versus non-responders, but ORs with other latent groups as reference level can be numerically obtained by simply dividing one OR by another OR. For example, ORs comparing responders with phone related error versus responders can be obtained by dividing ORs comparing responders with phone related error versus non-responders by ORs comparing responder versus non-responder. Researchers must be mindful when interpreting these results.

Despite our innovative analytic approach through MHMM, several limitations should be noted. Due to the nature of the estimation process, cluster membership can be simultaneously affected by the response sequence and what type of covariates are added to the model. In our study, however, the cluster assignment is similar across models [40, 45]. In addition, like all latent class/cluster analyses techniques, cluster labeling relies on subject-matter expertise and is subject to mislabeling. Lastly, our study population was a selected sample of youth at high risk for HIV that may limit generalizability of the findings to broader populations of youth. However, as noted earlier, this population most closely resembles one of the most important target population for HIV prevention interventions.

CONCLUSIONS

This study represents an important step towards increasing the utility of SMS administered monitoring surveys in HIV-related studies. Rather than merely providing SMS response rates as is typically done, use of MHMM allowed us to characterize participants based on their underlying willingness to respond to brief SMS surveys administered weekly over 52 weeks and to profile these subgroups using baseline characteristics. This more nuanced information can be used to develop effective strategies for increasing response rates, one of the biggest challenges to using EMA in HIV prevention research. The emergence of a class of responders with phone-related issues suggests that by simply securing phone ownership and service one could improve response rates in this subgroup. Actively monitoring non-delivery notifications available from service companies offers another opportunity to catch participants when they are willing but unable to respond during follow-up. Perhaps most relevant to our findings is the notion that there are subsets of participants who will simply not respond to SMS-based surveys. Future researchers could consider tailoring EMA for different subsets of participants as their characteristics may reflect distinct needs. Such tailoring might save resources in the long run, result in fewer missing values, and improve the utility of EMA for HIV research and research in general.