Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Jun 1.
Published in final edited form as: Assessment. 2016 Sep 1;25(4):403–414. doi: 10.1177/1073191116667584

Patterns and Predictors of Compliance in a Prospective Diary Study of Substance Use and Sexual Behavior in a Sample of Young Men Who Have Sex With Men

Michael E Newcomb 1, Gregory Swann 1, Ryne Estabrook 1, Marya Corden 1, Mark Begale 1, Alan Ashbeck 1, David Mohr 1, Brian Mustanski 1
PMCID: PMC5565708  NIHMSID: NIHMS878093  PMID: 27586686

Abstract

Behavioral diaries are used for observing health-related behaviors prospectively. Little is known about patterns and predictors of diary compliance to better understand differential attrition. An analytic sample of 241 young men who have sex with men (YMSM) from a 2-month diary study of substance use and sexual behavior were randomized to complete daily or weekly timeline followback diaries. Latent class growth analyses were used to analyze data. Weekly and daily diary groups produced similar compliance patterns: high, low, and declining compliance groups. Black YMSM were more likely to be in the declining compared with the high compliance group. YMSM who were randomly assigned to receive automated feedback about risk behaviors did not differ in compliance rate compared with those who did not. Risk behavior engagement did not predict compliance in the daily condition, but some substances predicted compliance in the weekly condition. Implications for observational and behavior change methods are discussed.

Keywords: behavioral diary methods, differential attrition, substance use, sexual risk behavior, young men who have sex with men

Behavioral diaries are widely used as a survey method for prospectively observing health-related behaviors in order to identify event-level predictors of behaviors or to track behavior change over time (Bolger, Davis, & Rafaeli, 2003; Shiffman, Stone, & Hufford, 2008; Stalgaitis & Glick, 2014). This methodology has notable advantages over traditional retrospective methods that average behavioral engagement over a specified period of time (e.g., several weeks or months) because they are less prone to bias in recall and offer the opportunity to detect change in behaviors within persons in relatively brief windows. However, the accuracy and generalizability of such data hinges on high levels of participant compliance, which varies widely across studies (Shiffman et al., 2008; Stalgaitis & Glick, 2014). The goal of this study is to understand patterns and predictors of compliance in a diary study of substance use and sexual behavior in young men who have sex with men (YMSM) to better contextualize findings.

Diary methodology has proven to be very fruitful in identifying event-level predictors of health outcomes, as well as predictors of change in behavior over time. This observational approach has been applied to various health domains, including medication adherence and compliance (e.g., Gordon, Prohaska, Gallant, & Siminoff, 2007; Huber et al., 2013), medical symptom management (e.g., Burton, Weller, & Sharpe, 2007; Given, O’Kane, Bunting, & Coates, 2013), diet and food intake (e.g., Brunner, Stallone, Juneja, Bingham, & Marmot, 2001; De Castro, 1994), dimensions of affect (e.g., Hankin, Fraley, & Abela, 2005; Mustanski, 2007), substance use (e.g., Newcomb, 2013; Pachankis, Westmaas, & Dougherty, 2011), and sexual behavior (e.g., Newcomb & Mustanski, 2013; Shrier, Shih, & Beardslee, 2005). Behavioral diaries have often been used to study health behaviors in men who have sex with men (MSM) specifically (Stalgaitis & Glick, 2014), a group that is at high risk for HIV infection (Centers for Disease Control and Prevention, 2015) and that exhibits elevated rates of substance use compared with heterosexuals (Marshal et al., 2008; Newcomb, Birkett, Corliss, & Mustanski, 2014).

One of the primary advantages of behavioral diaries is that they minimize recall bias that may be present in retrospective surveys that ask participants to estimate frequency and patterns of health-related behaviors over longer periods of time. Retrospective designs are particularly sensitive to bias when there is substantial within-persons variability in the outcome of interest, as indicated by a low intraclass correlation coefficient or Kappa (Bolger et al., 2003; Mustanski, Starks, & Newcomb, 2014). Retrospective reports may over- or underestimate frequency of health-related variables compared with diary studies and electronic momentary assessment (EMA; Horvath, Beadnell, & Bowen, 2007; Leigh, Gillmore, & Morrison, 1998; Mustanski, 2007; Stone, Broderick, Shiffman, & Schwartz, 2004). Several variations of this methodology, including daily diaries and weekly timeline followback (TLFB; i.e., seven daily diaries completed retrospectively in one sitting), have been utilized to examine predictors of substance use and sexual risk behavior in young and adult MSM (Stalgaitis & Glick, 2014). However, little research has evaluated differences between daily and weekly TLFB approaches. Daily diaries may provide more accurate data than weekly TLFB diaries because they minimize memory bias, particularly when reporting on symptoms that may be less anchored by discrete events than behaviors (e.g., it may be more difficult to recall an affective state compared with a substance use event).

Despite the notable advantages of diary methods, participant noncompliance (i.e., attrition or inconsistent completion of diaries) may affect accuracy and generalizability of findings. Studies generally report high levels of compliance to diary protocols (Morren, van Dulmen, Ouwerkerk, & Bensing, 2009; Shiffman et al., 2008; Stalgaitis & Glick, 2014), but there is wide variation across studies. For example, a review of sexual behavior diary studies found that 69% of studies reported compliance rates of 80% or higher (range = 61.9% to 100%; Stalgaitis & Glick, 2014). Social cognitive theory of self-regulation describes the processes by which individuals monitor their behavior over time for the purpose of enacting and/or maintaining behaviors (Bandura, 1991). Self-regulation is dependent on self-monitoring, including the fidelity, consistency, and proximity of self-monitoring to the target behavior. In the absence of strong intrinsic motivation, individuals may need consistent reminders to maintain diary compliance. Indeed, most studies that achieve high rates of diary completion also employ fairly intensive strategies to optimize compliance, such as personalized reminders from study assistants and automated alarm systems set to participant preferences (Morren et al., 2009; Shiffman et al., 2008). It remains unclear whether this methodology is able to achieve such high levels of compliance in more real-world settings that do not utilize compliance-improving strategies (e.g., mobile phone apps for behavior change).

Social cognitive theory also asserts that certain factors may impede the ability to consistently self-monitor and thus maintain compliance to a diary study, such as demographics, personality characteristics, or behavioral patterns (Bandura, 1991). If certain types of individuals are more or less likely to be compliant compared with others, then findings from such studies may not adequately capture the experiences of the entire sample, thus compromising generalizability of findings. In their review of the literature, Shiffman et al. (2008) reported that EMA and diary studies have achieved high compliance across a variety of marginalized populations, including elderly, low socioeconomic status and severely mentally ill samples. However, few studies have reported on differences in compliance by participant characteristics and behavior within a given study, which means that we do not know whether certain groups are more likely to be compliant than others. Sokolovsky, Mermelstein, and Hedeker (2014) examined predictors of compliance to an EMA protocol examining adolescent smoking escalation and found that higher mean negative affect, smoking rate, alcohol use, and male gender predicted reduced compliance. Another EMA study found that individuals with marijuana dependence had the lowest rate of adherence compared with tobacco-, alcohol- and opiate-dependent participants (Serre et al., 2012). Finally, Rendina, Ventuneac, Mustanski, Grov, and Parsons (2016) found that less educated, younger, and substance-using participants were less compliant in a sexual diary study of adult MSM.

Some research has identified aspects of the diary method itself that influence compliance rates. One sexual behavior diary study compared various diary frequencies and found that participants were more adherent when assigned with less frequent diary schedules (e.g., 84% compliance for diaries completed once every 2 weeks vs. 69.2% for diaries completed twice a week; Glick, Winer, & Golden, 2013). Less frequent diary schedules may be perceived as less burdensome, which could improve compliance. However, less frequent diary schedules often increase the length of individual surveys to gather data from each day in the assessment period, which also increases participant burden. A more recent review of 15 sexual behavior diary studies found no differences in compliance by frequency of administration or study length (Stalgaitis & Glick, 2014), but other EMA and diary studies have found that compliance tends to decrease over the observational period (Courvoisier, Eid, & Lischetzke, 2012; Glick et al., 2013; Heyer & Rose, 2015). For example, Glick et al. (2013) found in their sexual behavior diary study that compliance decreased from 77.9% in the first 6 months of their year-long study to 54.0% in the second 6 months. Understanding patterns and predictors of compliance, including what characteristics of individuals, behavioral patterns, and variations in the diary methodology, influence adherence will help: (a) better contextualize the accuracy and generalizability of findings from diary studies and (b) develop strategies for improving completion rates in less adherent groups.

Analytic concerns are also pertinent to the selection of sampling intervals. Diary methods are most commonly used on behaviors believed to vary within individuals, and the period of time between observations should be small enough to capture that variability. In the case of cyclical patterns where individuals vary around their mean or long-term trend, the Nyquist limit requires that at least four observations be present within that individual’s period or cycle (Luke, 1999). For example, if the behavior in question is more frequent on weekends, weekly measurement will not only fail to detect important within-person variation but will falsely show differences in mean level between individuals depending on where measurement falls during the week. Alternatively, spreading out the same number of measurements over a longer period of time can increase power to detect long-term trends (e.g., Growth Rate Reliability; Rast & Hofer, 2014).

The goal of this article was to examine patterns and predictors of participant diary compliance in the context of a 2-month online diary study of sexual behavior and substance use in a sample of YMSM. We aimed to (a) use latent class growth analysis (LCGA) to examine prospective patterns of participant compliance to daily and weekly diary survey completion and (b) examine demographic (e.g., race/ethnicity) and behavioral (e.g., baseline substance use and sexual behavior) predictors of compliance group membership. Based on the limited existing literature, we hypothesized no differences in diary compliance by demographic characteristics, including age, race/ethnicity, and relationship status. Based on previous research (Rendina et al., 2016; Serre et al., 2012; Sokolovsky et al., 2014), we hypothesized that YMSM who endorsed lower levels of substance use (i.e., alcohol, marijuana, and illicit drugs) and condomless anal sex (CAS) at baseline would be more compliant.

Method

Participants

Participants were a subset of YMSM from a 2-month prospective diary study of substance use and sexual behavior (total N = 370) that aimed to assess the conditions under which behavioral diary studies become self-monitoring interventions by comparing risk behavior change between those that completed diaries and a no diary control condition. Because the control group did not complete any diaries, these participants were excluded from the analyses (N = 129), leaving an analytic sample of 241. The mean age of the analytic sample at the start of the study was 22.90 (SD = 3.14). The largest racial group in the diary sample was White (41.4%), followed in order of size by Hispanic/Latino (25.3%), Black/African American (21.1%), multiracial (8.9%), Asian/Pacific Islander (1.3%), participants who reported their race/ethnicity as other (1.3%), and Native American (0.8%). The majority of participants identified as gay (87.2%) with a minority identifying as bisexual (12.8%). At the start of the study, 40.3% of the sample reported being in a serious romantic relationship.

Procedures and Design

Participants were recruited online via national advertisements posted on Facebook from August 2014 to April 2015. Inclusion criteria were (a) assigned male at birth and current male gender identity, (b) oral or anal sex with another man during the past 6 months, (c) any binge drinking (i.e., five or more drinks on a single occasion) or illicit drug use during the past 30 days, (d) between the ages of 16 and 29 years, and (e) HIV-negative or unknown serostatus. The protocol was approved by the institutional review board at Northwestern University with a waiver of parental permission under 45 CFR 46.408(c) for participants aged 16 to 17 years (Mustanski, 2011).

YMSM who clicked on the Facebook ads were linked to a landing page that described the study and interested individuals were directed to a screener that assessed the above inclusion criteria and collected various forms of demographic and contact information to aid in identifying potential duplicate or fake participants. Eligible participants were sent a link to complete a baseline assessment after which they were randomized to one of three diary conditions: daily diaries (N = 120), weekly TLFB diaries (N = 121), and no diary control condition (N = 129). Daily diary participants reported their substance use and sexual behaviors from the day immediately preceding the day on which the survey was sent, while weekly TLFB participants reported these behaviors retrospectively day-by-day for each day of the preceding week. Participants in each of the active diary conditions were allotted 48 hours to complete each diary survey, though the dates referenced in each diary survey did not change if the participant did not complete the survey within the first 24 hours. Automated e-mails were sent to participants as a reminder to complete diary surveys but very little personalized contact was utilized to encourage participation. Surveys were not accessible 48 hours after they were originally sent.

Within each of the two active diary conditions, half of the participants were randomly assigned to receive automated feedback each week (61 participants in the weekly TLFB condition and 60 participants in the daily condition; total feedback N = 121). Feedback was delivered in the form of graphic illustrations of their frequency of substance use and sexual behavior, as well as change in these risk behaviors from the prior weeks. This feedback was developed to reflect the types of feedback typically provided on self-monitoring mobile applications, and it did not involve personalized contact with the participant. All participants (including control condition) completed a 2-month follow-up survey to assess change in risk behaviors. Participants were paid up to $60 for participation, prorated for participation level. Participants were paid $15 for completing the baseline assessment and $15 for the 2-month follow-up. With regard to diary completion, participants were paid $15 if they completed at least 50% of their diary surveys (regardless of condition) and an additional $15 if they completed at least 85% of their diary surveys. All payments were made upon study completion.

Measures

General Demographics

The demographic questionnaire assessed participants’ age, race/ethnicity, self-reported sexual orientation, and relationship status.

Baseline Substance Use and Sexual Risk Behavior

We created quantity–frequency (QF) indices of alcohol and marijuana use utilizing two items from the Alcohol Use Disorders Identification Test (Saunders, Aasland, Babor, de la Fuente, & Grant, 1993) and Cannabis Use Disorders Identification Test–Revised (Adamson et al., 2010), respectively. The frequency item asked, “How often do you [have a drink containing alcohol/use marijuana]?” Response options ranged from “never” (coded 0) to “4 or more times a week” (coded 4). The alcohol use quantity item asked, “How many drinks containing alcohol do you have on a typical day when you are drinking?” Response options ranged from “1 or 2” (coded 1) to “10 or more” (coded 5). The marijuana use quantity item asked, “How many hours were you ‘stoned’ on a typical day when you had been using marijuana?” Response options ranged from “less than 1” (coded 1) to “7 or more” (coded 5). QF indices were calculated by multiplying the quantity items by the frequency items (alcohol QF: range = 0–16, M = 2.74, SD = 3.10; marijuana QF: range = 0–16, M = 2.84, SD = 4.28). Alcohol QF indices are frequently used as an estimate of the total number of drinks consumed within a given time frame (Greenfield, 2000), and this approach has been extended for measuring marijuana use (Compton, Saha, Conway, & Grant, 2009; Stephens, Wertz, & Roffman, 1993). We created a dichotomous variable to indicate lifetime use of any of the following other drugs: cocaine/crack, heroin, meth, opiates, prescription depressants, prescription stimulants, ecstasy/MDMA (methylenedioxymetham-phetamine), GHB (gamma-hydroxybutyrate), ketamine, poppers, erectile medications, inhalants, or other illicit drugs (39.9% endorsed use of at least one illicit drug). Baseline sexual risk behavior, or condomless anal sex (CAS), was assessed with the item: “On average, how often did you use condoms when you had anal sex with male partners in the past 2 months?” Response options ranged from “none of the time” (coded 1) to “all of the time” (coded 5; range = 1–5, M = 2.82, SD = 1.62).

Diary Completion

Participants who were randomly assigned to the daily diary condition received an e-mail link each day for 56 consecutive days to complete their diary survey, while participants in the weekly TLFB condition received an e-mail link each week for 8 weeks (seven daily reports per weekly survey) to complete their weekly diary survey. Participants were scored as compliant for each survey that was completed within 48 hours of receipt of the e-mail link.

Statistical Analyses

We used LCGAs in Mplus to identify different patterns of diary compliance. In the LCGA approach, within-class variance in growth terms are fixed to zero. We ran models separately for participants who were asked to respond to diaries on a daily and weekly basis in order to potentially identify different patterns of compliance that may be associated with the frequency of receiving the diaries. Each time point was coded as a 0 if participants did not respond to that diary and 1 if participants did respond. We began by determining the best overall fit in a latent curve analysis (i.e., linear, quadratic, or cubic). We took the best model and began comparing the solutions using different numbers of classes to identify which model best fit the compliance data. Models were compared on Akaike information criterion, Bayesian information criterion, entropy, Vuong–Lo–Mendell–Rubin Likelihood Ratio Test (VLMR), Lo–Mendell–Rubin Adjusted Likelihood Ratio Test (LMR), and Parametric Bootstrapped Likelihood Ratio Test (PB). The VLMR, LMR, and PB tests are specifically used to identify the ideal number of classes in class analysis.

After identifying the best-fitting class solution for both the daily diary and weekly TLFB groups, we included demographic predictors in the model to test which demographic factors predicted compliance group membership. Demographic predictors were added by regressing the predictors onto the latent intercept, slope, quadratic term (where applicable), and class membership in Mplus. The demographic predictors included age, relationship status (single = 0, in a relationship = 1), feedback condition (no feedback = 0, feedback = 1), and race/ethnicity. Race/ethnicity was dummy coded so that White would be the comparison group. Participants who identified as Asian/Pacific Islander, Native American, and “other” were combined into a single “other” category due to low representation of these groups in these data.

Finally, for the last models we ran for each diary condition, we included retrospective reports from the baseline interview of alcohol use, drug use, marijuana use, and CAS in the previous 2 months. For all predictor models, we calculated an adjusted p-value threshold using a Bonferonni correction to account for the multiple comparisons in each model, because Bonferonni is the strictest possible adjustment. We also make note of findings that would be significant at traditional cutoffs.

Results

Diary Completion Rates by Condition

The average diary completion rate for the entire diary sample was 56.11% (SD = 37.95%), and the median completion rate was 64.29%. There were 39 participants (16.2%) who never completed a diary (29 in the daily condition and 10 in the weekly condition). Excluding those who never began their diaries, the mean completion rate for the entire diary sample was 66.94% (SD = 31.47%). The mean was 52.37% (SD = 39.82%) for participants in the daily diary condition and 59.81% (SD = 35.77%) in the weekly TLFB condition. With participants who never started excluded, the mean for daily diary participants was 69.05% (SD = 30.53%) and the mean for weekly TLFB participants was 65.20% (SD = 32.26%). Of note, participants who never started their diaries were included in all analyses. Demographic variables were entered into a single multivariate linear regression model and used to predict final diary completion percentages, F(8) = 1.92, adjusted R2 = .03. Only sexual orientation was a significant predictor (standardized beta = .18, p < .05), such that bisexually-identified YMSM had a higher completion rate than gay-identified YMSM. Age, race/ethnicity, and relationship status were unrelated to the final diary completion rate. Feedback and diary condition were also added to the model and neither predicted completion rate.

LCGA With Daily Diary Data

Cubic and quadratic terms were not significant in the overall daily diary model, so only the intercept and linear slope were included in the LCGA models. The best fitting model for diary compliance in the daily diary sample was a three-class solution (see Table 1). Average completion rates at each time point for each class are presented in Figure 1. Class 1 encompassed participants with consistently high compliance and represented 41.32% (N = 49) of daily diary participants. Average completion rates for this “High Compliance” class were above 90% for the course of the study. Class 2 represented participants with consistently low compliance and represented 33.88% (N = 41) of the daily participants. Completion rates for the “Low Compliance” class started below 20% and by day six dropped below 10% and remained there. Class 3 participants began the study with high completion rates that decreased throughout the study. This class represented 24.79% (N = 30) of the daily participants. Completion rates for the “Declining Compliance” class were above 70% on Day 1 but declined throughout the course of the study and were below 30% by Day 56.

Table 1.

Daily Diary Latent Class Growth Analyses.

AIC BIC Entropy VLMR LMR PB
Class 1 9236.42 9242.00 — — — —
Class 2 5171.78 5185.62 1.00 0.00 0.00 0.00
Class 3a 4428.47 4450.77 0.99 0.01 0.01 0.00
Class 4 4171.22 4201.88 0.99 0.08 0.09 0.00
Open in a new tab

Note. AIC = Akaike information criterion; BIC = Bayesian information criterion; VLMR = Vuong–Lo–Mendell–Rubin Likelihood Ratio Test; LMR = Lo–Mendell–Rubin Adjusted Likelihood Ratio Test; PB = Parametric Bootstrapped Likelihood Ratio Test. p Values are reported for the VLMR, LMR, and PB.

a

Best fitting solution.

Figure 1.

Figure 1

Open in a new tab

Daily diary compliance by latent curve growth analysis class.

Note. “Overall” refers to observed retention for entire daily diary sample. Each class refers to the observed average retention for members of that class.

Demographic factors predicting compliance group in a single multivariate model are presented in Table 2. Participants in the “other” race/ethnicity group were more likely to be in the “High Compliance” and “Low Compliance” groups than the “Declining Compliance” group compared with White participants. Age, relationship status, and feedback were not significantly related to class membership in the daily diary condition. Baseline risk behaviors predicting class membership using a multivariate approach are presented in Table 3. Alcohol use, drug use, marijuana use, and CAS at baseline were not associated with compliance class membership.

Table 2.

Multivariate Demographic Factors Predicting Daily Diary Class Membership.

Intercept Slope Reference high
Reference decline
Reference low
Low Decline Low High Decline High
Age (years) −0.01 (0.06) 0.00 (0.00) −0.03 (0.08) 0.03 (0.08) −0.06 (0.09) −0.03 (0.08) 0.06 (0.09) 0.03 (0.08)
Relationship 0.87 (0.44)* −0.03 (0.01)* −0.38 (0.47) 0.69 (0.59) −1.07 (0.60) −0.69 (0.59) 1.07 (0.60) 0.38 (0.47)
Feedback 0.89 (0.41)* 0.02 (0.02) −0.55 (0.46) 0.27 (0.59) −0.82 (0.58) −0.27 (0.59) 0.82 (0.58) 0.55 (0.46)
Black 0.03 (0.52) 0.01 (0.02) −0.01 (0.70) 0.87 (0.95) −0.88 (0.74) −0.87 (0.95) 0.88 (0.74) 0.01 (0.70)
Hispanic 0.29 (0.46) 0.00 (0.01) −0.63 (0.56) −0.27 (0.64) −0.36 (0.71) 0.27 (0.64) 0.36 (0.71) 0.63 (0.56)
Other −1.17 (0.58)* 0.03 (0.02) 0.48 (0.64) −24.72 (0.00)***† 25.19 (0.64)***† 24.72 (0.00)***† −25.19 (0.64)***† −0.48 (0.64)
Open in a new tab
†

Significant at Bonferonni-adjusted p value (p < .003).

*

p < .05.

**

p < .01.

***

p < .001.

Table 3.

Multivariate Baseline Risk Behaviors Predicting Daily Diary Class Membership.

Intercept Slope Reference high
Reference decline
Reference low
Low Decline Low High Decline High
Alcohol use 0.08 (0.06) 0.00 (0.00) 0.02 (0.08) 0.14 (0.09) −0.12 (0.10) −0.14 (0.09) 0.12 (0.10) −0.02 (0.08)
Drug use 0.27 (0.43) 0.01 (0.02) 0.18 (0.53) 0.13 (0.59) 0.05 (0.70) −0.13 (0.59) −0.05 (0.70) −0.18 (0.53)
Marijuana use 0.02 (0.04) 0.00 (0.00) 0.05 (0.06) −0.13 (0.10) 0.18 (0.10) 0.13 (0.10) −0.18 (0.10) −0.05 (0.06)
CAS −0.19 (0.10) 0.00 (0.00) 0.07 (0.11) −0.08 (0.18) 0.15 (0.18) 0.08 (0.18) −0.15 (0.18) −0.07 (0.11)
Open in a new tab

Note. CAS = condomless anal sex. This model adjusted for the following factors: age, race/ethnicity, relationship status, and feedback condition.

†

Significant at Bonferonni-adjusted p value (p < .002).

*

p < .05.

**

p < .01.

***

p < .001.

LCGA With Weekly Diary Timeline Followback Data

The cubic term was not significant in the overall weekly diary model. The terms included in the LCGA models were the intercept, linear slope, and quadratic term. The three-class solution was also the best fitting model for the weekly TLFB sample (see Table 4). Average completion rates at each time point for each class are presented in Figure 2. The three classes present in the weekly TLFB data are similar to those found in the daily diary sample. The first class is again best defined as “High Compliance,” which represented 33.61% (N = 41) of the weekly sample. Participants in this class began the study with over 90% completion rate and by Week 5 had perfect compliance for the remainder of the study. The second class is similar to the “Low Compliance” class from the daily sample and represented 35.25% (N = 42) of the weekly sample. Participants began with an average completion rate near 50% and by the end of the study had a completion rate below 5%. The third class is equivalent to the “Declining Compliance” class and represented 31.15% (N = 38) of the sample. Like the “High Compliance” class, the “Declining Compliance” class began the study with over 90% completion rate, but by the end of the study had declined to below 50%.

Table 4.

Weekly Timeline Followback Latent Class Growth Analyses.

AIC BIC Entropy VLMR LMR PB
Class 1 1277.25 1285.63 — — — —
Class 2 950.34 969.91 0.93 0.00 0.00 0.00
Class 3a 932.55 963.30 0.85 0.02 0.02 0.00
Class 4 927.87 969.81 0.83 0.03 0.04 0.02
Open in a new tab

Note. AIC = Akaike information criterion; BIC = Bayesian information criterion; VLMR = Vuong–Lo–Mendell–Rubin Likelihood Ratio Test; LMR = Lo–Mendell–Rubin Adjusted Likelihood Ratio Test; PB = Parametric Bootstrapped Likelihood Ratio Test. p Values are reported for the VLMR, LMR, and PB.

a

Best fitting model.

Figure 2.

Figure 2

Open in a new tab

Weekly timeline followback compliance by latent curve growth analysis class.

Note. “Overall” refers to observed retention for entire daily diary sample. Each class refers to the observed average retention for members of that class.

Multivariate demographic factors predicting weekly TLFB compliance class are presented in Table 5. Participants who identified as Black were more likely to be in the “Declining Compliance” than the “High Compliance” class compared with White participants, though this difference was not significant when the Bonferonni-adjusted p-value threshold was used. Participants in the “other” race/ethnicity group were more likely to be in the “Low Compliance” class than either the “Decline Compliance” or “High Compliance” classes compared with White participants. Age, relationship status, and feedback were not significantly associated with class membership.

Table 5.

Multivariate Demographic Factors Predicting Weekly Timeline Followback Class Membership.

Intercept Slope Quadratic Reference high
Reference decline
Reference low
Low Decline Low High Decline High
Age (years) −0.02 (0.05) −0.01 (0.04) 0.00 (0.01) −0.07 (0.08) −0.03 (0.07) −0.03 (0.08) 0.03 (0.07) 0.03 (0.08) 0.07 (0.08)
Relationship −0.26 (0.48) −0.37 (0.32) 0.02 (0.05) 0.10 (0.72) −0.36 (0.67) 0.46 (0.97) 0.36 (0.67) −0.46 (0.97) −0.10 (0.72)
Feedback −0.69 (0.44) 0.32 (0.29) −0.06 (0.04) −0.39 (0.64) 0.52 (0.55) −0.91 (0.67) −0.52 (0.55) 0.91 (0.67) 0.39 (0.64)
Black 0.39 (0.55) 0.33 (0.36) −0.04 (0.05) 2.22 (1.30) 3.28 (1.29)* −1.07 (0.75) −3.28 (1.29)* 1.07 (0.75) −2.22 (1.30)
Hispanic −0.73 (0.64) 0.17 (0.39) −0.03 (0.06) 0.22 (0.78) 0.19 (0.75) 0.03 (1.16) −0.19 (0.75) −0.03 (1.16) −0.22 (0.78)
Other −1.09 (0.53)* −0.57 (0.44) 0.06 (0.07) −17.08 (0.74)***† 0.77 (0.74) −17.85 (0.00)***† −0.77 (0.74) 17.85 (0.00)***† 17.08 (0.74)***†
Open in a new tab
†

Significant at Bonferonni-adjusted p value (p < .003).

*

p < .05.

**

p < .01.

***

p < .001.

The results for baseline risk behaviors predicting weekly TLFB class membership in a multivariate model are presented in Table 6. Participants who reported having used drugs in the previous 2 months at baseline were more likely to be in the “High Compliance” and “Declining Compliance” classes than the “Low Compliance” class compared with those who did not report using drugs. Participants who reported higher marijuana use at baseline were more likely to be in the “Low Compliance” class than the “Declining Compliance” class compared with those with lower marijuana use; however, this difference was not significant at the Bonferonni-adjusted p-value threshold. Baseline alcohol use and CAS were not associated with class membership.

Table 6.

Multivariate Demographic Factors Predicting Weekly Timeline Followback Class Membership.

Intercept Slope Quadratic Reference high
Reference decline
Reference low
Low Decline Low High Decline High
Alcohol use 0.18 (0.18) −0.07 (0.07) 0.01 (0.01) 0.09 (0.28) −0.07 (0.40) 0.16 (0.18) 0.07 (0.40) −0.16 (0.18) −0.09 (0.28)
Drug use −2.74 (0.71)***† −0.67 (0.55) 0.03 (0.07) −32.05 (4.00)***† −2.71 (4.00) −29.33 (0.00)***† 2.71 (4.00) 29.33 (0.00)***† 32.05 (4.00)***†
Marijuana use 0.05 (0.07) 0.03 (0.03) 0.00 (0.00) 0.46 (0.39) 0.23 (0.42) 0.23 (0.09)* −0.23 (0.42) −0.23 (0.09)* −0.46 (0.39)
CAS 0.17 (0.09) −0.10 (0.06) 0.01 (0.01) 0.06 (0.21) −0.15 (0.17) 0.21 (0.18) 0.15 (0.17) −0.21 (0.18) −0.06 (0.21)
Open in a new tab

Note. CAS = condomless anal sex. This model adjusted for the following factors: age, race/ethnicity, relationship status, and feedback condition.

†

Significant at Bonferonni-adjusted p value (p < .002).

*

p < .05.

**

p < .01.

***

p < .001.

Discussion

The current study provides insight into patterns and predictors of behavioral diary completion that will help behavioral scientists design diary studies that optimize compliance to best understand the context surrounding health-related behaviors. These analyses found strikingly similar patterns of diary completion across participants who were randomized to a daily or weekly TLFB diary condition for 2 months. Furthermore, we were able to identify demographic and behavioral predictors of diary compliance, which differed by diary schedule.

LCGA of diary completion rates in both daily and weekly TLFB diary conditions revealed very similar patterns of diary compliance across diary schedules. In both conditions, a three-class solution provided the best fit and generated groups that were very similar in completion rates: high compliance, low compliance, and declining compliance groups. Furthermore, each group had roughly equivalent sample size, and this was the case in analysis of both daily and weekly TLFB diary data. This consistent pattern indicates the choice between a daily or weekly schedule need not be made based on one diary schedule being more effective at retaining participants than the other. Instead, the choice between diary schedules should be made based on the target behavior of interest. Behaviors or symptoms that are more variable over shorter periods of time or are less bound by discrete events (e.g., affective states vs. substance use events) are more difficult to recall, so a daily approach may be more accurate. A weekly TLFB approach may be preferable for behaviors that are easier to recall and occur less frequently (e.g., sexual behavior is less frequent than affective states), as evidence suggests that participants are more compliant with less frequent diary schedules over longer periods of time (Glick et al., 2013).

The only notable difference between daily and weekly conditions in compliance lies with the low compliance group. The low compliance group in the daily condition started the study with below 20% completion rate that declined rapidly to less than 10% by the end of Week 1 (Day 7), whereas this same group in the weekly condition started with an approximate 50% completion that declined to less than 10% by the end of Week 4 (Day 21). In the weekly condition, 39.5% of those in the low compliance group completed at least 25% of their surveys, while none of those in the low compliance group in the daily condition completed at least 25%. For participants who are less likely to be compliant in a prospective diary study, weekly diaries may help engage a portion of noncompliant individuals early on, after which more intensive follow-up may help keep them engaged in study activities. It is possible that some may perceive brief daily surveys as more burdensome than longer weekly surveys, which may account for the observed differences in the low compliance groups. However, it should also be noted that others may find the somewhat longer weekly surveys to be more burdensome than a brief daily survey. Further research should disentangle for whom each approach would be a barrier to compliance. Regardless, our data indicate that noncompliant participants may produce more data using a weekly compared with a daily approach, particularly in the short term.

Our analyses also revealed several interesting predictors of compliance group that may provide insight into important targets for improving compliance, and therefore accuracy of diary study findings. In the weekly condition, Black participants were more likely to be in the declining compliance group compared with the high compliance group (a similar nonsignificant pattern was found in the daily diary condition). This indicates that Black YMSM, the group at highest risk for HIV infection in the United States (Centers for Disease Control and Prevention, 2015), are more likely to initiate diaries at high completion rates but decline to about 40% to 50% compliance by the end of 2 months. This decline in compliance may reflect structural barriers to engaging in studies that involve use of technology platforms, such as lack of consistent access to a personal computer or smart phone. Surveys administered via text message do not rely on Internet access and may improve compliance among individuals who have less consistent access to the Internet. Future work should examine technology usability data to understand the platforms typically used by YMSM of different racial groups to provide more context to this finding. We also note that other race participants were more likely to be in the high and low compliance groups relative to the declining compliance group in the daily model, and they were more likely to be in the high and declining groups relative to the low compliance group in the weekly model. Given the heterogeneity of the participants included in the other race group, it is difficult to make conclusions based on these findings.

Findings also revealed that there were no differences in patterns of compliance between participants in both daily and weekly conditions who were randomized to receive tailored feedback about their sexual and substance use behaviors each week. This indicates that self-monitoring interventions may need to provide more targeted strategies to increase diary compliance and maximize behavior change beyond simply providing automated feedback to participants. The feedback delivered in this study was three graphic visualizations demonstrating each participant’s risk behavior during the prior week and across the entire 2-month study period. Feedback did not provide participants with concrete behavior change strategies nor did it involve real-life personalized contact. Our findings indicate that using feedback to engage participants in a self-monitoring program, possibly for the purpose of behavior change, may require more personalized contact with participants that communicates concrete behavior change techniques. Self-monitoring programs and mobile phone apps that rely solely on automated participant contact should be particularly mindful of using strategies to enhance participant engagement in order to maximize their ability to change behaviors or symptoms.

Finally, we examined baseline rates of sexual and substance use behaviors as predictors of group membership. If individuals who engage in these risk behaviors are less compliant to diary protocols, then our estimates of rates and predictors of risk behavior may be inaccurate. Among daily diary participants, none of the target risk behaviors predicted group membership, including baseline alcohol use, marijuana use, other illicit drug use, and CAS. This finding is encouraging and indicates that there was not substantial differential attrition based on baseline engagement in risk behavior and that differential attrition is less likely to affect conclusions for daily diary data.

Several risk behaviors did predict group membership in the weekly condition. First, participants who endorsed illicit drug use (other than marijuana) were more likely to be in the high relative to low compliance group. While there were no observed differences between the high and declining groups, this finding indicates that those participants who use drugs more frequently are more likely to be compliant to weekly TLFB diaries, at least in the short term. Drug users may complete more diaries because they are more motivated to self-monitor and reduce their drug use, given that illicit drugs are perceived as more damaging to health and long-term functioning compared with alcohol and marijuana use. In contrast, marijuana users were more likely to be in the low compliance group compared with the declining compliance group (marijuana users were also more likely to be in the low compared with high compliance group but this did not reach significance), which is consistent with previous findings from EMA studies (Serre et al., 2012). It is also possible that marijuana users are less motivated to self-monitor and reduce their use than illicit drug users because the acute and long-term effects of marijuana use are less severe. It is important to note that our measure of baseline marijuana use was a QF index, meaning higher scores on this variable are more likely to tap into chronic marijuana use. Due to the neurocognitive effects of marijuana use, chronic marijuana users may have more difficulty engaging in activities that require attentional or executive functions, such as completing diary entries (Bolla, Brown, Eldreth, Tate, & Cadet, 2002; Pope & Yurgelun-Todd, 1996). Our measure of illicit drug use, on the other hand, was a dichotomous variable indicating lifetime illicit drug use. This dichotomous variable is less likely to tap into chronic drug use, which may have similar effects on compliance as chronic marijuana use.

Individual differences in attrition are an important substantive and methodological concern in future research. Broadly, attrition in diary studies can be addressed by the same methods as missing data in other contexts. Full-information maximum likelihood permits model estimation on all available data regardless of the amount and location of missing data, and returns unbiased estimates when covariates that predict missingness are included in the model (Enders, 2010; Enders & Bandalos, 2001). Multiple imputation and the expectation–maximization (Dempster, Laird, & Rubin, 1977; Rubin, 1987) algorithm can also account for missing data provided appropriate predictors of missingness or attrition are included. Regardless of what methods are employed, missing data must be appropriately accounted for in a researcher’s statistical plan, else individuals with more data will be given undue weight and the differences between individuals who remain in a study and those who drop out can become confounded with substantive results.

Together these findings indicate that one specific approach to diary studies (e.g., daily vs. weekly schedules) is not necessarily more effective at promoting compliance than the other. However, weekly TLFB diaries appear to encourage better compliance early on, which may allow investigators to more effectively target individuals who have declining compliance before they are lost to follow-up. Furthermore, our study purposefully did not provide intensive follow-up with participants to encourage compliance, and it is likely necessary to engage participants in more personalized contact to head off the substantial decline in compliance that we observed among the majority of participants (i.e., in both the declining and low compliance groups). These findings also have implications for self-monitoring protocols that aim to change behavior. Providing automated feedback to participants about their patterns of risk behaviors, an approach that is frequently use in mobile apps to reduce health-risk behaviors, did not improve diary compliance. If participants are not compliant to self-monitoring protocols, it becomes very difficult to engage in meaningful long-term behavior change. In order to encourage compliance and optimize behavior change in a self-monitoring protocol, it may be necessary to have more personalized contact with participants while delivering tailored strategies for encouraging behavior change.

Finally, the fact that there were no differences in compliance by baseline sexual and substance use behaviors in the daily condition indicates that this methodology captures these experiences across a range of engagement in risk behavior. However, more research is needed to understand the differential patterns of attrition among marijuana and illicit drug users in the weekly TLFB condition. Future studies may help provide deeper context to these findings. For example, qualitative interviews with individuals enrolled in diary studies who fall into each of the compliance groups would help better understand barriers to compliance. Furthermore, examining technology usability data, such as the platform used to access surveys (e.g., personal computer vs. smart phone) or time of day surveys were taken, as predictors of diary completion would help understand whether certain groups experience technological barriers to diary compliance. Indeed, the procedures of this study required that participants have access to the Internet to complete surveys, either on a personal computer or smart phone, which may have been a barrier to compliance. Mobile apps and text messaging strategies that do not require consistent Internet access may improve compliance.

These findings must be considered in the context of several important limitations. LCGA is most appropriate for samples of at least 300 participants. As a result, the findings may be less generalizable and subject to instability. We saw some evidence of instability with shifts in participants’ class membership as demographic and risk behavior predictors were added to models. Membership was largely consistent in the daily condition, with less than 5% of participants shifting classes with the addition of predictors. There was more instability in the weekly condition where up to 40% of participants shifted classes with the addition of predictors. When recruiting online, it is possible that participants may enroll multiple times or fake eligibility. We adhered to rigorous procedures in order to minimize these risks (Newcomb & Mustanski, 2014), but it is not possible to fully rule out these possibilities. Furthermore, all participants were recruited from a single online source (i.e., Facebook). While this approach may lead to some bias in sampling based on frequency of use of this platform, Facebook recruitment ads are advantageous in that they can target a specific demographic group (e.g., men who are attracted to men in a specific age range) nationally and evenly distribute ad placements across regions in order to obtain a less biased sample than venue-based approaches. The incentive structure in this study may have influenced findings. Graduated monetary incentives may improve compliance. However, waiting until the end of the study period to provide incentives may reduce this effect, and the use of microincentives (e.g., small payments for each survey completed) may improve compliance further (Mushtag, Raij, Ganesan, Kumar, & Shiffman, 2011). It should also be noted that certain baseline predictors of diary compliance may not have been stable across the prospective diary period, such as relationship status and average engagement in risk behaviors, which may have influenced findings. Finally, weekly TLFB diaries may be more prone to bias in recall of behaviors compared with daily diaries. However, weekly approaches are less susceptible to this bias when assessing discrete behaviors (as compared with mood states), particularly when the weekly TLFB consistently references the day of the week in its items, as was done in the current study.

In sum, the present findings indicate that daily and weekly TLFB diary approaches produce very similar patterns of diary completion. We found that some demographic variables predicted differential patterns of compliance (i.e., race). However, behavioral risk variables only predicted compliance patterns in weekly TLFB diaries, which provide some evidence that daily diary studies are not biased to capturing data from individuals with high or low engagement in sexual or substance use behaviors. In order to further examine these effects, we urge researchers to report diary compliance rates, as well as demographic and key behavioral outcome differences in compliance. Based on the current findings and previous literature, personalized contact with participants and briefer diary study durations may help improve compliance. Optimizing diary completion by targeting certain groups early who are more likely to be noncompliant will help provide the most accurate observational data, as well as promote compliance to self-monitoring interventions for behavior change.

Acknowledgments

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was funded by a grant from the National Institute on Drug Abuse (R03DA035704; PI: Newcomb).

Footnotes

Authors’ Note

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Drug Abuse.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

  1. Adamson SJ, Kay-Lambkin FJ, Baker AL, Lewin TJ, Thornton L, Kelly BJ, Sellman JD. An improved brief measure of cannabis misuse: The Cannabis Use Disorders Identification Test-Revised (CUDIT-R) Drug and Alcohol Dependence. 2010;110:137–143. doi: 10.1016/j.drugalcdep.2010.02.017. [DOI] [PubMed] [Google Scholar]
  2. Bandura A. Social cognitive theory of self-regulation. Organizational Behavior and Human Decision Processes. 1991;50:248–287. [Google Scholar]
  3. Bolger N, Davis A, Rafaeli E. Diary methods: Capturing life as it is lived. Annual Review of Psychology. 2003;54:579–616. doi: 10.1146/annurev.psych.54.101601.145030. [DOI] [PubMed] [Google Scholar]
  4. Bolla KI, Brown K, Eldreth D, Tate K, Cadet JL. Dose-related neurocognitive effects of marijuana use. Neurology. 2002;59:1337–1343. doi: 10.1212/01.wnl.0000031422.66442.49. [DOI] [PubMed] [Google Scholar]
  5. Brunner E, Stallone D, Juneja M, Bingham S, Marmot M. Dietary assessment in Whitehall II: Comparison of 7 d diet diary and food-frequency questionnaire and validity against biomarkers. British Journal of Nutrition. 2001;86:405–414. doi: 10.1079/bjn2001414. [DOI] [PubMed] [Google Scholar]
  6. Burton C, Weller D, Sharpe M. Are electronic diaries useful for symptoms research? A systematic review. Journal of Psychosomatic Research. 2007;62:553–561. doi: 10.1016/j.jpsychores.2006.12.022. [DOI] [PubMed] [Google Scholar]
  7. Centers for Disease Control and Prevention. Diagnoses of HIV infection in the United States and dependent areas, 2013. Vol. 25. Atlanta, GA: Author; 2015. [Google Scholar]
  8. Compton WM, Saha TD, Conway KP, Grant BF. The role of cannabis use within a dimensional approach to cannabis use disorders. Drug and Alcohol Dependence. 2009;100:221–227. doi: 10.1016/j.drugalcdep.2008.10.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Courvoisier DS, Eid M, Lischetzke T. Compliance to a cell phone-based ecological momentary assessment study: The effect of time and personality characteristics. Psychological Assessment. 2012;24:713–720. doi: 10.1037/a0026733. [DOI] [PubMed] [Google Scholar]
  10. De Castro JM. Methodology, correlational analysis, and interpretation of diet diary records of the food and fluid intake of free-living humans. Appetite. 1994;23:179–192. doi: 10.1006/appe.1994.1045. [DOI] [PubMed] [Google Scholar]
  11. Dempster AP, Laird NM, Rubin DB. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 1977;39:1–38. [Google Scholar]
  12. Enders CK. Applied missing data analysis. New York, NY: Guilford Press; 2010. [Google Scholar]
  13. Enders CK, Bandalos DL. The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling. 2001;8:430–457. [Google Scholar]
  14. Given JE, O’Kane MJ, Bunting BP, Coates VE. Comparing patient-generated blood glucose diary records with meter memory in diabetes: A systematic review. Diabetic Medicine. 2013;30:901–913. doi: 10.1111/dme.12130. [DOI] [PubMed] [Google Scholar]
  15. Glick SN, Winer RL, Golden MR. Web-based sex diaries and young adult men who have sex with men: Assessing feasibility, reactivity, and data agreement. Archives of Sexual Behavior. 2013;42:1327–1335. doi: 10.1007/s10508-012-9984-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Gordon EJ, Prohaska TR, Gallant MP, Siminoff LA. Adherence to immunosuppression: A prospective diary study. Transplant Proceedings. 2007;39:3081–3085. doi: 10.1016/j.transproceed.2007.02.100. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Greenfield TK. Ways of measuring drinking patterns and the difference they make: Experience with graduated frequencies. Journal of Substance Abuse. 2000;12:33–49. doi: 10.1016/s0899-3289(00)00039-0. [DOI] [PubMed] [Google Scholar]
  18. Hankin BL, Fraley RC, Abela JR. Daily depression and cognitions about stress: Evidence for a traitlike depressogenic cognitive style and the prediction of depressive symptoms in a prospective daily diary study. Journal of Personality and Social Psychology. 2005;88:673–685. doi: 10.1037/0022-3514.88.4.673. [DOI] [PubMed] [Google Scholar]
  19. Heyer GL, Rose SC. Which factors affect daily compliance with an Internet headache diary among youth with migraine? Clinical Journal of Pain. 2015;31:1075–1079. doi: 10.1097/AJP.0000000000000208. [DOI] [PubMed] [Google Scholar]
  20. Horvath KJ, Beadnell B, Bowen AM. A daily web diary of the sexual experiences of men who have sex with men: Comparisons with a retrospective recall survey. AIDS and Behavior. 2007;11:537–548. doi: 10.1007/s10461-007-9206-y. [DOI] [PubMed] [Google Scholar]
  21. Huber LR, Broel EC, Mitchelides AN, Dmochowski J, Dulin M, Scholes D. Comparison of prospective daily diaries and retrospective recall to measure oral contraceptive adherence. Contraception. 2013;88:492–497. doi: 10.1016/j.contraception.2013.02.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Leigh BC, Gillmore MR, Morrison DM. Comparison of diary and retrospective measures for recording alcohol consumption and sexual activity. Journal of Clinical Epidemiology. 1998;51:119–127. doi: 10.1016/s0895-4356(97)00262-x. [DOI] [PubMed] [Google Scholar]
  23. Luke HD. IEEE Communications Magazine. 1999. Apr, The origins of the sampling theorem; pp. 106–108. [Google Scholar]
  24. Marshal MP, Friedman MS, Stall R, King KM, Miles J, Gold MA, Morse JQ. Sexual orientation and adolescent substance use: A meta-analysis and methodological review. Addiction. 2008;103:546–556. doi: 10.1111/j.1360-0443.2008.02149.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Morren M, van Dulmen S, Ouwerkerk J, Bensing J. Compliance with momentary pain measurement using electronic diaries: A systematic review. European Journal of Pain. 2009;13:354–365. doi: 10.1016/j.ejpain.2008.05.010. [DOI] [PubMed] [Google Scholar]
  26. Mushtag M, Raij A, Ganesan D, Kumar S, Shiffman S. Exploring micro-incentive strategies for participant compensation in high-burden studies. Paper presented at the 13th International Conference on Ubiquitous Computing; Beijing, China. 2011. Sep, [Google Scholar]
  27. Mustanski B. The influence of state and trait affect on HIV risk behaviors: A daily diary study of MSM. Health Psychology. 2007;26:618–626. doi: 10.1037/0278-6133.26.5.618. [DOI] [PubMed] [Google Scholar]
  28. Mustanski B. Ethical and regulatory issues with conducting sexuality research with LGBT adolescents: a call to action for a scientifically informed approach. Archives of Sexual Behavior. 2011;40:673–686. doi: 10.1007/s10508-011-9745-1. [DOI] [PubMed] [Google Scholar]
  29. Mustanski B, Starks T, Newcomb ME. Methods for the design and analysis of relationship and partner effects on sexual health. Archives of Sexual Behavior. 2014;43:21–33. doi: 10.1007/s10508-013-0215-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Mustanski BS. Are sexual partners met online associated with HIV/STI risk behaviours? Retrospective and daily diary data in conflict. Aids Care. 2007;19:822–827. doi: 10.1080/09540120701237244. [DOI] [PubMed] [Google Scholar]
  31. Newcomb ME. Moderating effect of age on the association between alcohol use and sexual risk in MSM: Evidence for elevated risk among younger MSM. AIDS and Behavior. 2013;17:1746–1754. doi: 10.1007/s10461-013-0470-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Newcomb ME, Birkett M, Corliss HL, Mustanski B. Sexual orientation, gender, and racial differences in illicit drug use in a sample of US high school students. American Journal of Public Health. 2014;104:304–310. doi: 10.2105/AJPH.2013.301702. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Newcomb ME, Mustanski B. Racial differences in same-race partnering and the effects of sexual partnership characteristics on HIV Risk in MSM: A prospective sexual diary study. Journal of Acquired Immune Deficiency Syndromes. 2013;62:329–333. doi: 10.1097/QAI.0b013e31827e5f8c. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Newcomb ME, Mustanski B. Cognitive influences on sexual risk and risk appraisals in men who have sex with men. Health Psychology. 2014;33:690–698. doi: 10.1037/hea0000010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Pachankis JE, Westmaas JL, Dougherty LR. The influence of sexual orientation and masculinity on young men’s tobacco smoking. Journal of Consulting and Clinical Psychology. 2011;79:142–152. doi: 10.1037/a0022917. [DOI] [PubMed] [Google Scholar]
  36. Pope HG, Jr, Yurgelun-Todd D. The residual cognitive effects of heavy marijuana use in college students. Journal of the American Medical Association. 1996;275:521–527. [PubMed] [Google Scholar]
  37. Rast P, Hofer SM. Longitudinal design considerations to optimize power to detect variances and covariances among rates of change: Simulation results based on actual longitudinal studies. Psychological Methods. 2014;19:133–154. doi: 10.1037/a0034524. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Rendina HJ, Ventuneac A, Mustanski B, Grov C, Parsons JT. Prospective measurement of daily health behaviors: Modeling temporal patterns in missing data, sexual behavior, and substance use in an online daily diary study of gay and bisexual men. AIDS and Behavior. 2016;20:1730–1743. doi: 10.1007/s10461-016-1359-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Rubin DB. Multiple imputation for nonresponse in surveys. New York, NY: Wiley; 1987. [Google Scholar]
  40. Saunders JB, Aasland OG, Babor TF, de la Fuente JR, Grant M. Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO collaborative project on early detection of persons with harmful alcohol consumption– II. Addiction. 1993;88:791–804. doi: 10.1111/j.1360-0443.1993.tb02093.x. [DOI] [PubMed] [Google Scholar]
  41. Serre F, Fatseas M, Debrabant R, Alexandre JM, Auriacombe M, Swendsen J. Ecological momentary assessment in alcohol, tobacco, cannabis and opiate dependence: A comparison of feasibility and validity. Drug and Alcohol Dependence. 2012;126:118–123. doi: 10.1016/j.drugalcdep.2012.04.025. [DOI] [PubMed] [Google Scholar]
  42. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annual Review of Clinical Psychology. 2008;4:1–32. doi: 10.1146/annurev.clinpsy.3.022806.091415. [DOI] [PubMed] [Google Scholar]
  43. Shrier LA, Shih MC, Beardslee WR. Affect and sexual behavior in adolescents: A review of the literature and comparison of momentary sampling with diary and retrospective self-report methods of measurement. Pediatrics. 2005;115:e573–e581. doi: 10.1542/peds.2004-2073. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Sokolovsky AW, Mermelstein RJ, Hedeker D. Factors predicting compliance to ecological momentary assessment among adolescent smokers. Nicotine & Tobacco Research. 2014;16:351–358. doi: 10.1093/ntr/ntt154. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Stalgaitis C, Glick SN. The use of web-based diaries in sexual risk behaviour research: A systematic review. Sexually Transmitted Infections. 2014;90:374–381. doi: 10.1136/sextrans-2013-051472. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Stephens RS, Wertz JS, Roffman RA. Predictors of marijuana treatment outcomes: The role of self-efficacy. Journal of Substance Abuse. 1993;5:341–353. doi: 10.1016/0899-3289(93)90003-t. [DOI] [PubMed] [Google Scholar]
  47. Stone AA, Broderick JE, Shiffman SS, Schwartz JE. Understanding recall of weekly pain from a momentary assessment perspective: Absolute agreement, between- and within-person consistency, and judged change in weekly pain. Pain. 2004;107:61–69. doi: 10.1016/j.pain.2003.09.020. [DOI] [PubMed] [Google Scholar]

RESOURCES