Web-Based Sex Diaries and Young Adult Men Who Have Sex with Men: Assessing Feasibility, Reactivity, and Data Agreement

Abstract

We compared quantitative diary data with retrospective survey data collected from a cohort of young adult men who have sex with men (MSM) in Seattle, Washington. Ninety-five MSM, aged 16–30 years, completed web-based surveys every 3 months and were randomized to 4 diary submission schedules: every 2 weeks, once a week, twice a week, or never. We calculated diary completion rates and assessed agreement between daily diary data and aggregate retrospective survey data for sexual behavior measures. Over 6 months, 78% of participants completed at least 80% of their diary days, and the 2-week schedule had the highest and most consistent completion rate. The majority of sexual behavior and substance use measures had strong agreement between the diary and retrospective survey data (i.e., kappa>0.80 or concordance correlation coefficient ≥ 0.75), although we observed poorer agreement for some measures of numbers of anal sex acts. There were no significant differences in mean responses across diary schedules. We observed some evidence of reactivity (i.e., a difference in behavior associated with diary completion). Participants not assigned diaries reported significantly more unprotected anal sex acts and were more likely to be newly diagnosed with HIV or another sexually transmitted infection compared to those assigned active diary schedules. This study suggests that sexual behavior data collected from young adult MSM during 3-month retrospective survey—an interval commonly used in sexual behavior research—are likely valid. Diaries, however, may have greater utility in sexual behavioral research in which counts, timing, sequence, or within-person variation over time are of particular import.

Introduction

Self-reported data on sexual behavior are ubiquitous in epidemiologic studies of HIV and other sexually transmitted infections (STI). Retrospective surveys are the most common tools for collecting sexual behavior data in research studies, yet the potential for measurement error is high (Catania, Gibson, Chitwood, & Coates, 1990; Schroder, Carey, & Vanable, 2003; Weinhardt, Forsyth, Carey, Jaworski, & Durant, 1998). Because these surveys often ask participants to report aggregate behavior over a period of time, participants may forget or inadvertently misreport behavior that occurred outside of the window period (Armstrong, White, & Saracci, 1992; Catania et al., 1990). Diaries, usually in the form of brief daily surveys, are sometimes considered a close approximation to a gold standard for measuring sexual behavior (Schroder et al., 2003). The strengths of the diary method include the ability to determine temporal relationships between exposures and outcomes, the potential for more valid responses due to the short recall period, and the ability to analyze within-person variation and event-level data (Bolger, Davis, & Rafaeli, 2003; Catania et al., 1990; Okami, 2002). However, diary use may also result in attrition, non-representativeness, and reactivity (change in behavior or reporting of behavior associated with monitoring) (Bolger et al., 2003).

Given the epidemiology of HIV/AIDS in the U.S., men who have sex with men (MSM) are a priority population for sexual behavior research (Centers for Disease Control and Prevention, 2009). Recent studies of MSM have highlighted the importance of event-level analysis in identifying behavioral risk factors for HIV infection (Colfax et al., 2004; Irwin, Morgenstern, Parsons, Wainberg, & Labouvie, 2006; Lambert et al., 2011; Mansergh et al., 2008; Mustanski, 2008). Diaries have been used in a handful of sexual behavior studies of MSM, but prior studies comparing diary to retrospective survey data have reported inconsistent results. Some have observed no differences (Durant & Carey, 2000; Hoppe et al., 2008), some have found higher rates of reported sexual behavior in retrospective surveys (Coxon et al., 1993; Gillmore, Leigh, Hoppe, & Morrison, 2010; Graham, Catania, Brand, Duong, & Canchola, 2003; Horvath, Beadnell, & Bowen, 2007), others have observed higher rates in diaries (McAuliffe, DiFranceisco, & Reed, 2007), and still others have observed variable differences depending on the specific behavior measured (Downey, Ryan, Roffman, & Kulich, 1995; Garry, Sharman, Feldman, Marlatt, & Loftus, 2002; Leigh, Gillmore, & Morrison, 1998). Furthermore, in one study comparing MSM HIV risk behaviors with partners met online, data collected through retrospective surveys and daily diaries provided conflicting results (Mustanski, 2007). The retrospective data showed positive associations between seeking partners online and other sexual risk behaviors, which is consistent with other literature; however, the diary data showed that unprotected anal sex was less likely among the partners met online (Mustanski, 2007). This study demonstrated that certain risk behaviors vary within individuals over time and, consequently, the potential importance and utility of using diaries to collect these more nuanced, individual-level data.

Web-based diaries in particular are a promising means to collect detailed sexual behavior data, but the few studies that have compared web- based diaries to retrospective survey data have collected data for ≤ 4 weeks (Garry et al., 2002; Horvath et al., 2007; Miller et al., 2002), and only one specifically enrolled MSM (Horvath et al., 2007). It is not known if a sample of MSM would complete web-based diaries for an extended period of time or how often men should complete diaries. Perhaps more importantly, the value of using diaries is uncertain. For measures where agreement is high between data collected in diaries and periodic retrospective surveys, diaries may offer little advantage and could introduce problems if reactivity is common. On the other hand, if agreement between dairies and retrospective surveys is low and completion rates are high, diaries may offer behavioral scientists an important tool.

We sought to define the feasibility of using online diaries, assess the agreement of reported sexual behavior between diaries and retrospective surveys, and measure reactivity associated with diary completion in a cohort study of young adult MSM.

Method

Participants

The DASH (Development and Sexual Health) Study was a 1-year prospective cohort study. Participants were eligible if they were male, were age 16–30 years, spoke English, intended to remain in the Seattle metropolitan area for one year, and had ever had sex with another male (defined as mutual masturbation or oral or anal sex). Because the primary aim of the parent cohort study was to describe early sexual behavior patterns of MSM, the sample was further restricted to men who either reported ≤ 10 lifetime male partners or were within 5 years of their same-sex sexual debut. We recruited from paid advertisements on Facebook (36% of eligible participants), paid peer referral (21 %), local community- and college-based organizations (14%), clinician referral and flyers at the Public Health-Seattle and King County (PHSKC) STD Clinic (10%), general word-of-mouth (5 %), or other/unknown means (14%).

Procedure

Participants completed web-based diaries throughout the course of the study and web-based retrospective surveys at baseline and every 3 months. Participants also received screening for HIV and other STI (chlamydia, gonorrhea, and syphilis) at the PHSKC STD Clinic at baseline and every 6 months. At the baseline visit, participants provided informed consent and detailed contact information. Study staff created a password-protected online account that allowed participants to access the retrospective surveys and diaries. These tools were created and administered using Illume software and a customized study management system (DatStat, Inc., Seattle, WA). Participants received $50 for their baseline, 6-, and 12-month clinic visits and retrospective surveys; $20 for their 3- and 9-month retrospective surveys; and $1.50 for each week of completed diaries. The Human Subjects Division at the University of Washington approved all study procedures.

Measures

Retrospective Surveys

We required participants to complete the baseline survey on-site, while other surveys could be completed off-site or during an on-site study visit. All retrospective surveys included identical questions about aggregate sexual behavior during the past 3 months. Participants received an automated email invitation to complete each follow-up retrospective survey 1 week prior to the survey’s target completion date, and the survey remained open for 5 weeks. The survey system sent automated email reminders each week that the survey remained unopened.

Diaries

Study participants with regular Internet access completed diaries about their daily sexual behavior. Participants without Internet access were excluded from the diary study. Although the diaries asked about behaviors on each day, participants did not log into their diaries on a daily basis. Instead, to determine the best schedule for long-term diary submission, the survey software randomly assigned participants to complete diaries using one of four schedules: twice a week, once a week, once every 2 weeks, or never during the first 6 months of follow-up. Participants were then re-randomized to a diary schedule for the second 6 months, which could be the same as the first schedule. These diary schedules were chosen based on the retention success of local investigators who used a 2-week schedule in a longitudinal study of young heterosexual women (Baer, Saroiu, & Koutsky, 2002). We hypothesized that the frequency of sexual behavior may be higher among MSM, thus we also chose schedules that were more frequent. Although daily diaries are the most common schedule used in diary studies, we wanted to assess the feasibility of diary use over a longer period and opted not to use daily diaries due to potential fatigue and attrition.

Based on their diary schedule, participants received automated email invitations to complete their daily diaries for the past 2 weeks, 1 week, or 3–4 days. Participants who did not complete their diaries received an automated email reminder 3 days after the initial invitation and then every week for up to 3 weeks. After 3 weeks, incomplete diaries were closed and could no longer be completed.

After logging into the diary website, participants saw a calendar with the days of the diary period highlighted (Fig. 1). Each of these days displayed ‘Yes’ and ‘No’ links, which referred to these instructions: “Click ‘Yes’ on the days you had any type of sex; click ‘No’ on the days that you did not have sex.” The screen included definitions of a “day” and “any type of sex.” The survey asked the participant to enumerate each sexual partner (up to 6) on that day and included questions about the number and type of sexual acts, condom use, if this was the first time ever having sex with the partner, or if the partner was a previous partner but was being included in the diaries for the first time. Once the participant completed a diary day, he returned to the calendar to complete any additional open days. There were no restrictions on the sequence in which the participant could complete the diary days.

Fig. 1.

Screenshot of diary calendar for the DASH study, Seattle, WA, 2009–2010

Outcomes

We assessed the agreement of five sexual behavior measures between the diaries and retrospective surveys during analogous 3-month periods: number of partners; number of new male partners; number of anal sex acts; number of unprotected anal sex acts; and any unprotected anal sex act. Because most retrospective surveys were not completed at exact 3-month intervals, questions that asked about behavior during the past 3 months included the date of the participant’s last survey and asked him to consider his behavior since that date. To calculate the diary estimates for each measure, we summed responses across all partners and days in the same 3-month period. We then compared diary data to retrospective survey questions that asked about aggregate behavior during precisely the same, approximately 3-month, period of time.

Although participants received reminders, some participants did not complete all of their diaries. Ignoring these missing diary days would result in underestimates for all diary measures since these days were covered by the retrospective survey. Because our goal was to measure absolute agreement, rather than relative correlation, we adjusted the diary measures to account for these missing data similar to other diary studies (Gillmore et al., 2010). We calculated the mean daily value for the completed diary days for each individual and imputed this value for each missing day. If a participant completed <50% of available diary days during a 3-month period, we excluded them from analyses using diary data for that period.

Statistical Analysis

To assess the absolute agreement between the diary and retrospective survey data, we used the concordance correlation coefficient (CCC) for continuous outcomes (Lin, 1989) and the Kappa statistic for binary outcomes (Schroder et al., 2003). Kappa statistics ranging from 0.81 to 1.00 represented almost perfect agreement, 0.61–0.80 substantial, 0.41–0.60 moderate, 0.21–0.40 fair, 0.00–0.20 slight, and<0.00 poor (Landis & Koch, 1977). We also calculated the mean difference for each outcome using paired t tests and 95% confidence intervals.

To test for differences in diary data between the 3 active diary schedules (twice a week, once a week, once every 2 weeks), we use done-way ANOVA for continuous variables and χ² or Fisher exact tests for dichotomous variables. Finally, to assess whether completing diaries was associated with a change in behavior (reactivity), we used generalized estimating equations (GEE) to assess for differences in participants’ retrospective survey data over time, differences between participants who were assigned any active diary schedule to those who were randomly assigned no diary, and time-by-group interaction effects. The GEE models used an exchangeable correlation structure and robust standard errors. In exploratory analysis, we also examined the association between diary assignment and HIV/STI diagnosis at the 6-month exam using a Fisher exact test. All analyses were conducted using Intercooled Stata 11.1 (Stata Corporation, College Station, TX).

Results

Study Participants, Retention, and Diary Completeness

From April to December 2009, we enrolled 95 participants. The mean age was 21 years and 41% were Hispanic and/or non-White (Table 1). There were no significant differences in baseline characteristics between participants assigned to each of the 4 diary schedules, although participants assigned no diaries tended to report higher lifetime numbers of partners and recent partners (Table 1). One participant reported that he did not have regular Internet access, thus he was excluded from all diary analyses.

Table 1.

Baseline characteristics of DASH study participants by first assigned diary schedule, Seattle, WA, 2009–2010

Baseline characteristics	Diary schedule (months 1–6)
	Every 2 weeks n (%)	Once a week n (%)	Twice a week n (%)	Never n (%)
Age
<20 years	9 (36.0)	5 (29.4)	11 (42.3)	9 (34.6)
≥20 years	16 (64.0)	12 (70.6)	15 (57.7)	17 (65.4)
Race/ethnicity
White (non-Hispanic)	19 (76.0)	9 (52.9)	16 (61.5)	12 (46.2)
Non-White	6 (24.0)	8 (41.1)	10 (38.5)	14 (53.9)
Highest educational degree
≤High school diploma	18 (72.0)	10 (58.8)	16 (64.0)	17 (65.4)
College degree	7 (28.0)	7 (41.2)	9 (36.0)	9 (34.6)
Sexual orientation
Gay	20 (80.0)	15 (88.2)	23 (88.5)	21 (80.8)
Bisexual, queer, or other	5 (20.0)	2 (11.8)	3 (11.5)	5 (19.2)
Yrs since same-sex sexual debut
≤5 years	20 (83.3)	12 (70.6)	16 (61.5)	21 (80.8)
>5 years	4 (16.7)	5 (29.4)	10 (38.5)	5 (19.2)
No. of male sex partners (lifetime)
≤10 partners	16 (64.0)	12 (70.6)	13 (50.0)	11 (42.3)
>10 partners	9 (36.0)	5 (29.4)	13 (50.0)	15 (57.7)
No. of male sex partners (3 mos)
<3 partners	15 (60.0)	13 (76.5)	16 (61.5)	14 (53.9)
≥3–5 partners	10 (40.0)	4 (23.5)	10 (38.5)	12 (46.2)
Known HIV infection	0 (0.0)	1 (5.9)	0 (0.0)	0 (0.0)

Sixty-eight participants were randomly assigned an active diary schedule during the first 6 months of study. During months 1–3, 82% of these participants completed ≥ 80% of their diary days and 72% completed ≥ 80% during months 4–6 (Table 2). During months 7–12, 63 were randomly assigned an active diary schedule. Overall diary completion rates fell to<60% during months 7–12, so we restricted all subsequent analyses to months 1–6.

Table 2.

Proportion of DASH study participants who were assigned an active diary schedule and completed ≥80 % of assigned diary days in each 3-month interval, Seattle, WA, 2009–2010

Diary schedule	Months 1–6				Months 7–12
	n	Months 1–3 n (%)	Months 4–6 n (%)	Overall n (%)	n	Months 7–9 n (%)	Months 10–12 n (%)	Overall n (%)
Every 2 weeks	25	22 (88.0)	21 (84.0)	21 (84.0)	27	15 (55.6)	15 (55.6)	14 (51.9)
Once a week	17	15 (88.2)	11 (64.7)	14 (82.4)	18	12 (66.7)	11 (61.1)	11 (61.1)
Twice a week	26	19 (73.1)	17 (65.4)	18 (69.2)	18	9 (50.0)	9 (50.0)	9 (50.0)
Combined	68	56 (82.4)	49 (72.1)	53 (77.9)	63	36 (57.1)	35 (55.6)	34 (54.0)

Although there were no statistically significant differences, participants assigned the every 2 weeks schedule during months 1–6 had consistently high completion rates, with 88% and 84% completing ≥ 80% of diary days in months 1–3 and 4–6, respectively. The most frequent diary schedule, twice a week, had the lowest proportion of participants completing ≥ 80% of diary days during each 3-month period (73% and 65 %, respectively).

Among participants assigned an active diary schedule, diary data were available for 82% (10,078/12,350) of the days covered by the 3-and 6-month retrospective surveys, and 84% of these diary days were completed within 1 week of the corresponding invitation. Participants did not complete diaries for 16%(1,987)of the days during this 6-month period, and diaries were not available for 2% (285) of the days because some participants completed their 6-month survey after their diary schedule ended. Therefore, we imputed data for 18% of diary days. On average, we imputed 11.7 days per person(median = 0, range, 0–109) during months 1–3 and 21.8 days per person (median = 8.5, range, 0–113) during months 4–6.

The vast majority (86%) of participants completed both the 3- and 6-month retrospective surveys, although slightly more completed the 6-month (92%) than the 3-month (88%) survey.

Agreement between Diary and Retrospective Survey Data

Among participants with active diary schedules, the frequency of reported behavior was slightly higher in the diaries than in the retrospective surveys for most measures measured on a continuous scale (Table 3). For example, during the first 3 months of follow-up, participants reported, on average, 1.9 new male partners in their diaries and 1.7 new male partners in their retrospective surveys. However, there were no significant differences in the mean responses between the retrospective surveys and diaries for any measure. The CCC and Kappa statistics were mostly ≥ 0.80, indicating strong agreement. The exceptions were numbers of anal sex acts and unprotected anal sex acts during months 1–3, with CCCs of 0.53 and 0.57, respectively. However, these CCCs were ≥ 0.80 during months 4–6. Because diary completion was lower during months 4–6, we also restricted these analyses to those who provided diary and retrospective survey data for both 3-month periods, and we saw no change in the CCC estimates.

Table 3.

Mean differences and agreement in reported sexual behavior between DASH study diary and survey data, Seattle, WA, 2009–2010

Variable	Months	na	Mean or % in diary	Mean or % in survey	Diary-survey, mean difference (95 % CI)	Agreementc
Number of partners	1–3	59	2.6	2.8	−0.2 (−0.5–0.2)	0.91
	4–6	–	b	2.3	–	–
Number of new male partners	1–3	59	1.9	1.7	0.2 (−0.2–0.5)	0.88
	4–6	55	1.5	1.3	0.1 (−0.1–0.4)	0.86
Number of anal sex acts	1–3	56	8.1	6.8	1.3 (−1.9–4.4)	0.53
	4–6	52	6.3	6.2	0.2 (−1.3–1.7)	0.84
Number of unprotected anal sex acts	1–3	57	6.5	6.5	0.0 (−3.8–3.8)	0.57
	4–6	52	4.8	4.5	0.3 (−0.9–1.6)	0.88
Any unprotected anal sex (%)	1–3	57	0.47	0.47	0.00 (−0.07–0.07)	0.86
	4–6	52	0.46	0.44	0.02 (−0.05–0.09)	0.88

^aIncludes participants who completed the survey and ≥50 % of available diary days for each 3-month time period (n =61 for months 1–3, n =56 for months 4–6). Deviations from these sample sizes are the result of missing data for each measure

^bWe could not determine the overall number of partners in the diaries during the second 3-month period since some partners overlapped between the first and second 3-month periods, and our diary programming did not allow us to distinguish which partners in the second 3-month period were also partners during the first period

^cTo measure absolute agreement between the survey and diary measures, we used the concordance correlation coefficient for continuous variables and the Kappa statistic for dichotomous variables

In a sensitivity analysis, we assessed agreement only among participants who completed ≥ 80% of their diary days, and these findings were similar to the overall findings.

Differences between Diary Schedules

We assessed agreement separately by diary schedule and found no systematic differences from the overall findings. There were no significant differences in mean responses across the 3 active diary schedules for any of the sexual behavior measures reported in the diaries during months 1–6 (Table 4).

Table 4.

Differences in sexual behaviors reported in diaries between DASH study diary schedules, Seattle, WA, 2009–2010

Variable	Diary schedule (months 1–6)			F value or χ2 statistic	p
	Every 2 weeks, mean (SD) n =22 (of 25)a	Once a week, mean (SD) n =14 (of 17)a	Twice a week, mean (SD) n =19 (of 26)a
Number of partnersd	2.9 (3.8)	2.4 (1.9)	2.7 (3.5)	<1b	ns
Number of new male partners	3.9 (6.2)	3.6 (4.1)	3.5 (4.3)	<1b	ns
Number of anal sex acts	13.1 (18.2)	17.1 (21.4)	16.7 (31.5)	<1b	ns
Number of unprotected anal sex acts	8.4 (17.8)	14.2 (19.6)	13.4 (31.3)	<1b	ns
Any unprotected anal sex [N/(%)]	11 (50.0)	11 (78.6)	10 (52.6)	3.24c	ns

^aExcludes participants who completed ≤50 % of available diary days for either 3 month period (n =13)

^bANOVA F test

^cχ² test

^dData from months 1–3 only

Reactivity Associated with Diaries

We observed some reactivity, or a difference in behavior associated with the completion of diaries, in longitudinal(GEE)analyses that assessed for differences over time between participants with active diary schedules and those assigned no diaries (Table 5). In the retrospective survey data, participants assigned no diaries reported, on average, 0.9 more new male partners in the prior 3 months than those assigned an active diary schedule (β=0.9, p=.05), but this difference did not change over the course of the study. We observed significantly greater increases in reported behavior over time among participants assigned no diaries compared to those with active diary schedules for anal sex acts (β=4.8, p=.01), unprotected anal sex acts (β=5.2, p<.01), and any unprotected any anal sex act (β=0.8, p<.01). There were no appreciable changes in these estimates after adjusting for baseline differences in numbers of new male partners, and there was no evidence of a dose—response effect associating lower sexual risk with increasing diary entry schedule across the 3 active diary groups.

Table 5.

Frequency of sexual behaviors reported in dash study retrospective surveys at baseline, 3-months, and 6-months among participants assigned to complete diaries compared to participants assigned not to complete diaries, Seattle, WA, 2009–2010

Sexual behavior, past 3 months	Groupa	Time			Beta coefficients (95 % CI)b,c	p
		Baseline survey M (SD)	3-month survey M (SD)	6-month survey M (SD)
Number of partners	Any diary	2.2 (1.6)	2.7 (3.0)	2.2 (2.2)	Group =0.5 (95 % CI: −0.4–1.4)	ns
					Time =0.03 (95 % CI: −0.3–0.3)	ns
	No diary	2.8 (1.7)	3.2 (2.3)	2.3 (1.7)	Group*Time =−0.2 (95 % CI: −0.9–0.4)	ns
Number of new male partners	Any diary	1.3 (1.5)	1.6 (2.7)	1.3 (1.7)	Group =0.9 (95 % CI: 0.0–1.9)	.05
					Time =0.03 (95 % CI: −0.2–0.3)	ns
	No diary	2.2 (2.0)	2.5 (2.5)	1.4 (1.6)	Group* Time =−0.4 (95 % CI: −1.1–0.1)	ns
Number of anal sex acts	Any diary	7.0 (10.8)	7.1 (9.6)	5.9 (8.9)	Group =−2.1 (95 % CI: −7.1–2.9)	ns
					Time =−1.1 (95 % CI: −2.7–0.5)	ns
	No diary	6.6 (10.8)	9.5 (12.6)	13.8 (16.0)	Group* Time =4.8 (95 % CI: 1.2–8.4)	.01
Number of unprotected anal sex acts	Any diary	5.4 (10.5)	6.8 (15.9)	4.3 (8.7)	Group =−3.7 (95 % CI: −8.3–0.8)	ns
					Time =−0.9 (95 % CI: −2.4–0.7)	ns
	No diary	3.8 (9.5)	6.2 (12.1)	12.5 (15.9)	Group* Time =5.2 (95 % CI: 1.8–8.7)	<.01
Any unprotected anal sex [# (%)]	Any diary	31 (56.4)	26 (47.3)	25 (45.5)	Group =−1.0 (95 % CI: −2.0–0.03)	.06
					Time =−0.3 (95 % CI: −0.5–0.1)	ns
	No diary	7 (36.8)	8 (42.1)	12 (63.2)	Group* Time =0.8 (95 % CI: 0.3–1.3)	<.01

^aOut of 68 participants who were assigned an active diary, 62 participants completed the baseline, 3-, and 6-month surveys. Out of 27 participants who were assigned no diary, 20 completed all 3 surveys. For each measure, only participants who had data for each measure for all surveys are included in the analysis

^b“Group” beta compares no diary vs. any diary; “Time” beta is a linear term for each 3-month interval (baseline, 3-month, and 6-month surveys)

^cAdjustment for baseline differences in numbers of new male partners did not appreciably change the beta estimates

Six (26.1%) of 23 participants who were assigned no diaries were newly diagnosed with HIV/STI at their 6-month exam, compared to 3(4.8%)of 63 participants with active diary schedules (Fisher exact test, p=.01).

Discussion

In this prospective study of young adult MSM, we found that web-based diaries were a feasible method for collecting data on sexual behavior for a period up to 6 months, and measures obtained from diaries were generally in agreement with those obtained from web-based retrospective surveys that collected similar data concerning the prior 3 months. We observed poorer agreement for more frequent behaviors including numbers of sex acts. Most measures were comparable across diary schedules, and participants with the least frequent diary schedule had the highest overall completion rate, suggesting that data collection every 2 weeks may be superior to more frequent data collection. There was some evidence of reactivity associated with diaries, with more anal sex acts reported over time in the retrospective surveys among participants without diaries as compared to those assigned any active diary schedule. We also observed a significant positive association between HIV/STI diagnosis and random assignment to a group that did not complete diaries.

Previous research on the agreement between diaries and retrospective surveys in this population has been equivocal (Coxon, 1999; Downey et al., 1995; Horvath et al., 2007; McLaws, Old-enburg, Ross, & Cooper, 1990). We did not observe any systematic differences between methods, and we found high levels of absolute agreement for most measures across the two data collection methods. To our knowledge, this is the only study that compared sexual behavior measures using contemporary, web-based data collection tools for a period of time greater than one month. Therefore, our finding of agreement between the two data collection methods should provide a measure of confidence that the validity of sex diaries and retrospective surveys is similar for many of the most commonly used sexual behavior measures, at least when using a 3-month timeframe.

The exceptions that we identified are consistent with other research that has found better concordance with less frequent behaviors and dichotomous measures, but weaker agreement for high frequency behaviors, presumably due to increased recall error (Downey et al., 1995; Horvath et al., 2007; McLaws et al., 1990). For example, we observed weaker agreement for numbers of anal sex acts (i.e., a high frequency behavior) during the first 3 months. However, this agreement improved during the second 3 months. The frequency of this behavior declined between the time periods, and this may explain the stronger correlation over time. Alternatively, participants may have improved their recall ability in the retrospective surveys after completing diaries for several months. We do not know if the retrospective survey or diary data are closer to the truth, yet recalling numbers of sex acts over a few days or weeks is likely more precise than over 3 months. Therefore, for research questions in which numbers of sex acts is a key parameter, diaries may provide more valid data than 3-month retrospective surveys.

To identify the optimal diary period, we randomized participants to one of four diary schedules. The longest diary period (2 weeks) was associated with the highest overall completion rates. We initially chose this schedule based on the experience of local investigators, in whose study heterosexual female participants completed diaries every two weeks and had very high completion rates (Baer et al., 2002). Our finding that more frequent diary schedules had lower completion rates among MSM demonstrates that a 2-week diary period is appropriate for both MSM and heterosexual populations. Nevertheless, we also observed a steep decline in diary completion rates after 6 months of study. This suggests that in the MSM population, at least using the procedures and incentives employed in our study, this method is most feasible for research studies of short or moderate, rather than long-term, duration.

To our knowledge, this is the first study to assess reactivity associated with diaries by randomizing some participants to no diaries and collecting separate retrospective survey data from all participants. In a study in which all MSM participants completed diaries, there was an overall decrease in several sexual behaviors during one month of follow-up (Horvath et al., 2007). We did not observe a significant change in reported behavior over six months among our participants who were assigned to a diary, but we did see an increase in number of anal sex acts and number of unprotected anal sex acts over time among participants assigned no diary. An alternate explanation for these findings is that the no diary group overestimated their behaviors due to “telescoping” (Catania et al., 1990). This is a phenomenon observed in other diary studies where participants expand their recall window beyond what is included in the question and may have occurred in the present study if not completing regular diaries systematically impacted the ability to correctly recall behaviors. However, in an exploratory analysis, we observed a significantly higher rate of incident HIV/STI diagnosis among the no diary group, a biologic measure that is not subject to reporting bias. Thus, we believe that the measured effect on behavior was likely valid, though this requires additional study. For example, MSM could be recruited online or from clinics and organizations serving MSM clients, and diaries could be integrated into standard care services; in return, MSM could receive tailored educational information based on these behaviors.

Our study had several limitations. First, our sample size was relatively small. Because of this, some of our analyses may have been insufficiently powered to detect significant differences, particularly in sub-group analyses. Second, our sample may not be representative of all young adult MSM. We recruited many participants online via paid advertisements given the high prevalence of Internet use among MSM (Mark, Wald, Drolette, & Golden, 2008) and the popularity of social networking sites(e.g., Facebook) among young adults (Lenhart, Purcell, Smith, & Zickuhr, 2010). Furthermore, Seattle MSM may not be representative of MSM in other parts of the U.S. However, the estimates in our sample are similar to those observed among similarly aged MSM in a 2006 Seattle-based random digit dialing survey (S.N. Glick, unpublished analyses). Third, to keep diary completion manageable for participants, we limited the number of questions we asked in each diary and were only able to compare a limited number of measures between the diaries and retrospective surveys. Fourth, we needed to impute 18% of the diary data due to missing diary days and used an imputation method similar to other diary studies(Gillmore et al., 2010). This method assumed that behavior on the missing days was similar to the days for which diaries were completed, and we would have over estimated the level of agreement if participants were more likely to systematically skip their diaries when they had either more or fewer partners than what was typical. We assessed agreement only among those with very complete diary data and did not observe any systematic differences from the overall results.

Given the effort that both participants and researchers must provide in order to complete and collect web-based diaries, it appears that for many studies of young adult MSM, 3-month retrospective surveys are adequate. Diaries, however, appear to provide more precise counts for frequent behaviors such as sex acts and provide more detail about the timing, sequence, and within-person behavior variation than retrospective surveys. Our results, especially when combined with other MSM diary studies that demonstrated the potential importance of measuring within-person variation in sexual risk behavior over time (Mustanski, 2007), suggest that web-based diaries among MSM may have important utility in research studies in this population. To overcome the limitations observed in our study, future work should explore strategies to improve diary completion rates, particularly for study periods longer than 6 months (e.g., using smart phone-based applications, fewer overall questions, and/or escalating incentives). Furthermore, our preliminary findings that diaries may be associated with lower levels of reported sexual behavior and STI suggest that web-based diaries may be useful as a behavioral intervention to prevent HIV/STI among young adult MSM. This subject merits additional study, preferably in a larger randomized trial with a biologic outcome.