Internet based HIV prevention research targeting rural MSM: feasibility, acceptability, and preliminary efficacy

Abstract

Internet delivered primary prevention interventions for HIV risk reduction present significant challenges. Changing lifestyle behaviors, such as beginning to use condoms, is difficult and men seeking dates on line may want to avoid thinking about HIV risk which may lead to low initiation and high dropout rates. Many Internet delivered HIV risk reduction programs have mimicked face-to-face outreach programs, failing to take advantage of the Internet’s capabilities or did not conduct evaluation. This study focuses on examining the feasibility, acceptability, and efficacy of an Internet delivered HIV risk reduction program for rural men who have sex with men (MSM). The program included online recruiting, three intervention modules, each with two sessions, online questionnaires. The intervention was developed based on iterative research and the Information-Motivation-Behavioral skills model. Participants (N = 475) were randomly assigned to one of six module orders and data were collected automatically at pre-test and after each module. Data supports the feasibility and acceptability of the program as demonstrated by good retention and rapid program completion. Knowledge, self-efficacy, outcome expectancies and motivation increase in a dose response fashion. Post-intervention behavior changes included reduced anal sex and significant increases in condom use. Limitations include a short follow-up period, a predominantly young white rural sample, and the lack of an attention control. Overall the results of the study provide support for the efficacy of Internet-based interventions to reduce risk of HIV infection. Results also support traditional research methods to evaluate HIV prevention programs delivered exclusively through the Internet.

Introduction

The number of people utilizing the Internet as a source of health information has increased rapidly in the last decade (Fox and Fallows 2003) making it an excellent medium for delivery of preventive health information. Prevention programs delivered via the Internet have a number of advantages. Health-related information and resources can be provided inexpensively and the participant may remain anonymous (Williams et al. 2005). Once developed, interventions delivered via the Internet are accessible from any computer connected to the World Wide Web (Bowen et al. 2006). Additionally, individuals can visit Internet-based interventions in private, reducing the fear of embarrassment or discovery (Kling et al. 1999; Williams et al. 2005) at a time when the individual is ready to do so, and as often as he/she likes (Binik 2001; Mustanski 2001). In response to these advantages, Internet-based interventions have been developed to address a wide range of health concerns.

Internet-based secondary and tertiary prevention efforts have been developed to target behavioral aspects of chronic health problems such as obesity (Hunter et al. 2008), diabetes (Glasgow et al. 2003; McCoy et al. 2005; McKay et al. 2001), diabetes self-care (Wangberg 2008), back pain (Buhrman et al. 2004) and headache (Devineni and Blanchard 2005). Mental health-focused Internet intervention programs have addressed depression (Andersson et al. 2005), panic disorder (Carlbring et al. 2001, 2005), eating disorders (Paxton et al. 2007) and psychological distress (Andersson et al. 2005; Devineni and Blanchard 2005). Finally, online interventions for addictive behaviors include alcohol abuse (Saitz et al. 2004) and smoking cessation (Oenema et al. 2008).

Internet delivered primary prevention programs, focusing on lifestyle behaviors such as exercise, healthy eating, and safer sex, are less common. Examples of Internet delivered lifestyle interventions include 14 exercise programs (reviewed by Vandelanotte et al. 2007), nine adult healthy eating (reviewed by Norman et al. 2007) programs and an RCT that focused on a variety of behaviors including fat intake, physical activity, and smoking cessation (Oenema et al. 2008). Overall, results from these studies are variable. The programs generally combined the Internet with one or more traditional intervention methods (i.e., face-to-face, interactive media, and phone technology), thus reducing the convenience of an all Internet study. Attrition, while variable (ranged 5% to 69%), averaged over 20% and attrition increased with duration of study participation. Finally, outcomes were generally measured shortly after the intervention and results were mixed, with about half showing positive outcomes for the Internet program.

The potential utility of Internet-delivered HIV sexual risk reduction is regularly cited in the literature (Bull et al. 2001; Pequegnat et al. 2006), but little outcome research is available. The most commonly cited approach to Internet-based HIV prevention is providing one-on-one information in chat rooms. Employees of community-based organizations or AIDs service organizations passively lurk in chat rooms and provide health information and HIV testing referrals when asked (Benotsch et al. 2006; McFarlane et al. 2005). Chat room interventions are limited in a number of ways. The reach of the intervention is limited because they require one-on-one interactions similar to face-to-face interventions. Participation flexibility is limited because project staff must be online, thus limiting the times when participants can log-on. The format is limited because chat room interventions do not use graphics or interactive activities and the content of conversations is difficult to control. Finally, for most chat room interventions, evaluation is either non-existent or limited to counting site hits (Benotsch et al. 2006; McFarlane et al. 2005; Ybarra and Bull 2007).

Published studies of HIV interventions that take full advantage of the Internet, including automating most aspects of the study (i.e., recruiting, consent, randomization, and outcome surveys), use of graphics, and 24 h access, are limited. A search of the literature produced three articles describing interventions in various stages of development and evaluation (Bowen et al. 2006; Bull et al. 2004; Kok et al. 2006). Kok et al. (2006) have published a discussion of six steps they used to develop an online interactive HIV intervention, including a needs assessment, program objectives, translation from theory to strategies, integration of the strategies into the intervention, implementation, and evaluation. The program is currently running and evaluation is in progress. Bull et al. (2004) developed Smart-Quest, a set of tailored messages presented to MSM. Participants were recruited using one-on-one chat room lurking and banner ads. In an effort to maintain anonymity, participants were asked only for email addresses for follow-up. The intervention was evaluated using a randomized control design with a 3 month follow-up. After 3 months, Bull et al. found that the majority of email addresses were invalid, such that only 15% of the 1776 original participants completed the follow-up, precluding an assessment of intervention outcome.

The pilot study by Bowen et al. (2006) evaluated the feasibility and efficacy of a two-session Internet-delivered intervention using a randomized control trial design. The brief Internet-based intervention focused on increasing HIV-related knowledge among rural men who have sex with men. The participants were randomized into either an intervention group with a 1-week follow-up or a 1-week wait-list control. Ninety percent of the participants were retained in the study 1 week following enrollment. HIV-related knowledge, condom use outcome expectancies, and condom use self-efficacy increased contingent upon participation in the intervention. These results support both the feasibility and acceptability of an Internet intervention for rural MSM and encouraged further development and evaluation of a comprehensive multi-session Internet-based HIV/AIDS risk reduction intervention.

Rural MSM generally have not been targeted for HIV prevention and the internet has the potential to reach them (Bowen 2005; Bull et al. 2001; Pequegnat et al. 2006). In rural areas, social stigma attached to homosexual behaviors and HIV present significants barrier to accessing HIV-related services. Rural MSM have access to the Internet and use it regularly to socialize and seek information (Bowen 2005). The purpose of this study was to examine the feasibility, acceptability and initial efficacy of the Wyoming Rural AIDS Prevention Project (WRAPP), which is an expansion of the intervention tested in the Bowen et al. (2006) pilot study. Intervention development was guided by the results of iterative studies conducted with rural MSM (Williams et al. 2005) and the Information-Motivation-Behavioral skills model (Fisher and Fisher 1992, 1993). The multi-module intervention was designed to increase condom use during anal sex and decrease the frequency of anal sex via HIV risk reduction education, increased HIV risk reduction self-efficacy and outcome expectancy, and behavioral skills acquisition.

The current study had four primary goals. First, to examine the feasibility of a completely electronic research study where participants were recruited through banner ads, automatically randomized, informed of intervention procedures and completed multiple post-test questionnaires. The second goal was to assess the acceptability of the program in terms of retention across multiple sessions. The third goal was to identify module specific effects on IMB skills model cognitions. It was hypothesized that there would be a significantly greater increase in knowledge and knowledge related self-efficacy (i.e., mechanical self-efficacy) after participation in the Knowledge modules than either of the other two modules. Finally, a fourth goal was to determine if the modules had a dose response effect on cognitive variables including knowledge, outcome expectancies, self-efficacy and willingness to engage in risk reduction. It was hypothesized that all cognitive variables would increase significantly more after participating in three modules than one module.

Methods

Study design

The study design is depicted in Fig. 1. Participants who completed the screening, consent, and pre-test questionnaires were randomly assigned to one of six module orders. This procedure allowed for analysis of the between groups effects of completing one module, the potential differential effects of different module orders, and the cumulative effects of all three modules. Initially, the knowledge module was conceptualized as a control, but results of the pilot study (Bowen et al. 2006) indicated that this module had significant effects on all cognitive variables as well as knowledge. An attention control group was not used since this was an exploratory study designed to assess the acceptability and initial efficacy of the WRAPP intervention.

Fig. 1.

Diagram of the WRAPP Project. This diagram shows the flow of the WRAPP project and participant attrition from initial ‘‘hits’’ to the 69% of randomized participants who completed the study. Abbreviations include ‘‘I’’ for the Knowledge module, ‘‘P’’ for Partner module, and ‘‘C’’ for the context module, and ‘‘PT’’ for post-test

The project was approved by the University of Wyoming and The University of Texas Health Science Center at Houston Institutional Review Boards.

Participants

Rural MSM, who frequented a popular website for gay men, were recruited using Internet banner ads that ran nationwide. Men interested in the WRAPP study completed a brief screening questionnaire to determine eligibility. Eligibility criteria were: 18 years of age or older, had sex with a man in the last 12 months, and lived in a rural area. The definition of ‘‘Rural’’ may vary by the purpose of the research (Rickette and Gesler 1992; Rickette et al. 1997) and includes size of city, distance from an urban area and/or mean density per square mile. In this study rural was defined as ‘‘living in a town of 75,000 people or fewer and at least 60 min drive from an urban center’’ (Bowen et al. 2004).

Intervention

The intervention was developed using the IMB model as a theoretical framework. The procedures for identifying appropriate topics and content have been explained in Bowen et al. (2006). The intervention consisted of three modules, each included two 20-min interactive sessions and printable feedback tailored to the participant’s responses during the intervention. All three modules can be view at http://www.wrapphome.net/.

Knowledge module (K: living with HIV and HIV prevention sessions)

This module was developed as the Information component of the IMB model and is described in the Bowen et al. (2006) pilot study. Briefly, the module format was a scripted conversation between an HIV-positive gay man, representing the ‘‘expert,’’ and a second man who had recently had a high risk sexual experience. The first session deals primarily with topics related to living with HIV and the second focuses on prevention. Dialogue is interspersed with interactive activities. ‘‘Tell me more’’ buttons are included with different areas of information that, if clicked, provide web links to additional HIV/AIDS resources/information on the feedback page.

Partner module (P: new and casual partners sessions)

The Partner Module was conceptualized as the Motivational component of the IMB model. The modules contrast life goals with short-term high risk situations. The format is a scripted discussion about new and casual sex partners between ‘‘four men’’ and the participant. The New Partner session is presented first and focuses on the participant clarifying long-term life goals, identifying excuses for having unsafe sex with new partners. Scripted interactive discussions focus on excuses for unsafe sex, and determining if unsafe sex is consistent with the participant’s life goals. The participant is asked to choose possible approaches to insisting on condom use with a new partner and rating his willingness to try a chosen approach.

The Casual Partner session is presented second. It uses the same group format, but the focus is on re-initiating condom use with non-exclusive sex partners. The scripted dialogue is interactive, such that the participant can choose a number of different approaches to convincing his part-ner(s) to use condoms. The participant evaluates his satisfaction with the chosen dialogue and also evaluates his confidence that he could have the specific conversation. The participant’s personal ‘‘red flags’’ for unsafe sex are identified, such as being lonely or having had too much to drink. The participant chooses a personal ‘‘red flag’’ (e.g., loneliness) that might lead to justifying unprotected sex. He then engages in a scripted interactive dialogue with a fictitious partner that focuses on safer sex in the context of the red flag. Printable feedback at the end of each session provides a summary of the participant’s responses to interactive components.

Contexts of risk module (C: bar and internet sessions)

The two sessions of this module focus on building skills by identifying specific risk reduction behaviors when looking for sex partners in bars and on the Internet. Like the Partner module, the two sessions are scripted group discussions among ‘‘four friends’’ and the participant. The Bar session narrative begins with a discussion of safer sex, including carrying condoms to a bar, setting limits on drinking and sexual activities before going to a bar and refraining from anal sex with partners met at a bar or delaying anal sex until the partner is better known. The participant identifies personal ‘‘red flags’’ for unsafe sex, such as feeling lonely, and identifies ways to maintain limits. The interactive components include choosing preferred approaches to safer sex when planning to go out to a bar, after meeting a partner in a bar, and after leaving the bar with a partner.

The Internet session covers advantages and disadvantages of meeting sex partners on-line. Discussions center around posting safer sex preferences on personal profiles, rejecting potential partners whose profiles include unsafe sex preferences, and being aware that false information is posted in profiles. Internet ‘‘red flags’’ such as lack of HIV status in a personal profile, are identified for new users. The use of chat rooms is examined in terms of pros and cons. Interactive aspects of the module focus on using different approaches to increase the safety of face-to-face meetings with online contacts. Printable sheets for both sessions provide a summary of the participant’s choices on interactive activities.

Procedures

Procedures for obtaining informed consent have been described in the Bowen et al. (2006). Briefly, eligible men viewed online informed consent pages and indicated consent by devising a username and password to access a project ‘‘account’’. To activate the account, receive a small gratuity for participating in the study, and receive reminders, men were asked to provide a valid e-mail address. Men were asked to provide contact information that included: first and last name, telephone number, and a friend’s name, email and telephone number, although these were not required. Prior to activating each participant’s account, information provided by a participant was checked against data on prior participants to identify potential duplicates. Two coders examined the data for similar IP addresses, emails, passwords, usernames, zip codes, and telephone numbers. Submissions that had at least two identical detection variables were coded as multiple submissions and those with only one were examined more closely. For a more complete discussion of the fraud detection methodology, see Bowen et al. (2008). If there was no evidence of previous participation, a participant was sent an email saying that his username and password had been activated and a link was provided to the project.

Upon completion of the pre-test questionnaire, participants were randomly assigned to one of six module orders (Fig. 1) and shown a schedule of intervention activities, expected completion dates, and the reimbursement they would receive for completing each post-module questionnaire. Participants could begin their first assigned session immediately after completing the pre-test assessment.

The system was programmed so that participants were required to wait at least 48 h between sessions (i.e., minimum time to completion was 10 days). To encourage module completion, email reminders were sent at 5, 12 and 14 days after beginning a module. Participants were dropped from the study if they took more than 16 days to complete a module and the associated post-test. Participants were reimbursed for completion of each questionnaire with gift certificates to an online shopping site. Gift certificates were emailed within 48 h of completion of each questionnaire. All participants earned a $15 gift certificate for the pre-test questionnaire and then they were randomized into one of two reimbursement schedules. In the first schedule, participants were reimbursed $25 for each completed post-test questionnaire (maximum $90). In the second schedule the participants earned $20, $25, and $30, respectively for post-test questionnaires 1, 2, and 3 (maximum $90). Retention was identical for both schedules, so no further analyses consider the schedule effects.

Measures

Study questionnaires were administered at intake (pre-test), and after completion of each intervention module (post-tests 1, 2, & 3). The questionnaires were identical, except that participants reported demographic characteristics and behavioral histories in the pre-test only. In addition to demographic characteristics and HIV risk behavior histories, questionnaires assessed participants’ HIV/AIDS knowledge, cognitive precursors to risk reduction behaviors (i.e., self-efficacies, outcome expectancies, and willingness to change risk behaviors), and recent HIV sexual risk behaviors.

Demographic characteristics

The demographic characteristics reported by participants included age, sexual orientation, ethnicity, education, student status, work status, income, and religious affiliation. Participants reported age in years, which was recoded into a categorical variable. Initial choices for reporting race/ethnicity were Caucasian, African American, Asian/Asian Pacific Islander, American Indian, Hispanic, and other. Because only a small number of participants were minorities, ethnicity was recoded as Caucasian, Hispanic, and other. All other demographic characteristics were recorded as categorical variables and included education, student/non-student, work status, income, relationship status, and religion (see Table 1 for categories).

Table 1.

Comparison of demographics of randomized participants who did not complete post-test 3 (‘‘dropped out’’) to those who completed the study (‘‘completed’’)

	Dropped outaN = 131 (%)	CompletedbN = 294 (%)	Zc
Age			−.03
18–24	48.9	46.8
25–34	29	31.1
35–44	8.4	15.0
45–80	13.7	7.2
Sexual orientation			−.34
Gay	86.3	84.4
Bisexual	13.7	15.3
Heterosexual	0	.3
Ethnicity			−.79
Non-hisp white	80.2	77.2
Hispanic	9.9	8.8
Asian/API, African. Amer., Native. Amer., other	9.9	13.9
Education			−1.27
High school or less	25.2	19.7
Some college or more	74.8	80.3
Student			−.03
Full time	29.8	31.3
Part-time	14.5	12.2
Not a student	55.7	56.5
Work status			−.38
Full time	53.1	55.7
Part-time	20	17.9
Unemployed/retired	26.9	26.5
Income			−.77
< 15,000	42.7	36.7
15,000–24,999	21.4	25.9
25,000–49,999	26	27.9
≥50,000	9.9	9.5
Religion			−.88
Catholic	13.7	21.1
Protestant	23.7	21.1
Spiritual/non-affiliated	24.4	19
Atheist/agnostic	15.3	17
Other	22.9	21.8

^aDrop out = Did not complete questionnaire at time 4, post module 3

^bCompleted 3 modules and questionnaires at pre-test, time 2, 3, and 4

^cMann–Whitney U test, Z score

^*p < .05

HIV/AIDS knowledge (IMB information)

The HIV/AIDS knowledge questionnaire included 13 questions about facts identified in previous qualitative research (Williams et al. 2005) thought to be related to HIV risk in rural MSM. Eight of the questions were answered ‘‘true,’’ ‘‘false’’ or ‘‘do not know.’’ Five questions had multiple-choice responses. For all questions, correct responses were coded ‘‘1.’’ Incorrect or ‘‘don’t know’’ responses were coded ‘‘0.’’ Correct responses were summed, maximum 13, to achieve a knowledge score.

Condom use outcome expectancies (IMB-motivation; Fisher et al. 2002)

Items measuring outcome expectancy were posed as statements meant to elicit a response about using condoms. Participants were asked to rate whether the statement reflected a belief about using condoms that was important to them. Statements were rated on six-point Likert-type scales that ranged from ‘‘not important’’ to ‘‘very important.’’ Results were subjected to principal components analysis (PCA) with Varimax rotation. Two factors were produced. The first, condoms make sex feel ‘‘safer/cleaner’’ (SC-OE), included five items and accounted for 28.60% of the variance. (Cronbach’s α = .88). An example item in this factor was, ‘‘Sex (with a condom) would feel cleaner.’’ The second factor, ‘‘emotional’’ (E-OE), consisted of five items related to how sex would feel with a condom. This factor accounted for 28.61% of the variance (α = .88). Items in this factor were similar to, ‘‘Sex (with a condom) would be less exciting.’’ Two items did not load on either factor and were not used in the analysis.

Condom use self-efficacy. (IMB—behavioral skills; Kalichman et al. 2008)

Self-efficacy was assessed using 23 questions that asked participants how confident they felt using condoms. Participants responded to the questions using six-point Likert-type scales that ranged from ‘not at all confident’ to ‘very confident.’ Results were subjected to two PCAs. The first PCA produced two factors, a mechanical self-efficacy factor (MSE, 4 items) and a second factor that included the remaining questions. The mechanical self-efficacy factor accounted for 17.75% of the variance (α = .80). A sample item is: ‘‘How confident are you that you will remember to pinch the tip of the condom to remove any air bubbles?’’

The remaining 19 questions were subjected to a second PCA, and two factors emerged: Emotional (E-SE; 8 items), and Refusal (R-SE; 10 items). A factor measuring the emotions of using a condom (E-SE) was composed of eight items that accounted for 31.11% of the variance (α = .94). A sample question was, ‘‘How confident are you that you will make sex pleasurable with a condom?’’ The R-SE factor included 10 items measuring the ability to refuse to have anal sex if a condom was unavailable (R-SE). The factor accounted for 36.86% of the variance (α = .96). A sample item was, ‘‘If you don’t have a condom, how confident are you that you will refuse anal sex?’’ One question had a factor loading below .5 and was dropped from further analysis.

Willingness to reduce HIV risk behaviors (IMB-motivation; Fisher et al. 2002)

Participants’ willingness to reduce HIV risk was assessed using four questions rated on six-point Likert-type scales that ranged from unwilling (1) to willing (6). The four questions were ‘‘How willing are you to try the following: ‘Limit my sex partners to 1 per month;’ ‘Use a condom with every new partner all the time;’ ‘Stick to oral sex, until we have been monogamous for 6 months and test HIV-negative;’ and ‘Use condoms all the time, until we have been monogamous for 6 months and test HIV-negative.’’’

HIV risk behaviors

First, participants were asked: ‘‘In the last 30 days, how many sexual partners have you had?’’ (or at Post-test 3: ‘‘Since joining the WRAPP Project, how many sex partners have you had?’’) Next frequency of anal sex was assessed with the following question. ‘‘With how many of your (insert # sex partners) partners did you have anal sex?’’ Finally, condom use was assessed by asking, ‘‘Of the (insert # anal sex partners) men you had anal sex with, how many did you use a condom with EVERY time?’’ The anal sex index (ASI) was computed by dividing the number of anal sex partners by number of sex partners. The condom use index (CUI) was computed by dividing the number partners with whom condom was used every time by the number of partners with whom the participant had anal sex.

Analyses

Missing data

Given the electronic nature of the data collection, there was very little data missing from the 295 completers. Three participants skipped one knowledge question on the pre-test, one skipped one question on post-test 4, and one skipped a number of knowledge questions on the post-test 4. The modal response was entered on the skipped knowledge question for the four participants who missed only one question. The fifth participant was recoded as a ‘drop-out.’ There was no other missing data for the participants who completed all four interventions.

Demographics

All demographic comparisons were conducted using a chi square and p < .05 was considered significant. Participants who completed the screening questionnaire were divided into eligible (rural) and ineligible (urban) groups. The sample of ineligible men included only urban men because underage men, heterosexual men who had not had sex with other men, and women were immediately routed out of the survey and no data was saved. Age, ethnicity and sexual orientation were the only demographics assessed in the screening questionnaire.

Four hundred twenty-five eligible participants completed the pre-test questionnaire and were randomized into one of six module orders. These participants were divided into ‘‘completers’’ and ‘‘drop-outs’’ based on completion of post-test 3. In both groups nominal and ordinal demographics were compared using Chi Square tests and pre-test cognitive variables (knowledge, self-efficacy, and outcome expectancies) were compared using independent samples t-tests.

Completers (participants who completed all intervention and questionnaire components) were divided into three groups based on the number of sex partners reported in the 30 days prior to starting the project. The groups included participants who reported ‘‘Zero’’, ‘‘One’’, and ‘‘Two or more’’ sex partners. These groups were based on the notion that HIV risk was lowest if the participant had not had sex or was limiting sex to one partner. Additionally, condom use with one partner was likely to be different than with multiple partners. The three groups were compared on demographic characteristics using chi square and pre-test cognitive variables using a one-way ANOVA.

Intervention effects on cognitive variables

In all analyses of change in cognitive variables p < .01 was considered significant. The rationale for p < .01 was that experiment-wise error would be inflated by the large number of significance tests, so p < .05 was too liberal. On the other hand, a strict Bonferonni correction of the significance level might preclude identification of important differences in this exploratory study.

First, individual module effects were examined from pre-test to post-test 1 to determine whether the intervention modules had differential effects on any of the cognitive variables (knowledge, self-efficacies, outcome expectancies, and ‘‘willingness to reduce risk’’). The analyses included one 2 (repeated measures; pre-test to post-test 1) by 3 (between subjects; modules) mixed model ANOVA with knowledge as the dependent variable and three 2 (repeated measures; pre-test to post-test 1) by 3 (between subjects, modules) mixed models MANOVAs were used to examine self-efficacy, outcome expectancies and willingness, respectively.

Second, we examined the potential effects of module order on the cognitive variables. There were 6 modules orders (Fig. 1). One repeated measures ANOVA was used to examine change in knowledge. Three repeated measures MANOVAs were used with the dependent variables of outcome expectancies, self-efficacies and willingness, respectively. The independent variables were the six intervention orders (between subjects) and pre-test to post-test 4 (repeated measures).

Finally, we tested our hypothesis that participation in three modules would result in greater cognitive change than participation in only one module by comparing change scores after completion of one module (pre-test to post-test 1) to change scores after completing all three modules (pre-test to post-test 4). One mixed model ANOVA was used to examine change in knowledge. Three mixed models MANOVAs were used for outcome expectancies, self-efficacies and willingness, respectively. The independent variables were ‘‘change scores’’ (repeated measures groups = pre-test to post-test 1 and pre-test to post-test 3) by number of sex partners (between subjects groups = 0, 1, or ≥2).

HIV risk behaviors

In all analyses using HIV risk behaviors, p < .01 was considered significant (see rational above). HIV risk behaviors (number of sex partners, anal sex index, and condom use index), were assessed for the last 30 days before entering the project (pre-test) and during participation (post-test 3). Changes were examined using three mixed model ANOVAs with independent variables of time (repeated measures; pre-test to post-test 3) by number of sex partners(between subjects; 0, 1, or ≥2).

Results

Participants

‘‘Eligible’’ versus ‘‘ineligible’’ (Fig. 1)

There were 1900 cases in the original data set, but 500 were deleted because they were fraudulent cases. Of the 1400 remaining unique cases, 783 were ineligible. There were no differences between these groups on the screening variables of age, ethnicity, or sexual orientation.

‘‘Dropped-out’’ versus ‘‘completers’’

Of the 617 eligible (rural) men, 425 completed the pre-test questionnaire and were randomized into the first intervention. Retention was calculated from randomization to completion of the final post-test. The overall retention rate for the randomized participants was 88.2% (n = 375) at Post-test 1, 76.7% (n = 326) after post-test 2, and 69.2% (n = 294) after post-test 3. A comparison of the demographics characteristics of the participant’s who dropped out prior to post-test 3 completion (‘‘dropped-out’’) and those who completed the study (‘‘completers’’) are presented in Table 1. There were no significant demographic differences between the two groups. Comparison of the two groups on the cognitive variables indicated that there were no significant differences at intake in HIV knowledge, self-efficacy, outcome expectancies or willingness to engage in HIV risk reduction behaviors.

Completers by sexual risk category

The three groups of men who completed the study included 106 men who reported ‘zero’ sex partners, 102 who reported ‘one’ sex partner and 86 who reported ‘two or more’ in the 30 days prior to beginning the study (Table 2). No significant differences were found between the groups on age, sexual orientation, ethnicity, student status, work status, or religion. A significant difference was found for relationship status (χ² = 26.75, p < .000), income (χ² = 26.75, p < .000), and education (χ² = 6.62, p < .05). A smaller percentage of men who reported one sex partner considered themselves to be ‘‘single’’ and more of these men reported living with a same sex partner. Men with the lowest income tended to have fewer sex partners and the men who went to college tended to report fewer sex partners.

Table 2.

Comparing demographics of ‘‘completers’’ who, at pre-test, report zero, one, or two or more sex partners in the last 30 days

Number of sex partners 30 days prior to project initiation	Zero n = 106 (%)a	One n = 102 (%)a	Two or more n = 86 (%)a	χ2b
Age				1.85
18–24	46.2	47.6	47.7
25–34	32.1	28.2	32.6
35–44	13.2	16.5	15.1
45–80	8.5	7.8	4.7
Sexual orientation				2.52
Gay	85.8	82.4	84.9
Bisexual	13.2	17.6	15.1
Heterosexual	.9	0	0
Ethnicity				2.29
Non-hispanic white	75.5	81.4	74.4
Hispanic	10.4	7.8	8.1
Asian/API, Afr. Amer., Amer. Indian, other	14.2	10.8	17.4
Education				6.62*
High school or less	12.3	21.6	26.7
Some college or more	87.7	78.4	73.3
Student				3.05
Full time	34	28.4	31.4
Part-time	11.3	9.8	16.3
Not a student	54.7	61.8	52.3
Work status				3.53
Full time	54.3	55.9	57.1
Part time	15.2	16.7	22.6
Unemployed/retired	30.5	27.5	20.2
Income				21.49***
< 15,000	41.58	41.2	25.6
15,000–24,999	17	21.6	41.9
25,000–49,999	34	28.4	19.8
≥50,000	7.5	8.8	12.8
Religion				9.13
Catholic	16	23.5	24.4
Protestant	21.7	25.5	15.1
Spritual/non-affiliated	19.8	14.7	23.3
Atheist/agnostic	17	19.6	14
Other	25.5	16.7	23.3
Relationship status				26.75***
Single/never married	85.8	59.6	79.3
Living same sex partner	4.7	28.8	13.8
Opposite sex partner/divorced	9.4	11.5	6.9

^aColumn percent

^bChi Square

^*p < .05

^***p < .001

The groups were examined for pre-test differences in HIV knowledge, self-efficacy, outcome expectancies and willingness to engage in risk reduction behaviors. There were no significant differences between the three groups on any of the cognitive variables.

Intervention completion

The average number of days participants took to complete the entire intervention was 19.39 days (SD = 7.33, median = 19 days, mode = 11, range 10–59). The average time between completing the pre-test and the first post-test questionnaire was 5.22 days (SD = 3.20, median = 4.00 days). The average time between completion of post-test 1 to completion of post-test 2 was 6.89 days (SD = 2.98; median = 6.50 days) and 7.28 (SD = 3.84, median = 6.50 days) days were between completion of post-tests 2 and 3. The modal number of email reminders for all three interventions was zero. The mean number of email reminders was 1.24 (SD = 1.06), 1.19 (SD = 1.03), and .99 (SD = .98) for the Knowledge, Context, and Partner modules respectively.

Intervention effects on cognitive variables

Individual module effects

First, we examined the differential effects of the individual modules (Knowledge, Contexts, or Partner) on HIV knowledge, self-efficacies, outcome expectancies and willingness from pre-test to post-test 1. In terms of knowledge, the results of the mixed model ANOVA indicated that there was a significant module by time interaction (F(2, 288) = 44.38, p < .000, ${\eta }_{\text{p}}^2=.24$). This interaction was accounted for by a significantly greater increase in knowledge when the men participated in the Knowledge Module than the Context Module (p < .000) or the Partner Module (p = .004) and no difference between the Context and Partner modules. The main effects of time and intervention were significant.

Mixed models MANOVAs were used to examine self-efficacies, outcome expectancies and willingness. Results for the two outcome expectancy variables indicate that there was no significant time by module interaction or a main effect for the modules. The overall all effect of time was significant (F(2, 290) = 13.33, p < .000, ${\eta }_{\text{p}}^2=.08$), with the outcome expectancy of feeling safer accounting for the change (F(1, 291) = 26.56, p < .000, ${\eta }_{\text{p}}^2=.08$). Negative outcome expectancies did not decrease significantly across time.

The self-efficacy variables showed a significant time by module interaction effect (F(6, 580) = 3.64 p < .001, ${\eta }_{\text{p}}^2=.04$) as well as a main effect of time. The Interaction effect was accounted for by a significantly greater increase in mechanical self-efficacy after participating in the Knowledge module than the Context module (p < .007) and a tendency toward a greater increase in refusal self-efficacy during the Knowledge module than the Context module (p < .04). Within-subjects comparisons indicated that all three variables showed a main effect of time (Emotion, F(1, 291) = 75.72 p < .000, ${\eta }_{\text{p}}^2=.21$; Refusal, F(1, 291) = 109.86 p < .000, ${\eta }_{\text{p}}^2=.27$; and Mechanical, F (1, 291) = 63.60 p < .000, ${\eta }_{\text{p}}^2=.18$).

The four Willingness questions did not exhibit a significant interaction or main effect of time or module. Within-subjects contrasts indicated that one question, ‘‘Willingness to use a condom with every new partner all the time’’ did increase significantly from pre-test to post-test 1 (p = .007).

Effects of intervention order

Second we examined the potential effects of the order of participation in the interventions on the cognitive variables. There were no significant interaction effects and no significant main effects for intervention order. The main effect for time from pre-test to the final post-test was significant for all cognitive variables.

Cumulative effects of module participation

We hypothesized that participation in three modules would result in greater change than participation in only one module (Table 3). There were no significant interaction effects or between group main effects for the number of sex partners (0, 1, 2 or more). Change scores showed a significant main effect for number of modules for all cognitive variables. Specifically cognitive changes were greater after completing all three modules than after completing only one module.

Table 3.

Cognitive variables: comparison of change scores after completion of one intervention module versus all three intervention modules

Cognitive variable	Number of modules completed		Within-subjects
	One	Three	Contrasts
	MΔ(SD)a	MΔ (SD)b	F	$h_{\text{p}}^2$ c
HIV/AIDS knowledge	1.49 (2.12)a	2.84 (2.50)b	122.93***	.30
Outcome expectancies
Feel safer	.26 (84)a	.55 (.94)b	43.03***	.13
Negative	−.00 (1.19)a	−.27 (1.45)b	11.52**	.04
Self-efficacy
Mechanical	.60 (.98)a	1.24 (1.12)b	143.88***	.33
Emotional	.49 (.98)a	1.03 (1.19)b	101.97***	.26
Refusal	.44 (.93)a	.92 (1.08)b	99.98***	.26
Willingness to reduce risk
Limit sex partners to 1 per month	.04 (1.46)a	.29 (1.49)b	12.62**	.04
Use a condom with every new partner all the time	.17 (1.08)a	.41 (1.09)b	18.14***	.06
Oral sex until monogamous for 6 month and test negative	.02 (1.49)a	.52 (1.58)b	31.18***	.10
Use condoms all the time until monogamous for 6 months and test negative	.15 (1.29)a	.52 (1.36)b	29.76***	.09

^aMean change from pre-test to post-test 1

^bMean change from pre-test to post-test 3

^cPartial eta squared

^**p < .001

^***p < .000

HIV risk behaviors

Self-reported number of sex partners, anal sex index and the condom use index at pre-test and at post-test 3 can be seen in Table 4. The sample sizes differ because men with zero sex partners at pre-test lack data for anal sex and condom use. Likewise, men who reported zero sex partners or no anal sex at post-test 3 would be missing from the anal sex and condom use indices. The groups of men with zero and one sex partner at pre-test significantly increased their reported number of sex partners to a mean of 1.02 (SD = 1.35, p < .000) and 1.67 (SD = 2.28, p < .004), respectively. The men with one sex partner significantly reduced their reported percentage of partners with whom they had anal sex (p < .01) from 80% to 69% and increased the percentage of anal sex partners with whom they used condoms from 41% to 60% (p < .000). Finally, the group of men with two or more partners at intake decreased their number of sex partners slightly (p < .02), but the number of partners with whom they reported anal sex did not change. Condom use with anal sex partners, on the other hand, increased significantly (p < .003) from 48% to 72%.

Table 4.

Changes in HIV risk behaviors from pre-test (30 prior to starting project) to post-intervention 3 (since starting project)

Number sex partners at pre-test	N	Pre-test M (SD)	Post-test 3 M (SD)	Paired T
Zero
# Sex partners	106	0	1.02 (1.35)	−7.76***
Freq. anal sex/# sex partners	55a	n.a.	.45 (.44)	n.a.
Condom use/freq anal sex	31a	n.a.	.90 (.27)	n.a.
One
# Sex partners	102	1.00	1.67 (2.28)	−2.95**
Freq. anal sex/# sex partners	84	.80 (.40)	.69 (.40)	2.60**
Condom use/freq. anal sex	61	.41 (.49)	.60 (.46)	−3.82***
Two or more
# Sex partners	86	5.35 (10.06)	3.93 (11.39)	2.42*
Freq. anal sex/# sex partners	70	.60 (.37)	.56 (.41)	.86
Condom use/freq. anal sex	47	.48 (.41)	.72 (.40)	−3.20**
All participants
# Sex partners	294	1.92 (5.87)	2.09 (6.45)	−.91
Freq. anal sex/# sex partners	154	.70 (.40)	.63 (.41)	2.50*
Condom use/freq. anal sex	108	.44 (.46)	.66 (.44)	−4.95***

^aPost-test 3 sample size, not part of sample for ‘‘all participants’’ comparisons

^*p < .05

^**p < .01

^***p < 001

Discussion

The Internet has been proposed as an excellent medium for delivering HIV risk reduction interventions to marginalized groups MSM living in rural areas represent an ideal population for investigating such interventions because their geographic and social isolation may limit access to HIV risk reduction resources. Additionally, resources for prevention and treatment in rural areas are scarce and the Internet could provide inexpensive and easily accessible programs. Given the broad endorsement of the Internet’s potential (Pequegnat et al. 2006) and it’s use by AIDS service organizations (Benotsch et al. 2006), it is surprising that published studies evaluating theory-driven Internet-based HIV risk reduction interventions are scarce. The current study represents a first research study that documents the feasibility, acceptability, and preliminary efficacy of a comprehensive Internet-based HIV risk reduction intervention for a highly stigmatized and hidden population.

Bull et al. (2004) note that internet research does not appear to be a direct translation of ‘what works’ in face-to-face research in terms of the feasibility of implementing RCTs, multiple intervention modules, regular data collection, and retention procedures (i.e., obtaining identifying information for follow-up). Overall, the methodology used in this study and the initial pilot study indicate that RCTs can be successfully implemented online, at least in the short term. Specifically, we used automated procedures to recruit, screen, obtain consent, conduct the pre-test and randomize participants into the six-session intervention. Completion of each two-session module was followed by a post-test and efforts to retain participants included up to three automated email reminders at 5, 12 and 14 days after initiation of a module.

Evaluation of the feasibility and acceptability of a project may include recruitment and retention rates, duration of intervention participation and the need for prompts or reminders to complete the study, among others. Unidentified multiple submissions and low retention rates can compromise the internal validity of the study and affect the overall feasibility of Internet based intervention research (Pequegnat et al. 2006). The recruitment data for our participants was typical of many Internet studies (Bull et al. 2004) in that there were many men who enrolled in the study (i.e., completed the screener and consent), but turned out to be multiple submissions or who did not complete the pre-test survey (Konstan et al. 2005; Ross et al. 2004). The occurrence of multiple submissions is a threat to any internet study, but procedures that facilitate follow-up (e.g., valid emails, telephone numbers) and log-in procedures (usernames, passwords) also facilitate identification of fraudulent participants (Bowen et al. 2008). In terms of enrollment, similar to both the Konstan et al. and Ross et al. survey studies, approximately one-third of the eligible men who began the pre-test did not complete it. The consistency of recruiting data across studies suggests that a 30% loss at initiation may have to be the standard.

Retention of participants who have been randomized into an intervention is also an important parameter of internal validity, feasibility, and acceptability. Internet research participation appears to be motivated by reimbursement (Bowen 2005), but it may also encourage multiple submissions. The total reimbursement did not appear high, yet multiple submissions were common. We tried to increase retention by implementing an increasing reimbursement schedule, with a higher payment for the last survey as compared to identical payments for each survey. Unfortunately, retention was not differentially affected. The need to increase retention is clear, but research is needed to identify reimbursement rates and schedules that encourage study enrollment and completion and minimizes multiple submissions.

Internet studies are lauded for the fact that participants may interact with the material at any time of day or night and in any location where there is access to the web. These aspects of Internet research have some advantages; but, unlike typical face-to-face interventions, there is very little ‘‘social pressure’’ to remain in the study. As shown by Milgram’s obedience studies (1965) only one-third of the participants completed the study protocol when the experimenter was not physically present. Milgram’s finding suggests that high retention rates in any Internet study will be more difficult than face-to-face, thus placing the onus on the project developers to make the intervention engaging or to include high reimbursement rates. The six sessions and four questionnaires in this intervention represent a fairly high participant load, yet overall study completion rate was 69% with no differential attrition for any one module. While this retention rate is somewhat lower than desired in a typical RCT, it is quite high for a study where participants have no direct human contact and similar to or higher than many primary prevention studies. These factors support the notion that a multisession intervention with a range of foci is acceptable and possibly even engaging. Additional process evaluations are needed to determine specific aspects that engage participants.

The acceptability of the Internet as an intervention platform is also demonstrated in the procedures of this study. Contact information, in the form of a valid email address, was required and ensured our ability to send three automated electronic reminders as well as reimbursement and reduce fraudulent accounts (Bowen et al. 2008). In addition, we requested personal identification including participant names and telephone numbers, as well as names and telephone numbers of contacts. Although participants could skip these questions or provide false names, it didn’t seem to prompt the men to drop out as more than 1400 men completed the information section of the study. Acceptability of multiple sessions with required waiting periods between sessions is seen by similar rates of module completion. The rapid completion, low number of reminders and similar retention across modules suggest that the different formats of the sessions were equally engaging. Specifically, if the sessions were boring or not engaging, one might have expected the men to take long periods of time to complete them and to require regular prompts to start or complete a session. Supporting these methodological findings, the positive process ratings reported by Williams et al. (in press) lend additional support to the acceptability of the mode of presentation.

Initial results support the WRAPP Internet Intervention as an effective platform for providing HIV risk reduction to rural MSM. The IMB skills model poses that information and motivation (i.e., outcome expectancies and willingness) directly affect behavioral skills (self-efficacy) and behavior. Participation in the modules demonstrated both module specific and non-specific effects on the cognitive precursors of behavior change. As expected, knowledge and mechanical self-efficacy were primarily affected by the knowledge intervention. On the other hand, any single module or combination of modules effectively increased knowledge, outcome expectancies and self-efficacy for felling safer, and willingness to use a condom with every new partner all the time. Participation in all three modules was necessary to affect the remaining willingness variables and refusal self-efficacy. Although further research is needed to determine if specific components of the modules affected specific cognitions, the continued change across multiple modules supports the utility and probably the need for interventions with multiple sessions and different foci. It is unclear whether participation in the same module three times would have a similar effect, but it is likely that participant would become bored with viewing and interacting with the same material over and over and would drop out at a higher rate than in this study.

In general, HIV behavior risks showed positive changes over the course of project participation. The lowest risk group of men (i.e., those with zero sex partners in the 30 days prior to intake) had an average of one sex partner at the follow up assessment, but they reported anal sex with less than half their sexual partners, and 90% of the partners with whom they had anal sex they reported using condoms all the time. Men who reported one partner at intake increased their average number of sex partners, but the average was less than two partners. The number of partners with whom these men had anal sex decreased, and condom use with partners with whom they had anal sex increased. Finally, the highest risk group (i.e., men with 2 or more partners at intake) had fewer partners at follow-up and, while the percentage of partner with whom they had anal sex did not change, more of these episodes included condoms.

There are a number of explanations for the behavioral outcomes. First, the outcomes may represent participants’ desire to be ‘good subjects’ (Orne 1969) by reporting what they believed the experimenters wanted. Mediating this explanation is the lack of experimenter presence during the experiment or measurement, thus reducing any pressure to respond in a socially desirable manner. The social desirability explanation is further weakened by the report of more sexual partners at follow-up by the two lowest risk groups and no change in number of anal sex partners in the highest risk group. A second potential explanation is that, for some of the men, the time frame at post-test was too brief to allow the accumulation of additional sex partners. While this may be true, anal sex and condom use were measured as indices based on number of partners and number of partners with whom the participant reported having anal sex and number of partners with whom condoms were used divided by the number of anal sex partners. While total number of partners may be affected by time, the use of indices reduces the effect of time. Replication of the study with longer term follow-up is required to address these rival explanations of the change apparent in the data. Longer term follow-up would also allow an assessment of whether booster sessions might be needed and when they might be best timed.

Overall, the findings support the acceptability, feasibility, and efficacy of this primary prevention program for a hard to reach group. However, future research needs to address a number of limitations. The generalizability of the sample is limited both due to the focus of the study (i.e., rural MSM) and the recruiting strategy (i.e., volunteers who click Internet banner ads) These two factors resulted in a predominantly young, gay and white sample, thus making a generalization to other age groups, or ethnic or sexual minorities tentative at best. The participants also lived in small rural towns where chances for meeting sex partners or for participating in HIV prevention services are limited. The effectiveness or acceptability of a similar Internet-based HIV intervention targeting urban men may be very different.

The interpretation of the outcome data is weakened by the lack of an intervention control group and longer term follow-up. The comparison of the Knowledge Intervention to the wait list control in the WRAPP pilot study (Bowen et al. 2006) and the module specific knowledge change partially mitigate the concern that changes may be due to a Hawthorne effect or questionnaire completion. Generalizability is also limited by the nature of the Internet itself. The quickly changing environment of the Internet (i.e., increased connection speeds and band widths) may result in a specific intervention being technologically obsolete in 6 months.

In summary, this research provides some of the first research support for Internet-delivered HIV risk reduction interventions. There is need for further research to support and expand the findings of this study. Specifically, studies that include other samples, an attention control group, additional process evaluation, more behavioral outcomes and longer follow-up would greatly expand the findings. The preliminary success of the intervention suggests that dissemination or the program may be warranted, even as evaluation continues. The WRAPP intervention is currently available, without evaluation, at www.wrapphome.net, making it accessible to anyone who is interested. While the focus of the intervention is rural MSM, much of the content is appropriate to anyone who is sexually active. One approach to dissemination of the program might be for community-based organizations or state health departments to use the program as a training tool for outreach workers or as a discussion initiator in group interventions.