Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Jul 1.
Published in final edited form as: AIDS Care. 2012 Dec 18;25(7):805–811. doi: 10.1080/09540121.2012.748870

A reanalysis of a behavioral intervention to prevent incident HIV infections: Including indirect effects in modeling outcomes of Project EXPLORE

Lisa A Eaton 1, Seth C Kalichman 2, David A Kenny 2, Ofer Harel 3
PMCID: PMC3761806  NIHMSID: NIHMS421768  PMID: 23245226

Abstract

Background

Project EXPLORE -- a large-scale, behavioral intervention tested among men who have sex with men (MSM) at-risk for HIV infection --was generally deemed as ineffective in reducing HIV incidence. Using novel and more precise data analytic techniques we reanalyzed Project EXPLORE by including both direct and indirect paths of intervention effects.

Methods

Data from 4,296 HIV negative MSM who participated in Project EXPLORE, which included ten sessions of behavioral risk reduction counseling completed from 1999-2005, were included in the analysis. We reanalyzed the data to include parameters that estimate the overtime effects of the intervention on unprotected anal sex and the over-time effects of the intervention on HIV status mediated by unprotected anal sex simultaneously in a single model.

Results

We found the indirect effect of intervention on HIV infection through unprotected anal sex to be statistically significant up through 12 months post-intervention, OR=.83, 95% CI=.72-.95. Furthermore, the intervention significantly reduced unprotected anal sex up through 18 months post-intervention, OR=.79, 95% CI=.63-.99.

Discussion

Our results reveal effects not tested in the original model that offer new insight into the effectiveness of a behavioral intervention for reducing HIV incidence. Project EXPLORE demonstrated that when tested against an evidence-based, effective control condition can result in reductions in rates of HIV acquisition at one year follow-up. Findings highlight the critical role of addressing behavioral risk reduction counseling in HIV prevention.

Introduction

Although many examples of behavioral HIV prevention interventions exist in the current literature, analysis of these interventions has generally relied on statistical approaches that do not take into account critical aspects unique to these trials. Novel techniques of data analysis can adjust for how the impact of an intervention varies during the months and years post intervention and for indirect effects of intervention on disease through behavioral changes. Most statistical approaches have generally relied on mixed effects, repeated measures analysis of variance, generalized estimating equations, pre/post analyses, or analysis of difference scores.(Eaton, Cherry, Cain, & Pope, 2011; Fetterolf, Wennberg, & Devries, 2004; Jamieson, 2004; Kalichman, Cain, & Simbayi, 2010) Unfortunately, these models typically presume that the strength of an intervention does not change over time and also fail to simultaneously estimate indirect effects due to mediating variables, thereby limiting or biasing results and our interpretation of effects. For this paper, we analyzed a series of analyses with indirect effects and time included as parameters using data from Project EXPLORE.(Koblin, Chesney, & Coates, 2004) This large, multi-site, behavioral HIV prevention intervention was previously analyzed using generalized estimating equations with an independent working correlation - a commonly used and widely accepted statistical approach for longitudinal data with within-subject correlation. Results from the main outcomes paper demonstrate a non-significant effect of overall HIV incidence among intervention group participants compared to control group participants, and significant effects in regards to a reduction in unprotected anal intercourse and receptive unprotected anal intercourse with serodiscordant partners among intervention group participants compared to control group participants. Our reanalysis of this trial extends the original analysis to include parameters that more accurately capture the nature of behavioral interventions.(Lennon, McAllister, Kuang, & Herman, 2005; Rotheram-Borus, Rhodes, Desmond, & Weiss, 2010) As a result, we report estimates that more aptly represent the nuances of testing the long-term outcomes of a behavioral trial and draw conclusions that add to the original findings.

Project EXPLORE

HIV infection rates outside of Asia and Africa remain highest among men who have sex with men. For example, 61% of all new HIV infections in the United States occur among MSM.(CDC, 2011) However, despite the fact that a majority of HIV infections in North America, Europe and Australia are among MSM, there are surprisingly few evidence-based, behavioral interventions for this population.(CDC, 2011; effectiveinterventions.org) Indeed, only one behavioral intervention for MSM, Project EXPLORE, has been tested with HIV incidence as the primary endpoint.(Koblin, et al., 2004) In this randomized, controlled trial both conditions demonstrated declines in the highest risk behavior for HIV transmission, with the intervention condition reducing receptive unprotected anal intercourse with serodiscordant partners by 20.5% when compared to the control group. Over the 48 months of follow-up, the HIV incidence for men who received intervention counseling was 1.9 infections per 100 person years, compared to 2.3 infections per 100 person years for men who received control condition counseling, an 18% difference. However, over the course of four years (average follow-up 3.25 years), the difference between the conditions was not significant. It is important to note that the intervention session in Project EXPLORE was tested against an effective behavioral intervention (Project RESPECT) shown to reduce STI among heterosexual men and women attending STI clinics. (Kamb et al., 1998) Therefore, the efficacy of Project EXPLORE was established in relation to an effective risk reduction intervention with demonstrated efficacy above standard of care.

Omnibus model of intervention, behavior and outcomes

The “gold standard” in evaluating the effects of a behavioral intervention are observed differences in biological outcomes, i.e., in this case, do intervention group participants yield fewer cases of HIV infection over the course of study follow-ups compared to control group participants? For Project EXPLORE, HIV incidence was the gold standard and is a function of the intervention and resultant behavioral changes. In mathematical terms, the effect of the intervention on HIV incidence is represented as the total effect, which is the sum of the direct effect of intervention on HIV status, and the indirect effect of intervention on HIV status through behavior. Although the indirect effect is clearly more descriptive in explaining the outcome than the direct effect alone, this path is frequently omitted from omnibus tests of behavioral interventions, as was the case in the original Project EXPLORE outcome paper.(Koblin, et al., 2004) Investigating this indirect effect allows us to interpret the strength of the relationships regarding how changes in HIV incidence are driven by changes in behavior that result from receiving the intervention. Moreover, if the process through which the intervention works is totally due to behavior, then the direct effect should be estimated as zero

In the case of Project EXPLORE, unprotected anal intercourse is the mediating variable between intervention condition and HIV infection. The rational is that unprotected anal intercourse is the behavior most proximal to determining HIV status, and reducing rates of unprotected anal intercourse was a targeted goal in Project EXPLORE. Although the main evaluation of Project EXPLORE was reducing HIV incidence, how behavior mediated this relationship is a key factor in explaining the outcomes of Project EXPLORE.

Here we present a reanalysis of Project EXPLORE. This study was one of the largest HIV prevention behavioral studies to date and arguably the most cited and recognized trial of its kind. Ultimately, this trial- largely touted as the definitive test of a behavioral HIV prevention trial- was generally deemed as unsuccessful.(Abu-Raddad, Boily, Self, & Longini, 2007; Branson et al, 2006; T. Coates, 2008; T. J. Coates, Richter, & Caceres, 2008; Golden & Manhart, 2005; Wohlfeiler & Ellen, 2007; Zenilman, 2005) We began data analysis by replicating findings demonstrated in the original outcomes paper and assessing HIV incidence between groups at each time point and the overall effect of HIV incidence. We then modeled multiple simultaneous logistic regression analyses assessing indirect effects with parameter constraints on the effects of intervention on behavior at all follow-up time points. With these analyses we aim to provide a more complete explanation of the outcomes in the Project EXPLORE trial in respect to preventing HIV incidence.

Methods

Study Design

Project EXPLORE counseling was grounded in principles of motivational interviewing and social-cognitive, behavioral-skills building for health behavior change. Study participants were randomized to one of two arms: (1) a control condition consisting of twice yearly HIV pre-and post-test counseling based on Project RESPECT (Kamb, et al., 1998) -an efficacious risk reduction counseling session that included assessing personal risk, identifying barriers to risk reduction, and discussions around condom use barriers and facilitators, and (2) an intervention treatment condition including the counseling (HIV pre- and post- test counseling) provided in the control arm plus counseling in the form of ten behavioral counseling sessions addressing factors associated with risk taking among MSM. This counseling included discussions of sexual behavior, negative mood states, communication difficulties, use of alcohol or recreational drugs, and events that drive risk taking(Koblin, et al., 2004) (www.hptn.org). Intervention counseling sessions occurred within the first 4-6 months of the study. Intervention participants were also provided quarterly maintenance sessions throughout the course of the study, however there was a substantial drop in retention of these sessions after two years. Project EXPLORE data are in the public domain and were accessed through hptn.org. Participants were 4,296 HIV negative MSM reporting one or more anal sex partners who were recruited from six different US cities. Participants were HIV tested and completed behavioral assessments semi-annually for up to four years. Table 1 presents the baseline demographic characteristics for intervention and control group participants.

Table 1. Demographic characteristics of MSM in intervention and control groups.

Intervention (n=2144) Control (n=2151)
n % n %
Latino or Hispanic 322 15 330 15.3
Race
 White 1630 76 1634 76
 African-American 139 6.5 158 7.3
 Asian 65 3.0 63 2.9
 Native American 20 0.9 18 0.8
 Other 289 13.5 277 12.9
Education
 >high school 27 1.3 43 2.0
 high school 171 8.0 166 7.7
 some college 557 26.0 572 26.6
 college degree 1389 64.8 1368 63.6
Income
 <$6,000 107 5.0 114 5.3
 $6-11,999 173 8.1 168 7.8
 $12-29,999 579 27.0 587 27.3
 $30-59,999 839 39.1 817 38.0
 >$59,000 444 20.7 460 21.4
Open in a new tab

Study Measures

Men enrolled into Project EXPLORE were HIV negative at the baseline assessment. Post intervention HIV tests occurred every six months starting at the 6 month follow-up. We created two HIV status variables: one that allowed us to (a) replicate original study results and (b) one that allowed us to test the current model inclusive of time and indirect effects. For the replication model it is comprised of a single composite score of HIV status collapsing over all time points and for the time/indirect effects model as eight separate HIV status variables representing HIV status at each of the eight follow-up assessments. Consistent with original study data analysis, we used unprotected anal (UA) sex acts for analyses regarding behavioral data. UA was defined as the sum of insertive and receptive unprotected anal sex acts with partners of all HIV statuses from the past six months. Reporting UA was dichotomized into yes/no and assessed at each time point.

Analytic Approach

To replicate data findings in the original article we first conducted a parallel analysis to the original analysis to determine the effect of intervention on acquisition of HIV infection. In this model, we do not include the mediator and assume that the intervention has the same effect on HIV status at each time. We use Mplus(Muthén & Muthén, 2007) in all of our analyses. As in the initial study, we report the odds ratio of the likelihood of becoming HIV infected in the intervention group relative to the likelihood of becoming HIV infected in the control. We report the percentage of reduction in HIV infections between the intervention and control as being one minus the intervention odds ratio.

Next, we reanalyzed the data to include parameters that estimate the effect of the intervention on unprotected anal sex (proximal effect), the effect of intervention on HIV status through unprotected anal sex (indirect effect), and the effect of time all in a single model. In that model, we assumed that the effect of the intervention decayed over time by a factor we denote as δ. The effect of the intervention at time t is assumed to equal ωδt-1 where ω is the intervention effect at time 1. Note that if δ equals one, then there is no decay, which was the implicit assumption made in the original analyses.

Results

Figure 1 represents the paths tested in the main outcomes analysis from the original model (upper panel) and from the reanalysis (lower panel). The original analysis largely relied on reporting the treatment to HIV status path, OR=0.82, 95% CI 0.64-1.05 (18.2% reduction in rate of HIV acquisition), and the treatment to behavior (UA) path, OR=0.86, 95% CI 0.79-0.94 (13.9% reduction in reporting of UA). These findings represent the effect of treatment on HIV status and UA over the course of the entire follow up period or up to 48 months, and demonstrate a nonsignificant effect of treatment on HIV status and a significant reduction in UA among intervention group participants relative to control group participants. Additional tests of the path from treatment to behavior including serodiscordant unprotected anal intercourse and serodiscordant unprotected receptive anal intercourse were also tested in the original model and resulted in demonstrating a significant reduction in these behaviors among the intervention group compared to the control. Because these behaviors are a subset of UA, we only test UA in the reanalysis. As expected, the direct effect of the intervention was not statistically significant (OR = 0.64, 95% CI = 034-1.17) and so was not included in subsequent analyses. The model used presumes that the effect of the intervention on HIV incidence is totally mediated by UA acts. We also note that d or rate of decay is estimated to be 0.68 (95%CI = 0.30 to 1.05), a value much lower than one. This result signifies that the intervention effects on HIV incidence decrease overtime.

Figure 1. Path diagram of models tested in original paper and reanalysis.

Figure 1

Open in a new tab

Note: Original analysis provided overall ORs for the path from treatment to HIV status and the path from treatment to UA. In additional analyses, these paths were tested separately for each time point and paths were adjusted for study site and baseline characteristics associated with retention and randomization.

For the reanalysis models all paths are tested simultaneously.

Figure 1 (lower panel) also represents the paths tested in the reanalysis. For this model, multiple paths are simultaneously analyzed including the paths from treatment to UA, UA to HIV status, and the indirect effect of treatment on HIV status through behavior. Estimates for each path are determined for each follow-up time point. Moreover, for this model HIV status is predicted to be a function of the current time point's follow-up assessment controlling for the previous time point's follow-up assessment. Paths were characterized in this manner to acknowledge that, among individuals who test HIV positive, it is possible that the behaviors that led to seroconversion would be reported on in the previous time-point's follow-up assessment.

Findings for the overall model shown in Figure 2 demonstrate that the intervention resulted in significant reductions in the rate of HIV acquisition with a relative reduction in the number of HIV infections among intervention participants compared to control group participants at the 6 and 12 month follow ups. Additional findings showed a reduction in UA among men in the intervention group compared to men in the control group at 6, 12 and 18 month follow-up time points. The paths from the prior time point's assessment (OR=1.91, 95% CI=1.39-2.73) to HIV status and the current time point's assessment (OR=2.36, 95% CI=1.64-3.41) to HIV status were fixed paths and found to predict HIV status; i.e., reporting UA was significantly associated with testing HIV positive.

Figure 2. Estimates of proximal and indirect effects for a Project EXPLORE reanalysis model reported using odds ratios.

Figure 2

Open in a new tab

Discussion

Findings from the current study, in part, reiterate findings reported in the original study. However, we add to prior conclusions made by expanding on critical relationships. Outcomes observed in the current study add to our initial understanding of the outcomes in Project EXPLORE. Findings from the reanalysis reveal significant indirect effects not identified in the original outcome paper; i.e., Project EXPLORE significantly reduced UA acts -the highest sexual risk behavior for HIV-, and significantly reduced the number of HIV infections in the intervention group relative to the control group for 12 months and 18 months, respectively. These findings highlight the importance of using data analytic models inclusive of both direct effects, i.e., treatment on main outcome, and indirect effects, i.e., treatment on main outcome through behavior.

These results are particularly notable given the context in which Project EXPLORE was tested. Project EXPLORE was tested against an effective behavioral intervention (Project RESPECT) shown to reduce STI among heterosexual men and women attending STI clinics.(Kamb, et al., 1998) Interestingly, comparing the outcomes of Project EXPLORE to Project RESPECT shows that Project EXPLORE outperformed Project RESPECT in regards to reduction in rate of HIV acquisition, however, the conclusions drawn about whether or not the these two interventions worked are quite different. Meaning, at 12 month follow-up, Project RESPECT resulted in a 20% reduction in STI/HIV infection among intervention participants compared to standard-of care-control participants - concluding that STI based clinic counseling can be effective at reducing disease. Similarly, at 12 month follow-up, Project EXPLORE resulted in a 24% reduction in HIV infections among intervention group participants compared to control group participants - yet, the overall conclusions from Project EXPLORE suggest there was a non-significant effect on HIV acquisition and ultimately that the intervention failed. Furthermore, the 24% reduction that is observed at the 12-month follow-up in Project EXPLORE is compared to the Project RESPECT-based control group, which clearly out-performed the standard-of-care. Thus, the fact that we observe a 24% decrease in infections after 12 months when compared to Project RESPECT suggests that Project EXPLORE is a highly effective intervention for HIV prevention.(Safren, Wingood, & Altice, 2007)

Findings from the current study should be viewed in light of their limitations. Results are limited to men who were recruited for and screened into the current study, i.e., information garnered from this trial may or may not be generalizable to the larger population. Parts of the data collection relied on self-report of potentially stigmatizing behaviors, which could potentially bias responses. Data were also collected in the context of a randomized controlled trial which also limits the extent the data can be generalized to the population of MSM at large, i.e., men in both intervention arms received considerable counseling. Likewise, the study did not address prevention with men who are HIV positive-which leaves out a critical population for reducing the spread of HIV. Although the analyses presented here use a state-of-the-art longitudinal, multi-equation, logistic model, there are missing data and our model of missingness may be mis-specified. Moreover, we have not allowed for measurement error or invalidity in the UA measure.

The current state of HIV prevention is largely prioritizing biomedical forms of HIV/STI prevention.(Cohen et al., 2011; Grant et al., 2010) These strategies for prevention must incorporate aspects that attend to behavioral needs such as behavior change in response to intervention(Eaton & Kalichman, 2007) or medical regimen adherence.(Grant, et al., 2010) Although results from Project EXPLORE do not demonstrate behavioral changes lasting an indefinite period of time, we must not overlook beneficial outcomes of intervening with individuals when effects can be seen over many months. Furthermore, none of the proposed behavioral or biomedical interventions, with the exception of an effective anti-HIV vaccine which doesn't exist, would be considered a one-shot intervention. All proposed and currently available HIV prevention interventions require behavioral maintenance and/or would incur continual medical costs to maintain. Even among biomedical interventions that are considered partially effective, such as male circumcision, continued behavioral counseling needs to be practiced to ensure beneficial results.(Auvert et al., 2005; Bailey et al., 2007; Gray et al., 2007) Therefore, plans for HIV/STI prevention should focus on a combination of behavioral and biomedical forms of prevention that are complimentary to each other. Based on this notion and overall findings from the current paper, Project EXPLORE based-counseling adds to the current arsenal of options for preventing HIV and should be included in broader packages focusing on HIV prevention and treatment.

Acknowledgments

This project was supported by National Institute of Alcohol Abuse and Alcoholism grant R01 AA018074 and the National Institute of Mental Health grants R01MH094230 and K01MH087219.

References

  1. Abu-Raddad LJ, Boily MC, Self S, Longini IM., Jr Analytic insights into the population level impact of imperfect prophylactic HIV vaccines. J Acquir Immune Defic Syndr. 2007;45(4):454–467. doi: 10.1097/QAI.0b013e3180959a94. 10.1097/QAI.0b013e3180959a94. [DOI] [PubMed] [Google Scholar]
  2. Auvert B, Taljaard D, Lagarde E, Sobngwi-Tambekou J, Sitta R, Puren A. Randomized, controlled intervention trial of male circumcision for reduction of HIV infection risk: the ANRS 1265 Trial. PLoS Med. 2005;2(11):e298. doi: 10.1371/journal.pmed.0020298. 05-PLME-RA-0310R1 [pii]10.1371/journal.pmed.0020298. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Bailey RC, Moses S, Parker CB, Agot K, Maclean I, Krieger JN, Ndinya-Achola JO, et al. Male circumcision for HIV prevention in young men in Kisumu, Kenya: a randomised controlled trial. Lancet. 2007;369(9562):643–656. doi: 10.1016/S0140-6736(07)60312-2. Randomized Controlled TrialResearch Support, N.I.H., Extramural Research Support, Non-U.S. Gov't. 10.1016/S0140-6736(07)60312-2. [DOI] [PubMed] [Google Scholar]
  4. Branson BM, Hunter Handsfield H, Lampe MA, Janssen RS, Taylor AW, Lyss SB, Clark JE. Revised Recommendations for HIV Testing of Adults, Adolescents, and Pregnant Women in Health-Care Settings. MMWR. 2006;55(RR14):1–17. [PubMed] [Google Scholar]
  5. CDC. HIV and AIDS among Gay and Bisexual Men. 2011 http://www.cdc.gov/nchhstp/newsroom/docs/fastfacts-msm-final508comp.pdf.
  6. Coates T. What is to be done? AIDS. 2008;22(9):1079–1080. doi: 10.1097/QAD.0b013e3282f8afb0. [DOI] [PubMed] [Google Scholar]
  7. Coates TJ, Richter L, Caceres C. Behavioural strategies to reduce HIV transmission: how to make them work better. Lancet. 2008;372(9639):669–684. doi: 10.1016/S0140-6736(08)60886-7. S0140-6736(08)60886-7 [pii]10.1016/S0140-6736(08)60886-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Cohen MS, Chen YQ, McCauley M, Gamble T, Hosseinipour MC, Kumarasamy N, Fleming TR, et al. Prevention of HIV-1 infection with early antiretroviral therapy. N Engl J Med. 2011;365(6):493–505. doi: 10.1056/NEJMoa1105243. Multicenter Study Randomized Controlled TrialResearch Support, N.I.H., Extramural Research Support, Non-U.S. Gov't. 10.1056/NEJMoa1105243. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Eaton LA, Cherry C, Cain D, Pope H. A novel approach to prevention for at-risk HIV-negative men who have sex with men: creating a teachable moment to promote informed sexual decision-making. Am J Public Health. 2011;101(3):539–545. doi: 10.2105/AJPH.2010.191791. Research Support, N.I.H., Extramural. 10.2105/AJPH.2010.191791. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Eaton LA, Kalichman S. Risk compensation in HIV prevention: implications for vaccines, microbicides, and other biomedical HIV prevention technologies. Curr HIV/AIDS Rep. 2007;4(4):165–172. doi: 10.1007/s11904-007-0024-7. Research Support, N.I.H., ExtramuralReview. effectiveinterventions.org. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Fetterolf D, Wennberg D, Devries A. Estimating the return on investment in disease management programs using a pre-post analysis. Dis Manag. 2004;7(1):5–23. doi: 10.1089/109350704322918961. Comparative Study. 10.1089/109350704322918961. [DOI] [PubMed] [Google Scholar]
  12. Golden MR, Manhart LE. Innovative approaches to the prevention and control of bacterial sexually transmitted infections. Infect Dis Clin North Am. 2005;19(2):513–540. doi: 10.1016/j.idc.2005.03.004. S0891-5520(05)00023-1 [pii]10.1016/j.idc.2005.03.004. [DOI] [PubMed] [Google Scholar]
  13. Grant RM, Lama JR, Anderson PL, McMahan V, Liu AY, Vargas L, Glidden DV, et al. Preexposure chemoprophylaxis for HIV prevention in men who have sex with men. N Engl J Med. 2010;363(27):2587–2599. doi: 10.1056/NEJMoa1011205. Multicenter StudyRandomized Controlled TrialResearch Support, N.I.H., Extramural Research Support, Non-U.S. Gov't. 10.1056/NEJMoa1011205. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Gray RH, Kigozi G, Serwadda D, Makumbi F, Watya S, Nalugoda F, Wawer MJ, et al. Male circumcision for HIV prevention in men in Rakai, Uganda: a randomized trial. Lancet. 2007;369(9562):657–666. doi: 10.1016/S0140-6736(07)60313-4. S0140-6736(07)60313-4 [pii]10.1016/S0140-6736(07)60313-4. [DOI] [PubMed] [Google Scholar]
  15. Jamieson J. Analysis of covariance (ANCOVA) with difference scores. Int J Psychophysiol. 2004;52(3):277–283. doi: 10.1016/j.ijpsycho.2003.12.009. 10.1016/j.ijpsycho.2003.12.009. [DOI] [PubMed] [Google Scholar]
  16. Kalichman SC, Cain D, Simbayi LC. Behavioral changes associated with testing HIV-positive among sexually transmitted infection clinic patients in Cape Town, South Africa. Am J Public Health. 2010;100(4):714–719. doi: 10.2105/AJPH.2009.162602. Research Support, N.I.H., Extramural. 10.2105/AJPH.2009.162602. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Kamb ML, Fishbein M, Douglas JM, Jr, Rhodes F, Rogers J, Bolan G, Peterman TA, et al. Efficacy of risk-reduction counseling to prevent human immunodeficiency virus and sexually transmitted diseases: a randomized controlled trial. Project RESPECT Study Group. JAMA. 1998;280(13):1161–1167. doi: 10.1001/jama.280.13.1161. joc80102 [pii] [DOI] [PubMed] [Google Scholar]
  18. Koblin B, Chesney M, Coates T. Effects of a behavioural intervention to reduce acquisition of HIV infection among men who have sex with men: the EXPLORE randomised controlled study. Lancet. 2004;364(9428):41–50. doi: 10.1016/S0140-6736(04)16588-4. 10.1016/S0140-6736(04)16588-4S0140673604165884 [pii] [DOI] [PubMed] [Google Scholar]
  19. Lennon MC, McAllister W, Kuang L, Herman DB. Capturing intervention effects over time: reanalysis of a critical time intervention for homeless mentally ill men. Am J Public Health. 2005;95(10):1760–1766. doi: 10.2105/AJPH.2005.064402. AJPH.2005.064402 [pii]10.2105/AJPH.2005.064402. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Muthén LK, Muthén BO. Mplus User's Guide. Sixth. Los Angeles, CA: Muthén & Muthén; 2007. [Google Scholar]
  21. Rotheram-Borus MJ, Rhodes F, Desmond K, Weiss RE. Reducing HIV risks among active injection drug and crack users: the safety counts program. AIDS Behav. 2010;14(3):658–668. doi: 10.1007/s10461-009-9606-2. 10.1007/s10461-009-9606-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Safren SA, Wingood G, Altice FL. Strategies for primary HIV prevention that target behavioral change. Clin Infect Dis. 2007;45(4):S300–307. doi: 10.1086/522554. 10.1086/522554. [DOI] [PubMed] [Google Scholar]
  23. Wohlfeiler D, Ellen JM. The limits of behavioral interventions for HIV prevention. In: Cohen L, Chavez V, editors. Prevention is Primary: Strategies for Community Well-Being. 2007. pp. 351–370. [Google Scholar]
  24. Zenilman JM. Behavioral interventions--rationale, measurement, and effectiveness. Infect Dis Clin North Am. 2005;19(2):541–562. doi: 10.1016/j.idc.2005.04.002. S0891-5520(05)00033-4 [pii]10.1016/j.idc.2005.04.002. [DOI] [PubMed] [Google Scholar]

RESOURCES