Pilot Randomized Controlled Trial to Reduce HIV-Related Stigma among Clinicians in Malaysia via Project ECHO^®

Abstract

HIV-related stigma from clinicians contributes to global disparities in HIV testing and prevention. Scalable interventions, particularly those reaching clinicians beyond individual healthcare facilities, are needed to reach clinicians worldwide. These interventions are especially important in places like Malaysia where substantial HIV-related structural stigma exists. We tested an intervention that incorporated evidence-based stigma reduction tools into a popular teletraining program for clinicians, Project ECHO for HIV Prevention and Stigma Reduction (PEHP-SR). We compared PEHP-SR to the same program but without stigma-reduction tools (PEHP) and with an existing HIV education course, HIV Connect (HC). We randomized n=78 clinicians in Malaysia to the three intervention conditions for nine months. Participants completed surveys measuring implementation-, stigma-, and clinical-related outcomes before, mid-way, and after the interventions. A subgroup of participants engaged in qualitative focus groups after the interventions. Quantitative survey results suggested that participants found PEHP to be the most acceptable and appropriate condition, and that stigma- and clinical-related outcomes improved in all three conditions. Qualitative focus group results provided insights into components of PEHP-SR that may have been ineffective and identified potential benefits of PEHP-SR not measured by quantitative surveys. Future research should continue to explore scalable interventions to reduce HIV-related stigma worldwide.

Introduction

HIV-related stigma from clinicians contributes to disparities in HIV testing and prevention worldwide. Members of key populations who experience and/or anticipate stigma from clinicians are less likely to initiate and/or engage in HIV care, including HIV testing and prevention services (Dovidio et al., 2008; Earnshaw et al., 2018, 2013; Kushwaha et al., 2017). Interventions that reduce HIV-related stigma (i.e., stigma towards members of key populations and people living with HIV [PLWH]) among clinicians are therefore needed. In response to this need, HIV-related stigma interventions for clinicians have been tested globally as well as in low- and middle-income countries specifically, where the burden of HIV is often higher (Smith et al., 2020). Reviews of these interventions have been encouraging, suggesting that many interventions reduce stigma among clinicians (Ferguson et al., 2023; Kemp et al., 2019; Nyblade et al., 2019; Smith et al., 2020; Stangl et al., 2023). Yet, these reviews additionally suggest that continued innovation in HIV-related stigma interventions for clinicians is needed, including by bringing interventions to scale to reach more clinicians (Ferguson et al., 2023), leveraging technology for interactive learning (Nyblade et al., 2019), and evaluating the impacts of interventions on implementation-related outcomes (Kemp et al., 2019).

To date, HIV-related stigma reduction efforts often target clinicians working in one health facility at a time (Nyblade et al., 2019). Although these efforts are important, additional strategies may be needed to bring stigma interventions to scale and reach clinicians who may work outside of health facilities that have recognized the need for and invested in stigma reduction interventions. Project ECHO^® (Extension for Community Healthcare Outcomes) is a popular teletraining program that leverages technology (including videoconferencing) and interactive learning to train and support non-specialty clinicians in community settings to provide specialty care. The current study evaluates a pilot randomized controlled trial of an intervention to reduce HIV-related stigma among clinicians in community settings in Malaysia via an adapted version of Project ECHO^® for HIV Prevention.

HIV-Related Stigma and Stigma-Reduction Tools

Stigma is a social process supported by social power whereby certain attributes, characteristics, and/or identities become devalued and discredited (Goffman, 1963; Link & Phelan, 2001; Major et al., 2018). The Health Stigma and Discrimination Framework (HSDF) theorizes that stigma is facilitated by cultural norms, and aspects of the structural and legal environment (Stangl et al., 2019). In Malaysia, civil and religious laws that criminalize and carry risks of harsh punishments for same-sex sexual behavior, illicit drug use, and sex work, and limit the gender expression of and opportunities for gender-affirming medical care for transgender individuals facilitate stigma toward key populations (Anti Death Penalty Asia Network, 2021; Barmania & Aljunid, 2017; Barmania & Reiss, 2018; Mendos et al., 2020; Suppiah & Kaur, 2018; Syariah Criminal Offenses Enactment, 1997). The HSDF proposes that stigma is further driven by lack of knowledge, anxiety, fear, and perceived community stigma (Stangl et al., 2019). Individual clinicians may experience and enact stigma as prejudice (i.e., negative emotions or feelings), stereotypes (i.e., group-based beliefs applied to individuals), and discrimination (i.e., unfair or unjust behaviors) (Earnshaw & Chaudoir, 2009). Previous work has documented substantial prejudice, stereotypes, and intentions to discriminate against key populations, including men who have sex with men (MSM), transgender women (TGW), and people who inject drugs (PWID), among both practicing medical doctors and medical students in Malaysia (Earnshaw et al., 2014; Earnshaw et al., 2016; Jin et al., 2014).

Decades of research has identified evidence-based intervention strategies, or tools, that may reduce stigma among clinicians (Bohnet, 2016; Cook et al., 2014; Li et al., 2013; Nyblade et al., 2019; Rao et al., 2019; Stangl et al., 2013). These tools have been summarized by Cook and colleagues (2014), in their review of multilevel stigma intervention tools, and the National Academies of Sciences, Engineering, and Medicine (2016), in their recommendations on how to address stigma at the individual level. Education provides learners with fact-based information to refute stereotypes, reduce fear, and increase cultural awareness (Cook et al., 2014; Corrigan et al., 2012; Nyblade et al., 2019; Stangl et al., 2013). Shifting social norms involves changing perceptions of stigma endorsed by one’s peer group (Cook et al., 2014; Li et al., 2013). Finally, contact involves facilitating interaction with members of stigmatized groups to reduce prejudice and refute stereotypes (Cook et al., 2014; Corrigan et al., 2012; Flanagan et al., 2016; Nyblade et al., 2019; Pettigrew & Tropp, 2006, 2008; Stangl et al., 2013). Contact can happen either in-person or virtually (e.g., via television shows, radio programs, books) (Paluck & Green, 2009). Photovoice, a research strategy that enables individuals to represent their communities through photographs and stories about their lives (Wang & Burris, 1997), has shown promise as a tool for facilitating virtual contact between clinicians and PWID as well as reducing clinician prejudice toward PWID (Flanagan et al., 2016). Evidence further suggests that interventions implementing these tools should be: (a) longitudinal, because single session interventions are insufficient to generate lasting change in stereotypes, prejudice, and discrimination; and (b) multicomponent, because there is no known single “silver bullet” intervention strategy that can eliminate stigma but rather a toolbox of strategies that can be used collectively to reduce stigma (National Academies of Sciences, Engineering, and Medicine, 2016; Nyblade et al., 2019; Stangl et al., 2013).

Project ECHO^® for HIV Prevention

Project ECHO^® is a promising method to deliver stigma reduction tools to clinicians in diverse community settings. Project ECHO^® uses technology (i.e., videoconferencing) to enable non-specialty clinicians in community settings to learn from and collaborate with specialists at academic “hubs” (Arora et al., 2007; Arora et al., 2011; Wood et al., 2018). Project ECHO^® programs are longitudinal, with learning communities meeting on a regular basis (e.g., biweekly) for several months. Education is facilitated in several ways, including through brief didactics, case presentations, and group discussion. Social norms are established through group discussion between the non-specialist and specialist clinicians. Additionally, there is opportunity to facilitate contact through guest speakers and the sharing of other content (e.g., photovoice presentations). The National Academy of Medicine and Centers for Disease Control and Prevention endorse Project ECHO^® as a popular and highly innovative strategy likely to produce significant improvements in the quality and efficiency of patient-centered integrated care (Centers for Disease Control and Prevention, 2017; Institute of Medicine, 2012; Struminger et al., 2017).

Current Study

The current study evaluated a pilot randomized controlled trial to reduce stigma among clinicians in Malaysia through an adapted version of Project ECHO^® for HIV Prevention. The intervention targeted general practitioners and family medicine clinicians, who play a key role in HIV testing and prevention services. Primary aims were to evaluate implementation-related outcomes to inform a future efficacy study. Secondary aims were to explore the intervention’s impact on stigma and clinically-relevant outcomes.

Materials and Methods

Intervention Design

An individually randomized group trial was conducted, with individuals randomized to one of three conditions. Two comparator conditions were included. A standard, or non-adapted, Project ECHO^® comparator condition was included given that standard Project ECHO^® interventions may reduce stigma via education and shifting social norms. Moreover, HIV prevention education for Malaysian clinicians was offered through a webinar series called HIV Connect at the time of study. Given that a webinar series may also reduce stigma via education, an HIV Connect comparator condition was included. Participants were randomized to conditions at a 1:1:1 ratio, and all interventions ran for 9 months (July 2022-March 2023). The study therefore aimed to test whether a Project ECHO^® intervention explicitly adapted to reduce stigma led to greater stigma reduction than a standard Project ECHO^® intervention and/or a webinar series already available to Malaysian clinicians. Additional details regarding all three study conditions are provided below and further summarized and compared in Supplemental Appendix 1, and details regarding the specific stigma-reduction tools included in each condition are described in Table 1.

Table 1.

Summary of stigma-reduction tools included in study conditions

Tool	Definition	Condition(s)	Delivery Strategies
Education	Provide individuals with fact-based information to refute stereotypes, reduce fear, and increase cultural awareness	HIV Connect	Didactics presented on clinical topics through asynchronous videos
		Project ECHO: HIV Prevention	Didactics presented on clinical and social determinants of health topics through brief live lectures Case presentations
		Project ECHO: Stigma Reduction	Didactics presented on clinical, social determinants of health, and stigma-related topics through brief live lectures Case presentations
Social norms	Change perceptions of stigma endorsed by one’s peer group	Project ECHO: HIV Prevention	Discussions between participants and hub specialists during didactics and case presentations
		Project ECHO: Stigma Reduction	Discussions between participants, hub specialists, and guest speakers during didactics and case presentations as well as following photovoice presentations
Contact	Facilitate interaction between participants with members of stigmatized groups to reduce prejudice and refute stereotypes	Project ECHO: Stigma Reduction	Hub specialist from a key population group Guest speakers from key population and PLWH groups Photovoice presentations from key population and PLWH groups

HIV Connect (HC)

The HIV Connect (HC) condition consisted of an online course developed by the Malaysian Society of HIV Medicine to educate primary care physicians about HIV. HC included a series of nine asynchronous videos focused on clinical topics and featuring HIV infectious disease experts (length: M=18:34, range=13:44–27:28 minutes). Participants were sent a link to approximately one video per month and could watch the video at a time convenient to them. As shown in Table 1, education was the only stigma-reduction tool included in HC.

Project ECHO^® for HIV Prevention (PEHP)

The Project ECHO^® for HIV Prevention (PEHP) condition was designed to mirror a standard Project ECHO^® program without the addition of stigma reduction tools. Seventeen online sessions were held approximately biweekly and synchronously for one hour over lunchtime and featured a brief didactic training followed by a patient-case presentation and discussion. Didactics addressed clinical and social determinants of health topics. The hub specialists included five members, including three clinicians with expertise in infectious disease, one in family medicine, and one in mental/behavioral health. As shown in Table 1, stigma-reduction tools included in PEHP were education and social norms. Although we did not instruct the hub specialists to engage in discussion to change social norms related to stigma, we expected that discussions led by hub specialists about their experiences with and perspectives on working with members of key populations and PLWH may shift social norms.

Project ECHO^® for HIV Prevention and Stigma Reduction (PEHP-SR)

The Project ECHO^® for HIV Prevention and Stigma Reduction (PEHP-SR) condition was modeled after the PEHP condition with the addition of evidence-based stigma reduction tools. Before launching the pilot randomized controlled trial, we took several steps to incorporate evidence-based stigma-reduction tools into PEHP. We used the Promoting Action on Research Implementation in Health Services (PARIHS) framework to guide this work (Kitson et al., 2008; Rycroft-Malone, 2004). PARIHS highlights evidence, context, and facilitation as key elements that promote successful implementation of evidence-based interventions. We drew evidence about stigma among Malaysian clinicians and evidence-based stigma-reduction tools from previous work (Cook et al., 2014; Earnshaw et al., 2014; Earnshaw et al., 2016; Jin et al., 2014; National Academies of Sciences, Engineering, and Medicine, 2016). We supplemented this existing evidence by exploring members of Malaysian key populations’ and PLWH’s experience with stigma from clinicians using photovoice (Earnshaw et al., 2023). We created and shared videos incorporating the photovoice data with clinicians in the PEHP-SR condition to facilitate contact with members of key populations and PLWH. We addressed context by conducting an asynchronous focus group with clinicians to explore barriers to and facilitators of implementing Project ECHO^® among general practitioners and family medicine clinicians in Malaysia (Walters et al., 2022). Based on results of these focus groups, we revised intervention content (e.g., topics addressed by didactics) and added several strategies to promote participation (e.g., scheduled sessions over lunchtime, added incentives for attending sessions). Finally, we gained insight into facilitating PEHP-SR by conducting an abbreviated pre-test of the intervention with n=8 participants. We used feedback from pre-test participants to finalize intervention procedures, including the ordering and timing of content within sessions.

Similar to PEHP, 17 one-hour sessions were held approximately biweekly and synchronously for one hour over lunchtime. Sessions featured combinations of brief didactic trainings, patient-case presentations and discussions, photovoice presentations from key populations and PLWH, and guest speakers. In addition to the clinical and social determinants of health topics addressed during didactics in the PEHP condition, didactics in the PEHP-SR condition addressed stigma-related topics. The hub specialists included four clinicians, including one with expertise in infectious disease, one in family medicine, and two in mental/behavioral health. The hub specialists additionally included one epidemiologist with expertise in social determinants of health and stigma, and one community representative. Guest speakers representing community-based and advocacy organizations as well as key population and PLWH groups regularly joined the sessions. Guest speakers from key population and PLWH groups were invited to share about their experiences in healthcare settings. As shown in Table 1, stigma-reduction tools included in PEHP were education, social norms, and contact. Education included didactic topics specifically focused on social determinants of health and stigma. Social norms were facilitated through discussions between participants, hub specialists, and guest speakers. Hub specialists and guest speakers were encouraged to address stigma. Contact was facilitated through the inclusion of a hub specialist who was a community representative and member of a key population group, guest speakers from key population and PLWH groups, and photovoice presentations from key population and PLWH groups.

Quantitative Methods

Quantitative and qualitative data were collected to evaluate the intervention. A QUAN→qual approach was used wherein quantitative and qualitative data were collected sequentially (Palinkas et al., 2011). Greater weight was given to the quantitative data in hypothesis testing, and qualitative results were used to expand on results observed in the quantitative analyses. Quantitative methods are described first, followed by qualitative methods.

Participants

A target sample size of n=78 participants, or n=26 per condition, was set based on power analyses conducted in R using the power package for single-level generalized linear models (Champely, 2018). Results suggested that there would be 80% power to detect a statistically significant (p<0.05) medium effect size (Cohen’s d=0.15) with 64 participants. An additional 14 participants (~20% of the sample size) were added to account for potential attrition. Individuals were eligible to participate if they were a family medicine specialist or general practitioner in Malaysia. Online advertisements were posted to Malaysian medical groups between May and July 2022. The goal of the study was described as comparing internet-based HIV training programs. Interested individuals provided written consent to be randomized to a training program and their contact information via Qualtrics. Identities of clinicians were verified by referencing national registries (e.g., Medical Register Information and Technical System; MeRITS).

Participants were randomly assigned to study conditions using the group_by and slice functions in R’s dplyr package by the study statistician (Wickham, François, Henry, & Müller, 2018). After participants were assigned to conditions, the study coordinator contacted them with details regarding their meeting schedule (e.g., meeting dates and times if applicable). Participants were aware of whether they were assigned to a Project ECHO^® or HC condition. Participants assigned to Project ECHO^® conditions, however, were unaware of whether they were assigned to PEHP or PEHP-SR. Participants received Continuing Professional Development credits and a lunch voucher for each session that they attended as well as a certificate of completion at the end of the program.

Quantitative Measures

Quantitative surveys were conducted at baseline (i.e., prior to initiating the interventions), half-way through the interventions (i.e., month 4.5), and at the end of the interventions (i.e., month 9). Participants reported their socio-demographic (including age, gender, ethnicity, religion) and clinical (e.g., affiliation, number of years spent practicing medicine, clinical rank, HIV testing experience, PrEP prescribing experience) characteristics at baseline. Surveys measured constructs related to the primary outcomes (i.e., implementation-related) at month 4.5 and month 9, and outcomes related to the secondary outcomes (i.e., stigma- and clinical-related) at baseline, month 4.5, and month 9.

Primary Outcomes.

Our primary aim was to evaluate implementation-related outcomes, including participants’ perceptions of acceptability, appropriateness, and feasibility of the interventions, as well as the feasibility of our intervention and evaluation procedures.

Implementation: Acceptability, Appropriateness, and Feasibility.

Implementation-related outcomes were measured with the Acceptability of Intervention Measure, Intervention Appropriateness Measure, and Feasibility of Intervention Measure (Weiner et al., 2017). Each measure included four items rated on five-point Likert-type scales ranging from completely disagree to completely agree. Example items include: “The training program meets my approval” (acceptability), “The training program seems applicable” (appropriateness), and “The training program seems doable” (feasibility). The measures were reliable in the current sample (acceptability: Cronbach’s α=0.91–0.94, appropriateness: Cronbach’s α=0.95–0.95, feasibility: Cronbach’s α=0.92–0.94).

Teaching Effectiveness: Presence, Expertise, and Engagement.

As an additional indicator of acceptability, participants rated their impressions of teaching effectiveness using the Online Teaching Effectiveness Scale (Reyes-Fournier et al., 2020), which included subscales measuring presence, expertise, and engagement. The presence subscale included six items, and the expertise and engagement subscales each included three items, all rated on five-point Likert-type scales ranging from strongly disagree to strongly agree. Participants are asked the extent to which their instructors demonstrate key characteristics, such as “are enthusiastic about teaching” (presence), “have subject matter knowledge” (expertise), and “invite questions from the audience” (engagement). The subscales demonstrated reliability in the current sample (presence: Cronbach’s α=0.94–0.95; expertise: Cronbach’s α=0.90–0.93; engagement: Cronbach’s α=0.95–0.96).

CONSORT Data and Percent Session Completion.

Given that this was a pilot study, we examined the CONSORT flow diagram to assess the feasibility of our procedures to recruit, randomize, and retain participants in our evaluation procedures (e.g., completion of surveys).

We further examined our attendance data to assess the success of our procedures to retain participants in the interventions. Attendance at Project ECHO^® sessions was recorded by the study coordinator and confirmed through an attendance form submitted by participants for Continuing Professional Development credits. Completion of HC sessions was recorded by the website hosting the videos, which recorded video viewership. Percentage of attendance was calculated for each participant (percentage = number of sessions attended or viewed / total sessions available).

Secondary Outcomes.

Our secondary aim was to evaluate the intervention’s impact on stigma and clinically relevant outcomes. As noted below, several measures of secondary outcomes were adapted to either better fit the study context or for length. These adapted measures are included in Supplemental Appendix 2.

Stigma: Prejudice, Stereotypes, and Discrimination.

As guided by the HSDF (Stangl et al., 2019), stigma-related outcomes measured in the current study included prejudice, stereotypes, and discrimination intent towards key populations (i.e., MSM, TGW, PWID, FSW) and PLWH. Measures were pre-tested in previous studies with clinicians in Malaysia (Earnshaw et al., 2014; Earnshaw et al., 2016; Jin et al., 2014). Prejudice was measured by asking participants to rate how they felt towards members of key population groups and PLWH using feelings thermometers ranging from 0 (very negative) - 100 (very positive) (Alwin, 1997). For example, participants were asked “How do you feel about transgender women patients?” Items were reverse-scored such that higher scores indicated greater prejudice. For the main outcomes analysis, a composite score was created by averaging scores across all four key population groups and PLWH. These measures demonstrated reliability in the current sample (Cronbach’s α=0.91–0.92).

Stereotype and discrimination scales were shortened so that they could be feasibly repeated for multiple key population and PLWH groups, and revisions to the stereotype items were made based on our preliminary work (i.e., online focus groups with clinicians; see Supplemental Appendix 2). Stereotypes about each key population group and PLWH were measured with four items modeled after the Brief Opioid Stigma Scale (Yang et al., 2019) and discrimination intent towards each key population group and PLWH was measured using three items from the discrimination intent subscale of the multidimensional HIV stigma scale (Stein & Li, 2007). All items were measured on five-point Likert-type scales ranging from strongly disagree to strongly agree. Example items include: “Female sex workers often do not make good decisions” (stereotype), and “I am willing to work with people who inject drugs” (discrimination intent, reverse-scored). For the main outcomes analysis, composite scores were created by averaging scores across all key population groups and PLWH. These measures demonstrated reliability in the current sample (stereotypes: Cronbach’s α=0.85–0.86; discrimination intent: Cronbach’s α=0.95–0.98).

Clinical: Comfort and PrEP.

Clinical-related outcomes included indicators of comfort in key HIV-related clinical interactions as well as PrEP-related knowledge and willingness to prescribe PrEP (Ariffin et al., 2015). Participants’ comfort with taking sexual and drug use histories was measured with 10 items about the extent to which participants felt comfortable asking patients about behaviors related to HIV transmission (e.g., condom use, injection drug use, sex work) on a five-point Likert-type scale ranging from completely uncomfortable to completely comfortable. For example, participants were asked how comfortable they felt asking participants “about their sexual orientation (e.g., if they are gay, lesbian, or bisexual)”. This measure demonstrated reliability in the current sample (Cronbach’s α=0.95–0.96).

Comfort with HIV testing and PrEP prescribing was measured with ten items about the extent to which participants felt comfortable with procedures related to HIV testing and PrEP prescription (Petroll et al., 2017). Items were rated on a five-point Likert-type scale ranging from completely uncomfortable to completely comfortable. As examples, participants were asked to indicate how comfortable they were “screening for HIV”, “prescribing PrEP”, and “providing risk reduction counseling related to substance use (such as offering harm reduction)”. The measure demonstrated reliability in the current sample (Cronbach’s α=0.90–0.93).

Participants’ PrEP knowledge was measured with seven items about the extent to which participants felt familiar with information about PrEP (Carter et al., 2019). Items were rated on a five-point Likert-type scale ranging from strongly disagree to strongly agree. As examples, participants were asked “I am familiar with on-demand PrEP” and “I am familiar with current research on PrEP safety.” The measure demonstrated reliability in the current sample (Cronbach’s α=0.86–0.89).

Finally, participants were asked about whether they were willing to prescribe PrEP to members of key populations on five-point Likert-type scales ranging from strongly agree to strongly disagree (Petroll et al., 2017). For example, participants were asked “I am willing to prescribe PrEP to men who have sex with men.” For the main outcomes analyses, composite scores were created by averaging across all four key population groups. The measure demonstrated reliability in the current sample (Cronbach’s α=0.87–0.95).

Analyses

First, participant sociodemographic and clinical characteristics were explored using baseline data stratified by condition. Second, linear mixed models were conducted to examine most of the primary and secondary outcomes, with repeated assessments nested within participants. A general linear model was conducted to examine attendance. Models included the main effects of condition and time, as well as the interaction between condition and time. For primary implementation-related outcomes, main effects of condition were of particular interest given that they provided insight into whether perceptions of the intervention differed across conditions. For the secondary stigma- and clinical-related outcomes, interaction effects between condition and time were of particular interest given that they provided insight into whether outcomes changed more in any condition. Third, because many of the main outcome analyses collapsed across key population and PLWH groups, ancillary analyses were conducted that disaggregated key population and PLWH groups for stigma- and key clinical-related outcomes. These analyses were conducted to explore whether the interventions were more effective in changing outcomes in relation to any particularly key population or PLWH group.

Qualitative Methods

Focus groups were conducted by two Malaysian team members with participants from each intervention condition after the final survey. Guided by the QUAN→qual approach (Palinkas et al., 2011), focus groups were conducted following preliminary exploration of the quantitative data.

Participants

Focus group participants were randomly selected and invited from each intervention condition. Participants included n=6 clinicians from the HC condition, n=4 clinicians from the PEHP condition, and n=5 clinicians from the PEHP-SR condition.

Protocol

Focus group participants were asked questions related to their perceptions of the acceptability of their intervention overall (e.g., What did you think of the training program?) as well as specific components relevant to each condition (e.g., What did you think of the photovoice presentations?). They were additionally asked questions related to feasibility (e.g., How easy or hard was it to make time for the program?) and impacts of the intervention (e.g., How has participating in this impacted your clinical work?).

Analyses

Focus group discussions were transcribed and reviewed by a small working group of Malaysian and U.S. team members. A qualitative codebook including themes, theme definitions, and example quotes was drafted (Saldaña, 2011). The codebook was applied to the transcripts and iteratively revised based on the data. Final themes focused on the implementation of, recommended changes to, and outcomes of the interventions. Codes were reviewed and a final consensus was reached among the small working group (Saldaña, 2011).

Results

Quantitative Results

Participants

As shown in the CONSORT diagram (Figure 1), n=411 individuals accessed the registration link. Of these, n=257 were excluded because they declined to participate in the study (i.e., did not consent to be randomized to an intervention arm), were withdrawn because they provided incomplete registration data or their identity could not be verified, or were waitlisted. The remaining n=154 individuals were randomized to the three conditions, of whom n=78 were allocated to and received an intervention (n=26 per condition). Of the n=76 individuals who were randomized but did not receive an intervention, most (n=46, 60.5%) withdrew because they could not commit to the scheduled time for their Project ECHO^® sessions or preferred a different condition. A smaller number (n=11, 14.5%) became non-responsive to communication from the project coordinator, and several (n=19, 25%) were waitlisted in case someone else dropped out of the intervention before it began. These participants did not complete the baseline survey. One participant officially withdrew from the PEHP-SR condition after it began due to a health issue. All participants completed the baseline survey, and n=24–26 of participants per condition completed the month 4.5 and 9 surveys. The participant who withdrew from the PEHP-SR condition due to a health issue completed the month 9 survey (but not month 4.5 survey) and was included in analyses.

Figure 1.

CONSORT Flow Diagram

Table 1 includes participants’ sociodemographic and clinical characteristics by study condition. On average, participants were in their late thirties (M[SD]=37.38[6.02], range: 28–71) and 59.0% identified as women. Regarding ethnic and religious identities, 44.9% identified as Chinese, 35.9% as Malay, 17.9% as Indian, and 1.3% as another ethnicity, and 35.9% identified as Muslim, 26.9% as Buddhist, 16.7% as Christian, 11.5% as Hindu, and 9.0% as another religion. Additional analyses demonstrated that there was considerable overlap between ethnicity and religion. As examples, 96.4% of participants who identified as Malay also identified as Muslim and 60% of participants who identified as Chinese also identified as Buddhist. Despite randomization, there were differences by ethnicity and religion across conditions with more participants identifying as Malay and Muslim in the PEHP condition. In contrast, more participants identified as Chinese and Buddhist in the HC and PEHP-SR conditions. Moreover, more participants identified as Christian in the PEHP-SR condition.

Primary Outcomes: Implementation

Table 2 includes results from the primary outcome analyses. Given socio-demographic differences between conditions despite randomization, all analyses controlled for religious identification. Average scores on implementation-related measures were fairly high (i.e., above four on five-point scales) in all conditions, indicating that participants generally rated the interventions as acceptable, appropriate, and feasible. The effect of condition was statistically significant in linear mixed model analyses. Post-hoc analyses suggest that participants in the PEHP condition viewed the intervention as more acceptable, appropriate, and feasible than participants in the PEHP-SR or HC conditions. Participants in the PEHP-SR viewed their intervention as more acceptable and appropriate but not more feasible than participants in the HC condition. There were no effects of time or condition by time interactions, suggesting that perceptions of acceptability, appropriateness, and feasibility did not change from the midpoint to the end of the study overall or within any of the conditions. Regarding perceptions of teaching effectiveness, participants in the PEHP condition rated their instructors as having greater presence and expertise than participants in the PEHP-SR or HC condition. Participants in the PEHP-SR viewed their instructors as having greater presence and expertise than participants in the HC condition. Participants in the PEHP and PEHP-SR conditions rated their instructors higher in engagement than participants in the HC condition. The statistically significant interaction between condition and time suggested that perceptions of engagement changed differently between the midpoint and end of the study for participants in different conditions. Post-hoc analyses (also shown in Table 2) suggested that perception of engagement increased among participants in the Project ECHO^® conditions but decreased among participants in the HC condition.

Table 2.

Participant sociodemographic and clinical characteristics at baseline (n=78), M (SD) or % (n)

Characteristic	HIV Connect (n=26)	Project ECHO: HIV Prevention (n=26)	Project ECHO: Stigma Reduction (n=26)	F-test or Chi-Square
Socio-Demographic
Age	37.42 (6.37)	36.50 (4.05)	38.23 (7.28)	F(2,77)=0.53
Gender				χ2 (2)=1.38
Man	50% (13)	34.6% (9)	38.5% (10)
Woman	50% (13)	65.4% (17)	61.5% (16)
Ethnicity				χ2(6)=12.94*
Malay	15.4% (4)	53.8% (14)	38.5% (10)
Chinese	61.5% (16)	23.1% (6)	50% (13)
Indian	19.2% (5)	23.1% (6)	11.5% (3)
Other	3.8% (1)	0% (0)	0%(0)
Religion				χ2(8)=18.6**
Muslim	15.4% (4)	53.8% (14)	38.5% (10)
Buddhist	34.6% (9)	15.4% (4)	30.8% (8)
Christian	15.4% (4)	11.5% (3)	23.1% (6)
Hindu	15.4% (4)	19.2% (5)	0% (0)
Other	19.2% (5)	0% (0)	7.7% (2)
Clinical
Affiliation				χ2(4)=2.35
Ministry of Health	65.4% (17)	69.2% (18)	76.9% (20)
Ministry of Eduction	6.6% (2)	3.8% (1)	0% (0)
Private Sector	26.9% (7)	26.9% (7)	23.1% (6)
Years Practicing Medicine	11.77 (5.80)	10.96 (4.31)	12.77 (7.89)	F(2,77)=0.56
Clinical Rank				χ2(8)=6.8
House Officer	0% (0)	0% (0)	0% (0)
Medical Officer	23.1% (6)	26.9% (7)	26.9% (7)
Registrar	0% (0)	11.5% (3)	7.7% (2)
Specialist	53.8% (14)	34.6% (9)	46.2% (12)
Consultant	0% (0)	0% (0)	3.8%(1)
General Practitioner	23.1% (6)	26.9% (7)	15.4% (4)
HIV Testing Experience				χ2(4)=3.73
Novice	3.8% (1)	0% (0)	7.7% (2)
Proficient	69.2% (18)	61.5% (16)	50% (13)
Experienced	26.9% (7)	38.5% (10)	42.3% (11)
PrEP Prescribing Experience				χ2(4)=6.51
Novice	42.3% (11)	69.2% (18)	65.4% (17)
Proficient	46.2% (12)	30.8% (8)	30.8% (8)
Experienced	11.5% (3)	0% (0)	30.0% (1)

Note:

⁺p<0.10

^*p<0.05,

^**p<0.01,

^***p<0.001

An analysis of the CONSORT diagram suggested that we were able to successfully recruit, randomize, and retain participants in our evaluation procedures. Although some individuals withdrew from the study after randomization, we were able to meet our recruitment goals. Moreover, the response rate was 97% at each follow-up survey. Regarding intervention attendance (Table 2), participants in the HC condition, which was delivered asynchronously, completed a higher percentage of sessions (M=89.37%) than participants in the PEHP (M=68.94%) and PEHP-SR (61.33%) conditions, which were delivered synchronously. Finally, there were no harms or unintended adverse events associated with the study reported by participants in any condition.

Secondary Outcomes: Stigma and Clinical

Table 3 includes results of the secondary outcomes analyses. Given that intervention attendance differed between conditions, attendance was added to analyses as a control variable alongside religious identification. Regarding the stigma-related outcomes, the significant main effect of time suggests that participants’ prejudice, stereotypes, and discrimination intent scores decreased over time in all three conditions. There was no significant main effect of condition on prejudice or discrimination, suggesting that prejudice and discrimination scores were similar across conditions. There was, however, a significant main effect of condition on stereotypes. Post-hoc analyses suggest that participants in the PEHP condition reported lower stereotypes than participants in the PEHP-SR and HC conditions. There were no statistically significant interactions between condition and time on prejudice, stereotypes, or discrimination intent, however, suggesting that changes in stigma-related outcomes over the course of the study were similar across all conditions. That is, results suggest that participants in the PEHP condition both started and ended the study with lower stereotype scores than participants in the other conditions.

Table 3.

Linear mixed models and general linear model predicting primary outcomes

	HIV Connect (n=26)		Project ECHO: HIV Prevention (n=24–26)		Project ECHO: Stigma Reduction (n=24–26)		Condition	Time	Condition × Time
	Month 4.5	Month 9	Month 4.5	Month 9	Month 4.5	Month 9
Implementation Outcomes
Acceptability (1–5)	4.15 (0.09)	4.19 (0.09)ab	4.74 (0.09)	4.70 (0.09)ac	4.34 (0.09)	4.46 (0.09)bc	F(2,69.32)=11.83***	F(1,69.84)=0.55	F(2,69.82)=0.70
Appropriateness (1–5)	4.15 (0.09)	4.23 (0.09)ab	4.74 (0.09)	4.74 (0.09)ac	4.38 (0.09)	4.53 (0.09)bc	F(2,69.41)=12.15***	F(1,70.20)=1.83	F(2,70.18)=0.56
Feasibility (1–5)	4.22 (0.09)	4.21 (0.10)a	4.68 (0.10)	4.60 (0.11)ab	4.19 (0.10)	4.32 (0.10)b	F(2,69.41)=12.15**	F(1,68.97)=0.03	F(2,68.95)=0.80
Teaching Effectiveness
Presence (1–5)	4.13 (0.11)	4.24 (0.10)ab	4.68 (0.11)	4.80 (0.10)ac	4.43 (0.11)	4.50 (0.10)bc	F(2,71.95)=10.03***	F(1,72.56)=3.75+	F(2,72.54)=0.02
Expertise (1–5)	4.25 (0.08)	4.36 (0.90)ab	4.78 (0.09)	4.83 (0.09)ac	4.47 (0.08)	4.69 (0.09)bc	F(2,72.72)=10.16***	F(1,73.36)=5.81*	F(2,73.34)=0.86
Engagement (1–5)	3.93 (0.13)	3.64 (0.13)ab	4.70(0.13)	4.81 (0.13)a	4.51 (0.13)	4.65 (0.13)b	F(2,72.68)=17.74***	F(1,72.69)=0.04	F(2,72.67)=5.25**
Percent Session Completion (0–100)		89.37 (5.09)ab		68.94 (5.15)a		61.33 (4.95)b	F(2,77)=8.03***

Note: M (SE) and results of tests of fixed effects are shown.

⁺p<0.10

^*p<0.05,

^**p<0.01,

^***p<0.001. Condition differences detected by post-hoc tests are indicated on 9 month means, and values that share a letter are statistically significantly different at p<0.05. Analyses control for religious identification.

Similar findings were observed for the clinical outcomes. The significant main effect of time on comfort suggests that participants’ comfort with taking sexual and drug use histories and comfort with HIV testing and PrEP prescribing increased over time. There were no statistically significant main effects of condition, suggesting that comfort scores were similar across participants in the three conditions, and no statistically significant interactions between time and condition, suggesting that changes in comfort over the course of the study were similar across all conditions. Finally, the significant main effect of time on PrEP-related outcomes suggests that participants’ PrEP knowledge and willingness to prescribe PrEP improved over time. There was no statistically significant effect of condition on PrEP knowledge, suggesting that participants had similar PrEP knowledge across the three conditions. There was a statistically significant effect of condition on willingness to prescribe PrEP, however, with post-hoc tests suggesting that participants in the PEHP condition were more willing to prescribe PrEP than participants in the PEHP-SR condition. There were no statistically significant interactions between condition and time on PrEP-related outcomes, however, suggesting that changes in PrEP-related outcomes over the course of the study were similar across all conditions. That is, results suggest that participants in the PEHP condition both started and ended the study with greater willingness to prescribe PrEP than participants in the PEHP-SR condition.

Ancillary Analyses

Findings from ancillary analyses that disaggregated key population and PLWH groups were generally similar to the main outcome results (see Supplemental Appendix 3). Close attention was paid to interactions between condition and time, to determine whether outcomes changed more in any particular condition for any particular key population or PLWH group, as well as time, to determine whether outcomes changed over time for any particular key population or PLWH group. There was a statistically significant interaction between condition and time on PWID stereotypes, which appeared to change more in the PEHP (M_diff=0.64) and PEHP-SR (M_diff=0.63) conditions than the HC (M_diff=0.22) condition. There were no other statistically significant interactions between condition and time. There were statistically significant main effects of time on most outcomes, with several exceptions. There were no main effects of time on PLWH stereotypes, TGW discrimination intent, PLWH discrimination intent, or willingness to prescribe PrEP to MSM.

Qualitative Results

Results of qualitative analyses provided additional insights into participants’ experiences with the interventions. Regarding feasibility, participants in both Project ECHO^® conditions reported that it was difficult to attend synchronous sessions at lunchtime. They recommend moving the time and/or providing asynchronous content (e.g., recorded sessions) that could be accessed when they missed sessions. A participant of the PEHP condition said: “Now the timing ya’ll chose is really, really wrong.” Participants in the HC condition, however, liked that the sessions were asynchronous, with one noting: “It’s quite flexible and also convenient for us, especially with busy clinics.” Although they liked the asynchronous format, HC participants wanted the ability to interact with presenters to ask questions. Participants in the PEHP spoke highly of the case presentations. One participant noted:

I really enjoy the case sharing because we really learned a lot of real experience (from) what’s going on with others. So, we learn from one another. So, when I miss the part, I really feel like, ‘ahhh I’m missing that, the important part.’

A participant in the HC condition requested case presentations. Participants in the PEHP and HC conditions reported that the topics covered by the didactics were helpful and suggested the addition of content that was included in the PEHP-SR condition (e.g., information about stigma and key populations, including a focus on transgender populations).

Regarding the stigma-reduction tools in the PEHP-SR condition, participants reported liking content related to key populations. For example, one participant said: “I particularly like the session that introduced the neutral terms for transgender population… it helped in my clinical practice. Especially when I meet the transgender patients, it kind of help with building rapport.” Participants also reported that they liked the interdisciplinary nature of the hub specialists and the guest speakers, including members of key populations and PLWH. When reflecting on guest speakers living with HIV, one participant highlighted that “a real story told by the population itself makes it very real”. In contrast, participants reported that the photovoice presentations were ineffective. Several focus group members did not remember the photovoice presentations and others noted that they preferred live guest speakers. One participant acknowledged that photovoice presentations introduced some variety but were ultimately not impactful: “I think the real story may tell me more compared to a photovoice. Right? Because we know the chronological (arc) of the story… it’s something different, but for me, it’s not hitting me on the point.” Another participant noted that the photovoice presentations made the sessions too rushed.

Participants in all three conditions reported that they had greater confidence in prescribing PrEP at the end of the interventions. Yet, participants highlighted other barriers to PrEP prescription, including lack of access and cost. An HC participant reported that they wanted to prescribe PrEP to a patient but was stopped by their supervisor because “they’re not comfortable to start with. They asked us to refer back to the ID (infectious diseases) clinic.” Participants in the PEHP-SR condition additionally reported greater understanding of experiences of members of key populations in healthcare contexts. One participant noted that they were building relationships with and referring patients to local non-governmental organizations that serve key populations. Participants in this condition also noted that the intervention increased their confidence in interacting with members of key populations. One participant said:

Even my patient was surprised when I asked them, like what’s the name that you preferred me to call you? And they feel like, oh, wow. This is the first time I got this question from a healthcare provider. So it’s beneficial for me and my patients.

Another participant in the PEHP-SR condition noted that they were adopting harm reduction approaches after learning more about social determinants of health. They said:

I’m gonna speak something quite controversial. Okay. I mean, because I used to think that the high risk behavior can be changed. I mean, the instinct can be changed. And then after this session, I think is that we are moving to harm reductions because the behavior is difficult to vanish. But what we do is keep the patient with us and make sure they’re okay.. undetected, and then reduce transmission… That’s what I learned.

Discussion

Results of this pilot randomized controlled trial provide insight into the implementation and preliminary efficacy of an intervention incorporating stigma reduction tools into Project ECHO^® for HIV Prevention. Regarding implementation-related outcomes, participants rated PEHP-SR (i.e., Project ECHO^® for HIV Prevention with the addition of stigma reduction tools) as acceptable, appropriate, and feasible. Yet, PEHP-SR was rated as less acceptable, appropriate and feasible than a similar intervention without the addition of stigma reduction tools (i.e., PEHP, or Project ECHO^® for HIV Prevention). Similarly, participants rated PEHP-SR hub specialists as having strong teaching presence, expertise, and engagement. Yet, PEHP-SR hub specialists were rated as having less strong teaching presence and expertise than PEHP hub specialists. Differences in teaching styles, personalities, and interpersonal dynamics between hub specialists in the PEHP-SR and PEHP conditions could all have contributed to differences in implementation- and teaching-related ratings between the conditions. Qualitative results suggest that the PEHP-SR condition, which included more components than the PEHP condition, may have felt too rushed to participants. Participants may have also disliked some components of the PEHP-SR condition (e.g., photovoice presentations). We additionally found that although participants generally rated the Project ECHO^® conditions more positively than the HC (i.e., HIV Connect) condition in terms of implementation-related outcomes and teaching effectiveness, they attended more of the HC sessions. Project ECHO^® session attendance in this study was roughly similar to other Project ECHO^® interventions (Diffin et al., 2021). Yet, qualitative results suggested that it was easier for participants to attend an asynchronous rather than a synchronous intervention, and that synchronous sessions scheduled over lunchtime were particularly difficult to attend.

Regarding preliminary efficacy-related outcomes, results suggested that stigma decreased over the course of the interventions in all conditions. Reductions in prejudice, stereotypes, and discrimination intent were similar in magnitude across all conditions, suggesting that the addition of stigma reduction tools in PEHP-SR did not yield greater reductions in stigma. Similarly, participants’ comfort taking clinical and drug use histories, comfort with HIV testing and PrEP prescribing, PrEP knowledge, and willingness to prescribe PrEP improved in all conditions. Improvements were similar in magnitude across all conditions, again suggesting that the addition of stigma reduction tools in PEHP-SR did not yield greater improvements in clinical-related outcomes. Reductions in stigma and improvements in clinical-related outcomes may have occurred in all conditions because all conditions included at least some stigma-reduction tools. As shown in Supplemental Appendix 1, all conditions included education, and PEHP and PEHP-SR both addressed social norms. Although PEHP-SR was the only condition to include contact, an arguably powerful stigma-reduction tool, qualitative results suggested that not all of the contact strategies were effective. Participants reported that they liked the guest speakers but did not find the photovoice presentations to be effective. These results provide support for arguments that contact interventions may be more or less effective depending on how they are delivered, including under optimal conditions of equal status, common goals, intergroup cooperation, and support of authorities or custom (Paluck et al., 2021; Pettigrew & Tropp, 2006). Notably, evidence suggests that contact-based interventions conducted under these optimal conditions demonstrate promise for reducing stigma among clinicians and improving health outcomes among patients (Penner et al., 2013).

Qualitative results additionally suggest that the PEHP-SR condition may have had some impacts that were not measured by the quantitative surveys. For example, participants in this condition reported building relationships with local non-governmental organizations that serve key populations, felt more confident interacting with members of key populations, asked for and used patients’ preferred pronouns, and began to adopt harm reduction strategies. Moreover, participants in the HC and PEHP conditions requested content that was included in the PEHP-SR condition, including a greater focus on stigma and key populations. Given these findings, we caution against using the results of this trial to conclude that stigma-reduction tools should not be added to future interventions with clinicians. Instead, future research should continue to evaluate the optimum ways to deliver stigma-reduction tools to clinicians.

Strengths, Limitations and Future Directions

This pilot randomized controlled trial represented a rigorous evaluation of a Project ECHO^® intervention adapted to incorporate stigma-reduction tools. By choosing to compare PEHP-SR to standard PEHP, a popular program endorsed by leading healthcare organizations including the National Academy of Medicine and Centers for Disease Control and Prevention, we chose a formidable comparator condition for this early stage of intervention testing. Yet, a recent expert panel convened by the National Institutes of Health Office of Behavioral and Social Science Research warned against the use of excessively formidable comparators in Phase I trials because they can “decrease the chances of finding a signal, quash further work on promising interventions, and thereby deprive stakeholders of potential benefits” (p. 76; Freedland et al., 2019). Although quantitative results did not support that the addition of stigma-reduction tools to Project ECHO^® yielded additional reductions in stigma, qualitative results suggested that there may be benefits to some of the tools. Because there were several differences between PEHP-SR and PEHP, however, it is difficult to determine which tools show promise versus were ineffective. For example, differences in implementation-related outcomes between the conditions could be due to the inclusion of different hub members, different topics, guest speakers, and/or photovoice presentations. Intervention development guided by the Multiphase Optimization Strategy (MOST) may be ideal for determining which stigma reduction tools, or intervention components, should be incorporated into PEHP and at what levels (Collins et al., 2007). More tightly controlled trials that better isolate single stigma-reduction tools may also provide insight into whether and which stigma-reduction tools provide additional benefit.

It is also important to study a broader range of stigma- and clinical-related outcomes in future trials. Qualitative results suggested that participants in the PEHP-SR condition may have changed stigma-related behavior (e.g., by asking for and using patients’ preferred pronouns), but these behaviors were not measured. Moreover, previous studies demonstrate that Project ECHO^® interventions result in improved clinical outcomes (including treatment of infectious disease (Arora et al., 2011; Struminger et al., 2017) and therefore future trials should evaluate a wider range of clinical outcomes, including with objective health outcomes (e.g., collected from medical records). Participants in this trial may have differed from the larger population of Malaysian clinicians in key ways, limiting the generalizability of these results. Participants had lower stigma scores, including prejudice towards key populations, than has been observed in past studies in Malaysia (Jin et al., 2014). It is possible that stigma has decreased in Malaysia since the time of the previous work. It is also possible that the individuals who signed up for this study, which was advertised as an HIV education program, had lower levels of stigma than their colleagues and/or that the individuals who remained in the study after randomization had lower levels of stigma than those who withdrew or became non-responsive. Future intervention trials should aim to recruit a broader, more representative sample of Malaysian clinicians. Despite randomizing participants between study conditions, there were differences in religion and ethnicity between conditions. Religion plays a key role in HIV-related stigma (Barmania & Reiss, 2018), and future trials should seek to ensure equal distribution of religious identification and ethnicity across intervention conditions. Finally, results of this trial may not generalize to settings beyond Malaysia.

Conclusions

It is critically important to develop scalable interventions, or those that reach beyond single healthcare facilities, to reduce HIV-related stigma among clinicians worldwide. Quantitative results of this pilot randomized controlled trial suggest that stigma- and clinical-related outcomes improved in PEHP-SR, an adapted version of Project ECHO^® for HIV Prevention that incorporated evidence-based stigma reduction tools. Yet, stigma- and clinical-related outcomes also improved in PEHP, the non-adapted version of Project ECHO^® for HIV Prevention, and HC, an asynchronous online course about HIV prevention. Qualitative results suggest that there may have been some benefits of the added stigma-reduction tools. Future research, perhaps guided by MOST, should continue exploring the benefits of adding stigma-reduction tools to PEHP and other interventions that can reach clinicians worldwide.

Table 4.

Linear mixed models and general linear model predicting secondary outcomes

	HIV Connect (n=26)			Project ECHO: HIV Prevention (n=24)			Project ECHO: Stigma Reduction			Condition	Time	Condition × Time
	Baseline	Month 4.5	Month 9	Baseline	Month 4.5	Month 9	Baseline	Month 4.5	Month 9
Stigma Outcomes
Prejudice (0–100)	29.46 (3.49)	28.43 (3.22)	23.44 (3.52)	24.03 (3.38)	20.79 (3.12)	15.18 (3.48)	30.39 (3.37)	28.36 (3.10)	22.14 (3.40)	F(2,70.55)=1.99	F (2,75.40)=10.43***	F(4,75.40)=0.15
Stereotypes (1–5)	2.77 (0.12)	2.72 (0.12)	2.60 (0.13)a	2.39 (0.12)	2.13 (0.11)	1.96 (0.13)ab	2.67 (0.12)	2.50 (0.11)	2.30 (0.13)b	F(2,69.52)=6.17*	F(2,79.38)=4.79***	F(4, 79.38)=1.06
Discrimination intent (1–5)	1.71 (0.10)	1.68 (0.10)	1.60 (0.13)	1.46 (0.09)	1.34 (0.09)	1.38 (0.13)	1.60 (0.09)	1.46 (0.09)	1.33 (0.12)	F(2,76.41)=2.38+	F(2,77.62)=3.00*	F(4,77.62)=0.56
Clinical Outcomes: Comfort
Comfort Taking Sexual and Drug Use Histories (1–5)	3.88 (0.14)	3.74 (0.14)	4.24 (0.12)	4.15 (0.13)	4.38 (0.14)	4.60 (0.12)	3.99 (0.13)	4.12 (0.14)	4.27(0.12)	F(2,71.59)=3.34+	F(2,90.16)=17.40***	F(4,90.21)=1.58
Comfort with HIV Testing and PrEP Prescribing (1–5)	4.12 (0.10)	4.26 (0.10)	4.29 (0.10)	4.17 (0.10)	4.31 (0.10)	4.54 (0.10)	4.07 (0.10)	4.37 (0.10)	4.46 (0.10)	F(2,69.91)=0.42	F(2,79.60)=17.43***	F(4,79.74)=1.36
Clinical Outcomes: PrEP-Related
PrEP Knowledge (1–5)	3.36 (0.14)	3.90 (0.13)	3.96 (0.13)	3.11 (0.13)	3.90 (0.12)	4.16 (0.13)	3.00 (0.13)	3.70 (0.12)	3.92 (0.13)	F(2,70.71)=1.04	F(2,86.33)=76.70***	F(4,86.97)=1.64
Willingness to Prescribe PrEP (1–5)	3.99 (0.11)	4.21 (0.11)	4.20 (0.12)	4.22 (0.11)	4.31 (0.11)	4.61 (0.12)a	4.06 (0.11)	4.07 (0.11)	4.16 (0.11)a	F(2,68.55)=3.34*	F(2,90.52)=4.91**	F(4,90.67)=1.08

Note: M (SE) and results of tests of fixed effects are shown.

⁺p<0.10

^*p<0.05,

^**p<0.01,

^***p<0.001. Condition differences detected by post-hoc tests are indicated on 9 month means, and values that share a letter are statistically significantly different at p<0.05. Analyses control for religious identification and percentage of session attendance.