A Systematic Review of Intervention Studies That Address HIV-Related Stigmas Among US Healthcare Workers and Health Systems: Applying a Theory-Based Ontology to Link Intervention Types, Techniques, and Mechanisms of Action to Potential Effectiveness

Abstract

Background

To end the HIV epidemic, we need to better understand how to address HIV-related stigmas in healthcare settings, specifically the common theoretical bases across interventions so that we can generalize about their potential effectiveness.

Purpose

We describe theory-based components of stigma interventions by identifying their functions/types, techniques, and purported mechanisms of change.

Methods

This systematic review examined studies published by April 2021. We applied a transtheoretical ontology developed by the Human Behaviour Change Project, consisting of 9 intervention types (ITs), 93 behavior change techniques (BCTs), and 26 mechanisms of action (MOAs). We coded the frequency and calculated the potential effectiveness of each IT, BCT, and MOA. We evaluated study quality with a 10-item adapted tool.

Results

Among the nine highest quality studies, indicated by the use of an experimental design, the highest potentially effective IT was “Persuasion” (i.e. using communication to induce emotions and/or stimulate action; 66.7%, 4/6 studies). The highest potentially effective BCTs were “Behavioral practice/rehearsal” (i.e. to increase habit and skill) and “Salience of consequences” (i.e. to make consequences of behavior more memorable; each 100%, 3/3 studies). The highest potentially effective MOAs were “Knowledge” (i.e. awareness) and “Beliefs about capabilities” (i.e. self-efficacy; each 67%, 2/3 studies).

Conclusions

By applying a behavior change ontology across studies, we synthesized theory-based findings on stigma interventions. Interventions typically combined more than one IT, BCT, and MOA. Practitioners and researchers can use our findings to better understand and select theory-based components of interventions, including areas for further evaluation, to expedite ending the HIV epidemic.

Peer-reviewed interventions to address HIV-related stigmas in US healthcare settings often lack a theoretical basis but can be characterized using theory-based constructs that contribute to our understanding of why these interventions do or do not work.

Introduction

In the United States, stigma related to HIV has deeply impaired efforts to curb the HIV epidemic and to care for those living with the virus, proving deleterious to individuals’ mental health [1], substance use [2], medication adherence [3], and other aspects of health and healthcare access [1, 4]. Addressing stigma is imperative to ending the HIV epidemic, and this mitigation effort is a priority in federal and local policies [5–7].

Stigma is a dynamic social process involving labeling, stereotyping, and separating—resulting in status loss and discrimination among those with less power in society [8]. As such, stigma is a fundamental cause of health inequities [9]. HIV stigma is not solely about HIV itself as an infectious disease but also other associated marginalized attributes. There are long-standing disparities in HIV incidence, healthcare access, and health outcomes [10] among people of color, sexual and gender minorities, and people who use substances [11]. Multiple intersecting systems of power and oppression jointly exacerbate these disparities, a socio-structural phenomenon larger than HIV stigma, often termed intersectional stigma [12–14]. For this reason, ending the U.S. HIV epidemic requires understanding ways to address the set of stigmas that contribute to HIV disparities, particularly among healthcare workers and healthcare systems, where stigma interferes with realizing the potential of status-neutral behavioral and biomedical tools [4–6, 15].

Previous meta-analyses and systematic reviews have shown HIV stigma interventions to be effective, including at reducing negative attitudes toward people living with HIV and increasing knowledge about HIV/AIDS among healthcare workers [16–20]. However, studies among healthcare workers across the globe have primarily focused on single stigmatized attributes at the individual level (e.g. HIV status), obscuring the potential for stigma mitigation across attributes that often have similar drivers, manifestations, and outcomes [9, 20–22]. Furthermore, while reviews have identified commonalities across effective interventions in low- and middle-income countries [19, 21, 23, 24], there remains a gap in understanding their theoretical bases, the technical content of their activities, and the mechanisms underlying their reported effects, limiting our understanding of effectiveness, including in high-income countries [19, 24–26]. In the USA, little is known about the quality and empirical support of interventions to mitigate HIV-related stigmas among healthcare workers and healthcare systems [20, 27].

To explore how to address HIV-related stigmas among healthcare workers and systems in New York City, we conducted a systematic review of intervention studies to understand how these interventions work by identifying common theoretical bases across studies. This effort required the application of an ontology—formal, explicit definitions of theoretical constructs and their interrelations [28]—to identify common elements. We adopted an established transtheoretical ontological framework developed by the Human Behaviour Change Project (see Fig. 1 [29]) to characterize within each study the functions/intervention types (ITs), behavior change techniques (BCTs), and mechanisms of action (MOAs) that characterize the causal processes through which interventions were intended to alter behavior [30–32]. Distinguishing function, form, and mechanism is recommended for improving the comparative effectiveness of complex health interventions [33] and may elucidate creative options to achieve similar functions with alternative forms that operate on the same MOAs [34].

Fig. 1.

Model of the relationship between ITs, BCTs, and MOAs intended to reduce HIV-related stigmas among healthcare workers and improve health outcomes

Methods

Search Strategies

A subcommittee of the STAR (Stigma and Resilience) Coalition, a group of community members, providers, researchers, and policy makers in New York City funded by an Ending the HIV Epidemic Supplement Award, convened with a library informationist to identify search terms for stigmatized attributes that intersect in ways that exacerbate HIV disparities. Additional synonyms were solicited from the larger STAR Coalition via an online survey, based on their knowledge of lived experience, literature, clinical practice, and policy. An informationist (ML) searched PubMed, PsycINFO, Embase, CINAHL, and CENTRAL for peer-reviewed articles published by April 2021. The operational set of HIV-related stigmas included synonyms for stigma related to HIV, age, substance use, sex work, disability, mental illness, ethnicity, racism, immigration, poverty, homelessness, incarceration, sexism, gender identity, and sexual orientation (see Supplementary File 1).

Inclusion Criteria and Exclusion Criteria

Studies needed to be conducted among practicing healthcare workers in the USA, including Puerto Rico, and include inferential statistics assessing, after intervention, healthcare worker or healthcare system constructs. Study outcome measures could be either enacted HIV-related stigmas (e.g. discrimination) or MOAs in addressing stigma (e.g. knowledge, attitudes) [35]. We excluded studies (1) with only qualitative data that had not been quantified, (2) with measurement only at the patient or community level, (3) without peer review (e.g. dissertations), (4) written in a language other than English or Spanish, and conducted, (5) solely among workers-in-training (e.g. students, residents, fellows), or (6) in environments explicitly dedicated to ongoing training and mentorship (e.g. academic medical campuses; to better understand strategies for practicing rather than in-training providers).

Screening and Data Extraction Procedures

Subsets of two reviewers screened each abstract and title, reviewed full texts for inclusion, then extracted data from each full-text record using Covidence software [36]. Screening and extraction were conducted blind to the other reviewer. The first author arbitrated discrepancies, as needed with the larger team, until consensus was reached.

Outcomes

Study characteristics

We extracted the following data: Study aims, stigmatized attributes studied, location, sample occupation, theoretical framework, intervention format (including frequency of exposure and duration of each exposure), costs, study design, conceptual description, and operationalization of stigma-related outcomes, dates of data collection, stated analytic plan, analytic sample size (if assessments were matched from pre- to post-intervention), timing of assessments, and statistically significant effects (at p < .01).

We coded stigmatized attributes, intervention format (e.g. workshop, policy), and conceptual description of outcomes based on each article’s methods section. Theoretical frameworks were extracted from any mention of theory in the introduction or methods sections. We followed a well-established taxonomy to categorize pre-experimental, experimental, and quasi-experimental studies based on randomization, presence of a comparison group, and timing of data collection in relation to intervention [37].

Theory-based components of intervention strategies and outcomes: ITs, BCTs, and MOAs

To deductively code each study’s strategies and outcomes as theory-based components, we used a set of transtheoretical, evidence-informed taxonomies based on the Behaviour Change Wheel, which together form an ontology developed by the Human Behaviour Change Project [29, 32, 38, 39]. ITs refer to nine super-ordinate functions of an intervention strategy (e.g. “Education” intends to increase knowledge, “Training” intends to increase skills) [32]. An intervention may serve more than one function (e.g. to increase both knowledge and skill) and thus could be coded with more than one IT.

To achieve their functions, interventions contain sets of techniques, which we coded using the Behaviour Change Techniques Taxonomy (BCTTv1). This taxonomy comprises 93 BCTs and is intended to be a comprehensive set, akin to a periodic table, of the smallest ingredients within an intervention strategy intended to change behavior (e.g. “Instruction on how to perform a behavior”) [40]. Each BCT is subsumed under a larger category indicating a general approach to change (e.g. “Goal setting” and “Problem solving” are both categorized under “Goals and Planning”). Characterizing both ITs and BCTs elucidates a clearer picture of both the form of change strategies (i.e. techniques) and their intended functions (i.e. ITs).

Finally, we coded study outcomes based on 26 MOAs, as defined in the Theory and Techniques Tool Online [30, 38, 41], a comprehensive set of potential mediators between BCTs and behavior change [38].

Coders of ITs and MOAs (BAK, TGMS) and coders of BCTs (RG, FK) used a blinded, consensus-driven approach, with additional unblinded BCT coding (BAK) to resolve discrepancies. Because of the complexity of coding BCTs from intervention descriptions, all BCT coders first passed a free, six-session online course to correctly identify BCTs [39]. Per this course’s guidance, each IT, BCT, and MOA was counted only once per article when present, even if that theory-based component appeared multiple times within an article (Supplementary File 7 defines each IT, BCT, and MOA.)

Potential effectiveness of ITs, BCTs, and MOAs

To explore the effectiveness of each IT, BCT, and MOA, we calculated a measure of “potential effectiveness” [42]: The number of articles reporting significant effects out of all articles coded as having studied that IT, BCT, or MOA, expressed as both a ratio and a percentage. Each IT, BCT, and MOA had to be coded in at least two studies for an interpretation to be meaningful [42]. As a more rigorous estimate, we also calculated potential effectiveness among only those studies with experimental designs.

Study quality and intervention reporting quality

To assess study quality, a smaller team (BAK, MPV, RG, CR-H, TGMS) adapted two established checklists [43, 44] to be applicable across study designs, then finalized a 10-item checklist. To examine the quality of intervention reporting, we used the Template for Intervention Description and Replication (TIDieR), a set of reporting criteria focused on individual-level strategies [26, 45]. Two studies [46, 47] addressed structural and/or policy changes but also individual-level changes and, as such, were coded with the TIDieR.

Results

Search Yield, Screening, and Data Extraction

Our search strategy yielded 21,031 records. We excluded 7,781 records by visually confirming automated deduplication within a citation management program [48]. After screening 13,250 study titles and abstracts and reviewing 89 full texts, we deemed 28 articles eligible for data extraction (see Fig. 2 for PRISMA flow diagram) [49, 50].

Fig. 2.

PRISMA identification of eligible intervention studies of HIV-related stigma reduction among healthcare workers and systems in the USA[49, 50]

Study Characteristics

All extracted characteristics are detailed by study in Table 1 and Supplementary File 2. Sample populations and study designs are described in Supplementary File 3.

Table 1.

Characteristics of 28 Studies Aiming to Reduce HIV-related Stigmas among Health Care Workers and Health Systems in the USA^a

References	Study aims	Theoretical framework	Stigmatized attribute(s) studied	Intervention strategy (intervention duration)c	Study design	Analytic sample(s)	Analytic plan
Amodeo and Fassler [51]	To examine the influence of substance use training on caseloads and competencies	—	Alcohol and/or Drug Use	Weekly in-person 3-hr substance abuse training program taught over 9 months (84 hr)	Static-group comparison (pre-experimental)	Follow-up n = 23 (experimental) n = 22 (control)	Student t-test and chi square
Batey et al. [52]	To examine feasibility, acceptability, and preliminary impact of an adaption to FRESH, an HIV stigma reduction workshop	Social Cognitive Theory; Intergroup Contact Theory	Intersectional Attributes (e.g. Gender Identity, Race/Ethnicity, Sexuality) affecting HIV stigma	One 1.5-day in-person HIV-related stigma workshop (12 hr)	One-group pretest–posttest design (pre-experimental)	Post-intervention n = 17 (experimental)	Paired t-tests
Bristol et al. [53]	To examine the impact of an LGBT training on the knowledge, attitudes, and competency of emergency department team members	—	Sexuality/Gender Identity (LGBT People)	One 30-min e-learning module followed by a 2-hr in-person LGBT cultural competency workshop (2.5 hr)	One-group pretest–posttest design (pre-experimental)	—	Multivariate OLS regression with adjustment for covariates
Cadiz et al. [54]	To evaluate Fit to Perform, a training for supervisors to manage nurses with substance use disorder	Stigma Theory	Alcohol and/or Drug Use	One in-person training for supervisors of nurses in recovery from substance use (4.5 hr)	One-group pretest–posttest design (pre-experimental)	Post-intervention n = 97 (experimental)	—
Cagle et al. [55]	To examine the effect of EMPOWER, a multicomponent intervention to reduce barriers to pain management in hospice care	Social Cognitive Theory	Alcohol and/or Drug Use	In-person staff training, screening, and education for patients and family caregivers on pain management	Posttest-only control group design (experimental) with 3-month follow-up	Post-intervention n = 55 (experimental) n = 71 (control) Follow-up n = 21 (experimental) n = 26 (control)	Regression models with adjustment for covariates
Collyer Yuchs and Bonham [56]	To examine the effect of an educational intervention on attitudes toward poverty	Culture of Poverty	Poverty	One in-person lecture and question-and-answer session about the culture of poverty (1 hr)	One-group pretest–posttest design (pre-experimental)	Post-intervention n = 26 (experimental)	Paired sample t-tests
Crawford et al. [47]	To determine if a structural intervention improved pharmacist attitudes and increased consumer access to syringes for persons who inject drugs	Harm Reduction	Alcohol and/or Drug Use	One in-person training on substance use and syringe distribution followed by individual training on a quarterly basis and as needed	Pretest–posttest control group design (experimental) with 6- and 12-month follow-up	Post-intervention (pharmacies) n = 18 (experimental) n = 21 (control) n = 27 (control) Follow-up (pharmacies) n = 20 (experimental) n = 19 (control) n = 27 (control) The analytic sample of staff was not reported	Log-binomial regression with adjustment for covariates
Crawford Fletcher and Akakpo [57]	To assess the impact of the Knowing Your Lens-Cultural Humility Awareness Training on racism among social workers within the child welfare system	Attitudinal Theory; Social Constructionism	Race/Ethnicity	One in-person interactive anti-racism group training (3.5 hr)	Nonequivalent control group design (quasi-experimental)	Post-intervention n = 42 (experimental) n = 45 (control)	Paired sample t-tests and difference in difference test
Flanagan et al. [58]	To investigate the effects of Recovery Speaks, a photovoice intervention to reduce stigma from primary care providers toward mental illness and substance use	Stigma Theory	Intersection of Alcohol and/or Drug Use and Mental Illness	One in-person photovoice performance by people with mental illness and addiction, followed by a discussion (1 hr)	Pretest–posttest control group design (experimental)	Post-intervention n = 13 (experimental) n = 14 (control)	Linear mixed regression models with adjustment for baseline scores
Golembo-Smith et al. [59]	To assess the effectiveness of a training for mental health providers to improve their capacity to treat early psychosis among youth	—	Mental Illness	One in-person workshop on psychosis among youth (2 hr)	One-group pretest–posttest design (pre-experimental)	—	Paired samples t-tests
Harris et al. [60]	To evaluate It’s Just Us, a pilot program targeting culture of nondisclosure with continuous contact approaches to reduce stigma by raising awareness of the stigma providers hold toward both clients and other providers with mental health challenges	Culture of nondisclosure; Social contact	Mental Illness	Fifteen organization-wide events spanning 2 years, comprising 7 Leadership Trainings (>5 hr), 4 Grand Rounds, 3 Brown Bags, 1 Yearly Meeting	One-group pretest–posttest design (pre-experimental)	—	—
Hayes et al. [61]	To compare the impact of three trainings on stigma and burnout among substance use counselors toward recipients of substance use services	—	Alcohol and/or Drug Use	One in-person workshop on substance use, based on either Acceptance and Commitment Training or multicultural training (6 hr)	Pretest–posttest control group design (experimental) with 3-month follow-up	Post-intervention n = 30 (experimental) n = 34 (control) n = 29 (control) Follow-up n = 26 (experimental) n = 32 (control) n = 27 (control)	Paired samples t-tests
Huber et al. [62]	To investigate the impact of a training for long-term care employees on aging and care for older adults	—	Age	Three 1-hr in-person classes on aging (3 hr)	One-group pretest–posttest design (pre-experimental)	Post-intervention n = 123 (experimental)	Paired samples t-tests
Ingraham et al. [63]	To describe and evaluate two educational curricula to reduce weight stigma among providers toward lesbian and bisexual women	—	Intersection of Weight and Sexuality (Lesbian/Bisexual Women)	An in-person cultural competency training on caring for lesbian and bisexual women of size that was either clinical (3 hr) or academic (1 hr)	One-group pretest–posttest design (pre-experimental)	Post-intervention n = 26 (experimental)	McNemar’s test for paired samples and Fisher’s exact test
Irvine et al. [64]	To examine the effects of an internet-based mental illness training program for health professionals in nursing homes	Social Cognitive Theory; Person-Centered Care	Mental Illness	Four virtual courses (each ~10–30 min long), including knowledge and video demonstration of behavioral skills on caring for mental illness, provided two at a time, administered over a 2-week period (40 min–2 hr)	Pretest–posttest control group design (experimental) with 2-month follow-up	Post-intervention n = 167b Follow-up n = 157b	ANCOVA models adjusting for pre-training scores, ancillary analysis of dose response
Kemppamen et al. [65]	To test the effects of three approaches to increase nurses’ willingness to care for persons with AIDS	—	Intersection of HIV/AIDS and Sexuality (Gay Men)	Either an in-person 1-hr lecture followed by 3 group discussions or 3 individualized sessions, led by an experienced nurse over 3–4 weeks, while providing care for people living with AIDS	Pretest–posttest control group design (experimental) with 3- and 6-month follow-up	Post-intervention n = 12 (experimental) n = 12 (experimental) n = 12 (control)	Repeated measures ANOVA and multivariate regression
Michaels et al. [66]	To evaluate the Anti Stigma Project workshop to reduce mental illness stigma among persons with mental illness and mental health providers	—	Mental Illness	One anti-stigma workshop including media analysis, interactive exercises, and group brainstorming (3 hr)	Pretest–posttest control group design (experimental)	Post-intervention n = 131 b	Repeated measures ANOVA
Mittal et al. [67]	To test a contact intervention and an education intervention to reduce stigma toward serious mental illness among primary care providers	—	Mental Illness	Either an in-person 20-min lecture by a physician in recovery followed by 10–15 min of discussion or an in-person PowerPoint lecture about serious mental illness stigma, each followed 1 month later by identical but abbreviated booster sessions of ~20 min	Pretest–posttest control group design (experimental) with 3-month follow-up	Post-intervention n = 19 (experimental) n = 20 (control) Follow-up Sample sizes not reported	Repeated measures ANOVA
Mizock et al. [68]	To evaluate the effectiveness of an adaptation of the Transgender Awareness Webinar in reducing transphobia among mental health providers and undergraduate students	—	Gender Identity (Transgender People)	One webinar lecture on transgender awareness with images, videos and live quizzes (40 min)	One-group pretest–posttest design (pre-experimental)	Post-intervention n = 158 (experimental)	Paired samples t-test, repeated measures ANOVA, multivariate F tests
Rosa Vega et al. [69]	To examine the impact on knowledge of a training for pharmacists to provide care for transgender patients	—	Gender Identity (Transgender People)	One in-person workshop on transgender care (3 hr)	One-group pretest–posttest design (pre-experimental)	Post-intervention n = 54 (experimental)	ANOVA with Tukey post-hoc test
Santa et al. [70]	To study whether attitudes and dispensing of naloxone improved following SBIRT and naloxone training with pharmacists	Systems Transformation Framework	Alcohol and/or Drug Use	One daylong in-person training (lecture, discussion, role-play) on opioid addiction followed by an online lecture with simulated patient contact and knowledge tests	One-group pretest–posttest design (pre-experimental); interrupted time series design (quasi-experimental)	Post-intervention n = 22 pharmacists n = 9 pharmacies (experimental) Follow-up n = 9 pharmacies (experimental)	Paired sample t-tests
Schweiger-Whalen et al. [71]	To evaluate Converging Cultures, an LGBT cultural competency workshop for hospital employees and undergraduate nursing students	Cultural Competence	Sexuality/Gender Identity (LGBT People)	One lecture, panel and discussion workshop on LGBT cultural competency (4 hr)	One-group pretest–posttest design (pre-experimental)	Post-intervention n = 130 (experimental)	Paired sample t-tests, Mann-Whitney tests, linear regression
Seidel et al. [72]	To evaluate a multi-city training and technical assistance and capacity-building program on aging and HIV for senior service providers	—	Intersection of HIV/AIDS and Age	Four days of in-person training on aging with HIV, with an additional 3–5 days of observational learning for a subset of participants 1–5 months later	One-group pretest–posttest design (pre-experimental)	Post-intervention n = 32 (Year 2), n = 20 (Year 3), n = 37 (Year 4) (experimental)	Paired sample t-tests and chi-squared tests
Steed [73]	To examine the effects of a cultural competency training on White female occupational therapists	—	Intersection of Race/Ethnicity and Poverty	One in-person anti-racism workshop (6 hr)	One-group pretest–posttest design (pre-experimental)	Post-intervention n = 13 (experimental)	Paired sample t-tests and sign test
Timms and Fallat [74]	To investigate the effects of age bias training on home care aides	—	Age	A 1-hr in-person lecture on facts, stereotypes and myths about aging, followed ~1 month later by a 1-hr in-person group discussion (2 hr)	One-group pretest–posttest design (pre-experimental) with 1-month follow-up	Post-intervention n = 103 (experimental) Follow-up n = 61 (experimental)	Paired sample t-tests
Weech-Maldonado et al. [46]	To examine the impact of a structural cultural competency initiative on climate and workforce diversity within hospitals	Model of Organizational Performance and Change; Interactional Model of Cultural Diversity	Race/Ethnicity	Infrastructure development, executive coaching, training, individual level action plans, and other interventions for workforce diversity determined by each intervention hospital, delivered over 2.5 years	Pretest–posttest control group design (experimental)	Post-intervention n = 2 (experimental) n = 2 (control)	t-tests and chi-square tests
Wilson-Young [75]	To discuss the effects of an affective teaching method on nurses who attended a workshop on homosexuality and AIDS	—	Intersection of HIV/AIDS and Sexuality (Gay Men)	One in-person workshop on sexuality and AIDS	One-group pretest–posttest design (pre-experimental)	Post-intervention n = 22 (experimental)	Paired sample t-tests
Zapata et al. [76]	To evaluate Opioid Addiction Treatment ECHO (Extension of Community Health Outcomes), a knowledge and capacity-building program for community health workers	Stigma Theory	Alcohol and/or Drug Use	Eleven weekly ~90-min video conference case study sessions on opioid use disorder with group discussion facilitated by an expert (16 hr, 21 min)	One-group pretest–posttest design (pre-experimental)	Post-intervention n = 24 (experimental)	Paired sample t-tests

^a“—” indicates that a study characteristic was not reported or not reported in a way that met review criteria.

^bAuthors did not report separate sample sizes for the experimental and control groups.

^cLength, frequency, and duration of the intervention strategies are included only if reported in the published article.

Among the 28 studies, the most common stigma behavior change targets among healthcare workers and/or healthcare systems (Table 1) related to alcohol and/or drug use (7 studies), mental illness/mental health (5 studies), and HIV (4 studies). Seven studies addressed the intersection of two or more stigmatized attributes; only one referred to the term “intersectional.” None reported intervention costs. Fifteen studies (46%) did not report a theoretical framework. The most common reported theoretical frameworks were social cognitive theory (3 studies) and stigma theory (3 studies).

Samples ranged from n = 17 to n = 260. Some studies were conducted at a single type of healthcare facility; others included multiple hospitals or pharmacies. Samples included: nurses (9 studies), paraprofessionals (5 studies), pharmacy staff (4 studies), social workers (4 studies), rehabilitation therapists (4 studies), medical doctors (2 studies), and health systems (2 studies). Nearly half of studies reported the inclusion of unspecified professionals and staff (e.g. “mental health providers”). Three included students who participated alongside other practicing staff. In data not shown in tables, three-quarters of studies did not report whether their samples included healthcare workers labeled with the stigmatized attribute(s) being addressed by their intervention.

Intervention strategy format, exposure time, and duration of exposure

Almost all studies included workshops (9 studies), lectures (8 studies), discussion sessions (8 studies), and/or trainings (7 studies). In four studies, intervention strategies were delivered partially or entirely virtually. The total exposure time to intervention content ranged from 40 min to 89 hr. In terms of duration of continuous exposure, 13 studies conducted interventions over the span of a single day, and 2 over two consecutive days. Longer duration, noncontinuous interventions were held in one of two ways: 3 studies had a regularly recurring exposure (e.g. weekly, quarterly) and another 5 studies clustered days with a time lag between exposures (e.g. a 2-week lag, a variable one- to five-month lag). Five studies did not specify exposure time, though one reported a 2-year and another a two-and-a-half-year duration for the intervention window.

Conceptualization and operationalization of stigma-related outcomes

The most commonly studied stigma-related outcomes (Supplementary File 2) were conceptualized as knowledge, attitudes, competency, awareness, and beliefs. Operationalization generally aligned with conceptual definitions, although in some cases there were discrepancies (e.g. operationalizing the concept of competency by measuring confidence). Additional outcomes included behaviors, implicit bias, empathy, social distance, perceived risk, cultural competency, burnout, attributions, relevance of stigma to one’s own life, and willingness to intervene/support. A few studies reported directly observable outcomes (i.e. caseloads of clients who use drugs, observed stigma), but most did not specify whether assessments were self-reported or directly observed.

Study design and analytic plans

The most common study design was the pre-experimental one-group pretest–posttest (17 studies). Nine studies used an experimental design, mostly an experimental pretest–posttest control group design. Of the 21 studies that fully reported the timing of each assessment, 10 reported testing immediately before and immediately after intervention, and 3 reported variable time periods for their pretests and posttests (e.g. a 2-week pretest, a 6-month posttest). Seven reported follow-up assessments, with periods ranging from 1 to 6 months. More than half of the studies did not report dates of data collection; among those that did, most reported collecting data between 2009 and 2019.

The most common analytic plan for testing changes was bivariate analysis (e.g. t-tests, paired sample t-tests, ANOVA). Four studies conducted regression analyses that adjusted for covariates. A few assessed outcomes at the site- rather than the individual level and a few did not report the analytic sample for both pretest and posttest assessments.

Overall effectiveness of intervention strategies

A majority of studies (75%) reported a statistically significant change in a stigma-related outcome (p < .01). All reported changes were in the hypothesized directions (e.g. knowledge of a particular topic increased, attitudes improved). These changes were observed across sample populations, methodologies, and types of healthcare settings.

Theory-based Components of Intervention Strategies and Outcomes

Table 2 summarizes frequencies and proportion of studies with each IT, BCT, and MOA code, and Supplementary File 4 details each study’s combination of ITs, BCTs, and MOAs. (See Supplementary File 7 to filter studies by each IT, BCT, MOA, and study characteristic; to view linkages across these components; and for definitions and excerpts for each code.)

Table 2.

Frequency and Potential Effectiveness of Intervention Types (ITs), Behavior Change Techniques (BCTs), and Mechanisms of Action (MOAs) in 28 studies of HIV-related Stigma Reduction among Health Care Workers and Health Systems in the USA^a

Code #		All studies (N = 28)				Studies with experimental designs (N = 9)
		Frequency and percent of studies with each code		Improved outcomes reported (­potential effectiveness)b		Frequency and percent of studies with each code		Improved outcomes reported (­potential effectiveness)b
		n	%	Ratio	(%)	n	%	Ratio	(%)
	Intervention Types (ITs) c
1.	Education	28	100.0%	22/28	(78.6%)	9	100.0%	4/9	(44.4%)
5.	Training	16	57.1%	12/16	(75.0%)	6	66.7%	2/6	(33.3%)
2.	Persuasion	11	39.3%	9/11	(81.8%)	6	66.7%	4/6	(66.7%)
8.	Enablement	10	35.7%	7/10	(70.0%)	4	44.4%	1/4	(25.0%)
9.	Modeling	6	21.4%	4/6	(66.7%)	3	33.3%	1/3	(33.3%)
7.	Environmental restructuring	5	17.9%	3/5	(60.0%)	2	22.2%	0/2	(0.0%)
4.	Coercion	2	7.1%	1/2	(50.0%)	1	11.1%	0/1
	Behavior Change Techniques (BCTs) c
4.1	Instruction on how to perform a behavior	23	82.1%	18/23	(78.3%)	7	77.8%	3/7	(42.9%)
6.1	Demonstration of the behavior	19	67.9%	15/19	(78.9%)	6	66.7%	3/6	(50.0%)
5.3	Information about social and environmental consequences	15	53.6%	11/15	(73.3%)	5	55.6%	2/5	(40.0%)
9.1	Credible source	14	50.0%	13/14	(92.9%)	3	33.3%	2/3	(66.7%)
1.2	Problem-solving	12	42.9%	11/12	(91.7%)	4	44.4%	2/4	(50.0%)
8.1	Behavioral practice/rehearsal	12	42.9%	12/12	(100.0%)	3	33.3%	3/3	(100.0%)
13.2	Framing/reframing	10	35.7%	9/10	(90.0%)	3	33.3%	2/3	(66.7%)
3.1	Social support (unspecified)	8	28.6%	5/8	(62.5%)	3	33.3%	0/3	(0.0%)
12.2	Restructuring the social environment	8	28.6%	7/8	(87.5%)	1	11.1%	0/1
11.2	Reduce negative emotions	7	25.0%	6/7	(85.7%)	3	33.3%	2/3	(66.7%)
2.2	Feedback on behavior	5	17.9%	5/5	(100.0%)
8.2	Behavior substitution	4	14.3%	4/4	(100.0%)	1	11.1%	1/1
1.4	Action planning	3	10.7%	2/3	(66.7%)	1	11.1%	0/1
3.3	Social support (emotional)	3	10.7%	2/3	(66.7%)	3	33.3%	2/3	(66.7%)
5.2	Salience of consequences	3	10.7%	3/3	(100.0%)	3	33.3%	3/3	(100.0%)
13.3	Incompatible beliefs	4	14.3%	3/4	(75.0%)	2	22.2%	1/2	(50.0%)
2.1	Monitoring of behavior by others without feedback	2	7.1%	2/2	(100.0%)
2.7	Feedback on outcomes (of behavior)	3	10.7%	3/3	(100.0%)
4.2	Information about antecedents	2	7.1%	1/2	(50.0%)	2	22.2%	1/2	50.0%
7.1	Prompts/cues	2	7.1%	2/2	(100.0%)	1	11.1%	1/1
8.7	Graded tasks	2	7.1%	2/2	(100.0%)	1	11.1%	1/1
1.1	Goal setting (behavior)	1	3.6%	1/1
1.6	Discrepancy between current behavior and goal	1	3.6%	1/1
1.9	Commitment	1	3.6%	1/1
3.2	Social support (practical)	1	3.6%	1/1
4.4	Behavioral experiments	1	3.6%	1/1
5.1	Information about health consequences	1	3.6%	0/1
5.4	Monitoring of emotional consequences	1	3.6%	1/1
5.6	Information about emotional consequences	1	3.6%	0/1		1	11.1%	0/1
6.2	Social comparison	2	7.1%	1/2	(50.0%)	1	11.1%	0/1
6.3	Information about others’ approval	1	3.6%	1/1		1	11.1%	0/1
7.6	Satiation	1	3.6%	1/1
7.7	Exposure	1	3.6%	1/1
9.2	Pros and cons	1	3.6%	1/1
10.3	Non-specific reward	1	3.6%	1/1
10.11	Future punishment	1	3.6%	1/1
12.1	Restructuring the physical environment	1	3.6%	1/1
12.5	Adding objects to the environment	1	3.6%	1/1
13.1	Identification of self as role model	1	3.6%	1/1
13.4	Valued self-identity	1	3.6%	1/1
14.2	Punishment	1	3.6%	1/1
16.3	Vicarious consequences	1	3.6%	1/1
	Category of BCTs c
4.	Shaping knowledge	24	85.7%	19/24	(79.2%)	8	88.9%	4/8	(50.0%)
6.	Comparison of behavior	20	71.4%	15/20	(75.0%)	7	77.8%	3/7	(42.9%)
5.	Natural consequences	16	57.1%	13/16	(81.3%)	5	55.6%	3/5	(60.0%)
1.	Goals and planning	13	46.4%	11/13	(84.6%)	4	44.4%	2/4	(50.0%)
8.	Repetition and substitution	13	46.4%	13/13	(100.0%)	3	33.3%	3/3	(100.0%)
9.	Comparison of outcomes	13	46.4%	12/13	(92.3%)	3	33.3%	2/3	(66.7%)
13.	Identity	11	39.3%	10/11	(90.0%)	4	44.4%	3/4	(75.0%)
3.	Social support	10	35.7%	7/10	(70.0%)	5	55.6%	2/5	(40.0%)
12.	Antecedents	10	35.7%	9/10	(88.9%)	1	11.1%	0/1
2.	Feedback and monitoring	7	25.0%	7/7	(100.0%)
11.	Regulation	7	25.0%	6/7	(83.3%)	3	33.3%	2/3	(66.7%)
7.	Associations	3	10.7%	3/3	(100.0%)	1	11.1%	1/1
10.	Reward and threat	2	7.1%	2/2	(100.0%)
14.	Scheduled consequences	1	3.6%	1/1
16.	Covert learning	1	3.6%	1/1
	Mechanisms of Action (MOAs) c
25.	General Attitudes/Beliefs	19	67.9%	16/19	(83.3%)	7	77.8%	4/7	(57.1%)
1.	Knowledge	15	53.6%	13/15	(86.7%)	3	33.3%	2/3	(66.7%)
4.	Beliefs about capabilities	8	28.6%	7/8	(87.5%)	3	33.3%	2/3	(66.7%)
13.	Emotion	9	32.1%	7/9	(77.8%)	5	55.6%	3/5	(60.0%)
17.	Attitude toward the Behaviour	6	21.4%	3/6	(50.0%)	4	44.4%	1/4	(25.0%)
11.	Environmental context and resources	4	14.3%	4/4	(100.0%)
21.	Values	4	14.3%	4/4	(100.0%)	1	11.1%	1/1
8.	Intentions	3	10.7%	2/3	(66.7%)	2	22.2%	1/2	(50.0%)
2.	Skills	2	7.1%	2/2	(100.0%)
6.	Beliefs about consequences	2	7.1%	0/2	(0.0%)	2	22.2%	0/2	(0.0%)
12.	Social influences	2	7.1%	2/2	(100.0%)
19.	Self-image	2	7.1%	1/2	(50.0%)
26.	Perceived susceptibility/Vulnerability	2	7.1%	1/2	(50.0%)
3.	Social/Professional role and identity	1	3.6%	0/1
5.	Optimism	1	3.6%	1/1		1	11.1%	0/1
7.	Reinforcement	1	3.6%	0/1
14.	Behavioural regulation	1	3.6%	0/1		1	11.1%	1/1
16.	Subjective norms	1	3.6%	0/1

^aCodes with a frequency of zero are not shown (see Supplementary File 5 for all codes, including codes with zero).

^b“Potential effectiveness” refers to the percentage of studies reporting a statistically significant improved outcome among those coded for an IT, BCT, or MOA; potential effectiveness is only shown as a ratio and not a percentage when the denominator is a single study.

^cFor definitions of each IT, BCT, and MOA, see Supplementary File 5. BCT categories are not defined per se, but are grouped according to similar BCTs within the Behaviour Change Techniques Taxonomy v.1 [32]

ITs

All studies included “Education” to increase knowledge and/or understanding (e.g. implications of pharmacy syringe access; origins of healthcare disparities due to racism; distinguishing myths from facts). The next most common ITs were “Training” to impart skills (16 studies, e.g. acceptance, mindfulness, and cognitive defusion; motivational interviewing; role-playing); “Persuasion” by using communication to induce positive or negative feelings or stimulate action to address stigma (11 studies, e.g. a photovoice project; simulated hallucinations to increase empathy); “Enablement” to increase capability or opportunity (10 studies, e.g. written resources; visits to other agencies); “Modeling” to demonstrate an exemplar of behavior (6 studies, e.g. videos of correct techniques; shadowing staff; disclosure of mental illness); and “Environmental restructuring” to change the physical/social context (5 studies, e.g. new protocols for dispensing medication; provision of safe injection packets for pharmacists to distribute).

BCTs

The most commonly coded BCT was “Instruction on how to perform a behavior” (23 studies, e.g. case presentations; virtual patient simulation; didactic presentations). This BCT often occurred alongside “Demonstration of the behavior” (19 studies, e.g. video or in-person modeling or observation). The next most commonly coded BCTs were “Information about social and environmental consequences” (15 studies, e.g. videos about the impact of stigma; information on social determinants); “Credible source” (14 studies, e.g. official working groups; representatives of stigmatized groups); “Behavioral practice/rehearsal” (12 studies, e.g. role-playing); “Problem solving” (12 studies, e.g. developing protocols, action plans, or quality improvement plans); “Framing/reframing” (10 studies, e.g. viewing problematic behavior as an expression of unmet needs; cognitive defusion); “Social support (unspecified)” (8 studies, e.g. reflecting on feelings; constructive confrontation); “Restructuring the social environment (8 studies, e.g. selecting site champions; facilitating interagency communication); and “Reduce negative emotions” (7 studies, e.g. methods to cope with stigma and enhance positive feelings).

MOAs

The most commonly coded mechanism was “General attitudes or beliefs” (19 studies, e.g. bias; stereotypes; desire for social distancing). The next most common MOAs were “Knowledge” (15 studies, e.g. of policies and practices; barriers to care); “Beliefs about capabilities” (8 studies, e.g. competency; self-efficacy); “Emotion” (9 studies, e.g. exhaustion and depersonalization; pity, anger or fear); and “Attitudes toward the behavior” (6 studies, e.g. willingness to care for AIDS patients).

Potential Effectiveness of ITs, BCTs, and MOAs

Table 2 summarizes potential effectiveness of the interventions. Among the nine studies with experimental designs, “Persuasion” was the IT with the most studies reporting statistically significant improved outcomes (4/6 studies, 67%), followed by “Education” (4/9 studies, 44%), “Training” (2/6 studies, 33%), and “Modeling” (1/3 studies, 33%). BCTs with the most evidence for potential effectiveness were “Behavioral practice/rehearsal” and “Salience of consequences” (each with 3/3 studies showing improvements, or 100%), followed by “Framing/reframing,” “Credible source,” and “Social support (emotional)” (each with 2/3 studies, 67%). MOAs with the most evidence were “Knowledge” and “Beliefs about capabilities” (each with 2/3 studies, 67%), followed by “Emotion” (3/5 studies, 60%) and “General Attitudes/Beliefs” (4/7, 57%).

Intervention Reporting Quality and Study Quality

Based on the TiDieR criteria (Supplementary File 5), the most commonly coded aspects of intervention reporting were the rationale (100%); mode of delivery (92.9%); procedures (92.9%); timing (75%); and materials used (75%). The remaining TIDieR criteria were typically not reported; information regarding fidelity, tailoring, and modifying were each reported in fewer than 15% studies.

Of our 10 quality assessment criteria (Supplementary File 6), all studies stated a study question (100%), and most met the standards for clearly described findings (96.4%); significance testing (96.4%); pre-stated outcomes (96.4%); analytic sample retention (85.7%); intervention sample retention (75%); and validity and/or reliability of measures (75%). Fewer met the standards for eligibility criteria (60.7%) or, when applicable, adjustment for confounding (17.4%). Of the nine studies with control arms, most (88.9%) reported the same population across study arms.

Discussion

Our study characterizes intervention strategies based on their theory-based components, linking intervention functions, techniques, and MOAs to one another and to their potential effectiveness. In terms of functions, interventions with the most potential effectiveness went beyond providing information and skills building, typically common components of stigma reduction in health care [26], to the use of communication intended to induce emotions and/or stimulate action (i.e. “Persuasion”). All used education, but the most promising interventions included techniques of credible sources delivering content, skills practice, cognitive reframing, addressing emotions, informing providers of the consequences of stigma for their patients, and attempting to increase the mechanism of beliefs about one’s capability to change stigmatizing behavior.

Few studies assessed intersectional approaches that addressed systems of power and oppression, like racism or poverty-related stigma, key drivers of the ongoing HIV epidemic [5]. Most targeted change only at the individual level, with information and contact-related interventions dominating, despite calls for structural interventions [77, 78] because of their potential to have greater impact at scale and to impact individual-level stigma [79]. Only four reported structural strategies; of these, two targeted structural changes to health systems (i.e. laws to increase syringe access; hospital-wide policies) and thereby overcame the common pitfall of obscuring systems by limiting intervention to individuals [78]. Ultimately, reducing stigma requires addressing power [80]. Notably, only two studies reported the involvement of medical doctors and only two the involvement of administrators, both positions of relative power in healthcare settings. Future interventions would benefit from an intersectional lens, accounting for interlocking systems of power and privilege to address HIV-related stigmas [6, 77].

Most articles did not explicitly articulate their theoretical framework, however, for all but one article our team successfully identified at least one theory-based IT, BCT, and MOA. Typically, interventions focused on sharing knowledge of stigmatized conditions, exposure to stigmatized peoples’ stories, and role-play. Nearly 60% of studies employed training to improve skills, but few actually assessed skills; instead, they tended to assess knowledge, attitudes, and beliefs about capabilities. Few focused on problem-solving using tools, strategies, and needs assessments, and few applied monitoring and feedback of behavior or the use of incentives or rewards, BCTs that might have longer-lasting impacts when used together [81].

Most study designs were of less robust quality, with neither a comparison group nor randomization, and most did not assess changes at follow-up. Short data collection periods and reliance on pre/post testing may show short-term success, but there was a dearth of information on long-term effects and factors contributing to sustained change. Some interventions were comparatively briefer than others, and none included cost estimates, limiting data on feasibility. Stigma is a fundamental cause of disease with many possible pathways [9], and may well require a sustained approach. The low number of experimental designs and infrequent adjustment for confounding also indicate room for growth in determining causal processes and effectiveness. Additionally, few studies examined client-level impact and deriving conclusions about that impact would require even more robust and long-term designs. Still, seven of the nine studies with experimental designs were published within the past decade, increasing optimism that robust studies are underway that might further elucidate the causal processes of mitigation strategies.

In terms of the quality of reporting, most article texts described what authors planned to do and why, with comparatively little space to document the entirety of techniques actually implemented. The focus on rationale rather than content also typically lacked a cited basis in theory, inhibiting our ability to generalize knowledge about the strategies deployed within each study. Peer-reviewed articles typically constrain word counts, so may need to be supplemented by detailed manuals, websites, and the like. Most studies met minimum standards for reporting quality (e.g. clearly stated research questions and findings), however, our exploration revealed a misalignment between the reported conceptualization of stigma-related constructs and their operationalization. Consistent with other findings [20], studies often did not specify assessment (e.g. scale origins, self-report vs. observation), impeding an assessment of the validity of outcomes.

For those who plan, implement, or evaluate stigma mitigation programs in healthcare settings, our findings offer a resource to inform and monitor components of their activities. Practically, program staff can refer to Supplementary File 7 for a description of each study’s target population and target behaviors; definitions of ITs, BCTs, and MOAs with coded excerpts; linkages between these components; and to filter by any single component or set of components as well as study characteristics to consider whether to implement peer-reviewed strategies or devise their own. This may also help implementers explore the extent to which their existing strategies already include theory-based and potentially effective ITs, BCTs, and MOAs, and whether to emulate peer-reviewed strategies that target similar functions by way of using alternative techniques. For monitoring and evaluation, implementers can consider stigma-related evaluation tools from studies that measured MOAs of interest. Likewise, researchers may use a similar process for intervention development to identify promising and/or understudied ITs, BCTs, and MOAs.

Our findings should be interpreted with several limitations. Most importantly, we cannot definitively state which ITs, BCTs, and MOAs are more or less promising than others, given the variability across studies in target behaviors, quality, analytic approach, and measurement. Our own coding approach, based on the ontology developed by the Human Behaviour Change Project, accounted for only the presence or absence of an IT or BCT, not the density or the combination of ITs and BCTs in each strategy. Supplementary File 4 illustrates how each study typically served multiple functions and therefore included more than one IT (e.g. “Education” and “Persuasion”). Likewise, BCTs, as the smallest possible component of an intervention, rarely occurred in isolation and more often in concert with one another as a whole strategy. Further, we used reported effects of each overall strategy as the basis for determining the potential effectiveness of each IT, BCT, and MOA; in fact, we can only say that strategies that contained these components demonstrated positive effects, not that these components on their own were effective. Our set of studies also often lacked precise terminology, diminishing our ability to glean the actual format and content of their strategies. Future efficacy-focused studies should examine combinations of ITs and BCTs to determine which operate on specific MOAs to more or less effectively address stigma. For our purpose, conducting a meta-analysis would have been ideal but not possible with such high variability. Our goal of elucidating which components do and do not mitigate stigma is also inherently limited, as in all systematic reviews, by bias toward publishing significant findings over null findings.

Conclusion

This systematic review advances HIV-related stigma science by linking the effectiveness of interventions to their theory-based components, describing commonalities across studies that address HIV-related stigmas among practicing healthcare workers and healthcare systems. Given the magnitude of the problem, the number of studies to guide us is relatively small and their limitations are problematic. Nonetheless, our findings can guide future HIV research and promote the conduct of higher quality, theory-based studies, including to evaluate combinations of the functions, techniques, and mechanisms that underlie stigma interventions. This would advance our understanding of which techniques operate on which stigma-related mechanisms, similar to the existing Theory and Techniques Tool Online [38], a heatmap of triangulated evidence that links BCTs to MOAs, but which is generally about behavior change, not stigma reduction. Both standardization and creativity is needed in future practice and research, and HIV program implementers and scholars should draw on our findings to consider existing and novel strategies. We hope to see more specification of theory-based components, with authors using language that can be easily coded across studies and therefore more easily linked to theory-based components, so that we can 1 day understand how best to address HIV-related stigmas in healthcare and mitigate long-standing inequities in services for affected populations.

Compliance with Ethical Standards

Authors’ Statement of Conflict of Interest and Adherence to Ethical Standards Authors Bryan A. Kutner, Michael P. Vaughn, Rebecca Giguere, Cristina Rodriguez-Hart, Karen McKinnon, Farnaz Kaighobadi, Bimbla Felix, Attisso Akakpo, Francine Cournos, Matt Mikaelian, Justin Knox, Daria Boccher-Lattimore, Kimbirly A. Mack, Marian LaForest, and Theodorus G. M. Sandfort declare that they have no conflict of interest. All procedures, including the informed consent process, were conducted in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, as revised in 2000.

Authors’ Contributions Bryan Andrew Kutner, PhD, MPH (Conceptualization: Lead; Data curation: Lead; Formal analysis: Lead; Methodology: Equal; Project administration: Lead; Writing – original draft: Lead; Writing – review & editing: Lead), Michael P Vaughn (Formal analysis: Supporting; Writing – original draft: Supporting; Writing – review & editing: Supporting), Rebecca Giguere (Formal analysis: Supporting; Project administration: Supporting; Writing – original draft: Supporting; Writing – review & editing: Supporting), Cristina Rodriguez-Hart (Formal analysis: Supporting; Funding acquisition: Supporting; Writing – original draft: Supporting; Writing – review & editing: Supporting), Karen McKinnon (Conceptualization: Supporting; Formal analysis: Supporting; Funding acquisition: Supporting; Writing – original draft: Supporting; Writing – review & editing: Supporting), Farnaz Kaighobadi (Formal analysis: Supporting; Writing – review & editing: Supporting), Bimbla Felix (Formal analysis: Supporting; Writing – review & editing: Supporting), Attisso Akakpo (Formal analysis: Supporting; Writing – review & editing: Supporting), Francine Cournos (Formal analysis: Supporting; Writing – review & editing: Supporting), Matt Mikaelian (Formal analysis: Supporting; Writing – review & editing: Supporting), Justin Knox (Formal analysis: Supporting; Writing – review & editing: Supporting), Daria Boccher-Lattimore (Formal analysis: Supporting; Writing – review & editing: Supporting), Kimbirly Mack (Formal analysis: Supporting; Writing – review & editing: Supporting), Marian LaForest (Conceptualization: Supporting; Data curation: Supporting; Supervision: Supporting; Writing – review & editing: Supporting), and Theodorus G. M. Sandfort (Conceptualization: Supporting; Formal analysis: Supporting; Funding acquisition: Lead; Methodology: Supporting; Project administration: Supporting; Resources: Lead; Supervision: Lead; Writing – original draft: Supporting; Writing – review & editing: Supporting)

Transparency Statement This study was not formally registered. The analysis plan was not formally pre-registered. Data from this study are not available in a public archive but will be made available by emailing the corresponding author. There is no analytic code associated with this study. Some of the materials used to conduct the study are available through the Human Behaviour Change Project (https://www.humanbehaviourchange.org/), specifically the Behaviour Change Techniques Taxonomy Online (https://www.bct-taxonomy.com/) and the Theory and Techniques Tool for linking BCTs to MOAs (https://theoryandtechniquetool.humanbehaviourchange.org/).

Data Availability

This study was not formally registered. The analysis plan was not formally pre-registered. Data from this study are not available in a public archive but will be made available by emailing the corresponding author. There is no analytic code associated with this study. Materials used to conduct the study are not publicly available.