Development and Validation of an Updated Internalized HIV Stigma Scale among People Living with HIV

Abstract

HIV-related internalized stigma consistently predicts poorer mental health, well-being, and sub-optimal HIV outcomes, such as poorer antiretroviral (ART) adherence. However, existing measures of HIV-related internalized stigma are likely outdated and may not capture the current reality and concerns that people living with HIV have. This study aimed to develop and validate an updated and more acceptable tool for assessing HIV-related internalized stigma. The study included 249 people living with HIV who were participants of the multi-site cohort study Multicenter AIDS Cohort (MACS)-Women’s Interagency HIV study (WIHS) Combined Cohort Study (MWCCS) in the United States. Participants completed a new internalized stigma scale, namely HIV-related Internalized Stigma Scale (HISS), which was developed with extensive input from people living with HIV. Following this, the dataset was combined with contemporaneous data collected for the MWCCS annual visit, which included an older measure of internalized stigma (the negative self-image subscale of the HIV Stigma Scale-HSS), ART adherence, and mental health variables (i.e., depression and post-traumatic stress disorder [PTSD] severity). Bayesian confirmatory factor analysis indicated that the 8-item new HISS yields a single factor solution, with factor loadings ranging from .68 to .94. Significant associations were found between the HISS and the negative self-image subscale of the HSS, ART adherence, depression, and PTSD severity, providing evidence of its concurrent and convergent validity. The HISS, as an updated, reliable, and valid tool, can be useful for ongoing research in the fields of epidemiology and psychology, as well as for evaluating interventions aimed at addressing HIV-related internalized stigma.

Introduction

HIV-related stigma refers to negative attitudes, beliefs, and discriminatory behaviors toward people living with HIV, which may lead to devaluing, prejudice, social exclusion, rejection, and even physical violence.^1,2 Multiple dimensions of HIV-related stigma have been examined, including internalized, experienced, anticipated, and perceived community stigma.^3,4 Existing research suggests that experiences of stigma not only affect the quality of life of people living with HIV, but are also related to HIV-related health behaviors and poor outcomes, which compromise HIV treatment and prevention efforts.^5–10 People living with HIV who report elevated levels of HIV stigma have lower access to medical care,⁵ lower ART adherence,^11–14 lower utilization of HIV care,^15,16 lower likelihood of viral suppression,^12,17–20 and are more likely to experience adverse mental health outcomes (e.g., depressive symptoms)¹².

Societal stigma is a significant social problem and may fuel internalized HIV-related stigma.²¹, which has been found to be highly predictive of HIV health behaviors and outcomes.^16,22,23 Internalized stigma refers to the phenomenon that occurs when people living with HIV endorse and accept negative attitudes about having HIV held in the community, and believe and apply these negative attitudes to themselves.²⁴ This specific dimension of HIV-related internalized stigma has been shown to be a more robust predictor of sub-optimal HIV health behaviors and outcomes compared to other dimensions of HIV stigma (e.g., anticipated stigma, experienced, or perceived community stigma).^3,4 It is consistently found to be associated with poorer HIV outcomes across the care cascade,^13,14 including lower access to medical care,⁵ lower utilization of HIV care,^15,16 poorer antiretroviral therapy (ART) adherence,^{11,23,25–27} suboptimal retention in care,²⁸ reduced likelihood of ART initiation,²⁹ and lower rates of viral suppression,^17–19 as well as poorer neurocognitive functioning.³⁰ Research also suggests that internalized stigma may mediate the effects of other stigma dimensions on health outcomes.^31,32 For example, one study found that internalized stigma mediates the effects of enacted and perceived stigma on quality-of-life outcomes, such as disclosure concerns.³³ Another study suggested that perceived community stigma is associated with internalized stigma, which in turn is associated with anticipated community stigma, resulting in lower odds of ART adherence.³¹ Moreover, internalized stigma, followed by anticipated stigma, mediates the association between experienced stigma in healthcare settings and trust in physicians,¹⁸ which has been shown to be a predictor of ART nonadherence.^34,35 Thus, understanding and accurately measuring HIV-related internalized stigma is instrumental for elucidating how it impacts physical and psychological well-being of people living with HIV.

Multiple scales have been used to empirically assess HIV-related internalized stigma over the course of the HIV epidemic.^10,36–44 Among these, the negative-self subscale of the HIV Stigma Scale (HSS) developed by Berger and colleagues³⁶ and published in 2001 has emerged as the most extensively utilized measurement tool and previous research has established that it is a valid and reliable measure of HIV-related internalized stigma.^{12,31,37,45,46} For example, in their systematic review, Van der Kooji et al.¹⁰ report that the HIV Stigma Scale³⁶ was the most frequently used measure of HIV-related internalized stigma, being employed in 59 studies (33.5%)—substantially more than any other identified scale. Currently available measures of HIV-related internalized stigma, including the negative-self subscale of the HSS, may be outdated and not capture the reality and concerns of people living with HIV in the current era of treatment and prevention. For example, some items (e.g., “Having HIV makes me feel unclean” and “Having HIV in my body is disgusting to me,”) may be considered offensive and stigmatizing by some people living with HIV. Given that stigma is a highly negative experience, it is crucial to capture this very real experience in research and evaluation, but without causing unnecessary distress in participants. Additionally, several concerns about the existing measures (e.g., stigmatizing language of the items) have been raised and discussed by researchers and healthcare providers working with people living with HIV, including those who are members of the National Community Advisory Board (NCAB) for the Multicenter AIDS Cohort (MACS)-Women’s Interagency HIV study (WIHS) Combined Cohort Study (MWCCS). This highlights the need to revisit the existing measures and to develop and validate a newly updated measure of internalized HIV-related stigma.

The current study aimed to develop an updated and comprehensive measure that people living with HIV find more acceptable, has good reliability and validity, and is predictive of HIV health outcomes. In addition, having a psychometrically sound and up-to-date measure is critical for evaluating the effectiveness of interventions designed to reduce internalized HIV stigma. Creating such a measure will contribute to efforts to assess HIV-related internalized stigma, capturing its theoretically and clinically relevant aspects for use by researchers and health care providers. We developed the HIV-related Internalized Stigma Scale (HISS) by extracting, evaluating, and adapting items from existing scales or subscales of HIV-internalized stigma. Consistent with Stutterheim and Ratcliffe⁴⁷, we recognize that qualitative research is essential for understanding the complexity of stigmatization and ensuring that research addresses relevant aspects of stigma. Thus, we also used qualitative data from existing studies and projects conducted with people living with HIV and socio-demographically matched participants living without HIV, as well as items assessing stigma in other fields, to create new items. We then refined these items through cognitive interviews and subsequently evaluated the scale’s psychometric properties (including validity and internal consistency) in a larger survey sample.

Methods

Scale Development

Development Phase of HISS Items

An initial large pool of items was created. This process was informed by identifying specific theoretical/conceptual elements of HIV-related internalized stigma. To identify constructs, potential items, and wording related to internalized HIV-related stigma, we examined existing scales of HIV-related internalized stigma (see Table 1 for the list of the existing scales). We also analyzed qualitative data obtained from in-depth interviews and focus groups conducted with people living with HIV across several projects (e.g., Women’s Adherence and Visit Engagement [WAVE] study, NCT00000797; Integrating ENGagement and Adherence Goals upon Entry [iENGAGE] to Control HIV, NCT01900236; and Adapting the Finding Respect and Ending Stigma Around HIV [FRESH] Intervention, NCT04491539) or other studies conducted with people living with HIV.^48–56 Appendix A presents the coding tree derived from secondary data analyses that informed item generation. Specifically, we listed different subconstructs related to HIV-related internalized stigma with possible items assessing each subconstruct separately. The subconstructs included shame/embarrassment, bad person, inferiority/incompetence, worthless/unworthiness, unclean, disgusting, guilt/self-blame, blemished/stained, disappointed, different, and not lovable. Next, we selected or adapted items using theoretical frameworks of HIV-related internalized stigma and existing scales by taking into consideration the following: (a) items that have been shown to work well in previous research (those having higher factor loadings) and (b) items that can adequately reflect and capture both intrapersonal and interpersonal perceptions and experiences of internalized stigma, including diverse cultural and socio-demographic perspectives. Consequently, we developed an initial pool of 13 items representing these 11 different subconstructs (see Appendix B).

Table 1.

List of Internalized Stigma Scales Reviewed

Author	Scale
Berger et al.36; Bunn et al.37	HIV Stigma Scale (Negative self-image subscale)
Phillips et al.41	Internalized Stigma of AIDS Tool
Kalichman et al. 39	Internalized AIDS-Related Stigma Scale
Sayles et al.42	Multidimensional Measure of Internalized HIV Stigma
Molina, & Ramirez-Valles40	HIV/AIDS Stigma Scale (Internalized HIV/AIDS stigma subscale)
Hernansaiz-Garrido, & Alonso-Tapia38	HIV Internalized Stigma Scale (HIV-ISS)
Zelaya et al.71	HIV/AIDS Stigma Scale (Self [internal] stigma)
Dos Santos et al.72	People living with HIV Stigma Index (Internalised stigma)
Kingori et al.73	Felt Stigma Questionnaire
Holzemer et al.74	HIV/AIDS Stigma Instrument—PLWA (HASI-P) (Negative self-perception subscale)
Visser et al.75	Parallel Scales to Measure HIV-Related Stigma (Internalized stigma subscale)
Mak, & Cheung76	Self-Stigma Scale (Cognitive subscale)
Jimenez et al.77	HIV Felt-Stigma Scale (Negative self-image subscale)
Hasan et al.78	Internalized HIV/AIDS-related Stigma Scale
Pantelic et al.79	HIV-stigma Scale for Adolescents Living with HIV (ALHIV-SS) (Internalized stigma subscale)

Cognitive Interviews

Following the development of the initial pool of items of the HISS, cognitive interviews were conducted with 16 people living with HIV to obtain feedback on items as a means of further refining and modifying selected items. Cognitive interviews⁵⁷ are grounded in principles of cognitive psychology and serve as a means to investigate the cognitive processes individuals employ when they respond to questionnaire items. These cognitive processes encompass several aspects, including understanding the item’s meaning, recalling pertinent information from memory, decision-making processes (such as the desire for an accurate response and the influence of social desirability), and the process of formulating a response. The objective is to enhance and refine the wording of questionnaire items using this information. This iterative process aims to ascertain the acceptability of the items, identify items that should be dropped, eliminate potential alternative interpretations, facilitate effective retrieval of information, and promote accurate responses from the participants.

The study team recruited 16 participants: eight from Baltimore/ Washington, DC site and eight from the Birmingham MWCCS site. Cognitive interviews were conducted by three interviewers and employed think-aloud and verbal probing techniques to gain feedback on how participants would respond to each item, the utility of the items in accurately reflecting participants’ perspectives, and the clarity of instructions/questions to potentially identify items to be added, modified, or removed. This think-aloud process allowed the researchers to gain insight into how participants interpreted items. Specific probes allowed interviewers to tap into participants’ thoughts and feelings associated with each question.

All interviews were audio recorded with consent and professionally transcribed verbatim. Transcripts were reviewed for accuracy by the study team. Coding and thematic organization were conducted manually by the three interviewers using an inductive approach. Members of the study team independently reviewed and annotated the transcripts, extracting and organizing input shared during the interviews into topics based on the interview guide (see Appendix C). These included clarity of instructions and items, interpretation and emotional reactions to items, items with negative emotional reactions, comments on alternative items, and suggestions for alternative wording. After independent coding, the three interviewers met to compare interpretations, resolve discrepancies through discussion, and reach consensus on coding decisions. The research team then met to review these coded topics, discuss emerging themes about the instructions, scale, and items, and agree on necessary modifications.

Revised HISS Item Pool and Scale Modification

Based on the insights generated through this iterative review and consensus process, the initial HISS scale explored during the cognitive interviews consisted of 13 questions for which responses were given using a five-point scale ranging from 1 = Very untrue of me, 2 = Somewhat untrue of me, 3 = Neither true nor untrue of me, 4 = True of me and 5 = Very true of me. Two items included an “alternative” way of exploring the construct, and participants were asked to provide feedback on the alternatives and preferences among those presented. All items were negatively worded for uniformity and ease of understanding (e.g., “I feel ashamed of having HIV” instead of “I never feel ashamed of having HIV/AIDS”), with higher scores indicating higher HIV-related internalized stigma.

In cognitive interviews, some participants felt that the original five-point Likert response option did not fully capture their perspectives. Based on their feedback and after study team discussion, the response options were expanded to a seven-point Likert scale that included the following: 1 = Very untrue of me; 2 = Untrue of me; 3 = Somewhat untrue of me; 4 = Neutral; 5 = Somewhat true of me; 6 = True of me; 7 = Very true of me. This type of Likert scale is considered best practice to maximize the discriminative power of items without affecting reliability.⁵⁸ Items that elicited strong negative emotions from participants (e.g., items felt to be inappropriate or stigmatizing, such as “Having HIV makes me feel unclean”) or items that participants felt were redundant (e.g., “I feel worthless because I am living with HIV” and “I feel inferior because I am living with HIV”, which overlapped with “Having HIV makes me feel that I am not as good as others”) were removed or modified (see Appendix B for the full list of 13 items with excluded items).

The team also adjusted item wording for clarity and rearranged the order of items to reduce the likelihood of participants feeling the scale was burdensome or emotionally taxing—for example, placing an item like “I feel ashamed of having HIV” near the beginning and “I feel I don’t deserve to be loved because I am living with HIV” toward the end, spacing emotionally charged items apart. Beyond the cognitive interview feedback, we revisited the broader initial item pool to ensure that all relevant concepts were represented. During this review, we added one item (“I feel I don’t deserve to be loved because I am living with HIV”) to better capture the ‘not lovable’ construct. The revised scale was also presented to the MWCCS NCAB to obtain their feedback on the acceptability of the scale.

These steps—using coded interview insights, refining item content, re-ordering for participant burden, and adding overlooked constructs—constituted an iterative feedback loop between data, participant perspectives, and expert review, resulting in the final revised HISS. The current revised HISS consists of eight items (see Table 3), which were designed to tap into different sub-constructs associated with HIV-related internalized stigma: shame/embarrassment, inferiority/incompetence, worthless/unworthiness, guilt/self-blame, blemished/tainted/stained, disappointed, different, and not lovable.

Table 3.

HISS factor loadings and Item Response Theory (IRT) parameters

Items		λ	a	d1	d2	d3	d4	d5	d6
1.	I feel out of place because I am living with HIV.	0.835	1.517	−0.199	0.191	0.401	0.661	1.287	1.667
2.	I think less of myself because I am living with HIV.	0.919	2.334	0.147	0.581	0.718	0.965	1.457	1.776
3.	Having HIV makes me feel that I am not as good as others.	0.940	2.765	0.197	0.569	0.769	0.993	1.556	1.825
4.	I feel blemished or stained for having HIV.	0.682	0.933	−0.373	0.006	0.222	0.561	1.150	1.568
5.	I feel disappointed in myself for having HIV.	0.872	1.785	−0.038	0.354	0.533	0.812	1.292	1.658
6.	I feel ashamed of having HIV.	0.872	1.784	−0.152	0.182	0.425	0.618	1.122	1.513
7.	I blame myself for having HIV.	0.919	2.331	0.021	0.463	0.598	0.893	1.294	1.503
8.	I feel I don’t deserve to be loved because I am living with HIV.	0.751	1.137	0.661	1.341	1.503	1.952	2.575	2.873

λ: Standardized factor loading; a: Probit discrimination parameter; d1-d6: Probit location parameters.

Participants and Procedure (Survey Phase)

Participants in the survey included 249 people living with HIV from the multi-site cohort study⁵⁹ (i.e., MWCCS). Specifically, participants from four MWCCS sites located in San Francisco, California (N=84); Birmingham, Alabama (N=57); Jackson, Mississippi (N=87); and Baltimore/Washington, DC (N=21) completed the newly developed internalized stigma scale (i.e., the HISS) in October 2021-September 2022. Subsequently, this dataset was merged with the broader MWCCS dataset, which included established measures of internalized stigma (the negative self-image subscale of the HSS³⁷), ART adherence, and mental health variables (i.e., depression and PTSD). This study was approved by the MWCCS Executive Committee, the MWCCS NCAB, and the Institutional Review Board at each site.

Measures

HIV-related Internalized Stigma Scale (HISS)

The HISS was developed within the current study to assess HIV-related internalized stigma. Items are rated on a 7-point Likert-type scale, ranging from 1 (Very untrue of me) to 7 (Very true of me). Sample items are, “I think less of myself because I am living with HIV” and “I feel out of place because I am living with HIV.” Higher scores indicate higher levels of internalized stigma. Psychometric properties of the HISS are presented in detail in the results section.

HIV Stigma Scale (HSS)-Negative Self-image

Internalized HIV-related stigma was also assessed with the negative self-image subscale of the widely used HSS.^36,37 Participants responded to the 7 items of this sub-scale, which includes such items as “Having HIV/AIDS is disgusting to me” and “Having HIV/AIDS makes me feel unclean.” Items are rated on a 4- point Likert-type scale with response options including 1=Strongly disagree, 2=Disagree, 3=Agree, and 4=Strongly agree. Higher scores indicate higher internalized stigma. In the current study, Cronbach’s alpha was .84.

HIV-related Health Outcomes

ART Adherence

ART adherence was assessed through a 3-item self-report measure.⁶⁰ The items assess, 1) the number of days in the last 30 days on which participants missed at least one dose of their medication; 2) how often participants took their medication as prescribed with five ordered response options including never, rarely, sometimes, usually, almost always, and always; 3) participants ability to adhere to ART regimen over the past 30 days, rated on a 5-point scale including very poor, poor, fair, good, very good, or excellent. The item scores were transformed to a 0–100 scale, with zero as the poorest adherence, and 100 a perfect adherence level.⁶⁰

Mental Health Outcomes

Depression

The Center for Epidemiologic Studies Depression Scale (CESD)⁶¹ is used to measure depressive symptoms in the MWCCS. The CESD has 20 items rated on a 4-point Likert-type scale (0=Rarely or none of the time [less than one day]), 3=Most or all of the time [5–7 days]), with higher scores indicating greater depressive symptoms. In the current study, Cronbach’s alpha was .91.

Post-traumatic Stress Disorder Symptoms

Post-traumatic Stress Disorder Checklist: Civilian Scale (PCL-C)^62,63 was used to measure symptoms of PTSD. The PCL-C has 17 items rated on a 5-point Likert-type scale ranging from 1 (Not at all) to 5 (Extremely) with higher scores indicating greater PTSD symptoms. Sample items are “repeated, disturbing memories, thoughts, or images of a stressful experience from the past” and “feeling very upset when something reminded you of a stressful experience from the past.”

Statistical Analyses

We used Rstudio 2023.06.1 (packages included psych, and BayesFactor) and Mplus 7.2⁶⁴ to calculate descriptive statistics and examine the psychometric properties of the HISS. We examined factorial validity using a confirmatory factor analysis (CFA) combined with the inspection of eigenvalues greater than 1. Because we theorized a 1-factor model, we expected to find only one eigenvalue greater than 1. We used Bayesian estimation to perform the CFA (BCFA) and address data sparseness, which manifested as some of the cells in the bivariate matrix of the HISS items being empty or had low count. Data sparseness negatively affects model convergence, estimation accuracy, and might result in residual items’ covariance independent of underlying latent factors. Therefore, we used Bayesian estimation which has been shown to be robust against data sparseness.⁶⁵

BCFA with ordinal variables uses polychoric correlation matrix to estimate factor loadings and Item Response Theory (IRT) parameters, which consist of probit coefficients. IRT parameters include discrimination and location. Discrimination parameters quantify the accuracy of a given item to distinguish between individuals with different levels of the underlying construct (or trait, in line with the IRT framework). Optimal values for probit coefficients are less clearly defined, given their relatively less common use. As a practical approximation, probit discrimination coefficients ranging from 0.6 to 2.8 can be regarded as acceptable or optimal. This determination is based on the widely employed conversion approach, which applies a constant of 1.645 to convert between probit and logit parameters, where probit ≈ logit/1.645. Location parameters correspond to the underlying trait level above which the probability of endorsing a given item is greater than 50%. Standardized factor loadings express the strength of association between a given item and an underlying latent factor. They range between |0| and |1|, with values ≥ 0.7 being considered acceptable.

BCFA model goodness of fit is based on the Posterior Predictive P-value (PPP). The posterior distribution of the model parameters based on the observed data is used to generate new data, which in turn are compared to the actual observed data. PPP corresponds to the probability of obtaining the observed data under the model’s assumptions. A PPP lower than 0.05 suggests poorer model fit, while values greater than 0.05 and closer to 0.5 indicate acceptable to good model fit.

We calculated HISS internal reliability using Cronbach’s alpha and marginal reliability, with values greater than 0.8 indicating good internal reliability. Unlike Cronbach’s alpha, marginal reliability estimates internal reliability based on items’ communality (i.e., excluding uniqueness or error variance). Concurrent and convergent validity were examined using correlation coefficients, based on a Bayesian approach with 4 chains and 10000 iterations. We used the HSS-negative self-image to test the HISS’s concurrent validity, which determines how well a new scale compares to a well-established test. Convergent validity testing, which examines the degree to which two measures of the same construct are related and whether the new measure accurately captures the concept, involved CESD and PTSD variables. We tested the association between HISS and ART adherence using a negative binomial regression model, because ART adherence variable was not normally distributed and showed significant overdispersion. We estimated the correlation between the two variables calculating the square root of the negative binomial R² model.

Results

Descriptive Demographics

Descriptive statistics for the sample are presented in Table 2. Mean age was 50.6 (SD=11.45), ranging from 29 to 84. Of participants, 158 (63.5%) were female and 91 (36.5%) were male. The majority (68%) identified as Black or African American and 5% were Hispanic. 59% reported being straight/heterosexual, and 23% being gay. 32% reported some college, and 23% high school as their highest level of education. 17% reported an average annual income of $12,000 or less. Nearly 80% were virally suppressed, and 98.4% were on ART, with the majority (86%) being adherent to ART.

Table 2.

Descriptive statistics (N=249)

	N / M	% / SD
Age, years	50.6	11.5
Site
San Francisco, California	84	33.7
Birmingham, Alabama	57	22.9
Jackson, Mississippi	87	34.9
Baltimore, Maryland	21	8.4
Race
Black or African American	169	67.9
White	60	24.1
Other	20	8
Ethnicity
Hispanic	12	4.8
Non-Hispanic	237	95.2
Gender
Male	84	33.7
Female	156	62.7
Transgender man	1	0.4
Transgender woman	5	2.0
Gender queer/gender fluid	2	0.8
An identity not listed	1	0.4
Sexual orientation
Straight/heterosexual	147	59.0
Lesbian	2	0.8
Gay	57	22.9
Bisexual	24	9.6
Queer	2	0.8
Questioning	2	0.8
Asexual	2	0.8
An orientation not listed	2	0.8
Pansexual	2	0.8
Education level
Less than high school	52	20.9
Completed high school	57	22.9
Some college	80	32.1
Four-year college degree	33	13.3
Attended/completed graduate school	26	10.4
Insurance
No	16	6.4
Yes	216	86.7
No answer	17	6.9
Average annual income ($)
<=6000	26	10.4
6001–12,000	41	16.5
12,001–18,000	42	16.9
18,001–24,000	20	8.0
24,001–30,000	23	9.2
30,001–36,000	19	7.6
36,001–75,000	38	15.3
75,001–100,000	12	4.8
100,001–150,000	3	1.2
150,001–200,000	7	2.8
>=200,000	5	2.0

Psychometric properties of the HISS

Factorial validity:

Based on assessment of the eigenvalues, only one factor had an eigenvalue greater than 1, specifically 5.79 explaining 72.32% of the total variance. We tested the fit of the 1-factor model (four chains and up to 100000 iterations). The PPP of the 1-factor model was 0.002, suggesting poor model fit. PPP is sensitive to model misspecification, including residual covariances, which in CFA models is constrained to zero. Some items may show small residual covariance due to, for example, wording effect or data sparseness. Through BCFA we set items’ covariance to approximately zero (instead of exactly 0) using Inverse Wishart prior distributions. This improved the model fit significantly: PPP = 0.28. We found significant though small residual correlations between item 1 and 8 (r=−0.17, p=0.01); item 2 with Item 4 (r=−.27, p<0.001) and with Item 6 (r=−.20, p=0.006); and Item 4 and 6 (r=.29, p<0.001).

In Table 3, we report 1-factor standardized loadings and IRT parameters for the final items included in the HISS. Factor loadings ranged from 0.682 to 0.940, with most of them being above 0.8. The discrimination parameters indicated that all items accurately differentiated participants with lower vs. higher levels of internalized stigma. Most of the location parameters were greater than or around 0 (i.e., the sample mean). The locations of the first response option “Very untrue of me” (see column “d1” in Table 3) were close to 0, with three items showing negative values. This indicates that the HISS items accurately estimate average or high levels of internalized stigma, rather than low construct levels (e.g., values one or two standard deviations below 0).

Since the latent trait underling the HISS items is a normally distributed, standardized latent variable, 0 represents the sample mean of HIV-related internalized stigma and the standard deviation is 1. Following the normal distribution theory, the majority of the scores will range from −3 SD below the mean and + 3 SD above the mean. Within this range, the HISS provides a total of 95% of the total information, with most of it ranging between 0 and +3 (72%). Between −3 and 0, the amount of information produced corresponds to 23% (see Figure 1). With Cronbach’s alpha 0.91 and marginal reliability 0.83, the HISS demonstrated optimal internal reliability.

Figure 1.

Test Information Plot

Theta: Standardized latent trait score with M=0 and SD=1.

Validity of the HISS

Concurrent validity:

the correlation between HISS and HSS (r=.78, p<.001) suggested good concurrent validity. Convergent validity: As expected, HISS was significantly associated with both depression symptom severity (CESD) and PTSD (see Table 4). All variables were normally distributed (Skewness and Kurtosis <|1|), except for ART adherence. HISS internalized stigma was also significantly inversely correlated with ART adherence. The results in Table 4 support the convergent validity of the HISS.

Table 4.

Correlation coefficients of the HISS with validation measures

Variables	HISS	M/SD[range]
HISS	-	12.55/12.26[0–48]
HSS-internalized stigma	.78***	5.22/4.44 [0–20]
ART adherence	−.22**	99.5[0–100]●
Depression symptom severity	.37***	12.43/10.55[0–50]
PTSD severity	.36***	25.24/18.90[0–83.42]

^**p < .01

^***p < .001.

HSS: HIV Stigma Scale, HISS: HIV-related Internalized Stigma Scale, ART: Antiretroviral therapy, PTSD: Post-traumatic stress disorder;

^●because of ART adherence non-normal distribution, we reported only the median and range.

DISCUSSION

HIV-related internalized stigma is a significant public health concern that diminishes treatment, care, and quality of life for people living with HIV. Ongoing measurement of HIV-related internalized stigma is important as the HIV epidemic continues and evolves in relation to new developments in treatment and prevention, as well as changes in the populations most affected by HIV. However, most existing measures of HIV-related internalized stigma were developed over 20 years ago and may not represent the experiences of people living with HIV today. Many of these measures include items that may be outdated, offensive, or stigmatizing, potentially limiting their relevance and accuracy in capturing contemporary experiences of internalized stigma. We developed a new 8-item measure, the HISS, specifically for measuring HIV-related internalized stigma among people living with HIV. The HISS was developed based on extensive quantitative and qualitative formative research related to experiences of HIV-related internalized stigma. This stage of measure development was followed by community input using cognitive interviews with 16 people living with HIV. Finally, the scale was tested with 249 individuals in the MWCCS located in four geographically distinct sites to assess its factor structure, reliability, and validity. Feedback was also solicited from the MWCCS NCAB for acceptability of implementation among study participants.

Psychometric assessment of the HISS supports an 8-item, 1-factor measure of HIV-related internalized stigma that will be particularly useful for discerning individuals with moderate to higher levels of internalized stigma. The items were designed to assess multiple subconstructs of HIV-related internalized stigma; they were internally consistent and demonstrated very good reliability. The new HISS scale correlated highly with the previously validated measure of internalized stigma^36,37 that has been widely used, suggesting good concurrent validity; both scales are measuring the same underlying construct of HIV-related internalized stigma. Additionally, convergent validity for the HISS scale was assessed by examining its relationships with mental health including depression and trauma-related symptoms. Both depression and trauma symptoms were positively and significantly related to HIV-related internalized stigma as measured by the HISS. Relationships were also assessed with HIV outcome variables, previously shown to be associated with internalized HIV stigma, including ART adherence. The HISS was inversely related to optimal ART adherence as would be expected based on prior research on HIV-related internalized stigma and HIV health behaviors.⁴

A key strength of this study lies in its thorough process of literature review and community input. Beyond simply applying standard psychometric methods, we integrated evidence from prior quantitative measures with qualitative insights from people living with HIV and incorporated iterative community feedback through cognitive interviews and NCAB review. This approach ensured that the final set of items were conceptually grounded, culturally relevant, and acceptable to those most affected. Moreover, testing the scale across four geographically distinct MWCCS sites added diversity to the sample and strengthened the generalizability of the findings. These features distinguish the HISS from prior measures and highlight its potential as a robust and relevant tool for future research and practice.

Beyond its psychometric strengths, the HISS also has important practical utility. An updated and acceptable measure of internalized HIV stigma is critical for evaluating the effectiveness of stigma-reduction interventions. For example, interventions such as RESilience and Empowerment Training (RESET⁶⁶) in the Netherlands, Positive, Open, Proud (POP⁶⁷) for women in the United States, and ‘integrating ENGagement and Adherence Goals upon Entry’ (iENGAGE⁶⁸; originally developed for promoting ART adherence) have targeted internalized stigma and would benefit from sensitive measures like the HISS to capture changes over time. Moreover, although the HISS items are specific to HIV, the approach we used to update and refine this scale—integrating quantitative evidence, qualitative data, and community-partnered feedback—can be adapted for other stigmatized conditions, such as mental health, where interventions like the Honest, Open, Proud (HOP)-based intervention⁶⁹ have been evaluated. Using similar participatory methods to develop measures that reflect current stereotypes and lived experiences can advance effect evaluations across different fields of internalized stigma.

The results of this study should be interpreted in light of some limitations. The sample was a convenience sample of people living with HIV engaged in the MWCCS. Individuals in the MWCCS tend to be older, have been living with HIV for longer periods, live in metropolitan areas with more access to HIV resources, and are familiar with stigma measures. Their experiences of HIV-related internalized stigma may not represent those who are younger, newly diagnosed, or otherwise marginalized and less engaged in HIV-related research and care. We urge future researchers and clinicians to use the HISS to further assess its validity and acceptability in other populations affected by HIV. Additionally, detailed demographic information beyond site location was not collected for the 16 cognitive interview participants, which limits our ability to fully describe the diversity of perspectives represented. Furthermore, the present analyses are cross sectional and thus assessment of predictive validity is constrained by time. Future longitudinal work will assess the relationships between the HISS and HIV health outcomes over time to build upon this study. Lastly, the HISS items were refined in terms of wording and format to ensure they were interpreted as intended and answered accurately by respondents, based on cognitive interviews. While cognitive interviewing is a valuable method for developing and refining survey items, it alone may not be sufficient to ensure that a new scale is acceptable, valid, and reliable.

Following the formal psychometric evaluation of the HISS, the measure was proposed for use in the MWCCS for longitudinal measurement of HIV-related internalized stigma. The MWCCS NCAB’s response was mixed with respect to feedback on the measure and recommendation for future use. Some members voiced that the measure remained potentially stigmatizing and raised concerns that implementing it may adversely impact study participants. Other members shared that they felt that items reflected experiences of internalized HIV stigma and that it is important to explore, given that many people living with HIV do experience internalized stigma and that those who are newly diagnosed or those with less social support may be most impacted. Still other NCAB members elaborated that they felt the measure did not capture the full experience of internalized stigma, including how they have coped with, built resilience to, and moved through feelings of internalized HIV stigma over time. This mixed feedback reflects a broader, ongoing discussion in the HIV field: some community members and researchers express concern that asking about stigma might inadvertently reinforce or re-surface stigma-related distress, while others emphasize that not asking about stigma does not diminish its presence and may limit opportunities for intervention and normalization. In developing and proposing the HISS, we navigated this tension by carefully reviewing item wording, incorporating community feedback, and balancing the need to document stigma experiences with efforts to minimize participant burden or harm.

Ultimately, while the scale was found to be psychometrically reliable and valid, acknowledging these noted limitations of the current research, further work centering people living with HIV participant experience and using community-engaged participatory processes can be conducted to refine the scale to further improve its acceptability and relevance. This effort may also include the development of comprehensive internalized stigma measures that are inclusive of resilience and resistance to HIV stigma for a strengths-based approach that reflects the fullness of the lived experience. Other person-centered considerations when implementing stigma-based measures in research include ensuring that supportive resources are in place for individuals who are adversely affected during the research process and that there are opportunities that allow for candid and on-going participant feedback about stigma scales and their impact to further advance the development of robust person-centered measures and tools.

The need for an updated, relevant, and appropriate measure of HIV-related internalized stigma and examination of the mechanisms through which it exerts its effects is essential as the HIV epidemic evolves. Increasingly our understanding of HIV-related internalized stigma is shifting from one of individual psychopathology to one that is informed by the social and structural causes of stigma and discrimination and the intersectionality of HIV-related stigma.⁷⁰ Even though the HISS is not specifically designed to capture these insights, it could be applied and even further refined in future research to study the impact of structural and multi-level stigma interventions on HIV-related internalized stigma. This research could assess how different groups internalize HIV stigma from an intersectional lens to facilitate the adaptation and reach of future interventions for those who need them most.

Conclusions

The HIV-related Internalized Stigma Scale, the HISS, is an updated, reliable, and valid instrument for use in ongoing research on epidemiology, psychology, as well as assessment of interventions addressing HIV stigma. The measure was developed through a thorough review of existing HIV-related internalized stigma measures, qualitative work conducted with people living with HIV, and input derived from cognitive interviews with people living with HIV. The one-factor measure captures a range of experiences related to internalizing societal HIV stigma (e.g., shame, guilt/self-blame, disappointment) that continue to affect people living with HIV. HIV-related internalized stigma remains a major barrier to treatment, prevention, and quality of life for people living with HIV. The HISS can be used in future research and clinical practice to improve the assessment of HIV-related internalized stigma and to evaluate the effectiveness of multi-level interventions designed to reduce it. Moreover, the development process highlights the importance of participatory, community-partnered methods in creating and refining stigma measures, ensuring that they remain responsive to the lived experiences of diverse populations. Integrating these approaches can strengthen both research and practice aimed at reducing stigma and improving health outcomes for people living with HIV.

Appendix A. Coding Tree for Development of HISS Items

1. Shame / Embarrassment  • Qualitative evidence:   – “I feel ashamed of having HIV.”   – Quotes describing stigma as “a punishment,” “sinful,” or “yucky” (e.g., feeling ashamed of infection) [Jeffries et al.48, Logie et al.80, Lekas et al.56]  • Quantitative item sources:   – HIV Stigma Scale36: “I never feel ashamed of having HIV/AIDS” (reverse-coded)   – Internalized AIDS-Related Stigma Scale39: “I am ashamed that I am HIV positive.” 2. Bad Person  • Qualitative evidence:   – Feelings of being a “bad person” or morally flawed because of acquiring HIV.   – Participants in prior qualitative studies55,56,81 described HIV as tied to moral failure or wrongdoing, e.g., “I feel like a bad person for living with HIV” or “I believe that I am a bad person for getting HIV.”  • Quantitative item sources:   – Internalized AIDS-Related Stigma Scale39: “Having HIV/AIDS makes me feel that I’m a bad person” and “I feel like a bad person for living with HIV.”  • Coding outcome:   – Excluded from the final HISS because cognitive interview participants reported that this wording felt stigmatizing or overly harsh, even though it reflected a genuine theme in the secondary data. 3. Inferiority / Incompetence  • Qualitative evidence:   – Descriptions of being “damaged goods,” “less than human,” or “lifeless” [e.g., Wu et al.55; Tsai et al.82]   – “I feel like I am beneath other people.”  • Quantitative item sources:   – HIV Stigma Scale36: “Having HIV/AIDS makes me feel that I am not as good as others.”   – Multidimensional Measure of Internalized Stigma42: “I do not feel as capable as others because I have HIV/AIDS.” 4. Worthlessness / Unworthiness  • Qualitative evidence:   – “I felt completely worthless,” “I am diminished,” “I have low self-esteem” [Arrey et al.83]  • Quantitative item sources:   – Internalized AIDS-Related Stigma Scale39: “I sometimes feel worthless because I am HIV positive.” 5. Unclean  • Qualitative evidence:   – Participants in prior qualitative studies50,55 often linked HIV with feelings of being “dirty,” “contaminated,” or “unclean.”   – For example: “Having HIV/AIDS makes me feel unclean,” “Being HIV positive makes me feel dirty,” or “I feel contaminated inside because of HIV.”  • Quantitative item sources:   – HIV Stigma Scale36 included items like “Having HIV/AIDS makes me feel unclean.”  • Coding outcome:   – Excluded from the final HISS after cognitive interviews, as participants reported these statements as stigmatizing and evoking strong negative emotions despite being common in earlier measures. 6. Disgusting  • Qualitative evidence:   – Some individuals described internalized stigma as self-repulsion or disgust, e.g., “Having HIV/AIDS is disgusting to me,” “I am disgusted with myself for getting HIV,” or “I despise myself due to having HIV.”14,56  • Quantitative item sources:   – HIV Internalized Stigma Scale38 and similar instruments contained items like “Having HIV/AIDS is disgusting to me.”  • Coding outcome:   – Excluded from the final HISS because cognitive interview participants identified these items as emotionally harmful and excessively stigmatizing, leading to their removal despite their prevalence in prior scales. 7. Guilt / Self-Blame  • Qualitative evidence:   – “It’s my fault that I have HIV,” “I cannot forgive myself,” “self-blame” [Sadati et al.84]  • Quantitative item sources:   – HIV Stigma Scale36: “I feel guilty because I have HIV/AIDS.”   – HIV/AIDS Stigma Instrument—PLWA74: “I felt ashamed of having this disease.” (linked guilt/shame) 8. Blemished / Stained  • Qualitative evidence:   – Feeling “dirty,” “contaminated,” or “tainted” [Wu et al.55; Mo & Ng50]  • Quantitative item sources:   – Internalized Stigma of AIDS Tool41: “I feel blemished.”   – HIV Internalized Stigma Scale38: “I felt dirty or stained.” 9. Disappointed  • Qualitative evidence:   – “I am disappointed in myself for having HIV.” [Hosek et al.51]  • Quantitative item sources:   – HIV Internalized Stigma Scale38: “I felt disappointed with myself.” 10. Feeling Different  • Qualitative evidence:   – “I feel out of place in the world,” “I feel different from others,” “I’m an outsider” [Mutumba et al.85]  • Quantitative item sources:   – HIV Internalized Stigma Scale38: “I feel different from others because I have HIV.” 11. Not Lovable  • Qualitative evidence:   – “I thought I don’t deserve to be loved because of my HIV infection.” [Fongkaew et al.86]  • Quantitative item sources:   – HIV Internalized Stigma Scale38: “I thought I don’t deserve to be loved.” Coding outcome:   – Added after cognitive interviews. While reviewing feedback and revisiting the initial item pool, the team identified the importance of capturing feelings of being undeserving of love. This item was introduced post-interview to ensure the scale reflected this salient aspect of internalized stigma.

Appendix B. HIV-related Internalized Stigma Scale (HISS)

Instructions: Please read the following statements that people living with HIV have at times expressed about their experiences and feelings about HIV. We understand that these statements may or may not apply to you and are often tied to external discrimination and prejudice. Thinking about your experiences and feelings over the past year, how true are the following for you on average? 1 – Very untrue of me 2 – Untrue of me 3 – Somewhat untrue of me 4 – Neutral 5 – Somewhat true of me 6 – True of me 7 – Very true of me
	1 Very untrue of me	2 Untrue of me	3 Somewhat untrue of me	4 Neutral	5 Somewhat true of me	6 True of me	7 Very true of me
1. I feel ashamed of having HIV.
2. Having HIV makes me feel that I am not as good as others.
3. I think less of myself because I am living with HIV.
4. I blame myself for having HIV.
5. I feel blemished or stained for having HIV.
6. I feel disappointed in myself for having HIV.
7. I feel out of place because I am living with HIV.
8. I feel I don’t deserve to be loved because I am living with HIV.
Excluded items *I feel like a bad person for living with HIV. *Having HIV makes me feel unclean. *Having HIV is disgusting to me. **I feel worthless because I am living with HIV. **I feel inferior because I am living with HIV. (*) represents items excluded due to stigmatizing wording and the associated negative emotions reported by participants, while (**) represents items that participants felt were redundant.

Appendix C. Cognitive Interview Guide

Thank you again for your interest in helping us to develop a questionnaire to identify the personal experiences and feelings that may result from social stigma, discrimination, and stereotyping related to HIV. During this conversation we will review with you a draft version of the questionnaire. We will ask you for your feedback and opinion about the individual questions. Your input will help us finalize the questionnaire so that we can test it among a larger group of people living with HIV.

In the next hour we will: 1) Review the consent form and begin recording; 2) Review the current draft of the questionnaire to gather your feedback and thoughts.

Step 1: Obtain Consent

At this time, I will go over the consent form with you and document your verbal consent.

[Go through the consent form and obtain verbal consent on last page. If someone no longer wants to participate simply end the interview by thanking them for their time]

Step 2: Begin Question Probing

Thank you. I will now begin recording and reviewing the questionnaire with you. Please confirm that your consent to record. This will be a “think aloud” process, meaning that I welcome you to provide your thoughts or opinions about each question as I go through them with you. There are no wrong thoughts or opinions – so please offer your thoughts freely.

Do you have any questions before we get started?

At this time, please read through the instructions at the top of the questionnaire and tell me when you are done.

• What you think the instructions are asking?

• How easy or difficult was it to understand? What made it easy or difficult?

• Do you have any specific emotional reactions to these instructions? If so, can you share more about it? To the purpose of the questionnaire?

I would like you to read the first bolded item silently and provide a response on the survey, just as you would if you were taking the survey on your own. Please feel free to say anything out loud that comes to mind as you read. I will then ask you similar questions about your thoughts for this item or question.

[Item #1: Once the interviewee has read and answered the item, ask the following general questions]

• Meaning: Tell me what you think the question is asking? Probe: This question asks about “XXX.” What does “XXX” mean to you?

• Judgment: Why did you pick that number for your answer?

• Ease: How easy or difficult was it to answer that question? What made it easy or difficult? Did you have any other thoughts you had about how you would answer that question?

• Reaction: Do you have any other specific emotional reactions to this question? If so, can you share more about it? How comfortable were you with answering the question? Was there any discomfort?

• Other options: What are your thoughts about the other options listed below as a way to ask about this feeling or experience instead? Are there any of these that you think are better at reflecting this feeling or experience?

• Appropriateness/relevance: Does this question feel appropriate or relevant when exploring the personal impact of social stigma, stereotypes, and discrimination?

Repeat above for all items (or as many as can be covered in allotted time)

Step 3: Closing Questions

We are just about out of time. We have a few remaining questions that apply overall to your experience completing this questionnaire.

If there is not enough time to cover all items:

• Ask participant to read through the remaining items and share if there are any items or topics that are concerning or confusing.

For all interviews:

• Did you answer questions based on intensity of your experiences and feelings, frequency of experiences and feelings, mixture of both?

• What are your experiences with HIV-related social stigma, discrimination, and stereotyping?

• Are there any personal experiences and feelings related to the impact of social stigma, discrimination, and stereotyping that you think are missing from this questionnaire and should be added?

• What are your thoughts about how the personal feelings and experiences we have just discussed are related to social discrimination and prejudice for you?

• What type of efforts and interventions do you think are needed to effectively these external barriers and social stigma?

Step 4: Summary

We greatly appreciate your thoughts about these questions. It seemed that you had concerns about the following items (summarize). There were certain items or questions that really seemed relevant to you (summarize) and that we may want to add a question about (XXX). This information will be very helpful in finalizing this questionnaire about the impact of stigma for individuals living with HIV.

Thank you again. Is there anything else you would like to share before we end?

Data statement

Access to individual-level data from the MACS/WIHS Combined Cohort Study Data (MWCCS) may be obtained upon review and approval of a MWCCS concept sheet. Links and instructions for online concept sheet submission are on the study website.