Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2019 Jan 1.
Published in final edited form as: Arch Sex Behav. 2017 May 9;47(1):107–119. doi: 10.1007/s10508-016-0910-4

Evaluating HIV knowledge questionnaires among men who have sex with men: A multi-study item response theory analysis

Patrick Janulis 1, Michael E Newcomb 1, Patrick Sullivan 2, Brian Mustanski 1
PMCID: PMC5680146  NIHMSID: NIHMS875360  PMID: 28488126

Abstract

Knowledge about the transmission, prevention, and treatment of HIV remains a critical element in psychosocial models of HIV risk behavior and is commonly used as an outcome in HIV prevention interventions. However, most HIV knowledge questions have not undergone rigorous psychometric testing such as using item response theory. The current study uses data from 6 studies of men who have sex with men (MSM; n = 3565) to 1) examine the item properties of HIV knowledge questions, 2) test for differential item functioning on commonly studied characteristics (i.e., age, race/ethnicity, and HIV risk behavior), 3) select items with the optimal item characteristics, and 4) leverage this combined dataset to examine the potential moderating effect of age on the relationship between condomless anal sex (CAS) and HIV knowledge. Findings indicate that existing questions tend to poorly differentiate those with higher levels of HIV knowledge but items are relatively robust across diverse individuals. Furthermore, age moderated the relationship between CAS and HIV knowledge with older MSM having the strongest association. These findings suggest that additional items are required in order to capture a more nuanced understanding of HIV knowledge and that the association between CAS and HIV knowledge may vary by age.

INTRODUCTION

Knowledge of the transmission, prevention, and treatment of HIV has been a focus of research since the early days of the HIV/AIDS epidemic. This work has been particularly important given that several models of health behavior identify knowledge as a prerequisite to health behavior change (Fisher & Fisher, 1992; Hughes & Admiraal, 2012). Following early scales developed to assess HIV knowledge (Ferguson, Cox, Irving, Leiter, & Farnsworth, 1995; Mccown & Johnson, 1991), Carey, Morrison-Beedy, and Johnson (Carey, Morrison-Beedy, & Johnson, 1997) developed and validated a 45-item scale that became one of the most widely used measures of HIV knowledge. Subsequently, they developed a shorter 18-item version which had strong internal consistency, test-retest reliability, and correlation with the 45-item version (Carey & Schroder, 2002).

However, despite the continued inclusion of HIV knowledge as an outcome in the evaluation of HIV preventive interventions (Herbst et al., 2005; Medley, Kennedy, O'Reilly, & Sweat, 2009), studies assessing the association between HIV knowledge and HIV risk behavior have been decidedly mixed. For example, several studies (Kalichman, Picciano, & Roffman, 2008; Nelson et al., 2015; Newcomb & Mustanski, 2014a; Scott-Sheldon et al., 2010; Walsh, Senn, Scott-Sheldon, Vanable, & Carey, 2011) have found no significant direct relationship between HIV knowledge and condom use while other studies have found a positive association (Bazargan, Kelly, Stein, Husaini, & Bazargan, 2000; Villar-Loubet et al., 2013). The lack of consensus on this association may be partially explained by the wide variety of study populations included in these studies as well as the potential for indirect relationships between HIV knowledge and sexual risk behavior, such as mediating relationships through self-efficacy or intentions to use condoms (Eggers, Aaro, Bos, Mathews, & de Vries, 2014). Furthermore, a systematic review of HIV knowledge measures (Hughes & Admiraal, 2012) indicated that many studies examining HIV knowledge used scales that have not undergone psychometric testing. While Carey and Schroder’s (Carey & Schroder, 2002) scale has undergone psychometric testing and validation, one study revealed that the scale may not perform well (e.g., have lower internal consistency) for certain subpopulations such as men who have sex with men (MSM) (Oglesby & Alemagno, 2013). This is particularly problematic given the continued substantial impact of HIV on MSM in the US. For example, MSM represented 66.7% of estimated new HIV infections in 2014 (Centers for Disease Control and Prevention, 2015).

In addition, the scale has yet to be evaluated in a US sample using modern psychometric approaches such as item response theory (IRT), which could provide additional insights about the properties of individual items. Given that good measurement is fundamental to accurately gauging associations between constructs, this study uses IRT in the assessment and selection of items from an updated item pool reflecting new information about transmission (e.g., treatment as prevention) in an effort to further improve the measurement of HIV knowledge as well as identify items for a shorter versions of the HIV knowledge measure.

IRT is a technique to model categorical responses to reflect an underlying latent variable (i.e., HIV knowledge in the current study). As a specific type of more general latent variable modeling tools (Glockner-Rist & Hoijtink, 2003), IRT allows the exploration of item properties and identification of problematic items. This can facilitate reduced burden on participants by identifying redundant or unhelpful items that can be eliminated from the scale. Similarly, IRT aids identification of problematic aspects of scales (e.g., questions that are too easy or difficult) that, after modification, can improve the overall validity of the instrument. The current study uses a two parameter model where the relationship between each item and the latent variable is reflected by two parameters: discrimination and difficulty. Discrimination represents the association between each item and the underlying latent variable while difficulty represents the probability of correctly responding to an item relative to an individual’s estimated latent score. Estimates for these parameters can be used to identify poorly fitting or redundant items in a scale. Furthermore, when comparing composite scores across groups on a multi-item scale, an implicit assumption is made that these item properties are constant across each sub-group (Teresi, 2006). Differential item functioning (DIF) is lack of invariance of these parameters and must be accounted for before comparing scores across groups. For example, a question with differential item functioning would indicate that individuals in different groups with the same level of the latent trait (i.e., HIV knowledge) could have different probabilities of correctly answering that question. Without accounting for this differential item functioning, composite scores across groups cannot be compared because differing composite scores could reflect a measurement artifact rather than a true group difference (Zumbo, 2007).

In addition to establishing equivalence across covariates, DIF analysis enables direct comparison of scores across studies that use different versions of a measurement scale. Therefore, pooled analysis of multi-study data, commonly known as “integrated data analysis” (Bauer & Hussong, 2009), can also facilitate examination of research questions that could not be answered in the confines of a single study. For example, previous research has suggested developmental differences (i.e., a moderating effect of age) across covariates of sexual risk behavior such as alcohol (Newcomb, 2013) and drug use (Newcomb & Mustanski, 2014b). Yet, given that many studies of MSM focus on specific age groups (e.g., young or older MSM), few studies may provide a sufficient range of ages or adequate statistical power to examine how the relationship between HIV knowledge and risk behavior differs across age groups. Therefore, the current study leverages the wide range of ages provided in the pooled dataset to examine if age moderates the relationship between risk behavior and HIV knowledge, controlling for any measurement differences across studies.

Accordingly, the aims of the current study are to 1) examine the item characteristics of an HIV knowledge questions from a pool that was updated to reflect new understanding of HIV transmission, 2) test for differential item functioning on commonly studied characteristics conceptually related to HIV knowledge (i.e., age, race/ethnicity, and HIV risk behavior), 3) select items with the optimal item characteristics in effort to remove redundant items and reduce the burden of the measure to participants and 4) leverage this combined dataset to examine the potential moderating effect of age on the relationship between condomless anal sex (CAS) and HIV knowledge.

METHODS

Participants

The current study draws data from 6 distinct studies of MSM. These studies include: Keep it Up! 1.0 (Mustanski, Garofalo, Monahan, Gratzer, & Andrews, 2013), Keep it Up! 2.0 (Strauss et al., 2016), InvolveMENt (Sullivan et al., 2014), Crew 450 (Mustanski, Johnson, Garofalo, Ryan, & Birkett, 2013), a pilot study for the InvolveMENt study (Wagenaar, Sullivan, & Stephenson, 2012), and TRACK (Newcomb & Mustanski, 2014a). All 6 studies focused on health issues among MSM, were performed by the authors’ research groups, and included the four following variables: multi-item HIV knowledge measure, a measure of recent condomless anal sex (CAS), race/ethnicity, and age of the participant. Table 1 presents descriptive statistics for age, race, and CAS in each of these studies for participants that completed the HIV knowledge items. These studies employed a wide variety of recruitment techniques ranging from referrals by clinicians, targeted outreach, online advertisements, and peer recruitment. Similarly, these studies utilized substantially different inclusion criteria although all participants were required, with studies including both HIV positive and HIV negative participants. However, full details on participants and recruitment methods from each sample can be found in the respective citations for each study.

Table 1.

Demographic Characterstics by Study

Crew 450 KIU! 1.0 TRACK InvolveMENt InvolveMENt
Pilot		 KIU! 2.0
Sample Size 450 102 143 803 1294 773
Age 18.9 (1.29) 21.3 (1.8) 27.4 (7.3) 27.89 (6.7) 30.75 (12.3) 24.2 (2.9)
Race/Ethnicity
  African American 53.3% 16.7% 22.4% 56.5% 32.8% 17.2%
  Latino 20.0% 28.4% 27.3% - 9.8% 25.7%
  Other 8.7% 8.8% 13.3% - 9.3% 9.9%
  White 18.0% 46.1% 37.1% 42.5% 47.9% 47.0%
CAS 48.7% 72.5% 44.8% 65.0% 38.5% 57.2%
Open in a new tab

Measures

A total of 38 items were used across all studies to examine HIV knowledge. Eighteen of these 38 items were used from Carey and Schroder’s (Carey & Schroder, 2002) short form of the HIV-Knowledge questionnaire, which itself is derived from their longer form of this measure (Hughes & Admiraal, 2012). Other items (i.e., items 19–38) were added in effort to update the measure to include items that reflected new understanding of HIV transmission and more accurately reflect common misunderstandings found in previous research (Mustanski, DuBois, Prescott, & Ybarra, 2014; Mustanski, Lyons, & Garcia, 2011; Mustanski, Ryan, & Garofalo, 2014). For example, these items include questions on sexual positioning on HIV risk and the effects of HIV treatment on the risk of transmission. Overall, these new questions were intended to measure novel misconceptions in HIV transmission as well as capture the massive change that has occurred in recent years regarding biomedical prevention and treatment of HIV that must also be reflected in the measurement of HIV knowledge. Table 2 displays the wording for each questionnaire item, which items were used for each study, and the percentage of correct responses for each item, regardless of study.

Table 2.

Items and Study

Question %
Correct		 Crew4
50		 KIU
1.0		 TRACK InvolveMENt InvolveMENt
Pilot		 KIU
2.0
1 Coughing and sneezing do NOT spread HIV. 83.2 X X X X X
2 A person can get HIV by sharing a glass of water with someone who has HIV. 90.4 X X X X X
3 Pulling out the penis before a male climaxes/cums keeps a person from getting HIV. 87.7 X X X X X X
4 A person can get HIV from having anal sex. 90.9 X X X X X
5 Showering or washing one’s genitals/private parts after sex keeps a person from getting HIV. 88.8 X X X X X
6 All pregnant females infected with HIV will have babies born with HIV [or AIDS]. 69.1 X X X X
7 People who have been infected with HIV quickly show serious signs of being infected. 91.4 X X X X X X
8 There is a vaccine that can stop people (or adults) from getting HIV. 80.5 X X X X X X
9 People are likely to get HIV by deep kissing, putting their tongue in a partner’s mouth, if their partner has HIV [and cuts in their mouth]. 65.4 X X X X X X
10 It is possible to get HIV when a person gets a tattoo if the equipment is not properly cleaned. 82.7 X X X
11 Using a latex condom or rubber can lower a person’s chance of getting HIV. 93.9 X X X X
12 A natural skin [lamb skin] condom works better against HIV than does a latex condom. 59.5 X X X X X X
13 A person will NOT get HIV if [she or] he is taking antibiotics. 88.8 X X X X X X
14 Having sex with more than one partner can increase a person’s chance of being infected with HIV. 93.8 X X X X X X
15 Taking a test for HIV one week after having sex will tell a person if she or he has HIV. 81.4 X X X X X X
16 A person can get HIV by sitting in a hot tub or a swimming pool with a person who has HIV. 90.7 X X X X X X
17 A person can get HIV from oral sex. 78.6 X X X X X X
18 Using Vaseline or baby oil with [latex] condoms lowers the chance of getting HIV. 81.8 X X X X X X
19 Only the receptive/”bottom” partner is at risk of being infected with HIV during anal sex. 89.7 X X
20 If you only have sex once with a partner, you cannot get HIV from them. 91.8 X X
21 People on HIV medications cannot spread HIV. 81.1 X X
22 HIV/AIDS is the only incurable sexually transmitted infection. 73.7 X X
23 Using an enema or douching after anal sex will prevent HIV infection. 78.9 X X
24 Latex condoms protect against all sexually transmitted infections. 57.4 X X
25 Everybody who has HIV knows that they have HIV. 93.3 X X
26 People who are HIV positive almost always tell their partners before they have sex. 85.3 X X
27 You can’t get HIV from a guy who usually just has sex with females. 84.1 X X
28 If a person who is HIV positive and on HIV medications has an undetectable viral load (low HIV levels), they cannot transmit HIV. 69.3 X X
29 A person cannot get HIV from having sex with someone who is HIV+ but has an undetectable viral load. 65.0 X
30 When using condoms during sex it is safer to use water-based lubricants than oil-based lubricants. 54.2 X
31 A person cannot get HIV from being the “top” or insertive partner during anal sex. 76.8 X
32 A person is more likely to get HIV by having unprotected anal sex than by having unprotected oral sex. 75.4 X
33 A woman can get HIV if she has anal sex with a man. 87.2 X X
34 A woman cannot get HIV if she has sex during her period. 87.3 X X X
35 There is a female condom that can help decrease a woman's chance of getting HIV. 11.5 X X X
36 You can tell if someone has HIV by looking at them. 96.4 X
37 You can tell if someone has HIV by inspecting their penis. 94.7 X
38 HIV is not a real health issue. 97.4 X
Open in a new tab

Race was coded as African American, White, Latino, and Other. This four category code was chosen to provide a uniform race code across all studies. Finally, CAS was measured dichotomously if the participant had engaged in any condomless anal sex during the previous 6 months. Again, this dichotomous categorization was chosen to create a uniform code across all 6 studies given the slight variations in time period and wording of questions capturing sexual risk taking behavior.3

Analytic Approach

Item Characteristics

Item characteristics were examined before moving on to modeling potential DIF. A structural equation modeling approach was used that is mathematically equivalent to traditional 2-paramater IRT approaches (Muthen, 1984). As discussed, two characteristics of each item are examined using this approach: difficulty and discrimination. For tests of knowledge such as the one used in the current study, the difficulty parameter indicates the probability of correctly answering an item given an individual’s estimated level of underlying knowledge. Accordingly, questions that tend to be answered correctly only by individuals with the highest levels of knowledge have high difficulty while those that tend to be answered correctly by many individuals (i.e., individuals at varying levels of underlying knowledge) are low difficulty items. The discrimination parameter is equivalent to the factor loading. Therefore, it describes the degree to which that item is correlated with the underlying score. Items with high discrimination provide a large amount of information on participant’s level of knowledge while those with low discrimination provide less information. Accordingly, a unidimensional confirmatory factor analysis model was constructed using all 38 items to measure HIV knowledge. Estimates for each item’s difficulty and discrimination parameter were then examined in conjunction with the total information curve (TIC). Given the nature of using multiple studies with many non-overlapping items (i.e., missing data by design), maximum likelihood with robust standard errors was used for estimation. As such, we are unable to compare absolute model fit indices that are common in many psychometric evaluations of questionnaires and instead we focus on the parameter estimates themselves in our analysis of the performance of each item rather than measures of overall fit. Maximum likelihood with robust standard errors also has desirable properties with regards to missing data and tends to out preform other traditional methods for handling missing data such as list or pairwise deletion (Allison, 2003).

Differential Item Functioning

DIF was modeled using a series of multiple indicator multiple cause (MIMIC) models (Finch, 2005; Jones, 2006), a structural equation modeling approach to identifying DIF. This approach explores DIF by assessing the association between independent variables (i.e., sources of DIF) and factor loadings/intercepts while simultaneously controlling for the effect of the independent variable on the underlying latent trait. Two main types of DIF are examined in the current study. First, uniform DIF describes bias in the difficulty parameter. Uniform DIF suggests that the likelihood of answering that specific item correctly varies across levels of the independent variable even after controlling for overall levels of the latent trait. For example, if an item showed age based uniform DIF in the current study, this would indicate that participants of different ages are more or less likely to correctly answer that specific item even after controlling for their estimated level of HIV knowledge based on their responses to the other items. Using the MIMIC method, uniform DIF is modeled by examining the direct association between the independent variable(s) and each item. Second, non-uniform DIF describes bias in the discrimination parameter (i.e., factor loading). Non-uniform DIF indicates that the item is a stronger or weaker indicator of the underlying latent trait across levels on the independent variable. In the current study, age based non-uniform DIF would indicate that the specific item is more strongly or weakly correlated with the underlying latent trait among differing aged participants. Using the MIMIC method, non-uniform DIF is modeled by regressing the item on an interaction between the latent variable and the independent variable. Therefore, this interaction term indicates if the relationship between the latent variable and indicator varies across levels of the independent variable. Using this method, DIF is iteratively examined by regression one item at a time on all sources of DIF. Accordingly, the MIMIC method results in a very large number of statistical tests given that a significant test is performed for each independent variable for each type of DIF on every item (i.e., number of sources of DIF * types of DIF * number of tests items). In the current study, the Benjemini-Hochberg procedure (Benjamini & Hochberg, 1995) was used to control false positive rates given its superior performance over more traditional methods such as the Bonferroni method (Thissen, Steinberg, & Kuang, 2002). Other modern methods, such as scale purification (Wang, Shih, & Yang, 2009), were not chosen given that little previous research has been conducted using IRT on HIV Knowledge and therefore no “anchor” items could be identified a priori.

Two main sources of potential DIF were examined in the current study: DIF associated with our main independent variables of interest (i.e., age, race, and HIV risk behavior) as well as cross-study DIF. Simultaneously modeling DIF in our explanatory variables as well as cross-study DIF allows us to examine our research questions (i.e., if our independent variables show DIF across items on the HIV Knowledge scale) while also controlling for discrepancies of item characteristics that may have arrived solely from differences in administration, sampling, or other features that varied across studies and thereby confound the observed DIF in our main variables of interest.

Item Reduction

Following the review of item characteristics, item characteristic curves, information curves, and the results of the DIF analysis, items were then selected to be retained in the reduced form of the scales. More specifically, Items were chosen based on three criteria. First, does the item have a statistically significant and strong discrimination to provide substantial information on the underlying scale? Second, does the item represent a breadth of difficulty that is underrepresented in the scale overall? Third, does the item show robustness to variability in age, race, and risk behavior? Based on these criteria, items were selected and a final unidimensional model with the remaining items was examined. This included modeling the effect of each independent variable and study effects on the latent factor after controlling for any remaining DIF.

Age as a moderator of the association between CAS and HIV Knowledge

Finally, the lack of consensus regarding the association between risk behavior and HIV knowledge in previous research may be partially explained by variation in this association across a moderating variable. Therefore, we can leverage the integration of data across multiple samples to provide a much larger range of ages to directly compare the association between CAS and HIV knowledge by age groups. Accordingly, the final model will examine this association by modeling the interaction between Age and CAS on HIV Knowledge while controlling for variation HIV knowledge measurement across studies (through DIF analysis).

RESULTS

Item Characteristics

Table 3 displays the difficulty and discrimination parameter for a single factor model with all variables included (Model 1). For the discrimination parameter, all but one item (i.e., item 35) had a significant discrimination parameter suggesting the vast majority of items were significant predictors of the latent variable and were informative to the model. The items providing the most information were items 25 (“Everybody who has HIV knows they have HIV”), 20 (“If you only have sex once with a partner, you cannot get HIV from them”), and 23 (“Using an enema or douching after anal sex will prevent HIV infections”). The significant loading items that had the lowest discrimination and provided the least information were items 17 (“A person can get HIV from oral sex”), 33 (“A women can get HIV if she has anal sex with a man”), and 24 (“Latex condoms protect against all sexually transmitted infections”).

Table 3.

Item Characterstics for All Items (Model 1) and the Reduced Items (Model 2 & 3)

Model 1 Model 2 Model 3
Discrimination Difficulty Discrimination Difficulty Discrimination Difficulty
Est. SE Est. SE Est. SE Est. SE Est. SE Est. SE
1 1.07 0.08 −1.81 0.11
2 1.39 0.10 −2.11 0.11
3 1.46 0.09 −1.78 0.08
4 0.87 0.09 −3.06 0.25
5 1.80 0.12 −1.70 0.08
6 1.11 0.07 −0.89 0.06
7 2.17 0.14 −1.74 0.07 2.14 0.16 −1.76 0.07
8 1.29 0.07 −1.40 0.07 1.34 0.09 −1.38 0.07
9 1.11 0.06 −0.70 0.05
10 1.15 0.16 −2.17 0.17
11 1.99 0.19 −2.29 0.12 1.86 0.20 −2.35 0.13
12 1.21 0.07 −0.39 0.04 1.19 0.08 −0.40 0.04 1.21 0.10 −0.41 0.04
13 2.34 0.14 −1.48 0.05 2.39 0.17 −1.49 0.05 2.40 0.23 −1.51 0.06
14 1.27 0.10 −2.65 0.16
15 1.41 0.08 −1.38 0.06
16 1.91 0.12 −1.78 0.07 1.77 0.12 −1.86 0.08
17 0.55 0.05 −2.53 0.23
18 1.56 0.09 −1.32 0.06
19 2.30 0.23 −1.83 0.09 2.26 0.24 −1.84 0.09
20 2.55 0.30 −1.93 0.10 2.68 0.33 −1.91 0.09 3.30 0.55 −1.78 0.09
21 1.98 0.17 −1.40 0.07 2.13 0.19 −1.35 0.07 1.87 0.20 −1.39 0.08
22 1.52 0.13 −1.19 0.07
23 2.48 0.21 −1.20 0.06 2.13 0.19 −1.25 0.06
24 0.83 0.08 −0.67 0.09 0.79 0.08 −0.65 0.09
25 2.99 0.33 −1.98 0.09 2.95 0.37 −2.00 0.09 3.53 0.65 −1.87 0.09
26 1.69 0.15 −1.74 0.10
27 1.62 0.16 −1.70 0.10
28 1.49 0.13 −1.01 0.07 1.60 0.15 −0.95 0.06
29 1.48 0.46 −0.63 0.18 1.45 0.47 −0.57 0.19
30 1.05 0.33 −0.27 0.19 0.75 0.30 −0.23 0.26 1.01 0.48 −0.20 0.21
31 1.80 0.44 −1.00 0.19
32 1.42 0.40 −1.08 0.25
33 0.80 0.16 −2.37 0.46
34 1.70 0.13 −1.42 0.08
35 0.12 0.08 17.66 12.09
36 1.97 0.37 −2.25 0.26 2.09 0.40 −2.24 0.25
37 2.20 0.33 −1.89 0.18 2.22 0.36 −1.94 0.18
38 1.45 0.37 −2.95 0.55
Open in a new tab

Note. Bolded values have p < 0.05

Ignoring the poor fitting item 35, all item difficulty parameters fell below the mean (i.e., latent score of zero) indicating that most items were relatively easy to answer. This concurs with the finding that most items were correctly answered by most participants (e.g., see correct percentage in Table 2). This also indicates that the scale provides the most information for participants who are below the mean in their knowledge of HIV, as indicated by the total information curve for the full scale and factor scores with a left skew (Figure 1). Again excluding item 35, items with the highest difficulty were 30 (“When using condoms during sex it is safer to use water-based lubricants than oil-based lubricants), 12 (“A natural skin [lamb skin] condom works better against HIV than does a latex condom”), and 29 (“A person cannot get HIV from having sex with someone who is HIV+ but has an undetectable viral load.”). Items with the lowest difficulty were 4 (“A person can get HIV from having anal sex”), 38 (“HIV is not a real health issue”), and 14 (“Having sex with more than one partner can increase a person’s chance of being infected with HIV”).

Figure 1.

Figure 1

Open in a new tab

Total Information and Latent Variable Distribution for HIV Knowledge Scale

Differential Item Functioning

DIF was examined using an iterative approach by separately regressing each item onto all sources of potential DIF (i.e., age, race, CAS, and dummy codes for study). Subsequently, all p- values for parameter estimates for both sources of DIF were examined against the Benjemini-Hochberg adjusted p-value for multiple comparisons. P-values that were lower than the Benjemini-Hochberg adjusted p-values were considered significant sources of DIF. Using this threshold, nine items showed significant DIF on 12 different parameters. Furthermore, all significant DIF was study based rather than from the substantive independent variables. Compared to Crew 450, the InvolveMENt Pilot study showed a significantly lower difficulty parameter for items 2 (DIF estimate = −1.21, SE = 0.27) and 4 (DIF estimate = −2.36, SE = 0.35) as well as significantly higher difficulty for item 9 (DIF estimate = 0.98, SE = 0.24). Also relative to Crew 450, the InvolveMENt study showed significantly lower difficulty for item 3 (DIF estimate = −1.28, SE = 0.36) but higher difficulty for items 9 (DIF estimate = 1.10, SE = 0.30) and 15 (DIF estimate = 1.25, SE = 0.32). Item 10 showed significant DIF on the discrimination parameter when comparing KIU 1.0 (DIF estimate = −2.00, SE = 0.53) or TRACK (DIF estimate = −2.28, SE = 0.61) to the discrimination parameter in Crew 450. KIU 2.0 also had a significantly higher difficulty parameter for item 27 (DIF estimate = 1.11, SE = 0.31) compared to KIU 1.0. Finally, InvolveMENt Pilot had a significantly lower discrimination for item 4 (DIF estimate = −1.01, SE = 0.28), InvolveMENt had a significantly lower discrimination for item 7 (DIF estimate = −1.081, SE = 0.31), and TRACK had a significantly lower discrimination for item 11 (DIF estimate = −2.25, SE = 0.62) relative to Crew 450.

Item Reduction

Following examination of item characteristics and DIF analysis, we proceeded to select a reduced pool of items based on those that provided the most information on HIV Knowledge, represented a wide range of difficulty, and showed minimal to no sign of DIF in the full scale. Retained items can be found under Model 2 and Model 3 within Table 3. For Model 2, most items were retained because they had the highest discrimination parameters (items: 7, 11, 13, 16, 19, 20, 21, 23, 25, 36, 37), four items were retained because they had the highest difficulty (items: 12, 24, 29, 30), and two items (items: 8 and 28) were retained given the importance and potentially evolving nature of the question topics. Finally, in effort to make a short form of the scale that would substantially reduce the length of administering the survey, Model 3 included six items. In effort to maintain a breadth of difficulty and total information, four items were selected with the highest discrimination (items: 13, 20, 21, 25) and two items were selected with the highest difficulty (items: 12, 30). Comparison of the total information curves and estimated latent trait scores provided in the full model versus the two reduced can be found in Figure 1. Estimates for the discrimination and difficulty parameters in both reduced item models can be found in Table 3.

Reduced Models and Covariates

We also examined the association between all covariates and HIV knowledge for both of the reduced models, controlling for cross-study DIF in Model 2. Path coefficient results for Model 2 (Table 4) indicated that White (γ = 0.51, p < 0.001) and other race (γ = 0.21, p = 0.043) but not Latino (γ = 0.14, p = 0.067) participants had higher estimated levels of HIV Knowledge compared to African American participants. Older participants also had higher estimated levels of HIV Knowledge (γ = 0.03, p < 0.001) as did those reporting CAS (γ = 0.12, p = 0.023). Similarly, results for Model 3 indicated White (γ = 0.51, p < 0.001) and other race (γ = 0.25, p = 0.045) participants had higher estimated levels of HIV but no significant difference was found between Latino (γ = 0.17, p = 0.050) participants compared to African American participants. Older participants also had higher estimated levels of HIV Knowledge (γ = 0.04, p < 0.001) but no significant difference was found between participants reporting CAS (γ = 0.11, p = 0.057).

Table 4.

Predictors of HIV Knowledge in both Reduced Models

Model 2 Model 2a Model 3 Model 3a
γ (SE) p-
value		 γ
(SE)		 p-
value		 γ
(SE)		 p-
value		 γ
(SE)		 p-
value
Race/Ethnicity
  Black/African-American (Ref) - - - - - -
  White 0.58 (0.06) <0.001 0.57 (0.06) <0.001 0.51 (0.07) <0.001 0.51 (0.07) <0.001
  Latino 0.14 (0.08) 0.067 0.14 (0.08) 0.083 0.17 (0.09) 0.051 0.17 (0.09) 0.054
  Other 0.21 (0.10) 0.043 0.20 (0.10) 0.054 0.25 (0.12) 0.045 0.24 (0.12) 0.000
Age 0.03 (0.01) <0.001 0.03 (0.01) <0.001 0.04 (0.01) <0.001 0.03 (0.01) <0.001
CAS 0.12 (0.05) 0.023 −0.37 (0.20) 0.061 0.11 (0.06) 0.057 −0.32 (0.23) 0.175
Age X CAS - - 0.02 (0.01) 0.012 - - 0.02 (0.01) 0.061
Study
  Crew 450 - - - -
  KIU! 1.0 0.71 (0.15) <0.001 0.68 (0.17) <0.001 0.63 (0.18) 0.001 0.64 (0.18) <0.001
  TRACK 0.51 (0.15) <0.001 0.47 (0.26) 0.066 0.51 (0.17) 0.002 0.51 (0.17) 0.003
  InvolveMENt 0.88 (0.09) <0.001 0.76 (0.12) <0.001 0.83 (0.11) <0.001 0.80 (0.11) <0.001
  InvolveMENt Pilot 0.70 (0.09) <0.001 0.63 (0.11) <0.001 0.78 (0.10) <0.001 0.77 (0.10) <0.001
  KIU! 2.0 0.42 (0.08) <0.001 0.34 (0.09) <0.001 0.42 (0.09) <0.001 0.41 (0.09) <0.001
Open in a new tab

Note. Bolded values have p < 0.05

Age as a Moderator of the association between CAS and HIV Knowledge

Finally, using each reduced model (Model 2a and 3a) we also examined the potential moderating association of age on the relationship between CAS and HIV knowledge (Table 4). For Model 2a (i.e., 17 items), results indicated that the interaction term was significant (γ = 0.02, p = 0.012) suggesting that the association between CAS and HIV knowledge varied by age. Furthermore, age continued to have a direct association with HIV knowledge (γ = 0.03, p < 0.001) for those who did not engage in CAS. Looking more specifically for variation of the relationship between CAS and HIV knowledge across different age groups (Figure 2a), results indicated only individuals significantly above the mean age (i.e., ≥ 50 years old) tended to have a statistically significant positive association between CAS and HIV knowledge, while participants under the age of 20 had a small statistically significant negative association. The results for Model 3a were similar but the interaction term did not remain significant (γ = 0.02, p = 0.061). However, Figure 2b indicates that the point estimate of the interaction largely remained the same.

Figure 2.

Figure 2

Open in a new tab

Association of Condomless Anal Sex (CAS) on HIV Knowledge across Age

DISCUSSION

This study examined the psychometric properties of an HIV knowledge scale using a combined dataset from 6 different health related studies of MSM and proposed two reduced versions of the HIV knowledge scale based on the psychometric properties. The results indicated that most test items held up to psychometric testing with strong loadings on the HIV knowledge latent factor. However, the scale lacked substantial variability in item difficulty with all item difficulty estimates being below the mean latent score (i.e., items tended to be easy). The consequence of this lack of difficult items was a large negative skew of factor scores with the scale being less able to distinguish between participants on the higher spectrum of HIV knowledge and a subsequent restriction in range in estimated scores for participants with above average HIV knowledge. This finding concurs with previous evaluations of an HIV knowledge scale (Gomes, Batista, Ceccato, Kerr, & Guimarães, 2014). Accordingly, our analysis suggests that novel items are needed in effort to capture the full spectrum of HIV knowledge that exists among US MSM. Furthermore, to the extent that restricted range of total scores distorts true variability in HIV knowledge, it is possible that the lack of difficult items also distorts the relationship between HIV knowledge and other important variables such as sexual risk behavior.

At the item level, the most useful (i.e., higher discrimination and/or difficulty) items tended to be those that did not focus on a specific form of transmission. Rather, informative items tended to be more subtle questions such as contrasting potential risk reduction techniques (e.g., item 12: A natural skin condom works better against HIV than does a latex condom), inquiring about types of situations where a certain transmission would put an individual at risk (e.g., item 20: If you only have sex once with a partner, you cannot get HIV from them) or were unrelated to transmission altogether (e.g., item 25: Everybody who has HIV knows that they have it). Accordingly, it is likely that more sophisticated questions regarding the transmission, prevention, and treatment of HIV are required in order to assess the full breadth of HIV knowledge and provide the most informative items. For example, items that require nuanced understanding of pre or post-exposure prophylaxis may prove particularly useful for distinguishing those with higher levels of HIV knowledge.

We also proposed two shortened versions (i.e., 17 items and 6 items) of the HIV knowledge scale identifying items to be included in these scales based on the IRT analysis. While both reduced versions provided reduced total information and less nuanced estimation of factor scores (Figure 1a), both versions also produced very similar associations with the independent variables of substantive interest suggesting the reduced variability did not obscure the substantive relationship.4 The two exceptions to this were that CAS was significantly positively associated with HIV knowledge in the 17 item scale but not in the 6 item scale and, similarly, the interaction between age and CAS was also no longer significant in the 6 item scale. However, this discrepancy was likely because only 2 items (Items 12 and 13) of the 6 item scale were measured in the study (i.e., InvolveMENt Pilot) that had the oldest participants with the strongest association between CAS and HIV knowledge. Therefore, the change in significance for these associations can likely be explained by the decreased reliability of measuring HIV using two items and the subsequent attenuation of associations due to increased measurement error (e.g., three items are generally required to identify a latent variable measurement model). Nonetheless, one of the most common issues with item reduction is obscuring the underlying content of the previous longer form (Goetz et al., 2013) and the continued lack of breadth in item information (i.e., restricted range of item difficulties) in the reduced version likely remains more problematic than the reduced total item information. Accordingly, the addition of items with broader levels of difficulty remains an important objective of future research.

The study also examined sources of DIF such as age, race/ethnicity, and CAS. Within studies, the items appeared rather robust against item parameter non-invariance on common covariates of HIV knowledge. This suggests that our participants generally responded similarly to these items and these results should provide some reassurance to researchers attempting to compare disparate groups on their levels of HIV knowledge using these items. However, it should be noted given this study’s focus on US MSM, DIF analysis should be performed before comparing composite scores across subgroups of other sampled populations. Without performing DIF testing, the validity of group comparisons cannot be established given that differing item properties across groups could bias estimates of total scores.

The finding that CAS was positively associated with HIV knowledge among older participants also merits further discussion. While it is unclear what may cause a positive association between CAS and HIV knowledge among older participants, research has increasingly focused on the lack of a negative association between HIV knowledge and risk behavior. For example, given the widespread and longstanding public health prevention efforts to educate MSM about HIV, deficits in HIV knowledge may be less likely to explain remaining differences in risk behavior (Skinta, Murphy, Paul, Schwarcz, & Dilley, 2012). Accordingly, we suspect that a confounding variable (e.g., optimism about the efficacy of biomedical prevention and treatment) among older participants is most likely responsible for the positive association witnessed between HIV knowledge and CAS. For example, CAS may not be the ideal indicator of HIV risk behavior among older MSM with high levels of HIV knowledge given the many ways in which these individuals may practice safer sex (e.g., sex with a regular partner known to be HIV-negative) and recent biomedical advanced to HIV prevention (Jin et al., 2015). Nonetheless, future research should further explore why participant age among MSM may moderate the relationship between HIV knowledge and risk behavior. Similarly, other potential moderating effects such as race/ethnicity should be explored and may provide much needed nuance to the exploration of the association between risk behavior and HIV knowledge.

The current study also found that African American participants had lower estimated HIV knowledge scores as compared to white and Latino participants. This finding concurs with limited previous research suggesting that, among both the general population (Ebrahim, Anderson, Weidle, & Purcell, 2004) and MSM samples (Garofalo et al., 2015), African American participants tended to have lower scores on HIV knowledge. Given the substantially greater burden of HIV incidence experienced by African Americans, particularly among MSM (Prejean et al., 2011; Sullivan et al., 2015), targeted HIV knowledge interventions may be appropriate and such interventions may be particularly valuable in underserved areas where substantial misinformation regarding HIV may be most likely to be present (Djokic et al., 2009).

One unique aspect of an HIV knowledge scale, in contrast to more traditional psychometric measures, is the constantly shifting landscape of HIV prevention and intervention knowledge. Many relevant aspects of HIV prevention and treatment have changed considerably since the period when most HIV knowledge scales were initially developed. Additional refinement of these measures will be required as our understanding of HIV prevention and treatment of HIV continues to evolve. In fact, the evolving nature of the HIV epidemic may explain some of the extremely easy items that remain in many HIV knowledge assessments (e.g., item 2: A person can get HIV by sharing a glass of water with someone who has HIV.). While originally intended to accurately gage incorrect beliefs in HIV transmission risk, few people may continue to endorse these items given the accumulation of public knowledge of HIV and the continued promotion of HIV knowledge by health authorities, thereby rending these items less and less useful.

However, public knowledge of HIV varies considerably from one group to another (Peltzer, Matseke, Mzolo, & Majaja, 2009). Accordingly, one possible avenue for further investigation is the use of specific HIV knowledge scales, or subscales, for specific populations (Hughes & Admiraal, 2012; Oglesby & Alemagno, 2013). For example, knowledge appears to be much lower among MSM in middle and low-income countries (Adam et al., 2009) as compared the current sample and other studies with similar samples of US MSM (Wagenaar et al., 2012). Accordingly, the addition of more challenging questions regarding HIV knowledge may be less necessary in subpopulations such as MSM in lower or middle-income countries that display more variability in responses to existing measures of HIV knowledge.

Limitations

The current study had a number of limitations. First, while the study attempted to follow guidance on best practices in IRT, DIF analysis (Hambleton, 2006), and item reduction procedures (Goetz et al., 2013), there are areas where the current study falls short of these recommendations. For example, the study does not include replication of IRT analysis of the reduced item scales in an independent sample. Consequently, it is possible the item properties estimates would change in other similar samples of MSM. For example, the item properties that were found especially problematic (i.e., low difficulty) may be specific to the MSM samples used in this study. However, this finding concurs with several international studies that indicated low item difficulty of HIV knowledge questions across a wide range of participants (Aaro et al., 2011; Burke, Fleming, & Guest, 2014; Gomes et al., 2014) suggesting low difficulty is a problematic aspect of HIV knowledge scales in many populations. Second, despite these similarities with previous studies, other aspects of the item properties almost certainly vary from one group to another and inferences derived from these item properties (e.g., selecting ideal items for a reduced scale) are likely to change when studying other groups. For example, some items in the current scale were focused on transmission of HIV to women (e.g., “A women can get HIV if she has anal sex with a man”) and these items may be more informative in other populations. Third, item reduction necessarily includes subjective assessment of items and other researchers could have reasonably come to differing conclusions. Because of this, we suggest these reduced set of items as a guideline for future studies of HIV knowledge and recognize that other item combinations could be used, although empirical assessment of items should be used to guide this selection. Fourth, the assessment of associations between HIV knowledge and independent variables was limited by those variables that were consistent across all 6 samples. Accordingly, given the somewhat limited assessment of associations between the short forms and other variables, the criterion, discriminate, and convergent validity of these reduced scales could not be confirmed and future studies would be required to confirm the shortened versions maintain the same rigor of the previously validated versions. In addition, important variables may also have been left out, such as socioeconomic status, that could serve as important covariates of HIV knowledge. Fifth, a single study (Crew 450) utilized a modified version of respondent driven sampling and it is possible that un-modeled dependency across individuals could bias standard errors. Nonetheless, the inclusion of data from five other studies which did not employ peer recruitment suggest this limitation was unlikely to substantially impact our results. Similarly, despite the wide range of recruitment techniques employed within these studies, these studies do not represent a probability sample of MSM and should not be interpreted as such. Finally, the mean age for the entire analytic sample was 26.8 with few participants being below 18 or above 50, despite the importance of these populations for recent increases in incidence of HIV (CDC, 2016). Future studies should explicitly examine the applicability of HIV knowledge scales to these age groups, particularly given the interaction between age, CAS, and HIV knowledge witnessed in the current sample. In addition, they should examine other potential important moderators of this relationship, such as race or ethnicity.

Conclusion

HIV knowledge remains an important aspect of many studies of HIV and as an outcome for HIV preventive interventions, despite the continued lack of clarity in the association between HIV knowledge and HIV risk behavior. Nonetheless, valid measurement of HIV knowledge is the foundation to accurately understanding associations with this knowledge and examining change in knowledge over time. In this study we assessed the item characteristics of traditional and novel HIV knowledge items and proposed two shortened versions of this scale. Future studies are essential to validating the shortened versions and further development of this scale, such as the addition of more difficult items, is required to accurately gauging the full range of variability in HIV knowledge. Finally, the variation in association between HIV knowledge and CAS remains an important area for future research, particularly to inform studies or interventions that target specific age groups of MSM.

Acknowledgments

Funding: This work was supported by grant R01DA035145 from the National Institute of Mental Health and National Institute of Drug Abuse.

Footnotes

3

The TRACK study is the single study that does not fit into this categorization because only prospective measurement of CAS was collected. Accordingly, in order to avoid losing data, TRACK participants were coded as engaging in CAS if they reported that behavior in the 6 months after completing the HIV Knowledge questionnaire.

4

The relationships between the full scale (i.e., 38 items) and each of the substantive independent variables, including the interactions terms, is identical to that of the 17-item reduced scales.

Compliance with Ethical Standards

Ethical approval: All procedures performed in this study were in accordance with the ethical standards and approved by our institutional review board.

Conflict of Interest: Patrick Janulis has no conflict of interest to declare. Michael E. Newcomb has no conflict of interest to declare. Patrick Sullivan has no conflict of interest to declare. Brian Mustanski has no conflict of interest to declare.

Informed consent: Informed consent was obtained from all participants in this study.

References

  1. Aaro LE, Breivik K, Klepp KI, Kaaya S, Onya HE, Wubs A, Flisher AJ. An HIV/AIDS knowledge scale for adolescents: item response theory analyses based on data from a study in South Africa and Tanzania. Health Education Research. 2011;26(2):212–224. doi: 10.1093/her/cyq086. [DOI] [PubMed] [Google Scholar]
  2. Adam PC, de Wit JB, Toskin I, Mathers BM, Nashkhoev M, Zablotska I, Rugg D. Estimating levels of HIV testing, HIV prevention coverage, HIV knowledge, and condom use among men who have sex with men (MSM) in low-income and middle-income countries. Journal of Acquired Immune Deficiency Syndromes. 2009;52(Suppl 2):S143–151. doi: 10.1097/QAI.0b013e3181baf111. [DOI] [PubMed] [Google Scholar]
  3. Allison PD. Missing data techniques for structural equation modeling. Journal of Abnormal Psychology. 2003;112(4):545–557. doi: 10.1037/0021-843X.112.4.545. [DOI] [PubMed] [Google Scholar]
  4. Bauer DJ, Hussong AM. Psychometric Approaches for Developing Commensurate Measures Across Independent Studies: Traditional and New Models. Psychological Methods. 2009;14(2):101–125. doi: 10.1037/a0015583. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bazargan M, Kelly EM, Stein JA, Husaini BA, Bazargan SH. Correlates of HIV risk-taking behaviors among African-American college students: The effect of HIV knowledge, motivation, and behavioral skills. Journal of the National Medical Association. 2000;92(8):391–404. [PMC free article] [PubMed] [Google Scholar]
  6. Benjamini Y, Hochberg Y. Controlling the False Discovery Rate - a Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society Series B-Methodological. 1995;57(1):289–300. [Google Scholar]
  7. Burke HM, Fleming PJ, Guest G. Assessment of the psychometric properties of HIV knowledge items across five countries. AIDS Education and Prevention. 2014;26(6):577–587. doi: 10.1521/aeap.2014.26.6.577. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Carey MP, Morrison-Beedy D, Johnson BT. The HIV-knowledge questionnaire: Development and evaluation of a reliable, valid, and practical self-administered questionnaire. AIDS and Behavior. 1997;1(1):61–74. [Google Scholar]
  9. Carey MP, Schroder KE. Development and psychometric evaluation of the brief HIV Knowledge Questionnaire. AIDS Education and Prevention. 2002;14(2):172–182. doi: 10.1521/aeap.14.2.172.23902. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. CDC. HIV Surveillance - Men who have sex with men (MSM) 2016 Retrieved from http://www.cdc.gov/hiv/library/slideSets/index.html.
  11. Centers for Disease Control and Prevention. Diagnoses of HIV Infection in the United States and Dependent Areas, 2011. HIV Surveillance Report. 2015;2013:23. [Google Scholar]
  12. Djokic D, Englund J, Daum R, Martin R, Dozier T, Potts S, Marcinak J. HIV knowledge and attitudes toward HIV testing of South Side Chicago Housing Authority residents. AIDS Patient Care and STDS. 2009;23(1):23–28. doi: 10.1089/apc.2008.0110. [DOI] [PubMed] [Google Scholar]
  13. Ebrahim SH, Anderson JE, Weidle P, Purcell DW. Race/ethnic disparities in HIV testing and knowledge about treatment for HIV/AIDS: United States, 2001. AIDS Patient Care and STDS. 2004;18(1):27–33. doi: 10.1089/108729104322740893. [DOI] [PubMed] [Google Scholar]
  14. Eggers SM, Aaro LE, Bos AER, Mathews C, de Vries H. Predicting Condom Use in South Africa: A Test of Two Integrative Models. AIDS and Behavior. 2014;18(1):135–145. doi: 10.1007/s10461-013-0423-2. [DOI] [PubMed] [Google Scholar]
  15. Ferguson E, Cox T, Irving K, Leiter M, Farnsworth B. A Measure of Knowledge and Confidence in Relation to Hiv and Aids - Reliability and Validity. Aids Care-Psychological and Socio-Medical Aspects of Aids/Hiv. 1995;7(2):211–228. doi: 10.1080/09540129550126722. [DOI] [PubMed] [Google Scholar]
  16. Finch H. The MIMIC model as a method for detecting DIF comparison with Mantel-Haenszel, SIBTEST, and the IRT likelihood ratio. Applied Psychological Measurement. 2005;29(4):278–295. [Google Scholar]
  17. Fisher JD, Fisher WA. Changing AIDS-risk behavior. Psychological Bulletin. 1992;111(3):455–474. doi: 10.1037/0033-2909.111.3.455. [DOI] [PubMed] [Google Scholar]
  18. Garofalo R, Gayles T, Bottone PD, Ryan D, Kuhns LM, Mustanski B. Racial/Ethnic Difference in HIV-related Knowledge among Young Men who have Sex with Men and their Association with Condom Errors. Health Education Journal. 2015;74(5):518–530. doi: 10.1177/0017896914549485. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Glockner-Rist A, Hoijtink H. The best of both worlds: Factor analysis of dichotomous data using item response theory and structural equation modeling. Structural Equation Modeling. 2003;10(4):544–565. [Google Scholar]
  20. Goetz C, Coste J, Lemetayer F, Rat AC, Montel S, Recchia S, Guillemin F. Item reduction based on rigorous methodological guidelines is necessary to maintain validity when shortening composite measurement scales. Journal of Clinical Epidemiology. 2013;66(7):710–718. doi: 10.1016/j.jclinepi.2012.12.015. [DOI] [PubMed] [Google Scholar]
  21. Gomes RRdFM, Batista JR, Ceccato MdGB, Kerr LRFS, Guimarães MDC. HIV/AIDS knowledge among men who have sex with men: applying the item response theory. Revista de Saúde Publica. 2014;48(2):206–215. doi: 10.1590/S0034-8910.2014048004911. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Hambleton RK. Good practices for identifying differential item functioning - Commentary. Medical Care. 2006;44(11):S182–S188. doi: 10.1097/01.mlr.0000245443.86671.c4. [DOI] [PubMed] [Google Scholar]
  23. Herbst JH, Sherba RT, Crepaz N, Deluca JB, Zohrabyan L, Stall RD, Lyles CM. A meta-analytic review of HIV behavioral interventions for reducing sexual risk behavior of men who have sex with men. Journal of Acquired Immune Deficiency Syndromes. 2005;39(2):228–241. [PubMed] [Google Scholar]
  24. Hughes AK, Admiraal KR. A Systematic Review of HIV/AIDS Knowledge Measures. Research on Social Work Practice. 2012;22(3):313–322. [Google Scholar]
  25. Jin FY, Prestage GP, Mao LM, Poynten IM, Templeton DJ, Grulich AE, Zablotska I. "Any condomless anal intercourse" is no longer an accurate measure of HIV sexual risk behavior in gay and other men who have sex with men. Frontiers in Immunology. 2015;6 doi: 10.3389/fimmu.2015.00086. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Jones RN. Identification of measurement differences between English and Spanish language versions of the Mini-Mental State Examination - Detecting differential item functioning using MIMIC modeling. Medical Care. 2006;44(11):S124–S133. doi: 10.1097/01.mlr.0000245250.50114.0f. [DOI] [PubMed] [Google Scholar]
  27. Kalichman SC, Picciano JF, Roffman RA. Motivation to reduce HIV risk behaviors in the context of the Information, Motivation and Behavioral Skills (IMB) model of HIV prevention. Journal of Health Psychology. 2008;13(5):680–689. doi: 10.1177/1359105307082456. [DOI] [PubMed] [Google Scholar]
  28. Mccown W, Johnson J. The Basic Hiv Disease Knowledge Questionnaire - a Rasch-Scaled Instrument to Measure Essential Hiv Knowledge. Psychological Reports. 1991;69(2):543–549. doi: 10.2466/pr0.1991.69.2.543. [DOI] [PubMed] [Google Scholar]
  29. Medley A, Kennedy C, O'Reilly K, Sweat M. Effectiveness of Peer Education Interventions for Hiv Prevention in Developing Countries: A Systematic Review and Meta-Analysis. AIDS Education and Prevention. 2009;21(3):181–206. doi: 10.1521/aeap.2009.21.3.181. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Mustanski B, DuBois LZ, Prescott TL, Ybarra ML. A mixed-methods study of condom use and decision making among adolescent gay and bisexual males. AIDS and Behavior. 2014;18(10):1955–1969. doi: 10.1007/s10461-014-0810-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Mustanski B, Garofalo R, Monahan C, Gratzer B, Andrews R. Feasibility, acceptability, and preliminary efficacy of an online HIV prevention program for diverse young men who have sex with men: the Keep It Up! intervention. AIDS and Behavior. 2013;17(9):2999–3012. doi: 10.1007/s10461-013-0507-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Mustanski B, Johnson AK, Garofalo R, Ryan D, Birkett M. Perceived likelihood of using HIV pre-exposure prophylaxis medications among young men who have sex with men. AIDS and Behavior. 2013;17(6):2173–2179. doi: 10.1007/s10461-012-0359-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Mustanski B, Lyons T, Garcia SC. Internet use and sexual health of young men who have sex with men: a mixed-methods study. Archives of Sexual Behavior. 2011;40(2):289–300. doi: 10.1007/s10508-009-9596-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Mustanski B, Ryan DT, Garofalo R. Associations of sexually transmitted infections with condom problems among young men who have sex with men. Sexually Transmitted Diseases. 2014;41(7):427–432. doi: 10.1097/OLQ.0000000000000150. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Muthen B. A General Structural Equation Model with Dichotomous, Ordered Categorical, and Continuous Latent Variable Indicators. Psychometrika. 1984;49(1):115–132. [Google Scholar]
  36. Nelson LE, Wilton L, Agyarko-Poku T, Zhang NH, Aluoch M, Thach CT, Adu-Sarkodie Y. The Association of HIV Stigma and HIV/STD Knowledge With Sexual Risk Behaviors Among Adolescent and Adult Men Who Have Sex With Men in Ghana, West Africa. Research in Nursing and Health. 2015;38(3):194–206. doi: 10.1002/nur.21650. [DOI] [PubMed] [Google Scholar]
  37. Newcomb ME. Moderating effect of age on the association between alcohol use and sexual risk in MSM: evidence for elevated risk among younger MSM. AIDS and Behavior. 2013;17(5):1746–1754. doi: 10.1007/s10461-013-0470-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Newcomb ME, Mustanski B. Cognitive influences on sexual risk and risk appraisals in men who have sex with men. Health Psychology. 2014a;33(7):690–698. doi: 10.1037/hea0000010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Newcomb ME, Mustanski B. Developmental change in the relationship between alcohol and drug use before sex and sexual risk behavior in young men who have sex with men. AIDS and Behavior. 2014b;18(10):1981–1990. doi: 10.1007/s10461-014-0766-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Oglesby WH, Alemagno SA. Psychometric properties of an HIV knowledge scale administered with populations at high risk for HIV infection. Health Promot Pract. 2013;14(6):859–867. doi: 10.1177/1524839912470453. [DOI] [PubMed] [Google Scholar]
  41. Peltzer K, Matseke G, Mzolo T, Majaja M. Determinants of knowledge of HIV status in South Africa: results from a population-based HIV survey. BMC Public Health. 2009;9:174. doi: 10.1186/1471-2458-9-174. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Prejean J, Song R, Hernandez A, Ziebell R, Green T, Walker F, Group HIVIS. Estimated HIV incidence in the United States, 2006–2009. PloS One. 2011;6(8):e17502. doi: 10.1371/journal.pone.0017502. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Scott-Sheldon LAJ, Carey MP, Vanable PA, Senn TE, Coury-Doniger P, Urban MA. Predicting Condom Use among STD Clinic Patients Using the Information-Motivation-Behavioral Skills (IMB) Model. Journal of Health Psychology. 2010;15(7):1093–1102. doi: 10.1177/1359105310364174. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Skinta MD, Murphy JL, Paul JP, Schwarcz SK, Dilley JW. Thoughts, Attitudes, and Feelings of HIV-Positive MSM Associated With High Transmission-Risk Sex. Health Education & Behavior. 2012;39(3):315–323. doi: 10.1177/1090198111427390. [DOI] [PubMed] [Google Scholar]
  45. Strauss BB, Greene GJ, Philips GL, 2nd, Bhatia R, Madkins K, Parsons JT, Mustanski B. Exploring Patterns of Awareness and Use of HIV Pre-Exposure Prophylaxis among Young Men Who Have Sex with Men. AIDS and Behavior. 2016 doi: 10.1007/s10461-016-1480-0. In Press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Sullivan PS, Peterson J, Rosenberg ES, Kelley CF, Cooper H, Vaughan A, Sanchez TH. Understanding Racial HIV/STI Disparities in Black and White Men Who Have Sex with Men: A Multilevel Approach. PloS One. 2014;9(3) doi: 10.1371/journal.pone.0090514. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Sullivan PS, Rosenberg ES, Sanchez TH, Kelley CF, Luisi N, Cooper HL, Salazar LF. Explaining racial disparities in HIV incidence in black and white men who have sex with men in Atlanta, GA: a prospective observational cohort study. Annals of Epidemiology. 2015 doi: 10.1016/j.annepidem.2015.03.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Teresi JA. Overview of quantitative measurement methods - Equivalence, invariance, and differential item functioning in health applications. Medical Care. 2006;44(11):S39–S49. doi: 10.1097/01.mlr.0000245452.48613.45. [DOI] [PubMed] [Google Scholar]
  49. Thissen D, Steinberg L, Kuang D. Quick and easy implementation of the Benjamini-Hochberg procedure for controlling the false positive rate in multiple comparisons. Journal of Educational and Behavioral Statistics. 2002;27(1):77–83. [Google Scholar]
  50. Villar-Loubet OM, Cook R, Chakhtoura N, Peltzer K, Weiss SM, Shikwane ME, Jones DL. HIV Knowledge and Sexual Risk Behavior Among Pregnant Couples in South Africa: The PartnerPlus Project. AIDS and Behavior. 2013;17(2):479–487. doi: 10.1007/s10461-012-0360-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Wagenaar BH, Sullivan PS, Stephenson R. HIV knowledge and associated factors among internet-using men who have sex with men (MSM) in South Africa and the United States. PloS One. 2012;7(3):e32915. doi: 10.1371/journal.pone.0032915. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Walsh JL, Senn TE, Scott-Sheldon LAJ, Vanable PA, Carey MP. Predicting Condom Use Using the Information-Motivation-Behavioral Skills (IMB) Model: A Multivariate Latent Growth Curve Analysis. Annals of Behavioral Medicine. 2011;42(2):235–244. doi: 10.1007/s12160-011-9284-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Wang WC, Shih CL, Yang CC. The MIMIC Method With Scale Purification for Detecting Differential Item Functioning. Educational and Psychological Measurement. 2009;69(5):713–731. [Google Scholar]
  54. Zumbo BD. Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language assessment quarterly. 2007;4(2):223–233. [Google Scholar]

RESOURCES