Measuring HIV Risk Perception and Behavior: Results from Round 1 of the Cognitive Interviewing Project with young women and men who have sex with men in South Africa

Abstract

Self-reported HIV risk perception and behaviors are used in a variety of settings for diverse purposes, such as HIV prevention program planning and screening. Careful consideration of how youth in high HIV prevalence areas interpret these kinds of questions warrants attention. The Cognitive Interviewing Project (CIP) conducted cognitive interviews on common risk survey items with 30 cis-female and 20 MSM youth (18 to 24), who had recent sex with a male partner, in Cape Town and Vulindlela, South Africa. Results identified a number of potential issues including 1) confusing text; 2) mismatches of terms with local usage; 3) confusion with items requiring self-tailoring; 4) presentation concerns limiting selection of full range of answers; and 5) challenges reporting on information dependent on partner (eg., HIV risk, HIV status of partner). Self-report Items used to identify those at elevated risk for HIV should be evaluated with local populations to optimize shared understanding.

INTRODUCTION

Efforts to prevent HIV infection and estimate HIV-related risk often rely on self-report data to identify, direct and evaluate interventions and programs. Similarly, research attempting to focus on populations at elevated risk for HIV often rely on self-report surveys to identify eligible participants. Survey items that ask about HIV risk perception (eg., chances of getting HIV) and risk behavior (eg., number of partners, ages of partner(s), sex events, event type and use of condoms)¹ have a limited validation evidence base, despite common use. Work conducted to date suggests that many of these items may be particularly prone to under-reporting due to self-presentation bias² and under-reporting of socially “undesirable” behaviors (eg., remaining sexually active during STI treatment) may in fact increase in the context of added pressure (eg., counseling promoting the desired behavior)³. Other issues such as problems in comprehension of items⁴ or responses or their translation and local interpretation⁵ may also erode accuracy in reporting. A critical aspect of validation is the establishment of a shared, clear understanding of what is being asked and what the answer options mean, as well as minimal demand for altering answers to manage self-presentation or mitigate perceived negative outcomes of choosing one answer over another.⁶ One method to determine the extent to which survey items and answer choices have a clear and consistent meaning to those answering the questions is cognitive interviewing.^7–10

Cognitive interviewing is a strategy used in survey science to understand comprehension of questions, demands in terms of memory retrieval or cognitive tasks to move from question to answer selection, decision making processes involved which may include modifying initial answer choices to manage potential consequences of one choice over another.^7,9,10 This strategy has been used with a wide variety of topics within public health research, including adaptation of surveys of sex behavior to youth in Zimbabwe¹¹ and pediatric adherence items in Kenya¹². This and related work has helped to position cognitive interviewing as a crucial part of survey development and adaptations in sub-Saharan Africa particularly for behavior that may be sensitive in nature and uncommonly openly discussed, like sexual behavior^2,13,14.

HIV risk and behavioral items that require respondents to aggregate across months of behaviors (eg., sex events over the past 3 months), estimate percentages (eg., percent of time condoms were used) or relative proportions (e.g., about how much of the time were condoms used) may also create challenges due to numeracy and cognitive demands inherent in abstract and summative exercises. For youth and young adults, who have unique social and developmental characteristics, assessing sex behavior and HIV-risk is particularly worthy of additional exploration especially in high HIV-incidence areas.

To explore the meaning youth make of commonly used sexual behavior and HIV-risk assessment items, we conducted cognitive interviews with young (18–24) women and men who have sex with men (MSM) in two high-incidence, highly researched areas in South Africa- Cape Town and rural KwaZulu-Natal (KZN).

The Cognitive Interviewing Project focused on a variety of assessment items to explore comprehension, understanding, and self-presentation concerns, and collected recommendations for improvements. The full project included three rounds of cognitive interviews. Here we present the findings from items assessing sexual behavior and HIV-risk perception from our first round of interviews.

METHODS

The Cognitive Interviewing Project (CIP) is a joint research project involving the University of Michigan, Michigan, USA; Center for the AIDS Programme of Research in South Africa (CAPRISA), Kwazulu-Natal; and the Desmond Tutu HIV Centre (DTHC), Cape Town, South Africa. The objective of CIP is to explore the meaning young women and MSM ascribe to commonly used survey items in HIV-research studies in high research areas. The first round of interviews focused on a set of items assessing HIV risk behavior and estimation of HIV risk taken from audio computer assisted self-interviews (A-CASI), CASI without audio, and interviewer collected tools from trials and research in the HIV Prevention Trials Network (HPTN) and local studies at CAPRISA and DTHC.^15,16 Both CAPRISA and DTHC served as recruitment and implementation sites. Each are active clinical research sites, located in areas with ongoing HIV epidemics, with robust histories of engagement in large-scale HIV prevention and treatment studies, and have longstanding relationships with communities of youth who presently or have engaged in HIV-related.

Population and Procedures.

Youth (ages 18 to 24) were recruited at outreach and site events, consented and engaged in one audio-recorded cognitive interview (see measures below). Inclusion criteria included: self-reported female or male sex, between the ages of 18–24 years, self-reported sexual activity with a male partner within the past 3 months, willing/available for a 60-minute interview, willing to be audio recorded, and speaks English or local language (isiZulu or isiXhosa). Age and sex were recorded among those who enrolled. Although current or prior experience as a research participant was not an enrollment criteria, recruitment strategies (through site based activities and inquiring among youth attending visits for other studies) enriched for youth how had experience with study research procedures, such as interviewer or computer administered surveys focused on sexual health and behavior.

Enrollment targeted 15 cis-females from each of the two participating sites (total n=30) and 10 cis-males identifying as MSM (total n=20), for a total of 50 interviews. Interested youth were consented using IRB study procedures and forms. Interviews were conducted in private locations, recorded, transcribed and translated (to English). Participants were reimbursed 70 RAND (equivalent to slightly under $5.00 USD) and provided snacks during the interview. No identifying information (name or contact number) was retained after scheduling and completing the interview, and not collected at all for individuals interviewed immediately following consent. De-identified transcripts were shared with the coding team. Audio recordings, only retained locally for cross-checking transcription, were destroyed at the completion of the final round of data collection in CIP. All procedures and material were reviewed and approved by all participating site IRBs.

Interviewers and Training.

A total of 5 interviewers (2 in Cape Town and 3 in KZN) were trained to conduct cognitive interviews, and each interviewer completed a series of mock interviews, for which feedback was provided before study launch. Neutral interviewing approaches were emphasized with multiple opportunities for practice.

Measures

Survey Questions and Responses.

Items for cognitive interviewing included sets of items assessing perception of HIV risk, sexual behavior (events), partner types, and other items known to be associated with higher risk for HIV (transactional sex and age of first sex). Items and response options were selected to reflect commonly used risk assessment questions and answer choices, which we drew from measures used in a number of computer assisted surveys from various studies.^15–17 Item selection for inclusion underwent a series of core team and external expert reviews. For each set of items (assessing risk perception and general risk and assessing behaviors associated with elevated HIV risk), 5 primary items were selected, although some items had a series of follow up questions that were translated from English to Zulu and isiXhosa with cross-translation and refinement for language accuracy by the research team. Items within each set were then randomized to create unique interview guides that maintained the same order in terms of set presentation (risk perception questions followed by risk behavior questions), but within each set the 5 items had a random order of presentation to minimize potential order effects. Items and response options are detailed in the results.

Cognitive Interviewing.

A semi-structured interview guide was used to assess concordance between item intent and meaning attributed to each item (including specific question, response options and any lead-in or explanations provided for or within the questions). The interviews were conducted in two phases with practice prior to each phase. In the first phase, participants were asked to use the “think aloud” process as they read and responded to each of the questions. Interviewers would provide guidance and support for articulating one’s thoughts during this process but would not probe for explanation or interpretations. After completing all 3 sets, the interviewer implemented phase 2, where participants returned to each question and responded to probes regarding: comprehension, retrieval, judgment and response. Comprehension reflected the participants’ understanding of the question, which included identifying the information being sought. Retrieval involved the way in which respondents recall information. Judgment encompassed the way in which participants summarize the information including determining relevance of memories and drawing inferences to provide a response. Response emulated the way participants report back the information, including mapping summarized information onto the response category. For participants reporting living with HIV at the time of the interview, their perceptions and recommendations for HIV risk-perception items were collected in phase 2 of the interview. These two main processes, (think-aloud and active-probing) are common cognitive interviewing strategies, often used in the improvement of survey items.¹⁸ At the conclusion of the probing section of the interview, the participants were asked whether they would prefer to complete the survey as an in-person interview, or whether they believed using a tablet or computer would be better to address the survey items.

Analyses.

De-identified translated transcriptions of each interview were analyzed by a trained team of three coders and the project PI using an adaptation¹⁹ of framework analysis²⁰. Content of discussions were first organized into frames containing responses to each item presented in the cognitive interview. These frames were then combined across the three distinct item sets (risk perception items, HIV risk behavior items and adherence items). Item sets were then iteratively reviewed to create a codebook identifying content themes within each set through thematic analysis^21,22. Identification of types of issues emerging in the areas of comprehension, retrieval, judgment and response formed the basis of summaries across items, followed by detailing item specific issues.²³ Coders met regularly and discussions of themes and refinement of them involved the full CIP team (all investigators, coders, and on-site interviewers). Coders met minimum criteria for applying the first framing pass (80% or greater agreement via the Dedoose program’s test sub-program). Consistency in application of thematic codes was achieved through team discussion to reach consensus on any identified discrepancies to ensure negotiated agreement²⁴. Main themes were then further discussed, distilled as having enough saturation to suggest a main (versus less consistent) theme, and main themes were characterized with example quotes. Emerging content that presented opportunities for insights despite being less common in discourse were also considered, although these are presented separately.

RESULTS

Sample.

A total of 50 interviews were collected between July and November 2016 for round 1 of the cognitive interviews. Participants were an average of 21 years of age (standard deviation of 1.76), ranging between 18 and 24. About 2/3 were female (60%) per protocol, and 26 interviews were conducted at CAPRISA and 24 at DTHC. Most participants (82%) disclosed an HIV negative status, 8 (16%) disclosed living with HIV, and 1 (2%) interviewee’s HIV status was not clear. Most participants (73%) had current or recent experience with participating in HIV-focused research studies, of these the most common type of study (72%) was in biomedical prevention (eg., oral PrEP).

Themes identified.

Across items focused on risk perception and risk behavior, 5 main observations emerged (Table 1): (1) unclear and confusing terms or specific words; (2) language or phrasing of questions mismatched to local language or common in-group language creating misunderstanding or misinterpretation of questions; (3) confusion over and dislike of items that required self-tailoring (selecting the term that matched the person’s sexual practices or anatomy); (4) social desirability and presentation concerns; and (5) reflecting on partner’s behavior and trustworthiness when asked to estimate one’s own HIV risk, particularly for young women. Each theme is detailed with example quotes in Table 1.

TABLE 1:

THEMES (across items)

Theme	Subtheme	Definition/Explanation	Example Quotes
Unclear and Confusing Terms/Words	Unclear Vocabulary	Use of “hole” in explaining type of sex was not well understood; options for current practices people may use as prevention (eg., “withdrawal” before ejaculation) caused confusion.	P: “On the hole part. […] My issues was that I had a confusion as to what a front and back hole are. […] They are referring to the female. I got confused as to what a back hole was.” -Female participant, 23, Vulindlela P: “No, the only withdrawal I know is the one for withdrawing money.” -Female participant, 20, Cape Town P: Withdrawal means, “let’s just drop it and not have sex”. I: So it is the same as abstinence? P: Yes -Male participant, 21, Vulindlela
	Divergent Interpretation	Abstinence was interpreted by some as a sporadic, non-permanent behavior and by others as long term celibacy. Interpretation of “partner” differed among participants. Some perceived partner to only refer to a romantic relationship, while others interpreted partner as someone who engages in sexual activity with them. Participants expressed varied interpretations of “sexual debut,” including or excluding forced sex and rape, and for MSM whether or not to report first time of heterosexual activity or first sex with another man.	P: “[I use] Abstinence somewhat, because I didn’t have exchange with this guy I think it was last week we didn’t have sex.” -Male participant, 24, Vulindlela P: It [abstinence] is when you refrain from having sex. It is refraining from sex or behaving yourself. I: You don’t have sex at all? P: Yes, you don’t have sex at all. -Male participant, 20, Vulindlela P: Those that have been raped? How would I change it and put it…? But it does need changing cos there are those people who have been raped, maybe a person has been raped at 12 years maybe, I won’t say I started having sex at 12, yes. I’ll speak about the age that I enjoyed it. - Female participant, 19, Vulindlela
	Long Questions are Confusing	Questions that were long, had long lead-in explanations or multiple clarifiers in brackets were more difficult to understand.	P: “It’s like a book the paragraph is too much. Just write anal sex is penis to anus and vaginal sex penis to vagina that’s it.” - Male participant, 24, Vulindlela
Language mismatched (with local language, words used commonly in one’s community)	“Untranslatable” for certain sexual terms	Participants (females) expressed there not being a word for anal sex in isiZulu.	P: “I would change it by saying “all in all how many times do you think you got involved in sexual intercourse performing anal sex”, it is just that I do not know what I am going to say in IsiZulu when it comes to anal sex” I: “Hmm. The problem is there is no term for Anal sex that you know.” P: “Yes for IsiZulu.” -Female participant, 24, Vulindlela
	Words fail to reflect common terms in MSM community	Male participants expressed that there are specific terms that the gay community uses to refer to sexual practices that are different than standard vocabulary	P: And it says where the penis is put inside the anus, meaning it’s talking about the person who inserts and then it says vaginal sex where the penis is put inside, which means it’s talking about the tops, not us bottoms. I: So how would you, what do you think this question is asking? P: Like it’s asking if you have ever had sex, I have never had sex I am a receiver, I never insert my thing on someone, that’s why I say this thing is talking to the giver. I: So in your understand, this question is only for top people? P: Yes. -Male participant, 20, Vulindlela
Self-tailored items	Combining terms to allow MSM and women to answer the same question by selecting the term that applied to them was disliked	Some MSM felt questions were only relevant to tops, not bottoms, and vice versa. MSM who read a question that didn’t apply to them often expressed confusion as to why the researcher would ask the question.	P: Was it important for them to say anal sex? Well…for someone who reads this they’ll just know you are talking about gay guys, because it’s just stated as anal sex, and that’s only gays who do anal sex, ok not only gays but the way it’s asked, it’s straightforward, because look it says anal sex where the penis is put inside the vagina…yes, it’s not relevant in gays. I just feel like it shouldn’t even say anal sex, it should just be ‘have you had sex?’ -Male participant, 21, Vulindlela
Social Desirability	Invasion of Privacy	Participants expressed worry about the lack of confidentiality and the possibility that the interviewer may tell others.	P: “Ok, I have nothing to hide, yeah but someone else would feel uncomfortable.” […] P: “Because it’s like you are invading their privacy and stuff like that, maybe they don’t even trust if you [the interviewer] that will not go outside and tell people that yhoo that one has 10 partners […]” -Male participant, 21, Cape Town, SA
	Managing impressions	Participants expressed worry about what others might think of them due to implied or overt stigma	P: Yes, there are some men who don’t want people to know that they practice anal sex. I: Only men? Is there any other way that men can practice sex besides anal sex? P: I mean guys who practice “both.” I: So you mean a guy who is bisexual or straight but practice anal sex with their girlfriend, for insist? P: Both actually. But especially straight guys, they don’t want to admit it. -Male participant, 23, Cape Town P:[Laughs] “someone would not want to choose 100 because they would feel like people will say they already have HIV but the question says what are your chances of getting HIV.” […] P: “They wont feel comfortable, they will think that the interviewer will think that they sleep around.” -Female participant, 20, Cape Town
	No social script for how to respond	For heterosexual women, asking questions about anal sex is not in accordance with social and cultural norms	P: “I don’t have a problem with it, it’s easy for me to answer it because I don’t do anal sex, but I think someone who does it will be a little bit uncomfortable because as I said people like me don’t believe in anal sex…but maybe someone who does it knows that the people who don’t do it, let’s say if I was a guy and I was interviewed by another guy and the interviewer is straight (does not do anal sex) and then he asks if I ever did anal sex, it would not be easy for me to answer because I know in our culture, community, and according to our parents it is wrong to do so, so I would be uncomfortable even though maybe I know I am bisexual, maybe I am not gay, I have a girlfriend but I do anal sex, I would be uncomfortable even if I think it’s right, but I would be knowing that people think it’s wrong.” -Female participant, age unknown, Vulindlela
Estimation of risk involves estimation of partner behavior and trust	Partner trust impacts risk perception and risk behavior particularly for young women-tension between behavior as the cause of risk and impact of partner behaviors on risk	Chances/percentages of HIV depend on partner trust Condom use depends on trust	P: “It is asking me what I think my chances of getting HIV are. I choose number four because I trust myself and I have only one partner. I do not cheat on my partner, I would get HIV if he would cheat on me.” -Female participant, 20, Vulindlela P: “Because that’s [choosing 40 or 50% risk] when there is no trust in the relationship and maybe your partner has cheated on you before so that’s why there are greater risk of the person to be affected by the virus.” -Female participant, 21, Vulindlela P: […] “If I don’t trust someone or they cannot give me an answer to a question of cheating then we have to use a condom when we have sex, that’s my way of protecting myself from them.” -Female participant, age unknown, Cape Town

P: Participant; I: Interviewer

By item, specific concerns and recommendations were identified (Table 2) during the think-aloud and active probing phases of the interview. Although all items had some potential for improvements per participant discussions, items varied in terms of whether the main concerns were word choices or phrasing and the extent to which social desirability concerns would erode accuracy in selecting responses. Items using mixed terms to allow for a single question to be used for females and males who have sex with men that required “self-tailoring” were generally disliked. Additionally, the manner in which participants assumed their answers would be interpreted by others as reflections on one’s personality or enduring traits was noted throughout, suggesting that response options would be driven in part by self-presentation concerns (what a given response would “say” about the person, their values or attributes).

TABLE 2.

ITEM SPECIFIC CONCERNS

ITEM
Different protective strategies have different levels of protection against HIV and STIs. Please indicate the protection methods you use?
(Select all that apply)
□ Condoms
□ Withdrawal
□ Abstinence
□ Other
If other, please specify:
SPECIFIC CONCERNS FROM INTERVIEWS
● Inconsistent interpretation of abstinence
● Withdrawal not understood
● Options not consistent with practices in MSM community
● Long lead-in
● “None” not an option
ITEM
Please indicate your level of RISK of being infected with HIV.
Response options: 0% to 100% in 10% intervals presented horizontally with additional explanation of:
0% is no risk and 100% is certain
SPECIFIC CONCERNS FROM INTERVIEWS
● Concern that reporting high levels of risk reflects poorly on participant- as an indicator of recklessness or not caring about oneself
● When with one partner, discomfort in focus on self rather than recognizing contributions of one’s partner to risk (females particularly)
● When with one partner, noted difficulty in selecting a response as high risk would suggest distrust of partner
● Different mental strategies were used to “calculate” risk, including relative contributions of own and partner(s) behaviors, desire to manage impressions, and estimates of how common HIV is/was in one’s community.
● Dislike of scale as percentages as they have absolute meaning and are not arbitrary-concerns that 100% would be for people living with HIV already.
ITEM
What is the chance that you will get HIV at some stage of your life?
Response options: 0 to 8 in intervals of 1 presented horizontally with additional explanations at
0 (no chance (definitely will not happen)),
2 (Low change (probably will not happen)),
4 (Maybe (Could happen/Could not happen)),
6 (High change (will probably happen)) and
8 (Very high change (will definitely happen))
SPECIFIC CONCERNS FROM INTERVIEWS
● Similar concerns to those voiced in response to previous item.
● Despite using a more arbitrary scale, participants had similar concerns about selecting responses in the upper range (reflects poorly on self-care, requires knowledge of partner behavior and challenges trust).
● Preference noted for defining each number of the scale rather than every other.
ITEM
Have you ever had sex? (When we say sex, we mean anal sex– where the penis is put inside the anus– and/or vaginal sex – where the penis is put inside the vagina)?
□ Yes
□ No
SPECIFIC CONCERNS FROM INTERVIEWS
● As with item 4, general dislike for self-tailoring (needing to pick the part of the statement that applies).
● As with item 4, confusion over whether or not the question is asking about insertive and receptive anal sex for MSM, or both.
ITEM
How old were you when you first had sex?
Response option: Open text field
SPECIFIC CONCERNS FROM INTERVIEWS
● For MSM, some confusion over whether to report first sex with a man (versus sex with a woman).
● For participants who had first sex in early teens, some self-presentation concerns over open space response-preference for ages to be listed starting with very young to allow for early teens to appear more in the middle of the options (to appear “average”).
● Concern/desire for clearer instructions for participants whose first sex event was forced (rape, molestation) in terms of whether or not to present that age or the age of first consenting sex, and related concerns that having to report the age of first sex inappropriately makes that participant “appear risky”.
ITEM
Have you used condoms (male or female condoms) every time you had sexing the past 3 months? (meaning, a condom was on at all times when your penis/your partner’s penis was inserted in your anus/your partner’s anus or vagina)
□ No
□ Don’t think so
□ Not sure
□ I think so
□ Yes
SPECIFIC CONCERNS FROM INTERVIEWS
● General dislike for self-tailoring (needing to pick the part of the statement that applies).
● Language mismatch on options that are more subtle than yes or no-poor translation and concept of slight gradations of yes or no not applicable.
● Some confusion over whether or not the question is asking about insertive and receptive anal sex for MSM, or both.
● Assumes knowledge of condom use which some participants noted they could not tell for certain if partners used condoms throughout sex event(s).
ITEM
Have you had a partner 5 years or more, older than you in the last 3 months?
□ No
□ Don’t think so
□ Not sure
□ I think so
□ Yes
SPECIFIC CONCERNS FROM INTERVIEWS
● As with item 4, gradations between Yes and No not well understood or conceptually relevant.
● Ease of comprehension challenged by having two time constraints listed in close proximity in one sentence.
● Self-presentation concerns, as most reflected on feeling a “yes” response reflected poorly on the person.
● Social norms include not asking a partner’s age, particularly male partners—forcing a “guess” with some discrepancies between participants in how they come to an estimate of partner age.
ITEM SET
These next questions ask about you sexual behaviors over the past 3 months. *When we say sex we are talking about an exchange of any duration of time between you and one or more other people where at some point in that exchange your/your partner’s penis was inserted in your/your partner’s anus or vagina regardless as to whether or not ejaculation was involved.*
a. About how many different partners have you had sex with over the past 3 months? It is OK for you to take your best guess for this number.    Response Option: Open text field
b. With about how many of these partners did you have sex without a condom? It is OK for you to take your best guess for this number.    Response Option: Open text field
c. About how many of these partners with whom who you had sex without a condom in the past 3 months were HIV-positive? It is OK for you to take your best guess for this number.    Response Option: Open text field
d. All together, about how many times did you have sex (times when you had insertive or receptive vaginal and/or anal sex) over the past 3 months? It is OK for you to take your best guess for this number.    Response Option: Open text field
e. All together, about how many of these times did you use a condom (meaning, a condom was on at all times when your penis/your partner’s penis was inserted in your anus or vagina/your partner’s anus or vagina)? It is OK for you to take your best guess for this number.    Response Option: Open text field
SPECIFIC CONCERNS FROM INTERVIEWS
● General dislike for self-tailoring (needing to pick the part of the statement that applies).
● As with item 4 and 6, confusion over whether or not the question is asking about insertive and receptive anal sex for MSM, or both-preference for items to be asked separately for women and MSM (receptive separate from insertive).
● Variable interpretation among participants of “It is OK for you to take your best guess” with some feeling this was an invitation to edit self-report to “less risky”.
● Perceived as invasive-preference for an explanation of why this information is needed or how it is used.
● Interval of recall too long for sexually active participants.
ITEM
Have you ever had anal sex (Anal sex–where the penis is put inside the anus)?
□ Yes    □ No
SPECIFIC CONCERNS FROM INTERVIEWS
● Language/translation issues with having words to convey anal sex.
● Misinterpretation of anal sex as “sex from behind”.
● Desire among MSM to have this ask separately for receptive and for anal sex.
● Social desirability concerning cultural taboo of anal sex for females.
● Phrasing fails to capture coerced or forced anal sex-females noted greater potential comfort with item if it included text explicitly recognizing anal sex may have been in those contexts
ITEM
Have you had sex with someone for food, airtime, money, clothes, a place to stay etc. in the past 3 months?
□ No
□ Don’t think so
□ Not sure
□ I think so
□ Yes
SPECIFIC CONCERNS FROM INTERVIEWS
● As with other items the gradations between Yes and No not well understood or conceptually relevant.
● Phrasing should clearly state examples as such-some discussion over some examples being more common and creating difficulty selecting a response when some but not all examples fit with one’s experiences.
● Self-presentation concerns included clear recognition that this is associated with someone who it “risky”.
● Future items should emphasize the survival aspect of transactional sex to allow for more accurate responses

Discussion

Results suggest that despite frequent use of items the same as, or similar to those we evaluated, discrepancies between intended and interpreted meaning and variability between participant interpretation of items may erode accuracy in data collection around HIV risk perception and behaviors. Issues identified largely centered on lack of clarity of language and phrasing, social desirability, and factors impacting confidence in one’s evaluation of their risk for HIV, such as partner trust or lack of access to information about what partners are actually doing that may subsequently elevate participant risk. Specific recommendations for changes to items and answer options were provided or extracted from discourse, which are the focus of subsequent rounds of cognitive interviewing in CIP and could be explored in large sample studies for potential impact on accuracy. Current results suggest potential promise in the following modifications and recommendations:

• Move from open-field (enter an age or a number) to a range or numeric selection for counts or reports that evoke self-presentation concerns (e.g., provide age ranges for age of first sex that intentionally start below expected lowest age).

• Avoid self-tailored or combined items and instructions (e.g., “meaning, a condom was on at all times when your penis/your partner’s penis was inserted in your anus or vagina/your partner’s anus or vagina”) and use sex and partner specific item sets (e.g., filter participants to appropriate items about insertive anal sex, receptive anal sex, insertive vaginal sex, or receptive vaginal sex separately).

• Consider offering definitions of various sex behaviors only as needed (e.g., click a button or some other action to receive specific definition) versus provided to all to reduce long explanations.

• Exercise caution when using “permission statements” (e.g., “It’s OK to take your best guess”) as these may be interpreted as permission to present in a favorable manner.

• Consider shorter recall periods for youth when asking for counts of events (e.g., use 1 month rather than 3 months).

• Work with local communities to better represent gradations of agreement and disagreement to use for response options and consider response scale modifications if gradations lack cultural relevance.

• Consider alterations to the phrasing of HIV risk perception items (e.g., ask about general feelings of safety and/or ask about worry or concern around HIV) to lower self-presentation demands.

Even with these suggestions, validation in terms of comparison of item response to objective measures is require. While cognitive interviewing provides valuable information about comprehension and meaning or interpretation of items and response options, it is limited by not addressing concurrent or predictive validity of items. It is difficult to imagine strong validity for items that have substantial issues in comprehension or interpretation, however, it is possible to have items that raise some concerns but nonetheless perform well in predicting outcomes.

For well over a decade, single variable and predictive models of risk for HIV infection among MSM have used self-report variables (eg., condomless receptive anal intercourse, number of partners).^25–28 Because reliable biomarkers of HIV-risk remain unavailable in most settings,²⁹ self-report, with its well-recognized inaccuracies,^29,30 will likely continue to represent a key measure of HIV risk. As such, there is a critical role for cognitive interviewing to improve one of the most basic principles in survey development--a respondent should know what is being asked and what the answer choices mean³¹.

It is additionally important to note that participants in the CIP were all young women or MSM from South Africa. Findings could lack generalizability, however observations from other studies conducted in sub-Saharan Africa do appear to be in line with our findings^11,32–34. The role of cultural context and self-presentation in understanding and responding to questions about anal sex among women¹¹, frustration with confusing items and challenges in estimating events and partners³² as well as risk of HIV infection³³ has been identified in other work including other groups and countries. Results of reviews of sexual behavior assessments focused on adolescents, largely in the US³⁴, also support several of the situational and interpretation findings we identified. Thus, we do have a degree of confidence in the generalizability of the issues we identify as relevant to many young adult populations. However, concerted efforts to combine cognitive interviewing results from diverse populations may help in the creation of measures that have wide-scale appeal. Attending to local sample and community while also prioritizing harmonization of measures can be particularly challenging. The alternative of using surveys that have consistent items but variable interpretation across communities in multi-site trials and projects may present an even greater challenge in data interpretation.

Recent work to identify predictive risk measures for adolescent and young adult populations include the evaluation of the VOICE risk score³⁵ among youth in participating in HPTN 068³⁶. Despite its validated performance with adult women in Africa, the tool, which uses a mix of demographic (age), observed (sexually transmitted infection) and self-reported factors (eg., whether one’s partner has other partners, alcohol use), did not perform well in their sample of young women as a predictor of HIV infection or as a method to identify PrEP candidates.³⁷ The authors called attention to the need for additional work in the area of identifying more germane risk items, and further suggested that even if objective risk tools could be optimized, questions concerning perceptions of HIV-risk remain critically important for engaging youth in prevention. Questions and answers that could prompt discussion, screen for potential interest in new or better prevention strategies, or gauge one’s experience of HIV risk remain important to implementation and scientific understanding of risk perception dynamics. Our results from cognitive interviews with youth living in HIV endemic, highly researched areas provides some guidance for item and answer construction that can be used in ongoing efforts to develop valid risk assessments and screening tools. Moreover, attending to how participants receive, experience and react to questions often used in HIV-related research has considerable merit in its own right. Efforts to phrase items and offer response options or instructions that minimize non-essential content associated with negative, anxiety provoking or stressful reactions should be prioritized.

Conclusion.

Cognitive interviewing to disentangle areas where HIV risk survey items may create confusion, variable interpretation or social demands is an important step in building accurate approaches to identify and monitor HIV risk. Based on the current results, revised items and additional probes are suggested and will be further evaluated in subsequent rounds of cognitive interviewing in the CIP. Developing risk perception items that explicitly recognize risks outside of one’s control (e.g., partner(s) behavior), use of number selection versus fill-in options, avoidance of terms that lack cultural relevance or create translation issues, and careful attention to potential misinterpretation of add-in definitions or permission statements are important areas for future consideration. Cognitive interviewing provides important information for item construction but does not ensure accuracy. Large sample evaluations of accuracy between objective behaviors and self-report for items that have demonstrated consistent and clear concordance through cognitive interviewing is needed.