Behavioral Assessments in Russian Addiction Treatment Inpatients: A Comparison of Audio Computer-Assisted Self-Interviewing and Interviewer-Administered Questionnaires

Abstract

Purpose:

Assess agreement between reported sex and drug use behaviors from audio computer-assisted self-interviewing (ACASI) and interviewer-administered questionnaire (IAQ).

Method:

Participants (N = 180) enrolled in an HIV intervention trial in Russia completed ACASI and IAQ on the same day. Agreement between responses was evaluated.

Results:

Of the 13 sex behavior questions, 10 items had excellent agreement (kappas/ICC 0.80–0.95) and 3 items had moderate agreement (kappas/ICC 0.59–0.75). The 3 drug behavior questions had excellent agreement (kappas/ICC 0.94–0.97). Among HIV-specific questions asked of HIV-positive participants (n = 21) only, 2 items had excellent agreement (kappas 1.0) and 3 items had moderate agreement (kappas 0.40–0.71).

Conclusions:

Assessment of drug and sex risk behaviors by ACASI and IAQ had generally strong agreement for the majority of items. The lack of discrepancy may result from these Russian subjects' perception that computers do not ensure privacy. Another potential explanatory factor is that both interviews were delivered on the same day. These data raise questions as to whether use of ACASI is uniformly beneficial in all settings, and what influence cultural factors have on its utility.

Obtaining information on self-reported risk behaviors related to HIV transmission has been both a critical and a challenging factor throughout the HIV epidemic. In HIV-related studies, patients may need to disclose sensitive, embarrassing, or even illegal information about sex and drug-related activities. Some individuals may deny engaging in what are perceived to be undesirable behaviors, resulting in social desirability bias.¹ This bias could over- or underestimate the effectiveness of intervention programs. Therefore, truthful reporting of risk behaviors is crucial for unbiased assessments.

Computer-assisted self-interview (CASI) and audio computer-assisted self-interviewing (ACASI) allow patients to complete questionnaires on their own via a computer. In an ACASI system, patients read the questions on a screen and listen to them through headsets, reducing potential literacy barriers.² Traditional methods of collecting sensitive information, such as interviewer-administered questionnaires (IAQ), require direct or indirect involvement by research staff. Research participants are thought to over-report desirable behaviors or under-report undesirable behaviors when interviews are done in person.²

Newer methods, such as CASI and ACASI, may increase privacy and reduce reporting bias compared to IAQ,³ although in the United States the evidence is mixed.² For example, among sexually transmitted infection clinic patients who completed both ACASI and IAQ, participants reported higher rates of sex risk behaviors to ACASI but there was no difference in rates of drug risk behaviors⁴; similar results were found among sexually transmitted infection clinic patients who were randomized to ACASI or IAQ.⁵ Studies of drug users generally have shown more reporting of sensitive behaviors to ACASI than IAQ.^6-8

Use of CASI and ACASI systems has been increasing internationally in diverse populations such as adolescents in Vietnam,⁹ Kenya,¹⁰ and India¹¹; alcohol and drug users in Brazil¹²; women in Zimbabwe¹³ and Kenya¹⁴; urban market workers in China¹⁵; and community volunteers in China, India, Peru, Russia, and Zimbabwe.¹⁶ All of these studies compared reporting by ACASI to IAQ. Four found higher prevalence of sensitive behaviors among those using ACASI. Adolescents in Vietnam who were randomized to ACASI were more likely to report risky sexual practices than those who were randomized to IAQ.⁹ Adolescent girls in Kenya who were randomized to ACASI reported significantly different sexual practices than girls who were randomized to IAQ.¹⁰ Drug users in Brazil who were randomized to ACASI were more likely to report multiple drug use and risky sexual behaviors compared to those who were randomized to IAQ.¹¹ Finally, breastfeeding women in Kenya who were assigned to complete both ACASI and IAQ in a randomized crossover design were more likely to report sensitive behaviors such as less time breastfeeding or earlier introduction of complementary foods to ACASI.¹⁴

The other three studies found mixed results. In India, adolescent girls were more likely to under-report sensitive sexual behaviors in ACASI than in IAQ. Adolescent boys' responses depended upon the type of sexual behavior; some were overreported in ACASI compared to IAQ, some underreported, and some the same.¹¹ In Zimbabwe, 76% of women had no differences between ACASI and IAQ responses.¹³ Urban market workers in China were more likely to report engaging in lifetime sexual intercourse during IAQ than ACASI, but there were no differences in the number of lifetime sexual partners or in responses to sexually transmitted disease (STD)-related questions.¹⁵ Finally, in a multi-country comparison of computer-assisted personal interviewing and ACASI, concordance varied by country, although most participants' responses did not differ by mode.¹⁶ In all of these studies, participants completed both ACASI and IAQ but were randomized to which they completed first.

Russia has the fastest growing HIV epidemic in Europe, with an estimated 940,000 infected since the mid 1990s.¹⁷ HIV transmission was initially predominantly through injection drug use¹⁸ but is spreading to the general population via sexual transmission.^17,19 Given the importance of the HIV epidemic in Russia, further research about disease transmission in this country will require understanding the advantages and disadvantages of research methodologies assessing risky behaviors.

The objective of this study was to assess the agreement between responses obtained from ACASI and those obtained from face-to-face interviews in Russian participants who completed both interview modes. Specifically, our goal was to evaluate whether participants would be more likely to overreport desired behaviors and underreport undesirable behaviors in face-to-face interviews compared to ACASI. Given the awareness of past collection of personal information on individuals with detrimental consequences or incomplete privacy assurances in some Eastern European countries, it was hypothesized that the use of recording devices would not yield more revealing information in the Russian setting. We compared responses to sexual and drug use behavior questionnaires collected as part of the Russian PREVENT Study (Partnership to Reduce the Epidemic Via Engagement in Narcology Treatment).

METHOD

Study Design and Participants

The Russian PREVENT study was a randomized controlled trial of men and women with alcohol and/or drug dependence recruited from two inpatient substance abuse treatment facilities (narcology hospitals) near St. Petersburg, Russia. Trained physician research associates approached patients after initial detoxification to assess eligibility, offer participation, and conduct assessments. Criteria for participant eligibility included the following: age 18 years and older, a primary diagnosis of alcohol or drug dependence, no alcohol and other abused substances for at least 48 hours, reported unprotected anal or vaginal sex in the past 6 months, willing to undergo HIV testing as per standard narcology hospital counseling and testing protocol or previously diagnosed as HIV infected, able to provide reliable contact information including a home telephone number, and an address within 150 kilometers of St. Petersburg. Patients not fluent in Russian or with cognitive impairment based on the research associates' clinical judgment were excluded from the study. All participants provided written informed consent prior to enrollment in the study. The Institutional Review Boards of Boston Medical Center and St. Petersburg Pavlov State Medical University approved this study. PREVENT had 181 enrolled participants, but 1 person was missing ACASI responses and therefore was dropped from these analyses. The current analyses focus on data collected at the baseline assessment.

Participant Assessment

Participant recruitment and follow-up occurred from October 2004 through December 2005. Baseline assessments measured risk behaviors by IAQ and an ACASI system. Interviewers were blinded to intervention group. All interviews were conducted in Russian, including those done by the ACASI system. Participants were compensated the equivalent of US$ 5 for the baseline assessment.

Instrument Design and Data Collection

Programmers at the Boston University School of Public Health Data Coordinating Center developed the ACASI system in Microsoft Access. Paper forms were created in English, translated into Russian by the Russian investigators, and back-translated by the Data Coordinating Center. The audio track was recoded with a male and a female voice, and the participants were allowed to choose which they preferred. Participants completed both the IAQ and the ACASI on the same day. There was not a protocol for a consistent pattern to whether the ACASI or the IAQ occurred first.

Interventionists (two psychiatrists and a psychologist trained in HIV and addiction) were trained to deliver the IAQ by US collaborators using a standard curriculum. The lead interventionist underwent an initial training in English in the United States. A subsequent 3-day training in St. Petersburg with simultaneous translation allowed multiple role-playing sessions to be observed and critiqued by a behavioral psychologist on the US study team.

Questions about sex and drug risk behaviors came from the RESPECT study (has primary sex partner, has secondary sex partner, mean number of times unprotected vaginal sex with primary partner past 3 months, mean number of times unprotected vaginal sex with secondary partner past 3 months²⁰) and the Risk Assessment Battery ([RAB]; heterosexual, multiple sex partners past 6 months, 4+ opposite sex partners past 6 months, same sex partner ever, buy sex, sell sex, used condoms past 6 months, unsafe drug use past 6 months, shared needles past 6 months, mean Drug Risk Assessment score²¹). Questions on HIV disclosure, asked only of HIV-infected subjects, did not come from a specific instrument.

Statistical Analysis

Descriptive statistics such as means, medians, standard deviations, and proportions were used to describe the study population. Agreement between responses obtained from IAQs and ACASI was assessed using the kappa statistic for dichotomous variables and the intraclass correlation coefficient (ICC) for continuous variables. We used the following guidelines to interpret values of kappa and ICC: <0.4, poor agreement; 0.4–0.75, moderate agreement; >0.75, excellent agreement.²² In addition, we compared participant responses from the two interview methods using McNemar's test for paired dichotomous variables and the Wilcoxon signed rank test for paired continuous variables. Exact p values were used for McNemar's test when the number of discordant pairs was <20. Additional exploratory analyses were conducted stratifying by drug diagnosis (yes vs. no) and HIV status (positive vs. negative). All analyses included only data collected at the baseline visit.

RESULTS

Study participants, described in Table 1, included the following: 75% male, 94% high school graduates, 33% married, and 15% HIV infected. Responses to sex behavior questions by ACASI and IAQs are presented in Table 2. Of the 13 sex behavior questions, 10 out of 13 (77%) items (heterosexual, primary or secondary sex partners, multiple sex partners, 4+ opposite sex partners, same sex partner, sex trade, any sexually transmitted diseases, any condom use) had excellent agreement (kappas 0.80–0.95). Three out of 13 (23%) items (prior HIV testing and number of unprotected sex episodes with primary and secondary partners) had moderate agreement (kappas/ICC 0.59–0.75).

Table 1.

Baseline characteristics of Russian narcology patients from the PREVENT cohort (N = 180)

Characteristic
Median age, years (IQR)	30 (26–40)
Male	135 (75)
Employed full time	89 (49)
High school graduate	169 (94)
Married	60 (33)
Diagnosis  Alcohol  Heroin  Alcohol and heroin	107 (59) 58 (32) 15 (8)
HIV infected	27 (15)

Note: Data are number (%) unless otherwise specified. IQR = interquartile range.

Table 2.

Baseline sex behaviors reported by audio computer-assisted self-interviewing (ACASI) and interviewer-administered questionnaire (IAQ) interview modes from 180 Russian narcology patients

	IAQ	ACASI	Agreementa	McNemar p value	Wilcoxon signed rank test p value
Heterosexual	96%	97%	0.95	.56
Primary sex partner	79%	76%	0.86	.02
Secondary sex partner	69%	67%	0.83	.17
Multiple sex partners, past 6 months	69%	69%	0.91	.71
4+ opposite sex partners, past 6 months	28%	26%	0.82	.37
Same sex partner, ever	7%	9%	0.89	.08
Buy sex	19%	19%	0.80	.76
Sell sex	9%	12%	0.85	.03
STDs, ever	43%	43%	0.90	.74
Used condoms, past 6 months	60%	56%	0.80	.06
Prior to this hospitalization, ever tested for HIV	89%	84%	0.59	.03
Number of times unprotected vaginal sex with primary partner, past 3 months (n = 122b)  Mean (SD)  Median (IQR)	26.2 (23.7) 17.0 (8-39)	26.5 (23.6) 20.0 (8-40)	0.75		.95
Number of times unprotected vaginal sex with secondary partner, past 3 months (n = 103b)  Mean (SD)  Median (IQR)	9.1 (12.2) 5.0 (1-10)	10.0 (14.7) 5.0 (2-10)	0.74		.85

Note: STDs = sexually transmitted diseases; IQR = interquartile range.

^aAgreement for categorical variables measured with the kappa statistic and for continuous variables with the intraclass correlation coefficient.

^bNumber of matched pairs with data for both questions.

The question “Prior to this hospitalization, have you ever been tested for HIV?” had a kappa statistic of .59. Of the 17 subjects with discordant responses on the ACASI and IAQ for this question, 13 people (76%) responded “yes” in the ACASI and “no” in the IAQ and 4 people (24%) responded “no” in the ACASI and “yes” in the IAQ, a statistically significant finding (McNemar p < .03). The question “Do you have a primary sex partner?” had a kappa of 0.86, suggesting excellent agreement. Among the nine discordant responses for this question, eight people (89%) responded “yes” in the ACASI and “no” in the IAQ and one person (1%) responded “no” in the ACASI and “yes” in the IAQ. McNemar's test was significant (p < .02), indicating that in instances of discordance participants were more likely to respond affirmatively with the ACASI.

Responses to drug use behavior questions by ACASI and IAQ are presented in Table 3. All three of the responses, which pertained to unsafe drug use, needle sharing, and the Drug Risk Assessment score, had excellent agreement (kappas/ICC 0.94–0.97).

Table 3.

Baseline drug use behaviors reported by audio computer-assisted self-interviewing (ACASI) and interviewer-administered questionnaire (IAQ) interview modes from 180 Russian narcology patients

	IAQ	ACASI	Agreementa	McNemar p value	Wilcoxon signed rank test p value
Unsafe drug use, past 6 months (n = 178b)	30%	31%	0.97	.16
Shared needles, past 6 months (n = 80b)	67%	70%	0.94	.32
Drug Risk Assessment Score (n = 80b)  Mean (SD)  Median (IQR)	11.0 (7.2) 11 (4-17)	11.5 (6.9) 12 (5.5-17)	0.94		.70

^aAgreement for categorical variables measured with the kappa statistic and for continuous variables with the intraclass correlation coefficient.

^bNumber of matched pairs with data for both questions.

The 21 HIV-infected participants were asked about disclosure of their serostatus; the 159 HIV-negative participants were not asked these questions (Table 4). Two (60%) questions had excellent agreement and three (40%) questions had moderate agreement. Twenty participants (95%) had concordant responses to the ACASI and IAQ when asked if they had ever told anyone if they were HIV infected and if they had told family members of their serostatus. The question about disclosing to friends had a kappa statistic of .70. The question about disclosing to any sexual partners in the past 6 months had a kappa statistic of .71. For each of the three discordant responses to these questions, participants responded “no” in the ACASI and “yes” in the IAQ. Finally, the question about disclosing to all sexual partners in the past 6 months had a kappa statistic of .40 and a significant McNemar's test p value (p < .01). Among the seven discordant responses, each responded “no” in the ACASI and “yes” in the IAQ.

Table 4.

Baseline HIV disclosure behaviors reported by audio computer-assisted self-interviewing (ACASI) and interviewer-administered questionnaire (IAQ) interview modes.

	IAQ	ACASI	Agreementa	McNemar test pvalue
Told anyone you are HIV infected	95%	95%	1.0	1.0
Disclosed HIV status to family	95%	95%	1.0	1.0
Disclosed HIV status to friends	62%	57%	0.70	.56
Disclosed HIV status to any sex partners in past 6 months	67%	52%	0.71	.08
Disclosed HIV status to all sex partners in past 6 months	43%	33%	0.40	<.01

Note: Questions asked only of participants who tested positive for HIV (n = 21).

^aAgreement measured with the kappa statistic.

In secondary analyses stratified by drug diagnosis and HIV status, agreement between the two interview modes appeared similar across subgroups (data not shown).

DISCUSSION

ACASI systems are used in research studies to improve truth telling concerning risky and potentially stigmatized behaviors.² ACASI has been used increasingly in recent years in HIV-related research studies in lieu of IAQ to improve privacy and reduce participants' inclination to give socially desirable responses. Seven published studies performed outside the United States compared responses to ACASI versus IAQ. Three found increased reporting of socially undesirable or stigmatized behaviors via ACASI.^9,10,12 These studies performed in Brazil, Vietnam, and Kenya compared the ACASI information to interview by randomizing participants to one method or the other. Four found mixed results.^11,13,14,15 In all of these studies, participants completed both ACASI and IAQ but were randomized to which they completed first.

For the most part, no strong reporting bias in responses to sexual and drug behavior questions was observed based on assessment methodology in this study of Russians with alcohol and/or drug dependence. There was excellent agreement between responses from ACASI and IAQ for the majority of responses.

A moderate level of agreement was observed on some questions related to HIV testing overall and on serostatus disclosure among HIV-infected participants. All participants were asked, “Prior to this hospitalization, have you ever been tested for HIV?” Participants with discordant responses were more likely to report “yes” to ACASI and “no” to the interviewer (McNemar p < .03). The HIV epidemic is of more recent onset in Russia compared to other countries. This area of discordance between interview approaches may be due to perceived stigma of being HIV infected and increased willingness to report a private experience such as HIV testing to ACASI. Additionally, the 21 HIV-infected participants were asked questions about disclosing their serostatus. When asked about disclosing to any or all sexual partners in the past 6 months, discordant participants were more likely to report “yes” to the interviewer and “no” to the ACASI. This discordance may be due to overreporting desirable behaviors to interviewers or to increased willingness to report less desirable behaviors to ACASI.

Overall, however, we observed strong agreement in the majority of items between the two interview modes. Two factors may explain this observation. The first is the study design, where participants completed both interview modes on the same day rather than being randomly assigned to a single interview mode. The participants, who already answered questions about their substance use and sexual behaviors to one mode of interview (IAQ or ACASI), may have provided consistent responses because they believed their answers would be compared.

A second factor is the potential perception of the study participants that computers do not ensure privacy. In Russia, databases with personal data on individual bank accounts, mobile phone numbers, passport information, and more are available for sale on the street. Many Russians do not believe that personal data reported to a computer and stored in the electronic form is more confidential and safer compared to paper form, especially because electronic information can be easily copied and distributed. In another study comparing ACASI and IAQ, only 23% of Russians responded “yes, absolutely” when asked whether the computer ensures sufficient privacy compared to 40% of participants in China, India, and Peru .¹⁶ This finding may be indicative of a larger mistrust of computers in Russia. Assessing study participants' preference of ACASI or IAQ would provide insight about use of this methodology in Russia.

In contrast to previous studies in the United States and other settings comparing ACASI and IAQ, this study observed generally strong agreement between responses to the two interview modes. The ACASI does not appear to substantially reduce reporting bias, compared to an IAQ, among inpatients in a substance abuse treatment facility in Russia. The advantages of advanced research techniques such as ACASI in one cultural setting may not be generalizable to other cultural settings. Based on this experience, the need to use the ACASI methodology in Russia for assessing risky behaviors is not compelling. Further studies utilizing a randomized controlled study design to compare responses from ACASI and IAQ in Russia would need to demonstrate a clear benefit of the ACASI methodology in order to make the case for these “advanced” methods to be utilized in Russia.