Test-Retest Reliability of a Sexual Behavior Interview for Men Residing in Brazil, Mexico, and the United States

Abstract

Understanding the natural history of sexually transmitted infections requires the collection of data on sexual behavior. However, there is concern that self-reported information on sexual behavior may not be valid, especially if study participants are culturally and linguistically distinct. The authors completed a test-retest reliability study of 1,069 men recruited in Brazil, Mexico, and the United States in 2005 and 2006. All of the men completed the same computer-assisted self-interview approximately 3 weeks apart. Refusal rates, kappa coefficients, and intraclass correlation coefficients were calculated for the full sample and by country, age, and lifetime number of female sex partners. Reliability coefficients for each study site and the combined population were high for almost all questions. With few exceptions, the authors found high test-retest reliability with a computer-assisted self-interview on sexual behavior used in 3 culturally and linguistically distinct countries.

Understanding the natural history of sexually transmitted infections and related disease requires the collection of data on sexual behavior. However, there is concern that study participants’ self-reports on sexual behavior may not be valid (1–6) because of measurement error from several sources, including the demands of the recall task and the survey method (7). In addition, measuring sexual behaviors with a common instrument across multiple countries may pose a threat to data quality (8), because such situations are affected not only by the survey method and burdens placed on the participant but also by differing population characteristics—for example, social attitudes toward disclosing sexual behavior (7, 9–11).

While validating human behavioral surveys, including sexual behavior surveys, is difficult, test-retest studies can be used to assess their reliability. These studies assess the consistency of participant responses between 2 time points (4). High consistency does not ensure validity of data, but low consistency can highlight potentially invalid data (7). In other words, reliability is necessary for validity but not sufficient (12).

Computer-assisted self-interviewing (CASI), including its audio version, audio-CASI, not only elicits higher-quality data on sensitive sexual behaviors than either face-to-face interviews or self-administered paper-and-pencil questionnaires (13–25) but also is more efficient and cost-effective when used with larger samples or for repeated studies (26). However, a number of studies have found that CASI methods may yield lower-quality data in some situations common to cross-national studies (17, 27–35). For example, data quality may suffer depending upon a participant's age, race, ethnicity, language ability, or familiarity with computers (17, 31, 33).

We are not aware of studies that have assessed CASI reliability in the context of cross-national populations. The objective of the current study was to assess test-retest reliability for a non-audio CASI that collected information on sexual health history and sexual behavior in 3 languages from men recruited in 3 different countries.

MATERIALS AND METHODS

Study population

Beginning in March 2005, men were recruited in Brazil (São Paulo), Mexico (Cuernavaca), and the United States (Tampa, Florida) for the HPV in Men (HIM) Study—a cohort study of the natural history of anogenital human papillomavirus (HPV). Men were enrolled if they were between the ages of 18 and 70 years; resided in one of the targeted recruitment areas; had no prior anal cancer, penile cancer, or genital warts; had no current diagnosis of a sexually transmitted disease, including human immunodeficiency virus; had no history of imprisonment, homelessness, or drug treatment in the prior 6 months; and were willing to engage in study visits every 6 months for 4 years. Additional details on the study design and population have been previously published (36, 37).

In Brazil, men were recruited from a large clinic in São Paulo that tests for human immunodeficiency virus and sexually transmitted diseases and from the general population through radio and print advertisements. In Mexico, men were recruited through a large health plan in the state of Morelos. In the United States, men were recruited from a large university campus and the general community in Tampa, Florida. Participants received a nominal monetary incentive for their participation. Men found to be illiterate or innumerate during the consenting or interview process were removed from analysis for the current CASI reliability study. All enrolled participants consented to the study protocol, which was approved by the human subjects protection committee of the Ludwig Institute for Cancer Research in Brazil; the ethical committee of the Center for Sexually Transmitted Diseases and AIDS in São Paulo; the National Institute of Public Health of Mexico; and the University of South Florida.

Recruited in 2005 and 2006, the first 1,069 men to complete their run-in (test) and baseline (retest) visits by CASI comprised the participants in the current reliability study. Age varied by study site, with the median age of participants in Brazil and Mexico (33 years in both countries) being higher than that of participants recruited in the United States (23 years). As expected, racial and ethnic characteristics also varied by study site. Overall, approximately one-half (50.8%) of participants reported a nonwhite race, while 41.4% reported a Hispanic ethnicity. Other population characteristics are provided in Table 1.

Table 1.

Characteristics of Participants at Retest (Baseline) in the HPV in Men (HIM) Study, Brazil, Mexico, and the United States, 2005–2006

Variable	Brazil (n = 338)		Mexico (n = 327)		United States (n = 404)		Combined Population (n = 1,069)
	No.	%	No.	%	No.	%	No.	%
Age, years
18–19	11	3.3	14	4.3	93	23.0	118	11.0
20–24	51	15.1	48	14.7	153	37.9	252	23.6
25–29	63	18.6	70	21.4	41	10.2	174	16.3
30–34	71	21.0	65	19.9	30	7.4	166	15.5
35–39	44	13.0	56	17.1	34	8.4	134	12.5
40–44	50	14.8	60	18.4	26	6.4	136	12.7
45–70	48	14.2	14	4.3	27	6.7	89	8.3
Refused to responda	0	0.0	0	0.0	0	0.0	0	0.0
Median age, years	33		33		23		30
Race
White	207	61.2	15	4.6	296	73.3	518	48.5
Black	101	29.9	0	0.0	53	13.1	154	14.4
Asian/Pacific Islander	3	0.9	0	0.0	22	5.5	25	2.3
American Indian	20	5.9	0	0.0	0	0.0	20	1.9
Mixed/other	0	0.0	311	95.1	33	8.2	344	32.2
Refused to respond	7	2.1	1	0.3	0	0.0	8	0.8
Ethnicity
Hispanic	63	18.6	327	100.0	52	12.9	442	41.4
Non-Hispanic	265	78.4	0	0.0	351	86.9	616	57.6
Refused to respond	10	3.0	0	0.0	1	0.3	11	1.0
Marital status
Single, never married	148	43.8	87	26.6	294	72.8	529	49.5
Married	99	29.3	186	56.9	58	14.4	343	32.1
Cohabitating	59	17.5	36	11.0	19	4.7	114	10.7
Divorced/widowed	31	9.2	18	5.5	31	7.7	80	7.5
Refused to respond	1	0.3	0	0.0	2	0.5	3	0.3
Education, years
<12	82	24.3	112	34.3	3	0.7	197	18.4
12	14	4.1	77	23.6	57	14.1	148	13.8
13–16	234	69.2	123	37.6	306	75.7	663	62.0
≥17	8	2.4	14	4.3	36	8.9	58	5.4
Refused to respond	0	0.0	1	0.3	2	0.5	3	0.3
Age at first sexual intercourse, yearsb
<15	93	30.7	32	10.6	54	14.1	179	18.1
15–17	137	45.2	129	42.9	189	49.4	455	46.1
18–19	42	13.9	71	23.6	92	24.0	205	20.8
20–42	24	7.9	68	22.6	38	9.9	130	13.2
Refused to respond	7	2.3	1	0.3	10	2.6	18	1.8
Median age at first intercourse, years	16		17		17		17
Lifetime no. of female sex partnersb
0	23	7.1	23	7.1	19	4.8	65	6.2
1	23	7.1	22	6.8	45	11.3	90	8.6
2–9	92	28.3	191	59.1	174	43.8	457	43.7
10–19	50	15.4	41	12.7	58	14.6	149	14.3
20–49	72	22.2	35	10.8	58	14.6	165	15.8
50–1,000	24	7.4	5	1.6	27	6.8	56	5.4
Refused to respond	41	12.6	6	1.9	16	4.0	63	6.0
Median no. of female sex partners	10		5		6		6
Ever having sex with a manb
Yes	110	34.5	39	12.8	42	10.7	191	18.8
No	204	63.8	266	87.2	348	89.0	818	80.5
Refused to respond	6	1.9	0	0.0	1	0.3	7	0.7

Abbreviation: HPV, human papillomavirus.

^aParticipant chose the “refuse” option for that specific question.

^bNumbers do not add up to 1,069 because of missing observations.

Procedure

Men expressing interest in the study came to the clinic for an initial visit. After consenting to the research and receiving instructions for using the CASI, participants completed the self-interview and then were sampled at anogenital sites for HPV. The CASI was written in the primary language of the region (Portuguese, Spanish, or English) and elicited information about participant demographic characteristics, substance use, sexual health history, and sexual behaviors implicated in the transmission of HPV. The men completed an identical CASI retest approximately 3 weeks later (the median test-retest interval was 21 days in Brazil, 25 days in Mexico, and 16 days in the United States). A Kruskal-Wallis test confirmed a statistically significant difference in test-retest interval by site (P < 0.0001). Per the protocol, men were not counseled or educated about HPV at either the test or the retest, although the informed consent form contained basic information on HPV and staff answered men's impromptu questions. Men did not receive their first HPV test results until 6 months later, at a subsequent clinic visit.

Interview measures

The interview contained 88 items. The majority of the questions had previously been administered to US men in a paper-and-pencil format and generally were found to have excellent reliability (38).

Participants’ sexual health was assessed with 18 questions about past sexually transmitted infections, the existence of a current sex partner, circumcision status, and the sexual health histories of their partners (ever having a partner with a sexually transmitted disease, genital warts, or an abnormal Papanicolaou smear). In addition, 45 sexual behavior items assessed incidence and frequency of penetrative sexual behaviors (vaginal, anal, and oral) with women and men; age at first intercourse; number of female and male partners; frequency of condom use for vaginal and anal sex; incidence and frequency of sex with “steady” and casual partners; time since last vaginal sex and anal sex; and history of paying for sex. Participants were asked to recall their frequency of sexual intercourse and numbers of sex partners for varying periods of time, including the last month, the last 3 months, and over a lifetime. Participants could choose to refuse to respond to any question by clicking a “refuse” button. Participants could answer “Don't know” or “Don't remember” for some nominal items—for example, regarding past sexually transmitted disease diagnoses and use of a condom; however, participants were not given the option of answering “Don't know” for interval and ordinal items—for example, regarding their lifetime number of sex partners.

A subset of interview items was selected for assessment of reliability, with preference being given to items for which reliability coefficients would be less affected by the test-retest interval; therefore, items with only a 1-month recall period were not assessed. A total of 38 variables were assessed, including 9 interval, 4 ordinal, and 25 nominal variables. Variables assessed included 14 sexual health history variables and 24 sexual behavior variables.

Data analysis

For each item, reliability coefficients were calculated for each of the 3 study sites. We calculated combined population coefficients by averaging study-site coefficients after weighting them by the inverse of their variances (39). Because age (40, 41) and number of sex partners (40, 42) are associated with increasing measurement error, reliability coefficients were stratified by age (<30 years vs. ≥30 years) and lifetime number of female sex partners (≤7 partners (median) vs. >7 partners) reported at retest.

For nominal variables, the kappa (κ) statistic was calculated (43). Because the κ statistic can be unstable in situations where there are sparse data (44), κ was not computed for variables where the number of cases or noncases was less than 5 (7). For ordinal variables, a weighted κ statistic was calculated (45) to allow credit for partial agreement. Benchmarks for interpreting κ and weighted κ values followed those of Landis and Koch (46): poor reliability, κ < 0.00; slight reliability, κ = 0.00–0.20; fair reliability, κ = 0.21–0.40; moderate reliability, κ = 0.41–0.60; substantial reliability, κ = 0.61–0.80; and almost perfect reliability, κ ≥ 0.81.

Interval variables were assessed using the intraclass correlation coefficient (ICC) (47). All ICCs created using nonnormal variables were transformed using Fisher's z transformation before calculation of confidence intervals (48). Confidence intervals were then transformed back to the original scale. ICCs approaching 1.0 indicate high test-retest reliability.

During exploratory analysis, extreme outliers were identified in 2 variables: number of different female sex partners in the past 3 months (a value of 11,111,109,632 on both test and retest) and age at first sexual intercourse with women (a value of 1,993 on test). Each observation was removed prior to analyses. Subsequent text regarding outliers identified in scatterplots does not include these 2 observations.

Refusal rates were assessed. Refusals were not included in reliability coefficient calculations.

RESULTS

With exceptions for skip patterns, participants at each study site answered virtually all of the 38 questions under study. The average refusal rate on retest for all questions was 1.0% in Mexico, 1.3% in the United States, and 2.5% in Brazil (data not shown).

For all nominal and ordinal questions, κ and weighted κ reliability coefficients for each study site and the combined population were almost perfect (κ ≥ 0.81) or substantial (κ = 0.61–0.80). Table 2 provides coefficients for 18 items for which reliability was less than 0.81.

Table 2.

Select^a Kappa and Intraclass Correlation Coefficients for Computer-assisted Self-Interview Items, by Study Site, HPV in Men (HIM) Study, Brazil, Mexico, and the United States, 2005–2006

Interview Item	Brazil			Mexico			United States			Combined Populationb
	No.c	κ or ICC	95% CId	No.c	κ or ICC	95% CId	No.c	κ or ICC	95% CId	No.c	κ or ICC	95% CId
Interval variables
Lifetime no. of male anal sex partners	94	0.50	0.33, 0.64	32	0.98	0.96, 0.99	35	0.99	0.98, 0.99	161	0.99	0.99, 0.99
No. of different female sex partners in the past 3 monthse	281	0.58	0.50, 0.65	292	0.88	0.85, 0.90	380	0.95	0.94, 0.96	953	0.94	0.93, 0.95
No. of sex partners other than a “steady” partner in the past 3 months	114	0.10	0.00, 0.28	86	0.61	0.46, 0.73	81	0.96	0.94, 0.97	281	0.95	0.94, 0.96
Ordinal variables
Frequency of condom use for anal sex in the past 3 monthsf	236	0.87	0.84, 0.90	178	0.83	0.78, 0.87	182	0.79	0.73, 0.84	596	0.84	0.81, 0.86
Nominal variables
Ever being diagnosed with a sexually transmitted disease	336	0.84	0.81, 0.87	327	0.80	0.76, 0.84	404	0.89	0.87, 0.91	1,067	0.85	0.83, 0.87
Ever being diagnosed with Chlamydia	338	0.74	0.69, 0.78	327	0.79	0.75, 0.83		—g		1,069	0.77h	0.74, 0.79
Ever being diagnosed with genital warts	337	0.74	0.69, 0.78		—g		404	0.81	0.77, 0.84	1,068	0.76h	0.73, 0.78
Ever being diagnosed with hepatitis B	337	0.79	0.75, 0.83		—g			—g		1,068	0.76h	0.73, 0.78
Ever being diagnosed with hepatitis C	337	0.71	0.65, 0.76	327	0.85	0.82, 0.88	404	0.85	0.82, 0.88	1,068	0.80	0.78, 0.82
Ever being diagnosed with human immunodeficiency virus	337	0.68	0.62, 0.73		—g			—g		1,068	0.68h	0.65, 0.71
Ever having a sex partner with a sexually transmitted disease	338	0.73	0.68, 0.78	327	0.78	0.73, 0.82	404	0.80	0.76, 0.83	1,069	0.77	0.74, 0.79
Ever having a sex partner with genital warts	337	0.78	0.73, 0.82	327	0.78	0.73, 0.82	404	0.76	0.72, 0.80	1,068	0.77	0.74, 0.79
Ever having a sex partner with an abnormal Papanicolaou test	336	0.82	0.78, 0.85	327	0.75	0.70, 0.79	404	0.79	0.75, 0.82	1,067	0.79	0.77, 0.81
Ever having vaginal, anal, or oral sex	327	0.74	0.69, 0.79	326	0.80	0.76, 0.84	404	0.80	0.76, 0.83	1,057	0.78	0.76, 0.80
Ever having oral sex	309	0.92	0.90, 0.94	302	0.88	0.85, 0.90	390	0.78	0.74, 0.82	1,001	0.88	0.87, 0.89
Ever exchanging sex for money/drugs	315	0.90	0.88, 0.92	305	0.85	0.82, 0.88	390	0.78	0.74, 0.82	1,010	0.86	0.84, 0.88
Ever paying a man for sex	124	0.63	0.51, 0.73		—g			—g		218	0.68h	0.60, 0.75
Sex with someone other than a “steady” partner in the past 3 months	257	0.93	0.91, 0.94	282	0.83	0.79, 0.86	283	0.96	0.95, 0.97	822	0.93	0.92, 0.94

Abbreviations: CI, confidence interval; HPV, human papillomavirus; ICC, intraclass correlation coefficient.

^aOmitted are reliability coefficients for 20 items for which all study site coefficients were greater than or equal to 0.81.

^bUnless otherwise noted, the total coefficient was derived by averaging the 3 study site coefficients after weighting them by the inverse of the variance of each site, as described by Fleiss (39).

^cNumber of participants, excluding refusals and missing observations.

^dKappa and ICC coefficients were z-transformed for estimation of confidence intervals, as described by Rosner (48).

^eOne outlier (a value of 11,111,109,632 on both test and retest) was removed before calculation of ICCs.

^fWeighted κ, following the method of Cicchetti and Allison (45).

^gThe κ value was unstable because of a low number of cases or noncases; likewise, a stable κ value could not be calculated for any study site for “ever being diagnosed with syphilis” and “frequency of condom use for paid anal/vaginal sex.”

^hBecause of instability in the κ value for at least 1 study site for this item, κ was derived directly from the total sample, rather than from the average of the κ coefficients from each study site.

Site-specific ICCs for all interval questions were 0.85 or more, with the exception of ICCs in Brazil and Mexico for 3 items asking men to report their numbers of sex partners. Scatterplots identified several extreme outliers in the bivariate distributions of all 3 items. Specifically, for the variable “number of sex partners other than a ‘steady’ partner in the past 3 months,” when 1 outlying participant in the Mexico sample was removed, the Mexico ICC increased from 0.61 to 0.84. For the same item, when 2 outlying participants in the Brazil sample were removed, the Brazil ICC increased from 0.10 to 0.79. For “lifetime number of male anal sex partners,” when 1 outlier identified in the Brazil scatterplot was removed from the data set, the ICC for Brazil increased from 0.50 to 0.99. For the variable “number of different female sex partners in the past 3 months,” when 1 outlier in the Brazil scatterplot was removed, the ICC for Brazil increased from 0.58 to 0.92.

After taking into account these outliers, test-retest reliability was generally high and consistent across sites: Reliability coefficients differed by no more than 17 percentage points among study sites.

All nominal and ordinal items had substantial or almost perfect reliability regardless of participant age. All interval variables had ICC scores greater than or equal to 0.85 for both age groups, with the exception of the older men's answers to the same 3 interval variables as those discussed above: lifetime number of male anal sex partners, number of different female sex partners in the past 3 months, and number of sex partners other than a “steady” partner in the past 3 months (Table 3). After removing the outliers discussed above (all of which involved men over age 32 years), the ICCs for these 3 questions increased to 0.84 or more.

Table 3.

Select^a Kappa Statistics and Intraclass Correlation Coefficients for Computer-assisted Self-Interview Items, by Age and Lifetime Number of Female Sex Partners, HPV in Men (HIM) Study, Brazil, Mexico, and the United States, 2005–2006

Interview Item	Age, years				Lifetime No. of Female Sex Partners
	<30		≥30		≤7		>7
	κ or ICC	95% CIb	κ or ICC	95% CIb	κ or ICC	95% CIb	κ or ICC	95% CIb
Interval variables
Lifetime no. of male anal sex partners	0.89	0.82, 0.93	0.50	0.34, 0.63	0.50	0.33, 0.64	0.62	0.45, 0.75
No. of different female sex partners in the past 3 monthsc	0.91	0.89, 0.92	0.60	0.54, 0.65	0.88	0.86, 0.90	0.93	0.92, 0.94
No. of sex partners other than a “steady” partner in the past 3 months	0.74	0.65, 0.81	0.12	0.00, 0.28	0.86	0.80, 0.90	0.29	0.15, 0.42
Ordinal variables
Frequency of condom use for anal sex in the past 3 monthsd	0.81	0.76, 0.85	0.88	0.85, 0.90	0.78	0.73, 0.82	0.85	0.82, 0.88
Nominal variables
Ever being diagnosed with Chlamydia	0.72	0.68, 0.76	0.80	0.77, 0.83	0.77	0.73, 0.80	0.80	0.76, 0.83
Ever being diagnosed with genital warts	0.81	0.78, 0.84	0.73	0.69, 0.77	0.70	0.66, 0.74	0.79	0.75, 0.82
Ever being diagnosed with herpes	—e		0.86	0.84, 0.88	0.77	0.73, 0.80	0.80	0.76, 0.83
Ever being diagnosed with hepatitis B	—e		0.76	0.72, 0.79	0.69	0.64, 0.73	0.78	0.74, 0.81
Ever being diagnosed with hepatitis C	0.79	0.76, 0.82	0.78	0.74, 0.81	0.72	0.68, 0.76	0.85	0.82, 0.87
Ever being diagnosed with human immunodeficiency virus	—e		0.72	0.68, 0.76	0.61	0.56, 0.66	0.76	0.72, 0.80
Ever being diagnosed with nongonococcal urethritis	—e		0.77	0.73, 0.80	0.79	0.76, 0.82	—e
Ever having a sex partner with a sexually transmitted disease	0.73	0.68, 0.78	0.74	0.70, 0.78	0.76	0.72, 0.79	0.77	0.73, 0.81
Ever having a sex partner with genital warts	0.82	0.79, 0.85	0.74	0.70, 0.78	0.81	0.78, 0.84	0.73	0.68, 0.77
Ever having a sex partner with an abnormal Papanicolaou test	0.78	0.74, 0.81	0.79	0.76, 0.82	0.77	0.73, 0.80	0.81	0.78, 0.84
Ever having vaginal, anal, or oral sex	0.81	0.78, 0.84	0.73	0.69, 0.77	0.81	0.78, 0.84	0.39	0.31, 0.47
Ever having oral sex	0.89	0.87, 0.91	0.82	0.79, 0.85	0.88	0.86, 0.90	0.76	0.72, 0.80
Ever paying a man for sex	0.71	0.58, 0.80	0.67	0.56, 0.75	0.54	0.39, 0.66	0.75	0.65, 0.83

Abbreviations: CI, confidence interval; ICC, intraclass correlation coefficient; HPV, human papillomavirus.

^aOmitted are reliability coefficients for 21 items for which all study site coefficients were greater than or equal to 0.81.

^bKappa and ICC coefficients were z-transformed for estimation of confidence intervals, as described by Rosner (48).

^cOne outlier (a value of 11,111,109,632 on both test and retest) was removed before calculation of ICCs.

^dWeighted κ, following the method of Cicchetti and Allison (45).

^eThe κ value was unstable because of a low number of cases or noncases; likewise, a stable κ value could not be calculated for any category for “ever being diagnosed with syphilis” and “frequency of condom use for paid anal/vaginal sex.”

Reliability coefficients were also stratified by lifetime number of female sex partners. Whether men reported numbers of partners above or below the median number of 7, reliability coefficients for all nominal and ordinal variables were substantial or almost perfect, except for 2: ever having vaginal, anal, or oral sex (κ = 0.39 for men with >7 partners) and ever paying a man for sex (κ = 0.54 for men with ≤7 partners) (Table 3). Two interval variables also showed lower reliability: lifetime number of male anal sex partners (ICC = 0.50 for men with ≤7 partners) and number of sex partners other than a “steady” partner in the past 3 months (ICC = 0.29 for men with >7 partners). Removal of the outliers identified above increased the ICC for these 2 interval variables to 0.85 or more.

DISCUSSION

In this test-retest reliability study of a CASI instrument, 1,069 men in Brazil, Mexico, and the United States were asked the same questions on sexual health history and sexual behavior at test and retest. For each study site and the 3 study sites combined, κ and weighted κ reliability coefficients for nominal and ordinal questions, respectively, were substantial (0.61–0.80) or almost perfect (≥0.81). However, while the combined population ICC scores for all interval variables were greater than or equal to 0.85, the study site-specific reliability of 3 interval variables was of concern: lifetime number of male anal sex partners, number of different female sex partners in the past 3 months, and number of sex partners other than a “steady” partner in the past 3 months. The apparently poorer reliability of these variables was due to the presence of a small number of outliers identified in scatterplots. When these outliers were removed, study site-specific ICCs increased to 0.79 or more. These outlying observations also distorted the ICCs for several interval variables after results were stratified by age and median lifetime number of female sex partners.

The ability of a small number of observations to distort a reliability coefficient is discussed in the literature (7, 49); however, to our knowledge, the impact on reliability in practice has rarely been described (50, 51). In the current study, 1 or 2 participants with highly discrepant test and retest answers were able to increase the variance of an item by more than 2 magnitudes. For example, 1 Brazilian participant reported 2,000 lifetime male anal sex partners at test and only 20 such partners at retest. Removing this individual decreased the variance in the item from 0.00602 to 0.00002 and increased the ICC from 0.50 to 0.98. Such scenarios also underscore the importance of weighting by the inverse of the variance when combining coefficients in order to reduce the contribution of less reliable coefficients.

In total, outliers were observed in data reported by 3 out of 338 men in Brazil, 1 out of 327 men in Mexico, and no men out of 404 in the United States. The numbers of outliers in the 3 countries did not differ significantly (P = 0.11). Nevertheless, a higher number of outliers in Brazil may have occurred if the Brazilian men had less lifetime exposure to computer technology. Since a substantially higher percentage of Brazilian participants were aged 45 years or older as compared with Mexican and US participants, it is possible that these older men had less comfort with the technology and therefore had more reporting errors (17, 30, 32). It is also possible that a higher number of outliers for Brazil, in comparison with Mexico or the United States, may have occurred if this cross-national instrument was less culturally appropriate for Brazilian men. The questionnaire from which the current study's CASI was created was developed in the southwestern United States near the Mexican border. Creation of the questionnaire in this region may have led to an instrument that was somewhat more culturally appropriate for US and Mexican participants and less so for Brazilian participants. If this was the case, this may also account for the somewhat higher rate of question refusal among Brazilians. A review of the 4 outlier participants’ responses for all interview items revealed that 2 of the Brazilian men also gave illogical answers to a number of other questions, while the remaining 2 outliers’ interviews were generally unremarkable.

Men found to be illiterate or innumerate during the consenting and interviewing procedures were not included in the current CASI study; however, staff may not have been able to identify all of the persons whose level of illiteracy increased their risk of providing unreliable responses. Reliability can also be affected by the number of days between test and retest (4). However, the outliers in Brazil had test-retest intervals of less than 22 days. In addition, reliability in this study was largely consistent across sites, even though median test-retest intervals varied from 16 days at the US study site to 25 days at the Mexican study site.

After stratification by lifetime number of female sex partners, lower reliability was also identified for 2 nominal variables: ever having vaginal, anal, or oral sex (κ = 0.39 for men with >7 partners) and ever paying a man for sex (κ = 0.54 for men with ≤7 partners). For the variable “ever having vaginal, anal, or oral sex,” 7 out of 441 men who reported more than the median number of 7 sex partners at retest reported at test that they had never had anal, vaginal, or oral sex. In addition to inviting concern about validity, this item's κ coefficient may have been rendered unstable because of the small number of cases. It is also plausible that the multiple sexual behaviors addressed in the question confused some men. For the second variable, only 6 men with fewer than the median number of sex partners acknowledged ever paying a man for sex, possibly lending instability to the κ coefficient. Variables with sparse data may simply reflect a lack of behavioral heterogeneity in the population or cultural stigmatization attached to certain behaviors (9). During the design of this study, we decided not to report coefficients where the number of cases or noncases was less than 5, in an effort to report only stable coefficients. In future evaluations of test-retest reliability, investigators may wish to consider increasing this minimum requirement.

However, absent the small number of extreme outliers and the presence of sparse data for 2 questions, virtually all coefficients indicated that the CASI interview items under study captured data on these men's sexual behaviors in a highly reliable manner. This result may have been obtained because participants in each country were primarily educated men living in an urban setting. In collecting data on abortions from Mexican women, Lara et al. (32) found audio-CASI more appropriate for urban and educated participants than for rural residents in Mexico. Success with audio-CASI has also been reported from Brazil, where it was not only acceptable to men recruited at a health clinic in Rio de Janeiro but also elicited more reports of sensitive sexual behaviors (24, 52).

To our knowledge, no studies have evaluated cross-national test-retest reliability for a sexual behavior survey administered by CASI; however, investigators in 3 reliability studies of adults have reported results for audio-CASI in more homogenous samples (34, 53, 54). These studies are difficult to compare with the current study because of different study populations, test-retest intervals, or reporting methods.

Schlecht et al. (40) assessed the reliability of sexual behavior data collected by face-to-face interview and self-administered questionnaire using pooled, multinational data from 6 studies. In that analysis, reliability suffered substantially when women in different countries reported their lifetime number of sex partners (ICC = 0.08–0.94). In contrast, the current study found almost perfect reliability when this question was asked of men in Brazil, Mexico, and the United States. These heterogeneous results may be due to the use of different survey methods or to the fact that Schlecht et al. assessed the reliability of separate and distinct studies that not only had different survey methods and study protocols but also different and lengthy test-retest intervals (6 weeks to 5 years) (40). Therefore, the current study may have been more suited for a comparison of cross-national reliability, since identical protocols with relatively similar and short test-retest intervals were used at all study sites.

Refusal rates were generally low, which has been found previously with surveys of human sexuality (28, 55). However, the question on lifetime number of female sex partners was refused by 11.7% and 12.6% of Brazilian participants on test and retest, respectively. This rate of question refusal may be due to the targeted recruitment of Brazilian men in a clinic setting. Higher refusal rates from participants and less preference for using audio-CASI in a clinic setting, as compared with a face-to-face interview, have been reported previously (14, 52). Also noteworthy is that items requiring the participant to provide a numeral, as opposed to a nominal, answer were approximately twice as likely to be refused in Brazil and Mexico as in the United States, where participants refused numerical items and nominal items at about the same rates (data not shown).

This study had limitations. Reliability cannot be used as a surrogate measure for validity, since item reliability is not sufficient for validity. Additionally, because of the targeted recruitment, these results should not be generalized to the entire populations of the 3 study countries.

Comparisons of sexual behavior by country may be helpful in attempts to deliver large-scale programs for the prevention of sexually transmitted diseases; however, if sexual behavior measures are to be used cross-nationally, they should produce reliable data for each locale (8). With few exceptions, we found high reliability using a single CASI instrument in 3 culturally and linguistically distinct countries. While not guaranteeing validity, these results indicate that for the current instrument, use of CASI among men in diverse settings produces reliable data on sexual behavior.

Glossary

Abbreviations

CASI

computer-assisted self-interviewing

HPV

human papillomavirus

ICC

intraclass correlation coefficient