Does it Matter How You Ask? Question Wording and Males’ Reporting of Contraceptive Use at Last Sex

Abstract

This paper reports results from a unique experiment conducted in the 2002 National Survey of Family Growth (NSFG) to gauge the effect of question format on men’s reports of contraceptive use at last sexual intercourse. Respondents received separate questions about their own and their partners’ contraceptive use or one combined question about either partner’s contraceptive use. We examine whether receiving separate questions, as opposed to one combined question, is related to higher reports of using any contraceptive method, specific methods, female methods in addition to male methods, and the number of methods reported. We find that reports of any contraceptive use at last sex and use of the most common methods, condoms and the pill, were stable across question formats. However, we find significantly higher reports of withdrawal, combining male and female methods, and multiple method use among men who received the separate-question format. We also find that characteristics of the sexual experience in question condition the effect of question format on men’s reports.

Researchers have increasingly acknowledged the importance of including men in studies of contraceptive behavior (Edwards 1994; Forste 2002; Gillmore et al. 2003; Landry and Camelo 1994). However, studies that ask men to report on contraceptive methods may suffer from greater reporting error than similar studies of women. Men often lack direct knowledge of their partners’ contraceptive use, relying instead on being told or seeing physical evidence of a method, such as a pill pack. Research has found that such “proxy reporting,” or reporting on another’s behavior, is more susceptive to measurement error (Moore 1988; Sudman, Bradburn, and Schwarz 1996; Todorov 2003).

Survey researchers may be able to reduce measurement error in men’s reports of contraceptive use with careful selection of question wording and format. Current large-scale surveys have varied in how they ask questions about contraceptive use. For example, the National Longitudinal Study of Adolescent Health (Add Health), the Youth Risk Behavior Survey (YRBS), and Cycles 1–5 of the National Survey of Family Growth (NSFG) asked a single question about the contraceptive methods a couple uses, while the National Survey of Adolescent Males (NSAM) asked separate questions about the respondent’s and his partner’s method. Studies have shown that variation in question format and wording may contribute to differences in estimates of method use (Catania et al. 1990; Santelli et al. 2000). This paper reports findings from an experiment on question format to help us understand whether and, if so, how these differences in question wording and format influence men’s reports of contraceptive use.

Evidence from survey research suggests that a separate-question format may help address the concern that men mis- or under-report their contraceptive method use. For example, asking multiple questions and focusing the second question on less memorable information aids recall (Belli 1998; Brown 1990; Cannell, Marquis, and Laurent 1977; Fowler and Cosenza 2008; Sudman and Bradburn 1982). Information about one’s own behavior tends to be more memorable and readily recallable than someone else’s, so asking men a second, separate question about their partners’ use of contraception may lend itself to better recall (Fowler and Cosenza 2008; Tourangeau, Rips, and Raskinski 2000). The separate-question format also takes longer to administer, which essentially would allow men more time to recall information about the methods they or their partner used (Burton and Blair 1991; Cannell and Marquis 1972; Tourangeau and Smith 1996).

Another way in which the separate-question format may help address concerns about men misreporting contraceptive use is by emphasizing the importance of reporting all methods used, not just the most common methods or male methods. Asking separate questions may better convey to the respondent that the interviewer wants greater completeness and precision in contraceptive use reporting than a single question implies (Fowler and Cosenza 2008; Loftus et al. 1990; Singer et al. 2010). Motivating respondents to be complete in their responses can be critical for producing accurate information (Fowler and Cosenza 2008; Groves et al. 2004; Krosnick, 1991, 1999). A question specifically about their partners’ method use, and the accompanying follow-up prompts, may motivate men to provide a more thoughtful or thorough answer to the larger question of what method of contraception was used during their last sexual intercourse. Last, in the separate-question format each question has fewer response options than in the combined-question format, which should be easier for respondents to read and process (Bishop and Smith 2001; Holbrook et al. 2006, 2007).

The NSFG, the largest national study of contraceptive use, provides an important resource for understanding the impact of question wording and format on men’s contraceptive reporting. In 2002, the NSFG included a split-sample experiment to allow analysis of how a single- or separate-question format affects men’s contraceptive reporting. Some male respondents were randomly chosen to receive a single, combined question, while others received separate questions about their own and their partner’s contraceptive use. This paper analyzes the results of the experiment, examining how men’s reports of contraceptive use vary between the single- and separate-question groups.

The lack of an objective truth against which to measure estimates of contraceptive use precludes evaluation of which question format yields the more accurate report. Rather, we investigate the degree to which men’s reporting of contraceptive use is sensitive to question format. We hypothesize that the separate-question format will be positively associated with reports of any method use, each specific method use, using female methods rather than only male methods or no method, and the number of methods reported.

Additionally, we test for interactions between question format and characteristics of the last sexual experience. It is important for researchers who are collecting or analyzing data on men’s contraceptive use to understand how the effect of question format may vary across subgroups. Failing to recognize subgroup differences in sensitivity to question format may lead to inaccurate conclusions regarding subgroup differences in contraceptive use (Ayhan and Isikal 2004; Catania et al. 1996; Holbrook et al. 2007).

In particular, we expect that the effect of question format may depend on the specifics of the sexual encounter in question. Recent events tend to be easier to recall (Rubin and Wetzel 1996; Stocke and Stark 2007; Wu, Martin, and Long 2001), so we expect that men whose last sexual experience occurred recently will be less sensitive to the question format.

Also, previous research has shown that the relationship between two sexual partners may influence both their contraceptive use (Manlove and Terry-Humen 2007; Manning et al. 2000; Santelli et al. 1996) and men’s knowledge and reports of the methods used (Moore 1988; Sudman et al. 1996). Of importance for this research, whether a couple resides together may exert a unique impact on men’s contraceptive reporting (Grady et al. 1996; Bachrach 1987). Studies have shown that the difference between partners’ contraceptive use reports decreases as the level of couple interaction increases (Menon et al. 1995; Koenig et al. 1984) and men who share living space with their partners are much more likely to have firsthand knowledge of their female partners’ use of non-coitally specific methods. Thus, we expect that men who are married or cohabiting with their partner will be less sensitive to question format than men who were not living with their last sexual partner.

Data and Methods

We use data from Cycle 6, the 2002 round of the NSFG, which included 4,928 completed interviews of men 15–44 years of age and an overall male response rate of 78%. For a detailed discussion of sampling procedures and study design see Groves et al. (2005) and Lepkowski et al. (2006). We analyze data from the 4,109 men who reported having had sexual intercourse with opposite-sex partners.

Measure of question format

Respondents were randomly assigned to receive either the combined-question or separate-question format. For each format, Figure 1 shows reproductions of the exact question wording, categories of contraceptives, and the cards shown to the respondents. We used a dichotomous measure for question format, coded 1 for the separate-question format (70%, by design) and 0 otherwise.1

Figure 1.

Questions and Response Show Cards

Measures of contraceptive method use at last sex

We created a series of dichotomous measures of contraceptive method(s) used at last sex. The first measure is coded 1 if the respondent reported using any method and 0 otherwise. Table 1, column 1 shows the weighted percent of men reporting each type of method and all other measures used in these analyses. In all, 71% of men aged 15 to 44 who had ever had sex used a contraceptive method at last sex.

Table 1.

Weighted percent reporting variables used in analysis, by question format

	Full Sample	Separate-question format	Single-question format
	1	2	3
Contraceptive method use at last sex
Used any contraception	71.1	72.0	68.9
Male methods
Used condom or rubber	31.7	31.7	31.9
Used withdrawal or pulling out	6.9	7.5	5.4*
Used vasectomy or male sterilization	5.3	5.2	5.4
Female methods
Used pill	25.6	26.2	24.2
Used tubal ligation or female sterilization	7.8	7.6	8.3
Used injection or hormonal implant	3.4	3.7	2.6+
Used spermicidal foam/jelly/cream/film/suppository	1.1	0.8	1.8
Used rhythm or safe period	1.1	1.2	0.8
Contraceptive use by gender-specific typea
No method	29.6	28.8	31.7
Male method only	31.0	31.1	30.8
Female method only	27.2	27.5	26.7
Male and female methods	12.1	12.7	10.8
Number of contraceptive methods reported
No method	29.6	28.0	31.1	*
One method	54.2	54.9	56.1
Two or more methods	16.2	17.1	12.7
Characteristics of last sexual experience
Last sex at least one month before interview	32.8	33.0	32.3
Married or cohabiting with partner	65.1	64.9	65.4
Controls
Race/ethnicity
Hispanic	17.2	17.1	17.6
Non-Hispanic white	64.8	65.0	64.2
Non-Hispanic black	12.3	12.4	12.1
Non-Hispanic other	5.7	5.5	6.1
Age at interview (mean)	31.3	31.3	31.3
Education (highest grade completed)
Less than high school diploma or GED	17.6	17.4	18.0
High school diploma or GED	33.3	32.1	36.1
Some college	27.8	28.2	26.9
Bachelor’s degree or higher	21.3	22.2	18.9
N	4,109	2,900	1,209

Chi-square tests where

⁺p<.10 and

^*p<.05. All tests were 2×2 tests examining the difference between columns 2 and 3 of the same row, except for “Contraceptive use by gender-specific type” and “Number of contraceptive methods reported” where we test the overall distribution of the variables’ attributes across columns.

^aN=4,032 because men who reported ‘other’ are not included in this variable.

We created three dichotomous measures for male-specific methods: (1) condoms; (2) withdrawal; (3) vasectomy or male sterilization. Each measure is coded 1 if the respondent reported using that specific method at last sex and 0 otherwise. Condoms were the most commonly reported male method at 32%.

We also created five dichotomous measures for each of the female-specific methods: (1) pill; (2) tubal ligation or female sterilization; (3) injection or hormonal implant; (4) spermicidal foam/jelly/cream/film/suppository; (5) rhythm or safe period.² Each method is coded 1 if the respondent reported his partner used that method and 0 otherwise. The pill was the most commonly reported female method used, with 26% of men reporting their partners used it. Only 1% of men reported that their partner used a spermicidal foam etc. or the rhythm method, which prevents us from including these measures in the multivariate analyses.

Our theory predicts that asking men separately about their partner’s contraceptive use will elicit more reports of female methods in particular, so we created a categorical measure of contraceptive use by gender-specific type. The categories for this measure are: no method used (30%), male method only (31%), female method only (27%), and both male and female methods (12%). The N for this measure is 4,032 because the 77 men who reported “something else” could not be categorized.

Finally, we created a measure of the number of methods a respondent reported using during last sex. This measure is coded: no methods (30%), one method (54%), and two or more methods (16%).

Measures of characteristics of the last sexual experience

The effect of question format may depend on the specifics of the sexual encounter in question, so we created two measures of characteristics of the last sexual experience. First, we created a dichotomous variable equal to 1 if the respondent’s last sexual encounter was at least one month prior to the interview and 0 if it occurred more recently. We also investigated a continuous measure of time since last sex. However, since almost 70% of men had sex within the same month as the interview, the dichotomous variable best operationalizes the concept and the respondents’ ability to recall details about contraception.

Second, we created a measure equal to 1 if the respondent was married or cohabiting with his last sexual partner and zero if he was not. Sixty-five percent of men were married or cohabiting with their last sexual partner.

Controls

The multivariate analyses control for three additional key sociodemographic characteristics: race/ethnicity, age, and education. We chose these three measures because they are key predictors of contraceptive use and may be related to survey responses (Ayhan and Isiksal, 2004; Holbrook et al., 2006, 2007). Self-reported race/ethnicity is included as a categorical variable with four categories: 1) non-Hispanic white (comparison group); 2) non-Hispanic black, 3) Hispanic black; 4) non-Hispanic other race. Age at the time of the interview is included as a continuous variable. We create a four-category variable for the respondent’s completed education: 1) less than a high school diploma or GED, 2) high school diploma or GED, 3) some college, and 4) Bachelor’s degree or higher.³

Analytic Strategy

Our analytic strategy has three main components. First, we use chi-square tests to assess overall differences in contraceptive use reporting by question format. Second, we estimate multivariate regression models of the relationship between question format and contraceptive use reporting, controlling for characteristics of the respondent and the last sexual experience. Because we are reporting the results of a random experiment, these regression models should reveal the same findings as the chi-square tests. We estimate logistic regression models with each method as a unique dependent variable and multinomial regression models for the categorical measures of the type and number of methods reported. Third, we assess whether the effect of question format varies by characteristics of the last sexual experience. For this component we estimate multivariate regression models with interaction terms between question format and time since last sex as well as question format and the respondent’s relationship with his last sexual partner.

For all analyses we account for the clustering resulting from the complex survey design (Lepkowski, et al. 2006). Specifically, following NCHS recommendation, we use the svyset command in STATA to account for the sampling weight, strata, and cluster.

Results

Columns 2 and 3 in Table 1 present the results from chi-square tests of the relationships between question format and the measures of contraceptive use. Question format does not significantly affect men’s probability of reporting any contraceptive use at the last sex. Of the specific methods, question format is positively related to reporting withdrawal and injections or hormonal implants.⁴ Men who received the separate-question format reported more withdrawal and injections/hormonal implants than men who received the single-question format. Logistic regression models confirmed these findings, and reveal that the difference between reports by question type for injections/hormonal implants is significant at the 0.05 level after controlling for characteristics of the respondent and last sexual experience.

We also found that question format affected the combination and number of methods reported. Chi-square tests revealed an overall difference by question format for the number of methods. We estimated multinomial regression models to identify which comparisons yielded different results. Table 2 shows the results from these multinomial regressions. Each column represents a separate model, and all models include the characteristics of the last sexual experience and the control measures described in the text. In Column 1 we see that men were significantly more likely to report both male and female methods than no method if they had received the separate question format. Columns 4 and 5 show the multinomial models for the number of methods reported. Men were significantly more likely to have reported two or more methods (versus none or one method) if they received the separate question format. It is worth noting that 78% of men who reported two or more methods reported a male and female method, so these two separate measures are capturing similar behavior.

Table 2.

Multinomial regression coefficients for the effects of question format on reports of multiple method use^a

	Type of method reported, compared to both male and female (N=4,032)			Number of methods reported, compared to 2 or more methods (N=4,109)
	None	Male only	Female only	None	1
	1	2	3	4	5
Separate-question format	−0.27* (0.15)	−0.17 (0.17)	−0.15 (0.20)	−0.44** (0.14)	−0.36** (0.13)

Note: Standard errors in parentheses.

^*p<.05;

^**p<.01 (one-tailed tests)

^aAll models include measures of time since last sex, relationship with last sexual partner, age at interview, race/ethnicity, education, and a constant term.

To determine whether the effect of question format depended on specific characteristics of the last sexual experience we estimated interaction effects. We estimated separate models for each measure of contraceptive use that included interactions between question format and time since last sex and relationship with last partner. In Table 3 we show the only statistically significant interaction—that between reporting that his partner used the pill and relationship status with partner.⁵ The interaction effect reveals that question format has a more positive effect on reporting pills for men who were single than for those who were married or cohabiting with their partner. This may be because single men may not see evidence of pill use (e.g. pill packs stored in a shared bathroom), making the knowledge of pill use harder for the respondent to access in his memory. The separate question and prompt related to women’s contraceptive methods may help access that information.

Table 3.

Logistic regression coefficients for the effects of question format and interaction with relationship status with last partner on reporting pill use (N=4,109)^a

Married or cohabiting with last sexual partner × Separate-question format	−0.43* (0.22)
Separate-question format	0.36* (0.17)
Married or cohabiting with last sexual partner	−0.06 (0.19)

Note: Standard errors in parentheses.

^aModel includes measures of time since last sex, relationship with last sexual partner, age at interview, race/ethnicity, education, and a constant term.

^*p<.05 (one-tailed test)

Table 4 presents the significant results from multinomial regression models with interactions between question format and characteristics of the last sexual experience. Although we mark effects significant at a .10 level, we consider those to be marginal and only interpret those significant at the .05 level or above. We find that the effect of question format on the combination and number of methods reported varies by time since the last sexual experience. In column 2, we find an interaction effect that is negative and significant for reporting only male methods versus both male and female methods. So, men who had sex more than one month ago were more likely to report only a male method (as opposed to both a male and female method), but the difference was less for those who received the separate question format than those who received the combined format. This suggests that the separate question format did increase reporting of combined female and male method use. In column 5 we see a similar effect when we consider the number of methods reported. Men whose last sex was at least one month prior to the interview were less likely to report multiple methods than those who had more recent sex. But the difference was diminished for those who received the separate question format as opposed to the combined question.

Table 4.

Multinomial logistic regression coefficients for the effects of question format on type of contraceptive method used^a

	Type of method reported, compared to both male and female (N=4,032)			Number of methods reported, compared to 2 or more methods (N=4,109)
	None	Male only	Female only	None	1
	1	2	3	4	5
Last sex at least one month before interview × Separate-question format	−0.47+ (0.32)	−0.65* (0.33)	−0.51+ (0.34)	−0.32 (0.29)	−0.47* (0.27)
Separate-question format	−0.09 (0.22)	0.08 (0.24)	0.04 (0.25)	−0.33* (0.19)	−0.20 (0.17)
Last sex at least one month before interview	0.82*** (0.25)	0.74** (0.26)	0.19 (0.26)	0.86*** (0.24)	0.56** (0.22)

Note: Standard errors in parentheses.

⁺p<.10;

^*p<.05;

^**p<0.01;

^***p<.001 (one-tailed tests)

^aAll models contain measures of time since last sex, relationship with last sexual partner, age at interview, race/ethnicity, education, and a constant term.

We did not find any significant interactions between question format and relationship with last sexual partner.

Discussion

This study used a survey methodology experiment on question format embedded in the NSFG to examine how manipulating question format affects men’s reports of contraceptive use. The findings are particularly relevant for designing future studies of contraceptive use and analyzing existing contraceptive use data. In addition, the findings may be important for studies of other behaviors that are commonly conceptualized at the couple level and asked about in proxy interviews, such as domestic violence.

We found that reports of any contraceptive use at last sex and use of the most common methods, condom and the pill, were stable across question formats. The stability of reports of condom use in particular is an encouraging indication of data quality for this method, which is essential for men’s roles in preventing pregnancy and STIs. Documenting this stability is important for researchers who collect data on contraceptive use from men. The consistency across question format means that researchers can consider other factors, such as costs and comparability, when designing instruments and still feel confident that they are obtaining reliable results. This information also is important for researchers who analyze existing data on contraceptive use, as surveys have used different question formats to collect reports of contraceptive use. Our findings suggest that these differences in question wording should not preclude researchers from making comparisons across surveys or across waves of the same survey.

On the other hand, we found significantly higher reports of withdrawal and injection or hormonal implant among men who received the separate-question format. Reporting withdrawal may be sensitive to question format because withdrawal is often used as a secondary method in combination with a female method, and when asked about male and female methods together, men may report the primary method only. Another possibility is that when asked a question about male methods only, men who did not use a condom may feel pressure to report using some method (Gregson et al. 2002; Wight and West 1999). Reporting injection or hormonal implant may be sensitive to question format because the methods offer no physical evidence and, therefore, men may need more assistance recalling that their partner has used the method. Given the relatively small number of men reporting injections, these findings should be interpreted cautiously and further research is needed to fully understand these findings. In the meantime, researchers who are particularly interested in capturing men’s reports of withdrawal or injection/hormonal implant may want to consider using the separate-question format.

We also found that the separate-question format was positively associated with reporting both male and female method use and with the number of methods reported. Regarding the reporting of male and female methods, we were somewhat surprised that we did not find evidence of an effect of question wording on reporting female methods in particular, only on reporting both a male and female method compared to no methods. This may be because separating the male and female methods highlights the importance of all methods, leading to an increase in reports of both types. Also, with the separate question format respondents receive twice as many probes asking if they used “anything else,” which may lead to more methods being reported. The increased reporting across methods is also found when we look at the number of methods reported, supporting the conclusion that the effect of question format is not specific to reporting female methods.

By examining interactions between question format and time since the last sexual experience, we learned that subgroup differences were diluting the effect of question format. In particular, the effect of the separate-question format on reporting male and female method use compared to only male methods and on reporting multiple methods compared to only one method was stronger among men whose last sexual experience was at least one month before the interview. This finding supports the existing literature that the separate-question format may help improve data quality by facilitating recall (Cannell and Marquis 1972; Cannell, Marquis, and Laurent 1977; Sudman and Bradburn 1982; Tourangeau, Rips, and Raskinski 2000; Tourangeau and Smith 1996) and that survey designers and analysts should attempt to limit questions or analysis to recent events when possible.

The analyses reported here provide evidence that the separate question format enhances data quality by simplifying the questions, aiding recall, and encouraging men to report their partner’s method use. Based on earlier preliminary analysis of this experiment, the two-question format was retained in the NSFG male questionnaire subsequent to 2002, and the single-question format discontinued. Since the influence of question format was relatively minor, the benefit of consistency with prior surveys (in particular the NSAM) of the measures of male contraceptive use could be allowed to weigh strongly in the decision. Additionally, there are reasons to believe that asking two simpler questions instead of one more complex question is enhancing data quality as simplicity is one of the basic tenets of survey design. Thus, although the results for data quality are not definitive, the benefits of keeping the two-question format outweighed those for the single-question format.

One concern is that the separate-question format may encourage men to over report contraceptive use due to social desirability bias. Earlier analysis of the NSFG’s preliminary data for this experiment showed that the difference between young men and women’s reports of condom use is greater for the separate-question format than the combined-question format (Abma, Porter, and Sonenstein 2003). More of this type of analysis, and perhaps analysis of couples’ reports, may help illuminate to what degree social desirability plays a role in sensitivity to question format.

Highlights.

• We examine how question format affects men’s reports of contraceptive use.

• We compare separate questions about male and female methods with a combined question.

• Reports of any contraception, condom, and pill use were stable across question formats.

• Reports of withdrawal and multiple method use were higher with the separate-question format.

• Characteristics of the sexual experience condition the effect of question format on men’s reports.