Men who have sex with men in Great Britain: comparison of a self‐selected internet sample with a national probability sample

Abstract

Objectives

To compare the characteristics of a self‐selected, convenience sample of men who have sex with men (MSM) recruited through the internet with MSM drawn from a national probability survey in Great Britain.

Methods

The internet sample (n = 2065) was recruited through two popular websites for homosexual men in Great Britain in May and June 2003. This sample was compared with MSM (n = 117) from the National Survey of Sexual Attitudes and Lifestyles (Natsal), a probability sample survey of adults resident in Great Britain conducted between May 1999 and February 2001.

Results

No significant differences were observed between the samples on a range of sociodemographic and behavioural variables (p>0.05). However, men from the internet sample were younger (p<0.001) and more likely to be students (p = 0.001), but less likely to live in London (p = 0.001) or report good health (p = 0.014). Although both samples were equally likely to report testing for HIV, men from the internet sample were more likely to report a sexually transmitted infection in the past year (16.9% v 4.8%, adjusted odds ratio 4.14, 95% CI 1.76 to 9.74; p = 0.001), anal intercourse (76.9% v 63.3%; p = 0.001) and unprotected anal intercourse in the past 3 months (45% v 36.6%; p = 0.064).

Conclusions

The internet provides a means of recruiting a self‐selected, convenience sample of MSM whose social and demographic characteristics are broadly similar to those of MSM drawn from a national probability survey. However, estimates of high‐risk sexual behaviour based on internet convenience samples are likely to overestimate levels of sexual risk behaviour in the wider MSM population.

Online surveys are becoming increasingly popular due to the rapid growth in internet use. They are particularly effective for conducting research among certain hard‐to‐reach minority populations and have been used in a number of surveys of HIV risk behaviour among men who have sex with men (MSM) in The Netherlands,¹,² Sweden,³ the UK⁴,⁵,⁶ and the US.⁷,⁸,⁹,¹⁰

Behavioural research among MSM has traditionally relied on convenience samples as a cost‐effective method of generating samples of sufficient size for detailed analysis of high‐risk sexual behaviour.¹¹ One approach is to recruit men who have contact with sexual health services.¹²,¹³ However, this restricts the sample to an at‐risk population of men who use these services. A second approach is to recruit probability¹⁴ or, more commonly, non‐probability samples¹⁵,¹⁶ of men from gay venues such as bars or clubs. This captures a more diverse population, but only reaches men who are affiliated with the gay community and attend such venues. The emergence of the internet provides yet another way of recruiting MSM for behavioural research.¹,²,³,⁴,⁵,⁶,⁷,⁸,⁹,¹⁰,¹⁷

A key advantage of recruiting MSM through the internet is the relative ease and speed with which large samples of MSM may be drawn from a wide geographical area. For example, the UK Gay Men's Sex Survey 2004, which was promoted on gaydar and gay.com (the two most popular websites for homosexual men in the UK) and on 35 gay community and health promotion websites, recruited a UK internet sample of nearly 12 000 men over a period of 4 months.⁶ The internet may also access men who are more geographically isolated, younger and more likely to be bisexually or heterosexually identified.⁴,¹⁸,¹⁹ These advantages suggest that the internet provides an attractive new venue for recruiting samples of MSM, especially as access to the internet is high in this group of men.⁵,²⁰ On the other hand, as internet‐based surveys are generally promoted through gay‐interest websites, this clearly restricts the sample to users of such sites.

In view of the potential benefits associated with recruiting MSM through the internet, it is important to evaluate the composition of the self‐selected samples that these surveys attract. All convenience samples are vulnerable to biases according to how and where they were recruited. This is reflected in studies that have compared self‐selected internet samples of MSM with other convenience samples.⁴,⁹,¹⁸,¹⁹ To produce generalisable findings, however, an examination of the composition of internet samples requires comparison with a sample of MSM identified through a random probability survey rather than with another convenience sample.

The purpose of this study was to compare a self‐selected convenience sample of MSM in Great Britain recruited through the internet with a sample of MSM identified from a national probability sample of adults resident in Great Britain.

Methods

The internet sample was recruited for an online survey of sexual behaviour among MSM that was conducted in 2003. The probability sample consisted of MSM who participated in the National Survey of Sexual Attitudes and Lifestyles (Natsal) 2000. The methods of both studies have been described in detail elsewhere.²⁰,²¹,²²

The research protocol was approved by the Royal Free Hospital and Medical School Local Research Ethics Committee, City University London Research Ethics Committee, the University College London, and North Thames Multi‐centre Research Ethics Committees and all the local research ethics committees in the Great Britain.

Internet sample

The internet sample was recruited via gaydar (http://www.gaydar.co.uk/) and gay.com (http://uk.gay.com/). Over a 5‐week period in May and June 2003, pop‐up and banner advertisements appeared in chatrooms and profile pages asking men to participate in the survey. No incentives were offered for participation. Clicking on a pop‐up or banner took respondents to the online survey, which they could complete and submit online. Only respondents who said they were at least 18 years old were allowed to answer the questionnaire. After questions on their sociodemographic profile, respondents were asked the sex of their sexual partners in the past year. At this point, only men who reported sex with another man over this period were asked to continue. Although the pop‐ups and banners were restricted to UK chatrooms and profiles, men from anywhere in the world using these sites could participate.

Natsal sample

Natsal 2000 adopted a multistage‐stratified probability design to identify a sample of men and women aged 16–44 years living in private households in the UK. After an investigation into the use of incentives in the early stages of fieldwork, respondents were offered a gift voucher of £5 regardless of whether they participated. MSM were defined as men reporting at least one male partner with whom they had genital contact in 5 years before the interview.²¹,²³ The Natsal sample used here refers to the subsample of men who reported sexual activity with a man in the past year. Fieldwork took place from May 1999 to February 2001, and interviews were conducted by trained interviewers using a combination of face‐to‐face interviews and a self‐completion section administered by computer‐assisted self‐interviews, which covered the most sensitive questions.

Sample comparability

To ensure the closest comparability between the two samples, only men aged 18–44 years living in Great Britain (England, Scotland, Wales) were included in the analysis. All men included in the analysis stated that they were sexually active with a man in the past year.

Variable comparability

The samples were compared on all variables for which the questions in each survey were substantially equivalent (table 1). Even small differences in question wording and format may influence response patterns.²⁴ There were notable differences between the two studies in questions about anal intercourse and unprotected anal intercourse (UAI). Internet respondents were asked if they had had anal intercourse and UAI with a male partner in the past 3 months, whereas Natsal respondents were asked if they had had anal intercourse with a male partner in the previous week, 4 weeks, 6 months and 1 year, but not 3 months. Natsal respondents reported UAI with a male partner in the 4 weeks and 1 year before the interview.

Table 1 Question wording.

Variable	Internet sample		Natsal sample
Age	How old are you?		In which month and year were you born?
Employment status	Which of the following best applies to you or your work in the last week?		Which of these descriptions applies to what you were doing last week?
Ethnic group	Which of the following ethnic groups best describes you?		To which of the ethnic groups on this card do you consider you belong?
Higher education	Could you tell us what your highest educational qualification is?		Have you passed any exams or got any of the qualifications on this card?
Social class	Coded on the basis of a similar series of questions in both surveys, according to the National Statistics Socio‐Economic Classification. Students were not asked these questions in the web survey and are therefore excluded from social class coding in both surveys.
Residence	Where do you currently live?		Coded on the basis of postcode sector.
Urban area	Which best describes the area you live in?		Coded by the interviewer.
Country of birth	Were you born in the UK? What country were you born in?		Were you born in England, Wales, Scotland, Northern Ireland/Eire or another country?
Self‐reported health	In general, how would you rate your health in the past 3 months?		How is your health in general?
Alcohol consumption	In the past 12 months, please indicate how often (on average) you have used alcohol.		How often have you had an alcoholic drink drink of any kind during the last 12 months?
Drug injecting	Have you injected drugs (other than anabolic steroids) in the past 12 months?		When was the last time you injected yourself with non‐prescribed drugs or other substances?
First sex with male	How old were you the first time you had sex of any kind with another male?		Have you ever had ANY kind of sexual experience or sexual contact with a male? How old were you the first time that happened?
Ever had an HIV test	Have you ever had an HIV test?		Apart from any occasion when you were donating blood, have you ever had a blood test that involved testing for HIV?
Had STI	Have you had a sexually transmitted disease (STD) in the last year?		Have you ever been told by a doctor that you had any of the following? When were you told by a doctor that you had ‘x'?
AI	In the past 3 months have you had anal sex with a condom? In the past 3 months have you had anal sex without a condom?		When, if ever, was the last occasion you had anal sex with a man?
UAI	In the past 3 months have you had anal sex without a condom?		In the last year, when you've had anal sex, how often have you, or your partner, used a condom?

AI, anal intercourse; STI, sexually transmitted infection; UAI, unprotected anal intercourse.

The questions listed are necessarily abridged because of space restrictions. A copy of the online survey may be obtained from ARE. A copy of the Natsal 2000 questionnaire may be obtained from the UK Economic and Social Research Council Question Bank at http://qb.soc.surrey.ac.uk/surveys/nssal/nssal00.htm.

Statistical analysis

Survey samples were compared using the survey analysis functions of the statistical software STATA V.7. The Natsal sample was weighted to adjust for unequal probabilities of selection and differential non‐response to make it representative of the population in terms of age, sex and region.²² Adjustment weights were not applied to the internet sample because probabilities of selection and level of non‐response cannot be calculated for a convenience sample.

Means or percentages are presented for all the variables examined for each sample. Confidence intervals (CI) of 95% are presented with the Natsal percentages to provide a measure of precision for these estimates. CIs for the internet percentages were narrow, and are not presented here because they add little to the interpretation of the data from this non‐probability sample.

The t test for independent groups and χ² test were used to examine significant differences between the means and proportions of background characteristics of the samples. Binary logistic regression analysis was applied to examine the association between sample and HIV testing, sexually transmitted infection (STI), anal intercourse and UAI. Where there was a significant association between the sample and any of these outcomes (p<0.05), multivariable binary logistic regression analysis with forward stepwise selection was used to examine whether the sample remained a significant predictor of sexual risk when confounding variables were included in the model. Crude and adjusted odd ratios (aORs) are presented with 95% CI and p values.

Estimating anal intercourse and UAI in past 3 months

The Natsal sample reported anal intercourse in periods of up to 1 week, 4 weeks, 6 months, 1 year and ⩾5 years before the interview, whereas the internet sample reported anal intercourse only in the 3 months before completing the questionnaire. Figure 1 is a plot of when Natsal respondents who reported anal intercourse in the past year said that anal intercourse had occurred during that time. It shows the cumulative percentage of whether they reported anal intercourse in 1 week, 4 weeks, 6 months or 1 year before the interview. A logarithmic curve is fitted to the plot (y = 14.2ln(x)+42.2; y = percentage of men reporting anal intercourse; x = number of weeks). The curve indicates that, of those Natsal respondents reporting anal intercourse in the past year (n = 94), an estimated 78.8% had anal intercourse in 3 months before the interview (y = 14.2×ln(13)+42.2). In this way, we estimated that 74 Natsal respondents had anal intercourse in 3 months before the interview (94×0.788), and used this to estimate the percentage of all Natsal respondents (including those who did not report anal intercourse in the past year) who had had anal intercourse in the 3 months before the interview (74/117×100 = 63.3%).

Figure 1 Cumulative percentage of National Survey of Sexual Attitudes and Lifestyles (Natsal) men who have sex with men (MSM) respondents reporting anal intercourse in the past year.

The Natsal sample reported UAI for periods of 4 weeks and 1 year before the interview, whereas the internet sample reported UAI in the 3 months before completing the questionnaire. Fitting a logarithmic curve to the cumulative percentage of Natsal respondents reporting UAI in the 4 weeks and 1 year before the interview (y = 0.141ln(x)+0.396) indicates that an estimated 75.8% of Natsal respondents reporting UAI in the past year (n = 56) had UAI in the 3 months before the interview. This can be used to estimate the number and percentage of Natsal respondents who had UAI in the 3 months before the interview (n = 56×0.758 = 42.4 and 42.4/116×100 = 36.6%, respectively).

The χ² test was used to examine differences in the percentage of respondents reporting anal intercourse and UAI over the past 3 months because crude and aORs cannot be calculated for data that are only available at the sample level.

Results

The internet sample consisted of 2065 MSM living in Great Britain (18–44 years) who completed the online survey. These men are thought to represent <1% of men using the gaydar and gay.com chatrooms and profiles over the survey period, on the basis of estimates of usage provided by the website owners. Natsal interviewed a total of 11 161 adults resident in Great Britain, with a response rate of 65.4%. The subsample of 135 MSM (18–44 years) that was identified for this analysis was equivalent to an effective subsample of 117 men, after adjusting for differing probabilities of selection and non‐response.

Table 2 shows the sociodemographic and behavioural characteristics of the two samples. There were no significant differences between the two samples in terms of reported ethnicity, education, social class, being currently employed, living in an urban area, country of birth, alcohol consumption, injecting drug use or age when they first had sex with a male partner (p>0.05). Regional distribution about the country was not significantly different, but there was strong evidence²⁵ that, compared with Natsal respondents, men from the internet sample were younger (p<0.001), and more likely to be students (p = 0.001), but less likely to live in London (p = 0.001). There was weaker statistical evidence that men from the internet sample were less likely to report good health (p = 0.014) or be unemployed, retired or otherwise not working (p = 0.007).

Table 2 Sociodemographic and behavioural characteristics by sample.

Denominator*	Sample				p Value‡
	Internet (2065)		Natsal (117 135)
	n	%	n†	% (95% CI)
Mean (SD) age (years)	29.4	7.6	31.9	6.2	<0.001
Employment status§					0.001
Student	351	17.0	6	5.2 (2.5 to 10.6)
Not working	141	6.8	16	13.7 (8.4 to 21.5)
Working	1573	76.2	95	81.1 (73.1 to 87.2)
Ethnic group (white)	1955	95.6	108	92.3 (86.7 to 95.7)	0.066
Holds degree or equivalent	898	43.6	51	43.6 (34.1 to 53.7)	0.994
Social class (non‐manual)	1298	82.3	87	80.0 (69.9 to 87.3)	0.593
Region of residence					0.291
England—North	442	21.9	29	25.0 (16.6 to 35.8)
England—Midlands	262	13.0	15	13.1 (7.3 to 22.2)
England—South	1034	51.2	60	51.6 (43.3 to 59.9)
Wales	69	3.4	7	5.8 (2.4 to 13.3)
Scotland	211	10.5	5	4.5 (1.9 to 10.1)
Lives in an urban area	1880	91.9	109	93.7 (86 to 97.3)	0.55
Lives in London	493	23.9	38	32.3 (28.4 to 36.4)	0.001
Country of birth (UK or Ireland)	1847	89.9	98	84.1 (75.5 to 90.1)	0.064
Health, self‐reported as good	1650	80.1	103	88.5 (82.1 to 92.8)	0.014
Drinks alcohol regularly	1427	69.4	87	74.2 (64 to 82.4)	0.336
Injected drugs, past year	38	1.8	1	0.9 (0.2 to 4.3)	0.393
Mean (SD) age at first sex with male partner (years)†	17.6	5.4	17.9	6.1	0.64

*Weighted unweighted denominator for men surveyed for Natsal.

†Weighted numerator for men surveyed in Natsal.

‡p Value associated with t test and χ² statistics from comparing means and proportions.

§Working versus others, p = 0.216; not working versus others, p = 0.007.

We found no significant differences between the two samples in whether they reported testing for HIV (table 3). However, the data strongly indicated that, after adjusting for confounding factors, men from the internet sample were more likely to report having had an STI in the past year (16.9% v 4.8%, aOR 4.14, 95% CI 1.76 to 9.74; p = 0.001) and more likely to report anal intercourse: 76.9% of men from the internet sample reported anal intercourse in the past 3 months compared with an estimated 63.3% of Natsal respondents (p = 0.001). Men from the internet sample were also more likely to report anal intercourse in the past 3 months than Natsal respondents in the past 6 months (76.9% v 69.3%, aOR 1.59, 95% CI 1.03 to 2.45; p = 0.036). The statistical evidence for an association between sample and UAI was weaker. Of the internet respondents, 45% reported UAI in the past 3 months compared with an estimated 36.6% of Natsal respondents (p = 0.064).

Table 3 HIV testing, sexually transmitted infection and sexual behaviour by sample.

Denominator*	Sample				Crude OR (95% CI)	p Value‡	aOR§ (95% CI)	p Value¶
	Internet (2065)		Natsal (117 135)
	n	%	n†	% (95% CI)
HIV tested, ever	1090	53.5	57	53.3 (42.8 to 63.4)	1.01 (0.66 to 1.55)	0.958	—	—
STI, past year	346	16.9	6	4.8 (2.2 to 10.4)	4.01 (1.74 to 9.25)	0.001	4.14 (1.76 to 9.74)††	0.001
Anal intercourse
3 months v 4 weeks	1587	76.9	60	51.8 (41.8 to 61.6)	3.09 (2.04 to 4.68)	<0.001	3.24 (2.12 to 4.96)‡‡	<0.001
3 months v 3 months (estimate)**	1587	76.9	74	63.3	—	0.001	—	—
3 months v 6 months	1587	76.9	81	69.3 (59.9 to 77.3)	1.47 (0.96 to 2.25)	0.073	1.59 (1.03 to 2.45)§§	0.036
Unprotected anal intercourse
3 months v 4 weeks	922	45	29	24.9 (17.4 to 34.2)	2.47 (1.56 to 3.91)	<0.001	2.67 (1.66 to 4.31)¶¶	<0.001
3 months v 3 months (estimate)**	922	45	42	36.6	—	0.064	—	—
3 months v 12 months	922	45	56	48.3 (38.7 to 58)	0.88 (0.59 to 1.3)	0.513	—	—

STI, sexually transmitted infection.

*Weighted, unweighted denominator for men surveyed for Natsal.

†Weighted numerator for men surveyed in Natsal.

‡p Value associated with crude OR derived from bivariate model and with χ² statistic from comparing proportions for data that are only available at the sample level.

§OR with Natsal 2000 sample as reference group adjusted in final model by †† not working, lives in London, health self‐reported as good, age of first sex with male partner; ‡‡ lives in London, country of birth, age of first sex with male partner; §§ student, lives in London, country of birth, age of first sex with male partner; ¶¶ student, not working, holds degree or equivalent, country of birth, age at first sex with male partner.

¶p Value associated with aOR derived from multivariate model.

**Natsal 3‐month estimate is calculated for the 13 weeks before the interview.

Discussion

This is the first study to compare a self‐selected convenience sample of MSM recruited through the internet with a nationally representative sample of MSM. Previous investigations have compared self‐selected internet samples of MSM with convenience samples that were recruited in gay venues.⁴,⁹,¹⁸,¹⁹

In our study, the self‐selected internet sample of MSM living in Great Britain was broadly similar to the sample of MSM drawn from a probability sample of the general population resident in Great Britain on a range of sociodemographic and behavioural variables. The internet sample contained more students and fewer respondents who were unemployed, retired or otherwise not working. This difference was also found in a Swedish sex survey, which compared men and women recruited via the internet with a probability sample.²⁶

However, we found strong statistical evidence of differences between samples in reporting STIs in the past year and anal intercourse in the past 3 months. The differences remained significant after adjusting for confounding factors, such as age at first sex or being a student. Men from the internet sample were also more likely to report UAI in the past 3 months, although the significant evidence for this was weak (p = 0.064), possibly owing to lack of power. These findings suggest that estimates of high‐risk sexual behaviour based on internet samples of MSM are likely to overestimate levels of risk behaviour in the wider MSM population.

Men recruited while actively seeking male partners through websites such as gaydar and gay.com might be expected to be more sexually active than those recruited from the general population. A similar differential was found in the Swedish study where men and women recruited through the internet reported more sexual partners than the probability sample with which they were compared.²⁶ However, the differential is not limited to convenience samples recruited via the internet. A comparison of a subsample of MSM from London recruited in Natsal 2000 with a convenience sample of MSM London recruited in gay bars, clubs and saunas found that men from the convenience sample were also more likely to report STIs, more male sex partners and more UAI partners.²⁷

The difference seen here between internet and probability samples of MSM in reporting high‐risk sexual behaviour suggests that adjustment weights may be usefully applied to internet samples of MSM to account for this potential selection bias. Although weights may be devised to successfully predict certain outcomes,²⁸ we advise caution in using the data presented here for this purpose. Their construction would be better investigated using a broad range of variables collected from internet and probability samples surveyed simultaneously using research instruments containing equivalent questions. In addition, a larger sample of MSM identified through a random probability survey would provide greater confidence in the generalisability of the weighting scheme.

Our analysis has some limitations. For example, it was not possible to compare identical measures of reported anal intercourse and UAI between the samples. However, the data clearly indicate that men in the internet sample were more likely to report anal intercourse and there is also more evidence that they were more likely to report UAI. Comparing UAI among internet respondents (in the past 3 months) and Natsal respondents (in the past 4 weeks) yielded an aOR >1. There was also significant evidence for a differential when comparing UAI among internet respondents (in the past 3 months) with the 3‐month Natsal estimate, although it was not strong (p = 0.064). However, circumstantial evidence suggests that the differences in reporting UAI were real. This is because a similar proportion of men from the internet sample reported UAI in the past 3 months, as was reported by the Natsal respondents in the past 12 months. This suggests that if the men from the internet sample had been asked, they would have reported more UAI in the past 12 months than Natsal respondents.

Data on the HIV status of the respondents and their UAI partners were available for men from the internet sample, but not from the Natsal sample. Consequently, we could not examine whether the difference in UAI between the two samples was with a partner of the same HIV status (serosorting) or with a partner of discordant or unknown HIV status.

The issue of question comparability arises in the case of other questions, which were not identical. Data were furthermore not gathered at the same points in time, which might account for the differences in estimates of risk behaviour. However, this is unlikely as no increase in high‐risk sexual behaviour with a casual partner was found among homosexual men in London between 2001 and 2003.²⁹

On the other hand, our study highlights an important advantage of recruiting national samples of MSM through the internet. The internet sample contained a total of 2065 men whereas the Natsal sample contained only 135 men (unweighted). The size of the internet sample facilitates detailed analysis of subgroups and multivariate analysis.

Overall, our findings suggest that the internet provides a valuable means of recruiting a large national sample of MSM whose social and demographic characteristics are broadly similar to those of a representative, probability sample. Internet samples are, however, likely to overestimate the prevalence of high‐risk sexual behaviour, and the data they generate should be interpreted in this context.

Abbreviations

aOR - adjusted odds ratio

MSM - men who have sex with men

Natsal - National Survey of Sexual Attitudes and Lifestyles

STI - sexually transmitted infection

UAI - unprotected anal intercourse