Respondent-driven sampling estimators under real and theoretical recruitment conditions of female sex workers in China

Abstract

We compare the performance of multiple respondent-driven sampling estimators under different sample recruitment conditions in hidden populations of female sex workers in the midst of China’s ongoing epidemic of sexually transmitted infections (STIs). We first examine empirically calibrated simulations grounded in survey data to evaluate the relative performance of each estimator under ideal sampling conditions consistent with respondent-driven sampling assumptions and under conditions that mimic observed respondent-driven sampling recruitment processes. One estimator, which incorporates respondents’ reports on their network of contacts, substantially out-performs the others under all conditions. We then apply the estimators to empirical samples of female sex workers collected in two Chinese cities which include unique data on respondents’ networks. These empirical results are consistent with the simulation results, suggesting that traditional respondent-driven sampling estimators overestimate the proportion of female sex workers working in low tiers of sex work and are likely to overstate the STI risk profiles of these populations.

Introduction

Hidden and hard-to-reach populations, such as female sex workers and others at high risk of acquiring sexually transmitted infections (STIs), lack a sampling frame but are of special interest to public health.¹ Traditional observation schemes for these populations—from direct observation to clinic-based inquiries to snowball sampling—are non-representative, impeding efforts to understand STI prevalence and transmission. Respondent-driven sampling has become the dominant method for collecting generalizable samples of hidden populations, with more than $150 million in federal funding.^2–4 Respondent-driven sampling uses peer referral and a dual incentive structure to recruit a sample, compensating respondents for participating and referring others. Several estimators have been suggested to reduce sampling biases in respondent-driven sampling, but each makes stringent assumptions about recruitment dynamics that may diverge from the real world.^5,6 Therefore, applied researchers have little guidance on which estimator to present in manuscripts and reports.

Here, we consider the relative performance of seven estimators under ideal sample recruitment conditions as well as real-world conditions observed in respondent-driven sampling studies of female sex workers in China, a hidden population with socially ordered tiers of sex work,^7,8 each related to different levels of STI infection in the backdrop of a growing Chinese STI epidemic.^9–11 We examine the following respondent-driven sampling estimators: 1) Naïve, 2) RDS1-SH¹², 3) RDS1-DS¹³, 4) RDS1-DG¹⁴, 5) RDS2-VH¹⁵, 6) RDS2-SS¹⁶, and 7) RDS1-LEN¹⁷ (these estimators are discussed in the eAppendix). We anchor our evaluation in empirically grounded simulations,¹⁸ which combine novel statistical techniques for network estimation and prediction^19,20 with data collected with respondent-driven sampling and a venue-based sampling approach among Chinese female sex workers. We follow up these analyses with an application of each estimator to two empirical samples of female sex workers in China.^7,18

Prior Evaluations of respondent-driven sampling

Prior empirical assessments of the methodological validity of respondent-driven sampling have compared it to alternative sampling approaches,^21–25 or benchmarked its estimates against known population parameters in non-hidden populations.^26–29 Prior studies show that respondent-driven sampling recruits samples more quickly, more cost-effectively, and with more confidentiality than other approaches, but that its estimates “are reasonable but not precise” compared to benchmarks.²⁷ Few comparative studies have examined the validity of respondent-driven sampling assumptions in the field, but recent work has shown that respondents in the real world recruit peers preferentially, violating crucial respondent-driven sampling assumptions.^7,30 Other evaluations have simulated respondent-driven sampling-style samples on synthetic or empirical social networks and evaluated robustness to violations of assumptions about social network structure,^4,5,31–33 data collection practices,^5,16 and recruitment dynamics,^6,18 but these studies typically only look at one or two estimators; more extensive comparisons have presented inconclusive findings.⁶

Methods

We focus on a variable of particular relevance to the expanding STI epidemic in China: the proportion of female sex workers in low tiers of sex work, who solicit clients in saunas, massage parlors, or streets, as opposed to high-tier female sex workers who solicit clients in karaoke bars, star hotels, and night clubs.⁷ Tiers are a distinctive feature of sex work in Asia³⁴ and STI infections and risky behaviors are concentrated among female sex workers in low tiers.^11,35,36 The tier-based social stratification of sex work has also been shown to bias recruitment dynamics in respondent-driven sampling, leading to overestimates of the proportion in low tiers.^7,18

We first consider how respondent-driven sampling estimators perform under realistic and idealized recruitment conditions using multiple data sources including (a) a population social network generated from data collected in the PLACE-RDS Comparison Study of female sex workers in Liuzhou, China^18,21, and (b) two sets of respondent-driven sampling chains simulated over this network, one that mimics recruitment patterns observed in the PLACE-RDS Comparison Study (“real-world scenario”), the other consistent with respondent-driven sampling assumptions (“ideal-world scenario”). These data sources are drawn from a previous study¹⁸ and are discussed in greater detail in the eAppendix. After examining simulation results, we also draw on empirical respondent-driven sampling data from the PLACE-RDS Comparison Study and the Shanghai Women’s Health Survey in an application section where we explore whether the simulation results generalize to empirical contexts. Respondents in both the PLACE-RDS Comparison Study and the Shanghai Women’s Health Survey provided verbal informed consent prior to participation. Surveys were administered face-to-face by trained interviewers in Mandarin Chinese or Zhuang. The PLACE-RDS Comparison Study protocol was approved by the Research Ethics Committee of the National Center for STD Control, China and the Institutional Review Boards at the University of North Carolina and Duke University. The Shanghai Women’s Health Survey protocol was approved by the Research Ethics Committee of the University of Wisconsin, Madison and of the Shanghai Institute of Planned Parenthood Research.

We examine both bias (accuracy) and efficiency (precision). Bias is measured as the mean difference between the statistic and parameter across simulated samples; it reflects tendencies toward over- or underestimation of population proportions of female sex workers in low tiers. To measure efficiency, we follow the respondent-driven sampling literature and examine design effects,^37,38 which are the ratio of the variance of the sampling distribution of RDS estimates to the sampling variance that would be obtained via simple random sampling. Because a biased estimator with low design effects may be preferable to one which is accurate but inefficient, we also compute the root mean square error, defined as the square root of the sum of sampling variance and squared bias.

Results

Performance of Respondent-Driven Sampling Estimators in Simulated Samples

Figure 1 focuses on how an estimator which is commonly presented in the literature, RDS2-VH, performs under real-world vs. ideal-world conditions.^5,18 Deviations from theoretical ideals in the real-world scenario lead to upward bias in the RDS2-VH estimator and to an overstatement of the fraction of the population in low tiers of sex work. However, estimates from the real-world scenario are less variable than estimates in the ideal-world scenario. Figure 2 shows the performance of all seven respondent-driven sampling estimators under the real-world scenario, where each tends to overstate the fraction of the population in low tiers of sex work. The RDS1-LEN estimator is least sensitive to these tendencies, followed by the Naïve mean, which has been reported to perform well in gold-standard evaluations.²⁸

Figure 1. Distributions of estimates of proportion in low tiers of sex work using the RDS2-VH estimator, by seeding and recruitment scenario.

Notes: The population proportion is computed from the largest connected component of the simulated population network and is shown with a dashed vertical line. Population social network is priorities for local AIDS control efforts survey data source with venue size adjustments and weak geographic distribution of ties assumption (see eAppendix for description of population social networks). Distributions are plotted by kernel density estimation.

Figure 2. Distributions of estimates of proportion in low tiers of sex work in the real world recruitment scenario, by estimator.

Notes: Dashed vertical line indicates the population proportion. Population social network is priorities for local AIDS control efforts survey data source with venue size adjustments and weak geographic distribution of ties assumption (see eAppendix for description of population social networks). Recruitment and seeding scenario is matched seeds and matched recruitments.

Table 1 extends these evaluations to both recruitment scenarios and quantifies estimator performance in terms of bias, design effect, and root mean square error. The ideal-world scenario has smaller biases but larger design effects than the real-world scenario. The RDS1-LEN estimator is the least biased and has the lowest design effects under both scenarios. Other estimators show substantial biases or else persistently high design effects, at levels consistent with prior literature^4,33,38, resulting in poor root mean square errors. These conclusions are robust to the underlying population distribution of female sex workers (see eAppendix).

Table 1.

Summary measures of estimator performance for proportion in lower tiers, by scenario.

	Bias	Design effects	Root mean square errors
Real world scenario
1. Naive	0.018	2.969	0.040
2. RDS1-SH	0.080	7.583	0.098
3. RDS1-DS	0.168	4.303	0.173
4. RDS1-DG	0.232	12.406	0.243
5. RDS1-LEN	0.013	3.251	0.040
6. RDS2-VH	0.159	3.990	0.164
7. RDS2-SS	0.153	3.913	0.159
Ideal world scenario
1. Naive	−0.092	20.170	0.131
2. RDS1-SH	0.006	13.046	0.075
3. RDS1-DS	0.024	29.728	0.115
4. RDS1-DG	0.028	74.102	0.180
5. RDS1-LEN	−0.018	6.267	0.055
6. RDS2-VH	0.023	25.849	0.108
7. RDS2-SS	0.018	25.557	0.106

Notes: Population social network data is drawn from PLACE data source with venue size adjustments and weak geographic distribution of ties assumption (see eAppendix materials for description of population social networks).

Performance of Respondent-Driven Sampling Estimators in Empirical Samples

Table 2 considers estimator performance in two empirical respondent-driven sampling samples of female sex workers in China, one recruited in Liuzhou for the PLACE-RDS Comparison Study in 2010,¹⁸ the other recruited in 2007 for the Shanghai Women’s HealthSurvey.⁷ Both studies gathered unique self-reports of network composition in follow-up interviews administered to recruiting participants when they collected incentives for successful referrals, which we use to provide the first empirical comparison of RDS1-LEN estimates to other RDS estimators.

Table 2.

Estimated proportions in low tiers of sex work in two data sets, by estimator.

	Liuzhou data		Shanghai data
	Whole samplea	Ego network subsetb	Whole samplea	Ego network subsetb
	(1)	(2)	(3)	(4)
Summary statisticsc
No. of observations	583	310	522	271
Proportion fully assigned		0.803		0.465
Earlier estimatorsd
Naïve	0.576	0.584	0.668	0.708
RDS1-SH	0.693	0.695	0.779	0.818
RDS1-DS	0.689	0.695	0.796	0.782
RDS1-DG	0.660	0.684	0.784	0.700
RDS2-VH	0.689	0.695	0.781	0.816
RDS2-SS	0.685	0.693	0.781	0.816
Linked ego networks estimator (RDS1-LEN)e
Accepted coupons		0.658		0.752
Offered coupons		0.626		0.751
Whole network		0.634		0.815

Notes:

^a: these columns show the traditional respondent-driven sampling estimators computed on the entire empirical sample; because only respondents who completed follow-up interviews reported on their ego networks, the RDS1-LEN cannot be computed for 273 of the Liuzhou and 251 of the Shanghai cases.

^b: these columns show the traditional estimators and the RDS1-LEN by type of network reported on the subset of respondents who completed follow-up interviews (see eAppendix).

^c: rows in this section show summary statistics of interest; “Proportion fully assigned” indicates the proportion of cases in that sample’s follow-up questionnaire reports where we were able to assign 100% of self-reported uninvited network peers to a tier of sex work.

^d: rows in this section show estimates based on estimators in the literature prior to 2012; for the RDS2-SS estimator, the assumed population sizes of sex workers in Liuzhou and Shanghai were 7,500 and 15,000, respectively.

^e: rows in this section show estimates obtained using different network types reported on in the follow-up questionnaire and described in the supplemental appendix with the RDS1-LEN estimator.

The top portion of Table 2 shows few large differences between estimates obtained in the full sample vs. the subset of respondents who were administered network composition questions. The bottom portion of the table presents RDS1-LEN results for three measured network types: (a) invited peers who accepted the invitation, (b) all invited peers, and (c) a respondent’s whole network comprising both invited and uninvited peers. The eAppendix provides details on the measurement of these network types. Table 2 yields the following conclusions: the RDS1-LEN estimator generates estimates that suggest a smaller fraction of female sex workers in low tiers than earlier estimators, albeit with a few discrepancies across network types discussed in the eAppendix. These results are consistent with the simulation results, indicating a smaller proportion of female sex workers in low tiers than that estimated by other estimators when applied to real-world data.

Discussion and Conclusions

We extended prior evaluations of respondent-driven sampling by comparing seven estimators under realistic sample recruitment scenarios that applied researchers are likely to experience in the field. We find that Lu’s linked ego networks (RDS1-LEN) estimator outperforms all others in simulated samples under both ideal and realistic sample recruitment scenarios, and that these results generalize to two empirical samples of female sex workers. Compared with earlier estimators, the RDS1-LEN estimator generates lower estimates of the proportion of female sex workers in low tiers in empirical settings where respondent-driven sampling recruitment assumptions are known to be violated.^7,18 By relying on self-reported personal network information, the RDS1-LEN estimator circumvents preferential recruitment biases, which confound other estimators.

This paper makes two contributions to the literature. First, we demonstrate that the RDS1-LEN estimator overcomes the two the most persistent problems plaguing other respondent-driven sampling estimators: bias in the face of realistic preferential recruitment and high design effects. We also demonstrate that it is possible to collect the type of data required by the RDS1-LEN estimator in empirical settings, and we consider three separate approaches to that data collection. We used a strong approach to derive these conclusions, relying on both simulated and empirical data about hidden populations of female sex workers and simulated and comparative approaches to data analysis, which enhances our confidence that the results can generalize, at least to other populations of female sex workers.

An important early paper on respondent-driven sampling estimators considered whether “to ask respondents what percentage of their friends fall into certain groups,”¹² but the authors dismissed this approach because respondents may misreport peers’ epidemiologically salient attributes, especially difficult-to-observe characteristics or behaviors such as STI/HIV status or condom use. The chief limitation of our paper is that the RDS1-LEN estimator cannot readily be extended to estimate such unobservable characteristics, which are of considerable interest to public health practitioners. However, RDS1-LEN’s outperformance of earlier estimators suggests that further attention to its development is warranted, especially gaining accurate self-reports about respondents’ personal network composition. We have highlighted measures of three different personal network types in this paper, showing that they produce consistent estimates, but more work is needed to develop protocols for accurate personal network data collection in respondent-driven sampling.^7,18 By comparing multiple respondent-driven sampling estimators, this paper will aid applied researchers in identifying which respondent-driven sampling estimator(s) should be presented in manuscripts and policy reports on the basis of their robustness to real-world sampling conditions.