A Critical Assessment of Bias in Survey Studies Using Location-Based Sampling to Recruit Patrons in Bars

Abstract

Location-based sampling is a method to obtain samples of people within ecological contexts relevant to specific public health outcomes. Random selection increases generalizability, however in some circumstances (such as surveying bar patrons) recruitment conditions increase risks of sample bias. We attempted to recruit representative samples of bars and patrons in six California cities, but low response rates precluded meaningful analysis. A systematic review of 24 similar studies revealed that none addressed the key shortcomings of our study. We recommend steps to improve studies that use location-based sampling: (i) purposively sample places of interest, (ii) utilize recruitment strategies appropriate to the environment, and (iii) provide full information on response rates at all levels of sampling.

Social ecological studies seek to understand the complex interactions between humans and the social, economic and physical environments in which they are located (Kreiger, 2001). Thus, survey research investigating social ecological relationships between respondents and any particular physical environment, such as patrons in bars, requires that participant responses be linked to those contexts. Since it is often impossible to obtain adequate samples of people who use specific contexts from general population surveys, location-based sampling, in which participants are recruited from those contexts, may offer an attractive alternative (Craun & Freisthler, 2008).

Applied to alcohol environments, social ecological approaches examine the ways in which interactions between people and drinking settings lead to greater alcohol use, abuse and related problems (Gruenewald, 2007). One of the theoretical claims of this approach is that bar patron characteristics stratify across settings, leading to the appearance of high-risk bars. We used location-based sampling to examine relationships between the individual characteristics of bar patrons (e.g. psychological profile, alcohol consumption), the environmental characteristics of the bars they patronize (e.g. proximity to patrons' homes), and alcohol-related problems in six cities in California. A census of bars and of patrons in bars was beyond the scope of the project, so to maximize generalizability our two-stage sample frame emphasized random selection of both bars and patrons. However, recruitment at both levels was poor (21.8% for bars; 7.0% for patrons). This motivated a systematic review of other studies that used location-based sampling in order to assess the feasibility of obtaining generalizable samples of patrons and bars using this sampling method.

Sampling Designs

For results from a multi-stage recruitment to be generalizable, eligible units must be representative of the source populations and selection and response bias must be limited at all stages (Henry, 1990). Representativeness is most efficiently achieved by randomly sampling from among the universe of potential participants (Kalton, 1983). Selection bias occurs when the probability that eligible units are selected into the sample is related to some characteristic which is also related to the dependent measure (Heckman, 1979). Response bias occurs when selected units do not participate and there are systematic differences related to outcomes between participant and nonparticipant groups (Kalton, 1983).

Commonly used approaches to sampling specific or hard-to-reach populations potentially yield non-generalizable samples. Even where initial eligibility is by random selection, generalizability may be impeded by selection and response bias. For example, convenience samples of easily accessible participants will be biased to the extent that selection is related to social networks, physical location, or other factors related to the outcome of interest (Kalton, 1983). Respondent-driven sampling (Heckathorn, 1997) will be biased by non-participation and respondents' preferences for recommending others, introducing systematic differences between branches of sampled networks that include participants and those that do not.

General population survey methods can minimize sample bias through multistage random samples of people within households (often within stratified geographic areas), or through other random sampling methods directed at forming representative samples. Examples include random-digit dialing (Kempf & Remington, 2007) and list assisted sampling (Tucker, Lepkowski, & Piekarski, 2002). Selection bias is minimized by the relative accessibility of samples at each stage (e.g. people within households within Census tracts). However, these survey methods are not adequate for social ecological research focused on human behaviors within specific contexts, because of the requirement that participant responses be linked to specific locations. In our case, bar patrons are relatively rare within the general population and compose a minority of drinkers (Gruenewald and Remer, 2014), so it is difficult to obtain sufficiently large numbers of respondents to assess patron characteristics of any particular place (e.g. “Bob's Bar and Grill”). Finally, these survey methods rely on respondents to report characteristics of the contexts of interest, a potentially substantial source of recall bias (as well as additional respondent burden).

Location-based sampling, in which researchers travel to randomly selected bars and recruit randomly selected patrons, intends to address these concerns by sampling people from the locations of interest. In this way survey responses can be directly linked to contexts, behaviors in these contexts assessed, and recall bias minimized. Selection and refusal bias remain a possibility; bar owners may refuse access to researchers and patrons may refuse to be interviewed. However, previous social ecological studies reported having good success recruiting participants using location-based sampling in bars and other similar contexts (e.g. Kelley-Baker, Voas, Johnson, Furr-Holden, & Compton, 2007; Voas et al., 2006).

The California Bar Study Survey

Six mid-sized cities of between 50,000 and 500,000 people in Northern California were selected from a larger ecological random sample of 50 cities examined in a study of alcohol availability, use and problems (Gruenewald and Remer, 2014). We selected the six cities based on geographic characteristics of interest to our project, and listed all bars within their city limits using data obtained from the California Department of Alcoholic Beverage Control. Bars were screened for possessing a valid license of type 23, 40, 42, 48, 61, or 75 (a bar/tavern license), or license type 47 (a Restaurant-General license) and a separate bar area (Ponicki, Gruenewald, Remer, Martin, & Treno, 2013). We then randomly selected bars from within the six cities. This yielded a sample of bars that was highly diverse in terms of size, patrons, locale, and operating characteristics (on which we report elsewhere). Research assistants approached bar staff requesting permission for a field team to return within two weeks to recruit patrons into a survey as they exited the bar. The assistants returned up to three times to obtain permission or refusal from a person of sufficient authority (e.g., a manager or owner).

Field teams attended the participating bars for two hours on Friday or Saturday nights to recruit patrons exiting the bars. The project assistants approached randomly selected exiting patrons with an offer of a $10 gift card for completing a brief demographic survey, and an offer of a further $30 gift card for calling to complete a 20-minute extended telephone survey during the following week (Protocol 1). When it was apparent that this approach was not yielding sufficient respondents for analysis, we abandoned the demographic survey and provided the randomly selected patrons with a wallet card containing instructions to complete the extended survey by telephone in exchange for a $40 check (Protocol 2). It was not possible to complete the extended survey on the street front due to the sensitive nature of some questions regarding drinking and related risks, and no confidential spaces were available at the bar exits.

Recruitment Outcomes

To calculate recruitment outcomes for our study and those included in the systematic review, we used the American Association for Public Opinion Researchers (AAPOR) standard definitions for in-person household survey outcomes (AAPOR, 2011). We calculated a response rate and a cooperation rate for both bar and patron recruitment phases. The minimum response rate is defined as the number of complete outcomes divided by the total number of eligible units. Possible outcomes are completion (C), partial completion (P), refused or reneged (R), not contacted (NC) or unknown eligibility (U).

$$\text{RR}=\frac{\mathrm{C}}{\mathrm{C}+\mathrm{P}+\mathrm{R}+\text{NC}+\mathrm{U}}$$ (1)

The cooperation rate is a more relaxed (though more commonly reported) measure than response rates, defined as the total number of completed and partial outcomes denominated by the total number of units approached for inclusion (i.e. excluding non-contacted units):

$$\mathrm{C}\mathrm{R}=\frac{\mathrm{C}+\mathrm{P}}{\mathrm{C}+\mathrm{P}+\mathrm{R}}$$ (2)

Of the 165 bars eligible for participation in the California Bar Study, bar recruiters contacted 112 and field teams were able to complete the patron survey recruitment at 36 locations (Table 1). The final bar response rate was 21.8%. Notably, 45 of the outlets were chains (or corporate franchises). The managers at 20 such locations all cited company policies prohibiting participation in research studies involving patrons. We did not approach the remaining 25 chain establishments, and removed one city from the sample (City 2) because 11 of its 19 bars were chains. In the 36 bars where we received permission to conduct patron recruitment, we received complete extended survey data from 94 of the 1,336 patrons whom we contacted, for a patron response rate of 7.0% (Table 2). The combined overall response rate for the universe of patrons in bars (i.e. the product of the bar response rate and the patron response rate) was 1.5%.

Table 1. Bar recruitment outcomes in 6 California cities.

Outcome	City 1	City 2	City 3	City 4	City 5	City 6	Total
Eligible	27	19	30	29	42	18	165
Unknown Eligibility (Ub)	0	0	0	0	0	0	0
Not Contacted (NCb)	9	19	7	8	5	5	53
Refused (Rb)	13	0	19	15	19	9	75
Partial (Pb)	0	0	0	0	1	0	1
Completed (Cb)	5	0	4	6	17	4	36
Bar Response Rate (BRR)	18.5%	n/a	13.3%	20.7%	40.5%	22.2%	21.8%
Bar Cooperation Rate (BCR)	27.8%	n/a	17.4%	28.6%	48.6%	30.8%	33.0%

Nb. There were 45 chains or corporate franchises among the 165 eligible bars. Our bar recruiters contacted 20 of these establishments, but all cited company policies prohibiting participation. We did not approach the remaining 25 chain establishments, and removed one city from the sample (City 2) because 11 of its 19 bars were chains.

Table 2. Patron recruitment outcomes in 36 bars.

	Protocol 1	Protocol 2	Total
Bars
Total	20	20	36
Total recruitment hours	35.0	42.0	77.0
Patrons
Eligible	593	1059	1652
Not randomized to inclusion	61	255	316
Unknown Eligibility (Up)	-	-	-
Not Contacted (NCp)	0	0	0
Refused (Rp)	397	408	805
Accepted wallet card only (Rp)	n/a	356	356
Completed exit survey only (Pp)	81	n/a	81
Completed phone survey (Cp)	54	40	94
Patron Response Rate (PRR)	10.2%	5.0%	7.0%
Patron Cooperation Rate (PCR)	25.4%	5.0%	13.1%

Bar response rates were low but within the expected range, however the patron response rate (and the patron cooperation rate) were well below what was expected. We were unable to assess the reasons for low patron participation. Field staff recruiting at bars reported that potential respondents were generally less likely to respond to recruiters of different demographic profiles to themselves (e.g., apparent ethnicity or age). Given the diverse range of patrons within the sampled bars, it was not possible to employ staff to match every conceivable demographic group. Another possible source of nonparticipation was insufficient remuneration. However, the incentives offered were within the typical range for our study area. Budget limitations and research ethics considerations precluded the team from offering higher incentives, and field recruiters reported that many potential respondents did not even stop to hear about the incentives. As the study protocols were designed to protect the anonymity of respondents, we did not collect telephone numbers, email addresses, or other means to contact potential respondents to follow up on reasons for nonparticipation.

Objectives of the Systematic Review

Though our approach minimized selection bias, the low bar and patron response rates impeded the planned statistical analyses and results could not be generalized beyond the sample. In that light, we conducted a systematic review to examine the effect of selection bias and response bias on generalizability in prior studies that have used location-based sampling to recruit bar patrons for survey research, and to determine whether it is indeed possible to recruit a generalizable survey sample using this approach.

Methods

Sample

Manuscripts eligible for inclusion in our systematic review reported the findings of studies that (i) investigated the social ecology of alcohol environments, where social ecology is defined as the interaction between patrons (agents) and licensed establishments (environments); (ii) used a location-based sampling strategy to recruit survey participants at venues licensed for on-premise alcohol sales; (iii) were full-length English language articles published in peer-reviewed journals; and, (iv) presented original research. Randomized controlled trials examining individual-level interventions (e.g., informing patrons of their blood alcohol concentration to reduce alcohol consumption; Meier, 1984; Croff, 2012) did not investigate social ecological relationships, so were not eligible for inclusion. Studies of informal dance events or raves were eligible only if the authors explicitly stated that data collection was completed exclusively at permanent licensed venues (i.e., not warehouses or festivals). Where two or more manuscripts reported results from the same data collection exercise, the article which provided the most complete information about sampling methods was selected for inclusion. Where additional relevant data was available in papers reporting secondary analyses, this information was extracted and is cited appropriately.

Search Strategy

We searched three electronic databases which were deemed relevant to this field and that provided sufficient combined breadth so as to capture the eligible studies: EMBASE (extensive), MEDLINE (all fields) and PsycINFO (keyword). In June 2013, combinations of the search terms ([bar OR nightclub OR club], [patron], and [survey OR recruitment OR portal]) presented in Figure 1 yielded 7,682 articles, of which 1,595 were removed as duplicates based on study author, year and title. We reviewed citations for the remaining 6,077 articles, and identified 204 for review of the abstract. Sixty-four were flagged for full-text review, of which eight were excluded as they were not studies of drinking in licensed venues, 14 did not use a location-based survey sampling strategy, ten were conference posters, books or brief reports, and 15 were secondary analyses of included studies. We examined the reference lists of the 17 articles eligible for inclusion in the study, and identified a further seven that met the inclusion criteria. In all, there were 24 original manuscripts that described 28 waves of data collection.

Figure 1. Sampling strategy and yield of systematic review.

Assessment Strategy

To assess selection bias we differentiated between bar and patron recruitment and categorized the strategies at each level as producing probability (i.e., random or purposive) or non-probability (i.e., convenient) samples. Purposive samples are used when researchers are interested in a particular subgroup (e.g., college bars, regular patrons). Results can be generalized to non-sampled members of that subgroup provided eligibility is objectively determined and both participants and locations are randomly selected from among all those that meet the predefined criteria (Kalton, 1983). Attempts to recruit all patrons in a bar during a defined study period were categorized as a census. This approach relies on random selection of bars in order for the results to be generalizable (Parsons, Grov, & Kelly, 2008). If studies used convenience samples at either the bar or patron level, results were deemed not to be generalizable beyond the included respondents.

Though necessary, a sampling frame that limits selection bias does not ensure generalizability; the most sound of approaches can still be undermined by poor recruitment outcomes. Therefore, for each study we extracted the bar response rate (BRR), bar cooperation rate (BCR), patron response rate (PRR) and patron cooperation rate (PCR) according to the AAPOR formulae defined above. Importantly, the necessary information for calculating these rates was not always outlined clearly in the manuscripts, but could be derived from the information provided in the prose, tables or figures. In such cases we extracted the raw count data or reported proportions and calculated the rates manually. In some instances the required information was sourced from both the primary and secondary manuscripts. Where sufficient information was not provided for readers to calculate outcomes, a study was assigned the category incompletely reported (IR). For publications that did not report whether permission was sought from bar owners, it is not clear if the bar outcomes are applicable. These were also coded as incompletely reported because the reader was not able to fully assess bias. In cases where the researchers did not seek permission from bar owners, the bar recruitment outcomes were considered not applicable (N/A). Except where publications stated otherwise, we assumed that they contacted all eligible bars or patrons.

Not all study designs fit our outcome definitions precisely. For studies where potential respondents were approached in groups, we extracted the cooperation rate for groups and the cooperation rate of respondents within groups, then calculated the patron cooperation rate as the product of the two. Other studies used two-step patron recruitment strategies. For example “portal” studies (in which respondents are recruited for a brief survey and/or biological sample collection at the threshold of a location and re-recruited for the same data collection procedures on exiting the location) have been used to precisely link respondents'intoxication to consumption of alcohol or other drugs within that location (Voas et al., 2006). This approach has the potential for respondents to be lost to follow-up, so where relevant, retention rates were also extracted. Patrons who conducted only the entry surveys were coded as partial outcomes, those who completed both the entry and exit survey were considered complete outcomes.

We used the bar and patron level assessments to obtain a global score for each paper. Studies were rated from 5 (highest) to 1 (lowest) based on the recruitment strategies and calculable outcome rates. Studies with randomly or purposively selected bars and patrons were rated 5 if both the bar response rate and the patron response rate were calculable, 4 if at least bar and patron cooperation rates were calculable, and 3 if only a patron cooperation rate was calculable. Studies for which the bars or patrons were selected conveniently (and for which the results are not generalizable) were rated 2. Studies for which the selection methods could not be assessed due to incomplete reporting or for which none of the response rates or cooperation rates were calculable were given the lowest score of 1.

Results

A summary of the 24 manuscripts included in this systematic review is presented in Table 3. Twenty (83.3%) used a cross-sectional design and four (16.7%) used the portal method. Four (16.7%) manuscripts contained two waves of patron recruitment utilizing different sampling strategies. In those cases, we disaggregated the response rates and extracted the results separately for each wave.

Table 3. Systematic review of studies using location-based sampling to recruit samples of patrons in bars (n = 24).

Study Details			Bar Recruitment			Patron Recruitment
Citation	Location	Study Design	Method	BRR	BCR	Method	PRR	PCR	Score b
Miller, 2009 Johnson, 2009	San Francisco Bay Area and Baltimore-Washington	Portal	Purposive	19.6%	62.5%	Random	37.1% a	49.7% a	5
Parsons, 2008	New York	Cross sectional	Random	IR*	IR*	Random (first wave) Census (second wave)	45.7% 62.2%	46.0% 62.5%	3
Clapp, 2009	California	Portal	Convenience	100.0%	100.0%	Random (Entry and exit) Random (Exit only)	36.0% a 46.7%	44.6%a 46.7%	2
Gueguen, 2012	France	Cross sectional	Convenience	NA	NA	Convenience	66.2%	84.5%	2
Harford, 1976	Boston	Cross sectional	Convenience	NA	NA	Convenience	29.2%	33.9%	2
Johnson, 2012 Miller, 2013	San Francisco Bay Area	Portal	Convenience	61.5%	69.2%	Random	19.9%	22.3%	2
Measham, 2009	Manchester, UK	Cross sectional	Convenience	IR*	IR*	Census	IR	91.5%	2
Thombs, 2008	Southeastern US	Cross sectional	Convenience	NA	NA	Convenience	IR	48.5%	2
Voas, 2006	East and West Coasts, USA	Portal	Convenience	IR	IR	Random	70.7%	78.9%	2
Degenhardt, 2006	Melbourne, Australia	Cross sectional	Convenience	IR	IR	Convenience	IR	IR	1
Duff, 2006	Melbourne, Australia	Cross sectional	Convenience	IR	IR	Convenience	IR	IR	1
Caudill, 2000	Maryland	Cross sectional	IR	53.0%	53.0%	IR	82.7%	89.0%	1
Meier, 1987	Washington State	Cross sectional	IR	IR	IR	Random	65.5%	87.3%	1
Ravn, 2012	Denmark	Cross sectional	IR	IR*	IR*	Convenience	88.0%	88.0%	1
Riley, 2001	Edinburgh, Scotland	Cross sectional	Convenience	IR	IR	Convenience	IR	IR	1
Thombs, O'Mara, Dodd, et al., 2009	Southeastern US	Cross sectional	Convenience	NA	NA	Convenience Random	IR IR	IR IR	1
Thombs, O' Mara, Tobler et al., 2009	Southeastern US	Cross sectional	Convenience	NA	NA	Convenience Random	IR IR	IR IR	1
Thombs, 2011	Southeastern US	Cross sectional	Convenience	NA	NA	Random	IR	IR	1
Measham, 2005	Manchester, UK	Cross sectional	Convenience	NA	NA	Random	IR	IR	1
Trocki, 2008	San Francisco Bay Area	Cross sectional	IR	IR*	IR*	IR	51.5%	56.0%	1
Jiang, 2013	San Diego	Cross sectional	Random	IR*	IR*	Census	IR	IR	1
Yacoubian, 2003	Baltimore-Washington	Cross sectional	IR	IR*	IR*	Random	85.4%	85.4%	1
Yacoubian, 2004	Washington, DC	Cross sectional	IR	IR*	IR*	Convenience	88.2%	88.2%	1
Yacoubian, 2007	Baltimore-Washington	Cross sectional	IR	IR*	IR*	Convenience	28.1%	76.5%	1

IR Incompletely reported

NA Not applicable (i.e. the authors did not seek permission from the bar owner, so the bar outcomes do not apply)

^*Authors did not report whether bar owner permission was sought, so it is not clear if the bar outcomes are applicable

^aAssumes consistent group size between recruited and refused groups, and no refusals for individuals within groups that agreed

^bReporting score defined according to the calculable applicable outcomes: (5) both the bar response rate and the patron response rate were calculable; (4) at least bar and patron cooperation rates were calculable; (3) only a patron cooperation rate was calculable; (2) bars or patrons were selected conveniently; (1) selection methods could not be assessed due to inadequate reporting or none of the response rates or cooperation rates were calculable.

Bar Sampling

Among the 24 studies, 14 (58.3%) selected bars conveniently, one (4.2%) purposively, and two (8.3%) randomly. There were seven (29.2%) studies in which the sampling strategy could not be assessed from the published reports.

Random selection of bars was rare. In one study (Miller, Furr-Holden, Johnson, Holder, Voas, & Keagy, 2009; Johnson, Voas, Miller, & Holder, 2009) researchers conducted unobtrusive observations to assess possible drug activity in 98 randomly selected nightclubs. Among a purposive sample of 51 “high” and “low” risk clubs, 10 were recruited for a bar response rate of 19.6%. However, only 16 eligible clubs were contacted so the bar cooperation rate was 62.5%. Reasons for non-contact are not provided, and generalizability may be reduced if contact was not randomly determined. Two (8.3%) studies used a time-space sample frame (Jiang & Ling, 2013; Parsons et al., 2008) in which opening hours for establishments were divided into sections, then weighted according to estimated patron throughput. The weighted time-space segments were then selected randomly for attendance by a field team, and either a random sample or a census of patrons was sought within the time-space units. This approach has the potential to yield a generalizable patron sample, however neither paper presented bar recruitment outcomes. If the researchers recruited patrons on the street front without seeking permission from bar owners then the bar recruitment outcomes were not applicable, but neither paper stated whether or not this was the case. This detail may have seemed irrelevant if the research teams did not interact with bar staff (it is not possible to report everything that researchers did not do), but it is nonetheless an important distinction.

Convenience samples of bars were typically chosen based on subjectively assessed characteristics of interest. Some papers described this method as purposive bar selection, however the lack of a random selection frame prohibits generalizability beyond the sampled bars. One study with a convenience sample of bars reported a bar response rate of 100% (Clapp et al., 2009) and one for which the bar recruitment strategy could not be characterized had a bar response rate of 53% (Caudill, Harding, & Moore, 2000). For another study, four refusals were received from among 13 conveniently selected nightclubs, for a cooperation rate of 69.2% (Johnson, Voas, & Miller, 2012; Miller, Byrnes, Branner, Johnson, & Voas, 2013). One of the recruited nightclubs was deemed too dangerous to complete the portal recruitment, so the response rate was 61.5%.

Seven (29.2%) studies used methods that did not require the bar owners'permission, so the bar response rate and bar cooperation rate were not applicable (N/A).

Patron Sampling

Of the 28 discrete patron recruitment waves reported in the 24 papers, 11 (39.3%) were convenience samples, 12 (42.9%) were random, three (10.7%) attempted a census of all patrons, and two (7.1%) had a sampling strategy that was incompletely described.

We calculated a patron response rate for 14 (50.0%) of the recruitment waves, and a cooperation rate for 18 (64.3%). All publications provided the total number of patrons for whom there were complete responses, but reporting of partial outcomes, refusals, and total approaches was inconsistent. Meaningful outcomes were only available for the random and census recruitment; in those studies we extracted eight patron response rates (range: 36.0% to 85.4%).

The calculations rarely accorded with (and generally appeared more favorable than) the AAPOR definitions. Several publications reported recruitment based on partial outcomes rather than complete outcomes (i.e. a cooperation rate) (e.g. (Caudill et al., 2000; Parsons et al., 2008; Trocki & Drabble, 2008; Yacoubian & Peters, 2007). For example, in the portal studies only entrance surveys were counted as complete outcomes. However, as the analyses entailed comparisons across the entrance and exit surveys, loss to follow-up should be reported. While these publications reported response rates of between 54% and 90%, we calculated the effective yield of all patrons eligible for inclusion to be between 19.9% and 70.7%.

In other instances we were unable to calculate outcome rates. For example, one study (Jiang and Ling, 2013) reported a 72.5% sample yield but did not state what value was used as the denominator (i.e., total eligible, total contacted), or how many invalid surveys were omitted (i.e., partial outcomes). Another study (Thombs, O'Mara, Dodd, Hou, Merves, Weiler, et al., 2009; Thombs, O'Mara, Tobler, Wagenaar and Clapp, 2009) reported response rates of 33.4% and 44.0% for patrons approached by a recruiter and directed to a research station (i.e., a retention rate), but did not provide the number of patrons initially approached by the recruiter.

Reporting Scores

One manuscript received a score of 5 (Miller, et al., 2009), and another (Parsons, et al., 2008) received a score of 3 because complete bar recruitment outcomes were not reported. Seven (29.2%) manuscripts with convenience samples of bar or patrons reported at least a patron cooperation rate, however the results were uninterpretable beyond the included bars or patrons due to the selection methods. Fourteen (58.3%) received scores of 1 because their selection methods were incompletely reported or because none of the bar or patron response or cooperation rates were calculable.

Discussion

This review of the research literature indicates that methodological and reporting shortcomings are common in studies using location-based sampling to investigate the social ecology of bars. After careful review, it appeared that only three of 24 research studies attempted to obtain probabilistic samples at both the bar and patron levels, and it was not possible to determine an overall response rate in two of these due to incomplete reporting of recruitment and response rates.

Considered in light of our own experience with location-based sampling, it may not be possible to obtain a survey sample of bar patrons generalizable to the universe of patrons or bars in any city or across any set of cities using this method. Recruitment of bars may be biased by structural features of alcohol sales within communities that preclude development of representative samples. For example, the sample frame for bars in our study included substantial numbers of chain and corporate franchises, but organizational policies at these venues precluded patron sampling. Recruitment of patrons may be biased by aspects of the survey process which attract some respondents but not others. None of the 24 studies included in our review reported addressing these limitations, and they have not been documented elsewhere in the literature.

Notwithstanding these recurring problems, the potential strengths of location-based sampling over other possible strategies remain. It may be the only feasible method for obtaining a respondent sample that can be reliably linked to specific contexts. Thus, the acceptable patron recruitment rates reported in some studies with random selection (e.g., Parsons, et al., 2008) give reason for optimism that selection and response bias can be minimized and generalizability achieved when certain conditions are met. In that light, we make three key recommendations for researchers seeking to use this strategy to study bar environments in the future:

First, survey researchers should select a purposive sample of locations. Our review indicates that location-based sampling is a viable approach for researchers seeking to generalize social ecological findings only to a particular subtype of bar or patron (e.g., high volume bars in high density areas, college students). Importantly, to maximize generalizability researchers should use systematic methods, beginning by enumerating all establishments within a defined study area. Where comprehensive lists are not available from licensing authorities, researchers should use alternate means (e.g., business directories, online searches, ethnography; Lipperman-Kreda, Grube, & Friend, 2012; Antin, Moore, Lee, & Satterlund, 2010; Morrison, Gruenewald, Freisthler, Ponicki, & Remer, 2014) to develop and verify a comprehensive list of eligible locations. A purposive sample frame should then be developed based on objective reproducible criteria related to the research aims, and if this list is too large to allow for a census, researchers should attempt to sample bars randomly from among the eligible locations. For example, in a randomized controlled trial of a staff-oriented intervention Graham, Osgood, Zibrowski, Purcell, Gliksman, Leonard, et al. (2004) limited the sample to large establishments which they had previously assessed as being at high-risk for aggression. These steps will focus the sample frame on locations of high relevance to the research questions but in which response bias is likely to be minimized at both the bar and patron levels, with the limitation that some types of locations (e.g., chain or franchise establishments) may be systematically excluded.

Second, recruitment strategies should be appropriate to the specific locations of interest. In our experience, “cold calling” (contacting bar staff in person, by telephone or by email without prior introduction) appears to be ineffective for recruiting bars to a research study. Graham, et al. (2004) had success (response rate = 69.2%) by employing third party recruiters who had personal relationships with bar owners (e.g. former alcohol sales representatives). At the patron level, matching recruiters to respondents on key demographic characteristics may improve outcomes. There may also be some benefit to priming patrons at their entry into the bar to expect an approach upon exit, as in portal studies.

Third, to enable readers to assess generalizability, authors should comprehensively describe their approach to recruiting both bars and patrons, and report complete recruitment outcomes at both levels. Imprecise terminology can mask biased results. Where selection of bars or patrons is non-probabilistic, it should be clear that results cannot be generalized beyond the study sample. The AAPOR definitions should be used to calculate outcomes for both levels of recruitment, and cooperation rates should be clearly distinguished from response rates.

In sum, location-based sampling has clear utility for studying social ecological relationships between bar environments and patrons when certain restrictions are imposed. For social ecological research outside bar environments, the strengths of location-based sampling remain and many of the limitations identified in this study may not apply. For example, selection bias may be a minimized when there are few enough locations or people within a location to allow for a census (e.g., gay men's bathhouses in a city; Woods, Euren, Pollack, & Binson, 2010). Response bias may be a minimized at the location level when researchers do not need to ask permission to recruit people associated with the locations (e.g., city parks, street corners) and at the person level when circumstances support participation (e.g., people waiting in an Emergency Department waiting room). Studies using location-based sampling in all contexts will benefit from researchers carefully defining sample frames and precisely reporting recruitment methods and recruitment outcomes.