Abstract
Objectives
We evaluated the ability of four sampling methods to generate representative samples of the Emergency Department (ED) population.
Methods
We analyzed the electronic records of 21,662 consecutive patient visits at an urban, academic ED. From this population, we simulated different models of study recruitment in the ED by employing two sample sizes (n = 200, 400) and four sampling methods: 1) true random; 2) random 4-hour time blocks by exact sample size; 3) random 4-hour time blocks by a pre-determined number of blocks; and 4) convenience or “business hours.” For each method and sample size, we obtained 1,000 samples from the population. Using chi-square tests, we measured the number of statistically significant differences between the sample and the population for eight variables (age, gender, race/ethnicity, language, triage acuity, arrival mode, disposition and payer source). Then, for each variable, method and sample size, we compared the proportion of the 1,000 samples that differed from the overall ED population to the expected proportion (5%).
Results
Only the true random samples represented the population with respect to gender, race/ethnicity, triage acuity, mode of arrival, language and payer source in at least 95% of the samples. Patient samples obtained using random 4-hour time blocks and business hours sampling systematically differed from the overall ED patient population for several important demographic and clinical variables. However, the magnitude of these differences was not large.
Conclusions
Common sampling strategies selected for ED-based studies may affect parameter estimates for several representative population variables. However, the potential for bias for these variables appears small.
INTRODUCTION
Background
Surveys and other observational study designs are common in emergency medicine (EM) research. However, little has been written about the strengths and limitations of various sampling methods. The ideal method would generate a sample that is representative of the population of interest with respect to patient- and disease-related variables. Random sampling or enrollment of consecutive eligible subjects from all encounters would seem ideal. However, it is costly, and often inefficient, to assemble random or consecutive patient samples in EM practice, because patients present for care seven days per week, 24 hours per day, albeit at different frequencies. This has led to alternative sampling strategies to obtain representative samples, including convenience samples, sampling during weekday “business hours” and sampling during randomly-selected time blocks.[1–5]
Importance
Business hours and other convenience samples are the easiest to collect, because patient recruitment occurs when research personnel are willing and most available. However, there is no guarantee that ED patients presenting during daytime hours are similar to patients who present at other times. In 1995 Lowenstein et al. first utilized random 4-hour time blocks in an attempt to generate a representative sample without requiring continuous emergency department (ED) coverage.[5] The authors assumed that random selection of enrollment periods, which included evenings, nights and weekends, would produce a representative sample of the ED population. This method was thought to prevent selection bias, while minimizing the off-hour time commitment of research staff. However, the representativeness of samples generated using random time blocks has never been confirmed.
Goals of This Investigation
The objective of this study was to evaluate the ability of four sampling methods to generate representative samples of the ED population. We evaluated four sampling methods (true random, random 4-hour time blocks stopping at the exact sample size needed, random 4-hour time blocks using a pre-determined number of time blocks and business hours) and two sample sizes (n = 200 and n = 400). We hypothesized that samples obtained using business hour sampling would differ systematically from the overall ED population, but that samples obtained using both of the random 4-hour time blocks methods and a true random sample would be similar to the overall ED population.
METHODS
Setting and study design
This study was conducted at a single urban, academic hospital-affiliated ED. The ED is a Level 2 trauma center with approximately 55,000 visits per year. Demographic and clinical data for each patient are entered into an electronic medical record at the time of ED care. During the study period (July 1, 2007 – November 30, 2007) 21,662 patients were treated in the ED. The study was approved by our institutional review board with a waiver of informed consent.
Data collection and processing
We retrospectively reviewed selected medical record data for all 21,662 patients. We extracted information pertaining to eight patient attributes: age; gender; race or ethnicity; triage acuity; mode of arrival; disposition; primary language spoken; and payer sources. We selected these because they are important sample characteristics that are commonly used in ED-based clinical research and may help to identify the potential for selection bias. We examined the distribution of these eight variables in samples of patients generated using four sampling methods that simulated different models of study recruitment in the ED. The four sampling methods were: 1) true random sampling; 2) random 4-hour time block sampling; 3) a fixed number of random 4-hour time blocks; and 4) business hour sampling (defined below). For each sampling method we also studied two sample sizes (n = 200 and n = 400). These sample sizes are used commonly in ED observational studies, as they provide estimates for proportions with 95% confidence limits of less than or equal to 7 percent or 5 percent respectively. The distribution of demographic and clinical variables for each sample was compared to the true distribution in the entire population of ED patients treated during the 5-month study period.
We generated 1,000 samples from the entire population using each method and sample size without replacement so that each individual or block could only be selected once for each sample. The true random sample was generated by having SAS (version 9.1; SAS Institute Cary, NC) randomly select the participants until the number of participants reached the desired sample size. Each participant in the data set had an equal chance of being selected and could be selected for multiple samples.
The random 4-hour time blocks were generated by dividing every day in the study period into 4-hour time blocks (0701–1100, 1101–1500, 1501–1900, 1901–2300, 2301-0300 and 0301–0700). For the random time blocks with a set sample size, the time blocks were randomized, and the sample included as many time blocks as required to accrue the desired sample size, assuming that all ED patients during each time block were recruited. If the sample size was reached in the middle of a time block, no further patients from that time block were included.
The random 4-hour time blocks method with the pre-determined fixed number of blocks used the same 4-hour time blocks as the random 4-hour time block method with the set sample size. Based on the average number of patients seen in each 4-hour period during the sample collection period and the odds of selection of each time block, we determined that a sample size of roughly 200 would include 9 blocks and a sample size of roughly 400 would include 17 blocks. All patients from each of the randomly selected 9 or 17 blocks were included in the samples.
Finally the business hour samples were generated by selecting random dates in the study period as the start date. Then, in an effort to mimic a standard work day (and excluding the lunch hour), ED patients who presented Monday to Friday (9 am until noon and 1pm until 4 pm) were included until the desired sample size was achieved.
Primary data analysis
The statistical analyses proceeded in an identical manner for each of the eight sampling methods - sample size combinations. One thousand samples were generated for each method and sample size; for each one, we compared the proportions of values for each variable (age, gender, ethnicity, etc.) to the proportions of values for the overall ED population using the 2-tailed chi-square test. Each comparison was considered statistically significant if p<0.05.
For each variable, we tabulated the number of samples where the distribution had a statistically significant difference from the distribution of the overall population. We compared the number of samples with statistically significant differences to the expected less than 50 samples (5%, based on p<0.05 for each comparison) that would be different, if the null hypothesis were true (i.e., the sampling method produced a sample equivalent to the overall population). We considered the sampling method/sample size representative with respect to precision if the proportion of samples that differed from the true distribution was less than five percent.
Testing for bias due to directional error
Differences between the samples and the overall ED population value could arise in one of two ways. Directional error, a type of bias, would occur if the sampling strategy selected patients who differed from the overall ED population in a systematic way (e.g. younger). We felt that this might occur with the business hour model. Non-directional error would arise if clustering occurs within the individual samples, but the clustering averages out when multiple samples are obtained by the sampling method. We believe that this may occur with the random block hours where clustering, or grouping of patients from some preexisting structure that causes more similarities among patients within the group than among patients if they were randomly selected from the entire population,[9] could occur within the blocks. To test for directional error for each sampling method - sample size combination, we converted each variable into binary categories: age (0–49 vs. 50 years or older); gender; race or ethnicity (Caucasian vs. non-Caucasian); triage acuity (emergent vs. non-emergent); mode of arrival (ambulance vs. non-ambulance); disposition (admitted to the hospital vs. not admitted); primary language (English vs. other); and payer sources (private insurance or Medicare vs. self-pay or other public source). Using the first category in each pair, we measured the difference in the percentage in this category for the sample and the percentage in this category for the overall population. If the 95 percent confidence interval (CI) of the difference between the sample population and the overall population excluded the population value, then we concluded that there was directional error for that variable, sampling method and sample size. Finally, to explore the clinical significance of the differences between the sampling methods, we determined the median and the range of the parameter estimates for each sampling method.
RESULTS
Characteristics of population
The overall population consisted of 21,662 consecutive ED patient visits. Fifty-seven percent of the patients were female, and 72 percent were 49 years of age or younger. As outlined in Table 1, 51% were non-Caucasian, and 10% were non-English-speaking. Fifty-seven percent had private or government insurance. Table 1 also includes information about transport to the ED, triage acuity and disposition after treatment in the ED.
Table 1.
Characteristics of the population (N = 21,662)
| Characteristic | Percent |
|---|---|
| Age in years | |
| 12 and under | 6 |
| 13–19 | 7 |
| 20–49 | 59 |
| 50–69 | 21 |
| 70 and older | 7 |
| Gender (male) | 43 |
| Race or ethnicity | |
| Caucasian | 49 |
| Black | 25 |
| Hispanic | 21 |
| Other | 5 |
| Language | |
| English | 90 |
| Spanish | 8 |
| Other | 2 |
| Triage acuity | |
| Emergent | 15 |
| Urgent | 51 |
| Non-urgent | 34 |
| Disposition | |
| Admitted | 16 |
| Discharged | 71 |
| Left without full treatment | 13 |
| Mode of arrival | |
| Ambulatory | 87 |
| Ambulance | 13 |
| Payer source | |
| Self-pay declared | 43 |
| Private insurance | 20 |
| Public insurance | 24 |
| Medicare | 13 |
Main results
Representativeness and precision of the sampling methods
Only the true random samples represented the overall population with respect to gender, race/ethnicity, triage acuity, mode of arrival, language and payer source for more than 95 percent of the samples (Table 2). True random sampling produced representative samples for all variables except disposition (n=200 sample size) and age (in the n=400 sample size; results of the n=400 analysis can be seen in a web appendix).
Table 2.
Estimates of precision: Percent of non-representative samples out of 1,000 samples.
| Characteristic | Sampling Method and Sample Size | |||
|---|---|---|---|---|
| True Random | Random 4-Hour Time Block by number of blocks | Random 4-Hour Time Block by exact sample size | Business Hour | |
| n = 200 | n ≈ 200 | n = 200 | n = 200 | |
| Age | 4.4 | 11.0 | 10.4 | 12.0 |
| Sex | 4.2 | 5.2 | 5.2 | 3.6 |
| Race | 4.9 | 8.3 | 9.4 | 4.3 |
| Language | 4.2 | 5.8 | 6.8 | 5.0 |
| Triage acuity | 4.6 | 10.5 | 10.3 | 7.1 |
| Disposition | 6.6 | 21.4 | 22.2 | 7.9 |
| Mode of | ||||
| arrival | 4.8 | 11.3 | 10.4 | 5.3 |
| Payer source | 4.9 | 7.3 | 9.7 | 9.6 |
Bolded numbers refer to the percent of samples that were statistically different from the expected 5.0 percent
The other sampling methods produced samples that lacked precision and were not representative of the overall ED population more often than true random sampling (Table 2). The business hours sampling method slightly outperformed the random 4-hour time block sampling methods; however, it generated samples with estimates for eight of nine variables that were systematically larger or smaller than their population values. Both of the random time blocks methods created samples that differed from the overall ED population at least five percent of the time for all variables. The percent of non-representative samples ranged from 5.2 percent to 21.4 percent for the random time blocks method with the fixed number of blocks and ranged from 5.2 percent to 22.2 percent for the random time blocks method using the set sample size.
For the random time block sampling methods, increasing the sample size did not materially improve sample representativeness. For the business hour sample, increasing the sample size actually increased the number of samples with estimates that differed from the true values for several variables (Table 2). The range and median for the proportion estimates for each variable are shown in Table 3.
Table 3.
Range and median of each variable for each sampling method compared to the overall ED population.
| Characteristic | Median %, Min-Max% by Sampling Method | ||||
|---|---|---|---|---|---|
| Percent in Population | True Random | Random 4-Hour Time Block by pre-set number of blocks | Random 4-Hour Time Block with set sample size | Business Hour | |
| N=21,662 | n = 200 | n ≈ 200 | n = 200 | n = 200 | |
| Age | |||||
| 12 & under | 6 | 7, 2–12 | 6, 0.5–13 | 6, 1–14 | 5, 0.5–10 |
| 13–19 | 7 | 7, 2–14 | 7, 2–13 | 7, 2–13 | 6, 1–12 |
| 20–49 | 59 | 59, 49–68 | 59, 48–70 | 59, 49–71 | 59, 47–70 |
| 50–69 | 21 | 21, 12–29 | 20, 10–30 | 21, 12–31 | 23, 13–32 |
| 70 & older | 7 | 8, 2–14 | 7, 1–16 | 8, 2–14 | 8, 2–16 |
| Sex (male) | 43 | 43, 32–53 | 43, 33–54 | 43, 32–54 | 43, 33–53 |
| Race | |||||
| Black | 25 | 25, 15–35 | 24, 15–35 | 25, 16–35 | 24, 14–36 |
| Caucasian | 49 | 49, 38–60 | 49, 34–64 | 49, 37–61 | 51, 39–61 |
| Hispanic | 21 | 21, 11–30 | 21, 11–32 | 21, 12–34 | 20, 10–28 |
| Other | 5 | 6, 1–11 | 5, 0.5–18 | 6, 1–20 | 6, 1–11 |
| Language | |||||
| English | 90 | 90, 83–98 | 90, 81–97 | 90, 82–97 | 91, 84–96 |
| Spanish | 8 | 8, 1–15 | 8, 2–15 | 8, 2–16 | 7, 2–13 |
| Other | 2 | 2, 0–5 | 2, 0–7 | 2, 1–8 | 2, 0–7 |
| Triage acuity | |||||
| Emergent | 15 | 15, 7–25 | 15, 7–25 | 15, 8–26 | 14, 6–21 |
| Urgent | 51 | 51, 40–62 | 51, 38–65 | 51, 37–66 | 51, 40–61 |
| Non-urgent | 34 | 35, 24–46 | 34, 20–47 | 35, 22–50 | 35, 25–45 |
| Disposition | |||||
| Admitted | 16 | 17, 9–26 | 16, 8–25 | 17, 9–27 | 17, 8–26 |
| Discharged | 71 | 71, 60–80 | 71, 52–85 | 71, 54–82 | 72, 62–82 |
| Left before complete visit | 13 | 13, 5–20 | 12, 3–29 | 12, 4–29 | 11, 5–21 |
| Mode of arrival | |||||
| Ambulatory | 87 | 87, 78–95 | 87, 75–96 | 87, 77–95 | 88, 81–97 |
| Payer source | |||||
| Self-pay | 43 | 44, 30–56 | 43, 30–57 | 43, 31–57 | 43, 30–56 |
| Private insurance | 20 | 20, 12–29 | 20, 10–30 | 20, 10–30 | 19, 11–27 |
| Public insurance | 24 | 24, 14–35 | 24, 12–36 | 24, 14–35 | 25, 15–35 |
| Medicare | 13 | 13, 7–22 | 13, 4–23 | 13, 5–21 | 14, 6–23 |
Bias due to directional error
When the variables were dichotomized, the magnitude of the directional error was small (less than 1%) for all methods and sample sizes except for the age, race and transport variables using the business hours sampling method (Table 4). Patient samples collected during business hours were biased; they were more likely to be older, Caucasian and to have arrived by ambulance.
Table 4.
Estimates of bias: Differences between estimated and population percentages of patients for each patient characteristic.
| Variable value* | True Random | Random 4-Hour Time Blocks with Set Number of Blocks | Random 4-Hour Time Blocks with Set Sample Size | Business Hours |
|---|---|---|---|---|
| n = 200 | n ≈ 200 | n = 200 | n = 200 | |
| Difference (95%CI) | Difference (95%CI) | Difference (95%CI) | Difference (95%CI) | |
| Female | 0.0% (−0.3%, 0.2%) | 0.0% (−0.2%, 0.2%) | 0.4% (−0.4%, 0.1%) | 0.4% (0.2%, 0.7%) |
| 50 years or older | 0.1% (−0.1%, 0.3%) | −0.3% (−0.5%, 0.0%) | −0.1% (−0.9%, 0.7%) | 2.7% (2.5%, 2.9%) |
| English as primary language | 0.1% (−0.1%, 0.3%) | 0.1% (−0.1%, 0.2%) | −0.6% (−1.2%, 0.1%) | 0.9% (0.8%, 1.0%) |
| Caucasian | 0.0% (−0.2%, 0.2%) | −0.3% (−0.6%, −0.1%) | 0.2% (−1.0%, 1.3%) | 1.5% (1.3%, 1.7%) |
| Non-urgent triage acuity | −0.2% (−0.3%, 0.0%) | 0.2% (0.0%, 0.4%) | 0.6% (−0.5%, 1.6%) | 0.8% (0.6%, 1.0%) |
| Admitted | 0.1% (0.0%, 0.3%) | 0.0% (−0.2%, 0.1%) | 0.7% (−0.2%, 1.6%) | 0.5% (0.3%, 0.7%) |
| Medicare or commercial payer source | 0.0% (−0.2%, 0.2%) | 0.1% (−0.1%, 0.3%) | 0.3% (−0.7%, 1.3%) | 0.1% (−0.1%, 0.4%) |
| Arrived by ambulance | −0.2% (−0.3%, 0.0%) | −0.1% (−0.3%, 0.1%) | −0.3% (−1.1%, 0.5%) | 1.3% (1.1%, 1.4%) |
Each variable was dichotomized: age (0–49 vs. 50 years or older); gender; race or ethnicity (Caucasian vs. non-Caucasian); triage acuity (emergent vs. non-emergent); mode of arrival (ambulance vs. non-ambulance); disposition (admitted to the hospital vs. not admitted); primary language (English vs. other); and payer sources (private insurance or Medicare vs. self-pay or other public source)
Sample size characteristics: Mean = 212.9 (std 34.1), min = 121, max = 309
LIMITATIONS
There are several important limitations to our study. First, we only examined eight demographic and clinical variables. We selected these because they are important sample characteristics and were easily extracted from the clinical and administrative dataset. We did not study other salient patient- and disease-related characteristics, such as vital signs, illness severity or co-morbidity. We also did not study social or behavioral attributes or chronic disease or injury risk factors. Our study was conducted in a single hospital-based ED during a relatively short (5 month) period of time. Accordingly, our results may not be generalizable to other ED settings or to other patient populations. Another limitation is that we limited our study to four sampling methods and two sample sizes. Our study also does not address several other methodological pitfalls and sources of enrollment bias. Research assistants may be less likely to enroll patients with certain characteristics (for example, those who are intoxicated, combative or reluctant to participate or who suffer from mental illness). If these characteristics are associated with the study question, selection bias may result, distorting the findings of the study, regardless of the sampling strategy employed. Proper training of research personnel and strict rules for inclusion and exclusion of patients may mitigate this source of selection bias.
DISCUSSION
This study demonstrates that true random sampling generates more samples that are representative of the true ED population than samples obtained by enrolling consecutive patients during business hours or during random 4-hour time blocks. Business hour and both random block samples differed significantly from the overall ED patient population in more than half of the eight demographic variables we evaluated. It is likely that these methods also result in differences in variables that were not measured, as these sampling schemes are less precise and more biased than true random sampling in ways that are difficult to predict. While the implications of these differences will depend on the study questions and the research design, researchers should be aware of these differences; if possible, researchers should compare known characteristics for differences between their sample and the population using information from ED electronic medical record databases or other available sources.
The finding that random 4-hour time blocks produced non-representative samples more than the expected five percent of the time was surprising. We had assumed that these techniques would produce samples that were representative of the true population. In our secondary analysis of the dichotomized categorization for each variable, we found that the mean differences between the estimates based on the 4-hour samples and the values of the true population were very near zero for all variables (Table 4). This suggests that the differences observed between the estimates based on the 4-hour random time blocks and the overall population values were not directional; that is, sample estimates were equally likely to fall above or below the true population value. The most likely explanation for this finding is clustering within time blocks. Clustering occurs when the subjects in the sample (i.e. time block) are more likely to share a characteristic than two subjects randomly selected from the population.[9] Clustering within observational and interventional studies results in a loss of statistical power and can produced biased samples.[10] When all of the samples are considered together, these differences in estimates average out and appear representative of the overall population (i.e. the magnitude of the directional error is small). However clustering produces a loss in precision for each sample as an estimate of the population parameters. Thus, clustering results in more than the expected five percent of samples that are statistically different from the true population values. There are analytical methods to account for clustering at different hierarchical levels.[9] Our results suggest that emergency medicine studies that incorporate time-block sampling methods should consider cluster effects when designing studies and utilize appropriate statistical methods in power calculations and data analysis.
We hypothesized that business hours sampling, which mimics a “convenience” sample, would produce systematically biased samples. Common sense and experience suggest that patients who present for care “between 9 am and 4 pm, excluding lunch,” will differ in demographic and clinical characteristics from those who present during weekends, evenings and nights. Consider the variable “alcohol use.” A study evaluating alcohol use would find different results if alcohol-related visits varied according to time of day and day of the week. Investigators should consider the relevant patient characteristics for each study and determine how the sampling procedure might be modified to ensure recruitment of a representative sample. Stratified sampling and overweighting are techniques that can improve sample representativeness.[7, 8] An alternative method is to employ concentrated consecutive patient enrollment over a short time period;[6] however, samples from this method may not generate representative samples based on the day of the week, time, month, season or year.
In the business hour samples, we found statistically significant (but numerically small) directional error in the estimates for almost all variables. The directional error actually increased with an increase in sample size, making the consequences of bias more substantial and worrisome. Directional error is important to recognize, because systematic differences between the samples and the population can produce a biased estimate that does not accurately reflect the true value in the population. When directional error is recognized, investigators can make some general statements like “using a business hour sampling we know that our population will be older and more likely to be Caucasian than the true ED population.” However, failure to recognize directional error may limit generalizability of results or lead to inaccurate conclusions.
Studies are frequently criticized for using business hour sampling. While we found statistical differences between the true population values and the estimates generated by business hour and both random 4-hour time block sampling methods, these differences were of questionable clinical meaning. Our study suggests that business hour sampling generally produced variable estimates that were numerically similar to those obtained using random sampling methods for these variables.
Conclusions
In this study of 21,662 consecutive ED patient encounters, patient samples obtained using random 4-hour time blocks and business hours sampling differed statistically from the overall ED patient population for several important demographic and clinical variables. However, for many research projects these differences may not be clinically significant.
Ultimately, the investigator’s choice of sampling methodology involves a trade-off between study resources and validity. Sampling methods that use the random time block or business hour model to recruit participants may increase feasibility at a modest cost to validity. Studies that devote more resources to achieve the ideal random sampling method will maximize validity but at a potentially large cost relative to the gain in validity, especially in clinical settings such as the ED.
Acknowledgments
Dr. Ginde is supported by grant KL2 RR025779 from the National Institutes of Health. Dr. Heard is supported by DA020573-03 from the National Institute on Drug Abuse. Dr. Lowenstein is supported in part by Grant Number R49/CCR811509 from the Centers for Disease Control and Prevention. The contents of this paper are solely the responsibility of the authors and do not necessarily represent the official views of the CDCP.
This manuscript is not under review at any other journal and has not been published previously, in whole or in part. We have not published or submitted any related papers. A similar abstract was presented as a moderated poster at the SAEM Annual National Meeting in 2008.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- 1.Alagappan K, McGowan J, Declaro D, Ng D, Silverman RA. Tetanus antibody protection among HIV-infected US-born patients and immigrants. Int J Emerg Med. 2008;2:123–126. doi: 10.1007/s12245-008-0020-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Carpenter CR, Scheatzle MD, D'Antonio JA, Ricci PT, Coben JH. Identification of fall risk factors in older adult emergency department patients. Acad Emerg Med. 2009;3:211–219. doi: 10.1111/j.1553-2712.2009.00351.x. [DOI] [PubMed] [Google Scholar]
- 3.Bijur P, Bérard A, Esses D, Calderon Y, Gallagher EJ. Race, ethnicity, and management of pain from long-bone fractures: a prospective study of two academic urban emergency departments. Acad Emerg Med. 2008;7:589–597. doi: 10.1111/j.1553-2712.2008.00149.x. [DOI] [PubMed] [Google Scholar]
- 4.Singer AJ, Garra G, Chohan JK, Dalmedo C, Thode HC., Jr Triage pain scores and the desire for and use of analgesics. Ann Emerg Med. 2008;6:689–695. doi: 10.1016/j.annemergmed.2008.04.017. [DOI] [PubMed] [Google Scholar]
- 5.Lowenstein SR, Tomlinson D, Koziol-McLain J, Prochazka A. Smoking habits of emergency department patients: an opportunity for disease prevention. Acad Emerg Med. 1995;3:165–171. doi: 10.1111/j.1553-2712.1995.tb03189.x. [DOI] [PubMed] [Google Scholar]
- 6.Ginde AA, Weiner SG, Pallin DJ, Camargo CA., Jr Multicenter study of limited health literacy in emergency department patients. Acad Emerg Med. 2008;15:577–580. doi: 10.1111/j.1553-2712.2008.00116.x. [DOI] [PubMed] [Google Scholar]
- 7.Cochran WG. Sampling Techniques. 2. New York, NY: Wiley; 1963. [Google Scholar]
- 8.Warwick DP, Lininger CA. The Sample Survey: Theory and Practice. New York, NY: McGraw-Hill; 1975. [Google Scholar]
- 9.Wears RL. Advanced statistics: statistical methods for analyzing cluster and cluster-randomized data. Acad Emerg Med. 2002;9:330–341. doi: 10.1111/j.1553-2712.2002.tb01332.x. [DOI] [PubMed] [Google Scholar]
- 10.Killip S, Mahfoud Z, Pearce K. What is an intracluster correlation coefficient? Crucial concepts for primary care researchers. The Annals of Family Medicine. 2004;2(3):204–208. doi: 10.1370/afm.141. [DOI] [PMC free article] [PubMed] [Google Scholar]