Study Designs and Evaluation Models for Emergency Department Public Health Research

Abstract

Public health research requires sound design and thoughtful consideration of potential biases that may influence the validity of results. It also requires careful implementation of protocols and procedures that are likely to translate from the research environment to actual clinical practice. This article is the product of a breakout session from the 2009 Academic Emergency Medicine consensus conference entitled “Public Health in the ED: Screening, Surveillance, and Intervention” and serves to describe in detail aspects of performing emergency department (ED)-based public health research, while serving as a resource for current and future researchers. In doing so, the authors describe methodologic features of study design, participant selection and retention, and measurements and analyses pertinent to public health research. In addition, a number of recommendations related to research methods and future investigations related to public health work in the ED are provided. Public health investigators are poised to make substantial contributions to this important area of research, but this will only be accomplished by employing sound research methodology in the context of rigorous program evaluation.

Emergency departments (EDs) are highly complex medical environments that serve as our society’s primary medical safety net. Emergency medicine researchers are well positioned to design and implement projects that produce generalizable knowledge to improve the public’s health. The design and execution of robust ED-based public health studies is challenging. Whether the focus is alcohol, tobacco, or other drug use, human immunodeficiency virus (HIV) or sexually transmitted infections (STIs), intimate partner violence, or other areas of health promotion and injury prevention, several common themes associated with both study design and model evaluation affect the quality of public health research in the ED.

This article was developed in the context of the 2009 Academic Emergency Medicine consensus conference entitled “Public Health in the ED: Surveillance, Screening, and Intervention” held on May 13, 2009. We report on the findings of a conference workshop intended to review study designs and evaluation models specific to ED-based public health research. This article reviews concepts related to ED-based public health research, including 1) study designs, 2) participant selection and retention, and 3) measurement and analyses. It frames the key concepts raised in discussion of these topics during the consensus workshop, with a broader goal of educating researchers and providing a more focused foundation for performing high-quality ED-based public health research.

STUDY DESIGNS

The choice of a valid study design is critical. Selection of a design and its features appropriate to the study’s context will minimize threats to internal validity by providing unbiased estimates of effect measures (all italicized words are defined in Data Supplement S1, available as supporting information in the online version of this paper). Research is generally grouped into four unique general categories, including experimental, quasi-experimental, preexperimental, and observational designs. Previous ED-based research has utilized all of these designs to achieve the goal of maximizing the internal validity of research performed to broadly improve the public’s health.

Emergency department–based public health research may also be separated into four specific categories that include surveillance, screening or testing, interventions, and economic evaluation. Ideal study designs depend primarily on the study question being asked and the type of investigation being conducted; however, some universal principles that increase the methodologic quality of the studies do apply.

Surveillance

Surveillance is a term that describes the systematic collection of population-based information to report primarily disease occurrence and their etiologies. Generally, surveillance has been divided into researchor non–research-related categories. Surveillance is likely to be nonresearch when it involves the regular, ongoing collection and analysis of health-related data, conducted to monitor the frequency of occurrence and distribution of disease or injury in a population. As such, these systems typically are under the purview of governmental and international organizations, including the Centers for Disease Control and Prevention (CDC) and the World Health Organization (WHO).^1,2

Surveillance conducted under a research purview, on the other hand, occurs when it involves the collection and analysis of health-related data conducted either to generate knowledge that is applicable to other populations and settings from those which the data were collected or to contribute to new knowledge about the health condition. Surveillance research in EDs is becoming increasingly common.^3,4

Surveillance research, by definition, is observational, and the most robust observational design includes prospective data collection. Surveillance research may also involve linking data collected in the ED to other sources of data, including those from national organizations, state health departments, or legal or welfare systems. Additionally, the validity and generalizability of surveillance research in the ED may be increased by use of multiple centers, standardized definitions of cases and outcomes, and around-the-clock recruitment. Excellent examples of such research include defining the prevalence of violence-related injury and STIs.^5–7

A large, high-impact ED-based surveillance study conducted near the beginning of the HIV epidemic was performed in the ED at Johns Hopkins University, using a then-novel identity-unlinked testing approach for characterizing the prevalence and demographic factors associated with HIV infection.⁸ This approach has now been widely adopted to help characterize various other public health issues in both ED and other health care settings.⁹

Linking data collected in the ED with other data sources related to major public health issues may improve our understanding of and perspective on these health issues. For example, Davidson et al.¹⁰ in 1997 linked ED visits for alcohol-related complaints with the Colorado Death Registry to learn that 5-year mortality rates among alcohol-intoxicated patients were 2.4 times that of an age- and sex-matched comparison group and that these alcohol-related visits were a significant predictor of increased morbidity and mortality. Similarly, a study of men with injuries inflicted by a female partner found that over 50% of the injured men had previous histories of arrest for domestic violence.¹¹

Real-time surveillance research has the potential for immediate reporting to the public health community as alerts of incipient and ongoing threats to the public’s health and it can also be used to assess the effectiveness of ongoing interventions.³ ED-based infectious disease surveillance for HIV and methicillin-resistant Staphylococcus aureus (MRSA), for example, were critical in early identification of the existence of these national epidemics and helped inform public health strategies directed at curtailing their spread.¹²

Audit Studies

Audit studies can be used to track access to outpatient care for vulnerable populations.^13–15 Lack of access to outpatient primary and specialty care is both a major public health concern and a contributor to ED crowding. Using experimental designs, auditing has been used as a research tool that employs an intentionally deceptive approach to uncover intentional or unintentional discrimination in a variety of markets (e.g., housing, credit, employment).^16,17 Asplin et al.¹⁴ incorporated this approach in the health care sector by having the same person call the same clinic twice with the same scripted request for an appointment following a visit to the ED. The investigators varied the person’s insurance status, however, to assess the impact of insurance status on access to care, demonstrating that patients with private insurance were more likely to receive an appointment than those with Medicaid or those without insurance.

Capture–Recapture

Capture–recapture is a relatively uncommon surveillance technique originally described for population biology. In recent years, however, it has been extended to epidemiologic investigations to estimate incidence or prevalence.^18,19 Although several capture–recapture approaches have been used, its general methodology includes examining the overlap in identification of cases from different data sources or populations. By calculating the expected number of persons in each combination of data sources, capture–recapture methodology allows for the estimation of persons identified by no data sources. The addition of persons identified by no source to the number of unique persons already identified provides an estimate of disease prevalence.^20–22

Interventions

An intervention is broadly defined as anything introduced into a research environment that is specifically controlled by the investigator. Examples of interventions in ED-based public health research may include a unique approach to screening, use of different types of counselors or different approaches to providing specific content to patients, or a multifaceted screening or testing program. Studies that attempt to evaluate the efficacy or effectiveness of an intervention typically use experimental designs; however, quasi-experimental and preexperimental designs have been used with some success. Experimental designs, also often referred to as randomized controlled trials, are considered the highest quality and most valid approach to assessing the impact of an intervention. This design provides the opportunity to create two or more study groups that are theoretically balanced across all measured and unmeasured characteristics with the exception of the intervention. As such, investigators are better able to assess the effect of the intervention while minimizing bias. There are a number of well-designed randomized controlled trials described or discussed in the ED-based public health research literature.^23–28

Randomized controlled trials are subject to certain biases, two of the more common being selection bias and measurement bias. Selection bias occurs when the study sample does not represent the population from which it was selected. This may occur when the sample size is too small, if refusal rates are high, or if eligibility criteria are too stringent. As an example, ED-based studies evaluating brief interventions for unhealthy alcohol use generally have strict inclusion and exclusion criteria. Many of these studies exclude patients with alcohol dependence, other drug use, or psychiatric illness, thus selecting a sample that is not representative of the broader group of patients with alcohol dependence. Also, choosing an ED as the site for recruitment limits the sample population and creates a potential bias relative to other clinical venues.

There are a number of quasi-experimental designs that have been used in public health ED research and are often used when the performance of a randomized controlled trial is too expensive, is premature relative to the maturation of the content area, or is simply impractical or not feasible. Specific quasi-experimental designs include nonequivalent control group, interrupted time–series, and the equivalent time–samples designs. Equivalent time–sample designs, for example, in which the intervention is alternated sequentially or in a randomized fashion with the control condition, allows the investigator to approximate true randomization. This approach attempts to balance two or more study groups (depending on the number of times the intervention and control are alternated, the length of each time period, and the number of subjects enrolled in each period). All quasi-experimental designs use some form of quasi-randomization to assess the effect of an intervention.

Preexperimental designs are considered the weakest experimental design in terms of ensuring internal validity. In general, two unique preexperimental designs exist, including one-group pretest–posttest (also commonly referred to as “before–after” or “pre–post”) and static group comparison (also commonly referred to as a “historical control group” or “nonconcurrent control group”) designs. These specific designs do not allow for the control for secular trends, and the groups being compared are likely to be dissimilar. The dissimilarity is most apparent when the physician is selecting patients for an intervention. When a preexperimental design is used, the two groups are likely not comparable at all, thus requiring multivariable modeling to adjust for variation between the two groups while assessing the independent effect of the intervention. This approach can result in erroneous conclusions about the usefulness of an intervention. Regardless of the study’s design, duplicate findings by different groups of investigators in different settings are required to provide evidence that an intervention is effective.

Economic Evaluations

Economic analyses are important components of program evaluation, providing integrated financial considerations related to public health interventions. Cost-effectiveness, or the costs per outcome (e.g., quality-adjusted life-years), must be distinguished from cost benefit, in which both costs and outcomes are expressed in monetary terms. In cost–effectiveness research, the denominator can be identified and enumerated. In cost–benefit analysis, the benefit is often nebulous and difficult to value in dollars. While many cost–effectiveness analyses related to public health interventions exist, there is still a need for critical economic evaluation of many specific ED interventions.^29–35 Most cost–effectiveness analyses are also constructed using theoretical models (i.e., combining data from multiple unique sources), which is useful, but use of actual clinical trial data to inform the economic evaluation is essential. Economic analyses performed concurrently with controlled clinical trials may provide a strong basis for understanding the financial impact of the intervention being studied and can be used to leverage funding to support research. A cost–benefit analysis, however, identifying actual costs of an intervention (the numerator in the analysis), is complex and generally requires the input of a health care economist.

PARTICIPANT SELECTION AND RETENTION

There are multiple methodologic challenges that influence results and limit generalizability, as has been seen in many of the public health ED studies reported to date.^36–38 Participant selection and retention may lead to bias, small sample sizes may reduce precision of estimates, and both are particularly important when designing clinical research. In total, the number of subjects included in public health research has been, in some instances, relatively small compared to other efficacy research for other medical conditions.³⁸ Recent studies, in particular, for alcohol use routinely carry high refusal rates, particularly among adolescents.³⁹

Participant Selection

It is critical that researchers clearly define the types of participants for inclusion in the study and make every attempt to recruit all such study participants presenting to the ED during the study period. Adequate assessment of unintentional selection bias requires comparisons of those who were included versus those who were missed.

Unintentional selection bias commonly results in a convenience study sample. The limited time periods when researchers or assistants are present in the ED may result in exclusion of a significant subsection of the population and because EDs are open 24 hours per day, this may represent an important but limiting methodologic challenge. To adequately assess this potential source of bias, it is crucial to report the number and characteristics of potential participants from time periods in which sampling was not performed.

Limitation of recruitment to specific risk groups is another problem with ED-based public health research. Although this strategy allows for easier recruitment and permits detection of modest intervention effects with smaller sample sizes, recruiting only a selected sample limits generalizability. In addition, certain groups may have a greater inclination to participate than others. Without assessing this propensity to participate, results may be biased by baseline openness to the intervention or screening method.⁴⁰

Finally, adequate enrollment is important in the context of a clinically meaningful outcome effect. While some public health research has the benefit of having large effect sizes, many others have modest to small effect sizes or suffer from confounding (which may mask a relatively larger effect size).⁴¹ In an attempt to identify modest effect sizes in the context of confounding, or within subgroups, there is a need for larger sample sizes to reduce the probability of type II error. Pilot testing and enrollment may serve as an important component of improving the precision of sample size estimates for larger, more definitive research.

Participant Retention

Loss to follow-up of ED patients has been a major challenge for ED researchers^42–45and a critique of emergency medicine public health research.⁴⁶ Study enrollment and follow-up protocols must be intentionally and thoughtfully designed and followed rigorously as the study proceeds. Using proven techniques (e.g., gift cards, multiple sources of contact) of tracking patients has demonstrated that the majority of ED patients can be located at 12 months.^46,47 Some studies have achieved as high as 95% follow-up at 3 months through use of multiple locators for their patients;⁴⁸ others report use of telephone cards and innovative compensation methods to increase follow-up.^46,49 These techniques must be replicated and further developed to ensure consistent, maximal follow-up for patients enrolled in prospective observational or experimental research.

MEASUREMENT AND ANALYSES

Assessment Effects

Assessment effect continues to plague screening and intervention research. Although difficult, there is a significant need to disentangle assessment from treatment effect. One particular problem with public health research is that control groups often receive a level of care exceeding that of true “usual care,” and as a result may do better than expected, falsely attenuating differences between groups. The screening or assessment can act as a type of intervention. McCambridge and Day⁵⁰ used an experimental design and the Alcohol Use Disorders Identification Test (AUDIT) questionnaire to demonstrate that just administering the AUDIT instrument alone decreased alcohol use in college students. Daeppen et al.⁵¹ included a control group without any assessments in their study of brief alcohol intervention among injured ED patients, yet still reported no difference between alcohol consumption between study groups.

An ongoing study by D’Onofrio et al. (unpublished data) to determine the efficacy of a brief intervention performed by ED practitioners for harmful and hazardous drinkers includes the evaluation of “assessment effect.” One-half of the control group received assessments similar to the two intervention arms; the other half received no assessments and will only be contacted at 12 months.

Limiting assessments and contacts with patients needs to be balanced with ability to contact patients over time. Greater attention to details in the characteristics of the control group and its selection and quantifying the assessment exposure will assist in accurately measuring outcomes and describing magnitudes of intervention effects. More research needs to be done in the area of limiting assessment effect, due to conflicting study results regarding the minimal dose of the intervention. This issue is extremely important and has the potential to have a large economic impact. If the screening alone has adequate power for behavior change, effort and personnel could be garnered for other important patient care issues.

Accurate and reliable measures of social desirability may differ for different racial and ethnic groups.^52–58 While computer-based interviewing facilitates greater patient anonymity and less social desirability bias, some assessment methods (e.g., face-to-face) may have considerable influence on measurable outcomes in study participants that reflect a bias toward greater social desirability. This may be of particular importance when non-English study participants are interviewed in their native language.^57,58

Consenting Effects

Great variability likely occurs in the consent processes that are used across public health research conducted in the ED. The process of consenting study participants may, in and of itself, introduce bias. To our knowledge, this is an area that has never been studied in the ED. While standardizing the consent or its process among various institutions may not be feasible, publishing the specific study consent, as well as describing the consenting process, may assist other investigators in assessing its potential effects. Attempts by a consortium of ED public health investigators toward standardizing consents and consenting protocols may assist in minimizing this as a moderating effect among public health research studies.^59,60 Publishing the consent forms as part of the electronic journal should be strongly considered.

Outcome Measures

Inclusion of the most meaningful, objective, and valid outcomes should take priority when designing a study. Too often researchers include variables without adequately considering how these will be used in the analyses and whether they are necessary to investigate the primary aims of the study. In most cases less is better; accordingly, evaluation tools should be thoroughly validated before beginning the study to ensure that they do not contain motivational or intervention content (i.e., content that has the potential to create an assessment effect, as previously discussed). Standardization of outcome measures is also important, as this provides comparability across studies. The National Emergency Department HIV Testing Consortium recently published consensus-based nomenclature and definitions related to reporting of such programmatic results.⁵⁹

For alcohol-related research, outcomes other than alcohol and drug consumption may be equally important. Examples include alcohol- or drug-related injuries, number of ED visits, or contacts with legal authorities. Uniformity in outcome measurement and reporting would also help to broaden the understanding of how intervention results could be effectively and broadly applied to population health.^61–64

Threats to External Validity

Using sound study design to minimize threats to internal validity is important, and conducting larger-scale research to enhance external validity and generalizability is required before widespread dissemination. Researchers have established relationships with colleagues at other institutions, so most studies are conducted at the same clinical sites where previous studies have been performed. As such, “contamination” of the study site and contamination of the data are practical and important concerns. Some examples of contamination in this context include 1) additional education of staff related to previous studies; 2) direct involvement of staff in previous studies at the same institution; and 3) differences in care provided above the standard as a result of these. Accordingly the “negative” effects found in some studies may be accounted for, in part, by the increase in the quality of “usual care” among those patients enrolled in the control groups. While difficult, ED public health researchers might consider doing sequential studies at different clinical sites to avoid this potential threat to validity. This may also improve collaboration and public health education in the community and generate interest in these important topics in nonacademic sites.

External validity is also threatened by the unique demographic characteristics of the patient populations at certain institutions. What is successful at a tertiary care urban hospital may not work in a suburban community ED. To address this concern, multicenter studies including nonacademic institutions, should be encouraged.

Screening Instruments and Assessment Tools

Just as a universal definition of what is being studied helps in interpretation of study design, agreement on common screening and assessment tools could improve comparability of public health intervention studies.^59,61–64 The wide variety of topics and populations studied by ED-based public health research makes the task of choosing standardized screening and assessment instruments challenging. Still, examination of the problems encountered in screening, brief intervention, and referral to treatment (SBIRT) studies can offer some instructive lessons for future researchers.

The SBIRT studies to date have used a wide range of screening and assessment instruments. While the AUDIT is the most frequently used screening instrument, there are many others.^{52–54,65,66} Several studies use a prescreen or brief screen to identify study participants who may meet inclusion criteria for further screening or assessment. As already mentioned, the screening alone can act as an intervention, thus modifying the results.⁵⁰

Choosing the “best-practice” screening approach can be challenging. Various screening tools have variable sensitivities and specificities and are influenced by such things as culture and sex.^52–54,66 The use of computerized screens could theoretically assist in this, as they could be preprogrammed to choose the most appropriate screen in that individual based on a few preliminary epidemiologic questions such as sex, race/ethnicity, or age.^{55,56,67–71}

SUMMARY

Several common themes resulted from the discussions related to study designs and program evaluation conducted as part of the 2009 Academic Emergency Medicine consensus conference, and several recommendations were formulated to improve the quality and translation of public health research in the ED (Table 1). Public health investigators are poised to make substantial contributions to this important area of research. This will only be accomplished, however, by employing sound research methodology.

Table 1.

Study Dsign and Program Evaluation Recommendations for Future ED-based Public Health Research

Include multiple centers and maximize use of electronic medical records to increase the validity and generalizability of surveillance, screening, and other public health programs. Develop approaches to maximize participation of groups of patients who are generally not likely to participate in screening and intervention research. Develop innovative methods of improving study subject retention and disseminate details related to follow-up methodologies. Develop simple and reliable protocols and procedures to minimize assessment effects in the context of needed initial screening. Outcome measurements should be meaningful, objective, and valid and when possible should be standardized within content or topic area. Standardize screening, assessment, and outcome measurement tools and nomenclature. Explore alternatives to screening, brief intervention, and referral to treatment (SBIRT) and brief negotiated interventions. Educate researchers about the importance of techniques to monitor and ensure treatment fidelity within studies. Train public health and emergency medicine investigators in formal research methods. Establish an emergency medicine public health research consortium or content-specific consortia.