Greater Knowledge and Appreciation of Commonly Used Research Study Designs

Abstract

In this article, we provide an overview of the different study designs commonly utilized in carrying out clinical and public health research and of the points to consider in reviewing these study designs. The design and conduct of cross-sectional health surveys, case-control, prospective, and case-crossover observational studies, and randomized controlled trials, are discussed in this review article. Careful attention to the concerns we have raised will hopefully lead to the design and conduct of high quality research projects and their write-up.

In this article we provide an overview of several of the basic study design approaches commonly utilized in contemporary clinical and public health research. These include the cross-sectional, case-control, and prospective studies, randomized controlled trials, and an important, but less appreciated observational study design, the case-crossover approach. We provide schematic representations of several of these study designs and a checklist describing the features of each study approach and points to consider in utilizing these designs. This overview is not meant to be exhaustive. For more extensive discussions about these study designs, there are several excellent textbooks that deal with the specific details and philosophies relevant to the design and conduct of clinical and epidemiologic investigations, including the more advanced study designs that will not be covered in this overview, such as propensity score matching and time series analyses^1–11. We do hope, however, that this overview will lead to a greater understanding of several commonly utilized study designs, the strengths and limitations of these designs, and the types of biases that might be encountered in carrying out investigations using these design approaches.

Cross-Sectional Studies

The cross-sectional study, also referred to as a prevalence (Kodak snapshot) study, creates considerable difficulties in interpreting any temporal or etiologic associations that may exist between an exposure or risk factor and a health outcome due to its cross-sectional “slice in time” perspective. This study design, however, can be used to generate hypotheses and ideas for further research using more rigorous study designs that are described subsequently.

In a cross-sectional study, prevalent (old and new) cases of disease are identified over an a priori selected time period and location. Exposure to a single risk factor, or set of risk factors, is ascertained and their frequency described. This type of study design is commonly employed in carrying out cross-sectional health surveys of the general population, as is done periodically in the nationally representative National Health Examination Survey, or of special population groups, such as health questionnaire surveys of the elderly or adolescents. Through secondary analysis of the data collected in these surveys, associations between various risk factors and disease occurrence can be broadly estimated. This study design is prone, however, to a number of potential biases, including information, response, and selection bias, and confounding by time and other covariates. These concerns are important to consider in using this design since they make it difficult to derive meaningful inferences from this type of investigation and one needs to be very careful in the interpretation of any findings thereof.

The temporality of the main observational study designs used in clinical and epidemiological research is shown in Figure 1. Cross-sectional studies do not establish the directionality of any associations observed between exposure factors and health outcomes due to blurring of the disease-risk factor relationship. In contrast, other observational studies that are discussed subsequently are able to establish the directionality of associations to a far greater extent.

Figure 1.

Temporal Direction of Different Observational Study Designs

Observational Studies

Prior to discussing the 2 major types of observational studies, it is important to understand how epidemiologists view the world of clinical and public health research, namely through a 2 x 2 lens of disease and its predisposing factors (Figure 2).

Figure 2.

Two by Two Representation of Key Observational Studies

In a case-control study, one begins with the selection of newly diagnosed, or incident and prevalent, cases of the disease under study, a suitable comparison group is subsequently selected, and differences in the frequency of the putative risk factor(s) are assessed and compared between these 2 groups. In contrast, in a prospective study, apparently healthy individuals free from the disease under study are initially classified according to the presence, or absence, of a suspected risk or exposure factor (as well as according to dose and duration of the risk factor to the extent possible). These individuals are subsequently followed over time for the development of disease (or other health outcome) with differences in the incidence rates of disease compared in those with, as compared to those without, the risk factor under study. Proportions and odds ratios with accompanying confidence intervals are calculated in a case-control study whereas the relative risk (with confidence intervals around this point estimate) is used to quantitatively measure the extent of harm or benefit associated with a risk factor in a prospective epidemiologic investigation. The relative risk is typically not able to be calculated in a case-control study since the prevalence of the outcome under study is determined by the study design and not by the characteristics of the population under study^12,13.

Case-Control (Retrospective) Studies

The essence of this design is the comparison of 2 distinct population groups. This includes the initial selection of a sample of individuals with the disease or health outcome of interest (cases) and a comparison group (controls) of those without the disease under study or a related clinical condition. For example, women with newly diagnosed (incident) cervical cancer (cases) would be initially identified and compared to women without cervical cancer (controls). Differences in the frequency of exposure to risk factor(s) of interest, such as their sexual or infectious disease history, in cases as compared with controls would be examined. The odds of being exposed to the risk factor(s) in cases with cervical cancer as compared with controls would be compared. An odds ratio > 1 implies that cases are more likely to be exposed to the putative risk factor than controls (e.g., exposure to Human Papilloma Virus) whereas an odds ratio less than 1 suggests that the exposure under study may in fact be protective (e.g., Human Papilloma Virus vaccination). The basic architecture of this study design is shown in Figure 3.

Figure 3.

Design, Conduct, and Analysis of Case-Control Studies

In designing and critically reviewing the results of case-control studies, particular attention needs to be devoted to selection of the case and control groups. One should ideally study incident cases of disease and a sample of persons without the disease under study from either the same hospital from which the cases were diagnosed, or of healthy non-diseased persons residing in a similar geographic area to that of the cases. The inclusion of prevalent cases of disease should be carefully considered since issues related to survival bias (selective survival of the cases under study) and recall bias may be operative; recall bias, namely the differential recall of cases as compared with controls, may be particularly problematic and difficult to overcome, especially if the cases have had the disease under study for many years and are unable to accurately recall when they were “exposed” to the key factor of interest. It is desirable to select controls that are as similar as possible to cases, with the exception of the key risk factor to be examined, who will broadly represent the frequency of the risk factor under study in the general population.

For example, in studying the association between regular coffee consumption and occurrence of acute myocardial infarction, with the hypothesis being that regular daily consumption of coffee is associated with an increased risk for acute myocardial infarction, an investigator would select persons with a first diagnosis of acute myocardial infarction and a comparison group of either hospital or neighborhood controls who have not had an acute myocardial infarction in the past. Differences in the frequency of regular coffee consumption would be assessed in these 2 comparison groups with the hypothesis being that cases of acute myocardial infarction would be more likely to be regular coffee drinkers than persons in the control group. One of the key aspects of this study would be to ensure that the control group accurately reflects the frequency of the key risk factor (e.g., regular coffee consumption) in the general population from which the cases arose.

Specific attention should be paid to measurement of the primary exposure variable as well as to the analytic methods utilized to adjust for potentially confounding factors (e.g., cigarette smoking, serum lipid levels) that may confound the primary associations under study.

Since often times case-control studies may suffer from unmeasured confounding, investigators may choose to “match” cases with controls to remove the effects of several potentially confounding factors. This might be done in lieu of, or in conjunction with, multivariable adjustment for potentially confounding variables. For example, investigators might match their case and control groups on the basis of age and sex to remove the confounding influence of these important risk factors; once successful matching has taken place, however, the investigator is no longer able to comment on the relation of age and sex (or any other matching factor) to the disease under study. Issues related to matching have been extensively discussed in epidemiology textbooks and published articles^1–7. Issues related to this study design are highlighted in Table 1.

Table 1.

Factors to Consider in the Design and Analysis of Case-Control Studies

Factor	Yes	No	Uncertain
Was the sampling method for the selection of case and control groups clear, understandable, and potentially replicable?	□	□	□
Have the case and control groups been selected without regard to the exposure factor(s) of interest?	□	□	□
Were incident (newly diagnosed) cases of disease being studied or were only more long-standing (prevalent) cases included in whom the etiology-disease association might be considered more questionable or misinterpreted?	□	□	□
Do the cases under study reflect a wide range of possible disease severity or were only select cases from a single hospital or ambulatory care clinic represented which may limit the generalizability of the study?	□	□	□
Was recall bias operating in characterizing the risk factor under study, is this bias potentially different between cases and controls, and how have the investigators dealt with this important concern both methodologically and analytically?	□	□	□
Were appropriate hospital, clinic, and/or neighborhood controls selected from the same population from which the cases were diagnosed?	□	□	□
Have a sufficient numbers of cases and controls been selected to adequately test the proposed hypotheses?	□	□	□
Have cases and controls been matched on a limited number of other relevant factors at either a group or individual level and have satisfactory matches been accomplished?	□	□	□
Has matching been taken into account in the analysis of the principal study findings?	□	□	□
Has categorization of the primary exposure factor(s) been carried out into appropriate dose and response categories and has any attempt been made to validate self-reported exposure status?	□	□	□
Were the individuals involved in either direct data collection or record abstraction blinded to the study hypotheses and/or to case-control status?	□	□	□
Were odds ratios and accompanying confidence intervals presented to quantify the disease-exposure association?	□	□	□
Have potential confounding variables been adequately considered and analytically controlled for either through the use or a priori matching or through the use of stratified or multivariable regression analyses?	□	□	□
Have the results been clearly presented with appropriate interpretation of the principal study findings?	□	□	□
Were the study strengths and limitations clearly acknowledged?	□	□	□

Longitudinal (Prospective or Cohort) Studies

In this study, healthy subjects free from disease are selected and are non-randomly categorized as to whether they were, or were not, exposed to a risk factor(s) of interest. Both groups are followed over time for purposes of examining differences in the incidence rates of the study outcome(s) in persons exposed to the risk factor of interest as compared to those without this risk factor (Figure 4). Relative risks (RR) are subsequently calculated with accompanying 95% confidence intervals. Odds ratios that are calculated in a case-control study are used as proxies for relative risks since incidence rates of disease are typically not able to be calculated in this type of investigation^12,13. A RR >1 suggests that persons with the risk factor are more likely to develop disease as compared to persons without the risk factor under investigation (e.g., Human Papilloma Virus and cervical cancer) whereas an RR < 1 suggests that the suspect risk factor may be protective (e.g., regular exercise and development of acute myocardial infarction). For example, in prospectively studying the association between regular daily coffee consumption and the development of acute myocardial infarction, the investigator would initially select individuals free from the disease under study, classify these persons into those who regularly drink coffee and those who do not, and follow these persons over time for the development of acute myocardial infarction through the use of various measurement tools.

Figure 4.

Design of Cohort Studies

In interpreting the results of this study, one needs to carefully characterize the risk factor(s) under study and its possible delineation into a dose-response relation (e.g., amount and strength of coffee regularly consumed), careful measurement of the disease (e.g., acute myocardial infarction) or other pertinent health outcome under investigation, and measurement and analytic control for various potentially confounding factors (e.g., cigarette smoking) that may also affect the risk of the disease/health outcome and may be differentially distributed between persons with and without the exposure factor under study.

There are a number of limitations/disadvantages of longitudinal studies, however. These studies typically require considerable time, money, and effort to accumulate a sufficiently large cohort to follow and have a large enough number of adverse outcomes develop to systematically examine the association between a risk factor and the development of disease. A person’s exposure status may change over time and a single assessment of exposure may be inadequate to accurately characterize one’s risk status. Losses to follow-up may be higher than expected, and there may be important confounding when patients self-select themselves into groups for further follow-up. Some of the biases and concerns that need to be considered in the design and conduct of prospective studies include information bias, potential misclassification of exposure and outcome, ascertainment bias, and losses to follow-up (Table 2).

Table 2.

Factors to Consider in the Design and Analysis of Longitudinal Studies

Factor	Yes	No	Uncertain
Have exposed and nonexposed study subjects been selected from similar at risk populations? Has the investigator presented a sufficient rationale for the choice of the study population and its potential generalizability?	□	□	□
Was exposure status been adequately measured, categorized, and independently validated?	□	□	□
Have possible changes in exposure status since the time of initial baseline classification been measured and taken into account in the analysis?	□	□	□
Have the study endpoints been determined without regard to exposure status and are the persons involved in the collection of these data unaware of the primary study hypotheses?	□	□	□
Was determination of the principal study outcome adequately measured and independently validated?	□	□	□
Have potentially confounding factors been measured and has the influence of these factors been controlled for analytically?	□	□	□
Was an acceptable means of determining subject follow-up used and was a high follow-up rate (e.g., >80%) been achieved?	□	□	□
If an acceptable follow-up rate of exposed and nonexposed cohorts was not achieved, have the characteristics of those unavailable for follow-up been compared with those remaining under follow-up to determine whether these groups are comparable?	□	□	□
Did the study have adequate power to detect differences in the principal study outcome(s) in exposed and nonexposed cohorts?	□	□	□
Were there an adequate number of individual study endpoints or was a composite endpoint created either after the study was completed or a priori?
Was the duration of follow-up sufficient?	□	□	□
Were incidence rates of disease calculated and relative risks and accompanying confidence intervals presented?	□	□	□
Were the study results clearly presented and adequately interpreted?	□	□	□
Were the study strengths and limitations clearly acknowledged?	□	□	□

Investigators may vary the time frame over which prospective studies are carried out depending on the availability of data, resources, and time needed to conduct the study. In the more typical concurrent longitudinal study, subjects are initially classified according to exposure status and followed for a period of sufficient duration to determine differences in the frequency of the principal study outcome(s) in exposed as compared with non-exposed individuals.

In a non-concurrent prospective study, individuals are classified in the past with regards to the exposure(s) of interest and followed up to the present. As configured, this study design can compress the amount of time needed for follow-up, but there may be problems with the standardized assessment of the risk factor(s) under study and losses to follow-up. In counter-distinction, the more conventional prospective cohort design allows for the careful characterization and standardized assessment of individuals with and without the risk factor of interest at the time of baseline enrollment; this design may, however, take many years of careful follow-up to collect an adequate number of cases of disease to systematically examine any postulated associations with adequate certainty and statistical power.

Nested Case-Control Study

Investigators may carry out a case-control study nested within the architecture of a prospective study for purposes of cost-efficiency and judicious use of valuable biologic samples (Figure 5). In this design, which is typically performed when the number of disease-related events accumulated in a prospective study is small, the intent is to carefully study the relatively small number of persons who have developed the health outcome of interest. These individuals are compared with the characteristics of controls drawn from the same study, matched on a limited number of other important factors, who have not developed the disease or outcome of interest at the time that the case did. Differences in the frequency of exposure to the putative risk factor(s) are then compared between these 2 groups.

Figure 5.

Design and Analysis of Cohort studies

Case-Crossover Studies

This study is used to assess the role of possible precipitating factors involved in the development of infectious or chronic diseases. In this study, cases serve as their own controls as they move in and out of hazard periods of risk. This design is utilized to evaluate the effects of brief exposures with transient effects on acute health outcomes when a traditional control group is either not readily available or is not considered necessary to address the hypothesis being tested. The primary advantage of the case-crossover design lies in its ability to help control for potential confounding. Self-matching subjects against themselves eliminates confounding between subjects from both measured and unmeasured covariates that do not change over time.

This design incorporates elements of both a case-control study and a nonexperimental crossover type experiment. The fundamental aspect of the case-crossover design is that each case serves as its own control, thereby minimizing the potential for unmeasured factors to modify observed associations between disease risk factors and health outcomes. Time-varying exposures are compared between intervals when the outcome occurred and intervals when the outcome did not occur within the same individual¹⁴. A factor may be considered to be a triggering agent if its occurrence is more likely to have taken place during the brief interval prior to the onset of disease as compared to other previous periods. Factors to consider in the use of this design are shown in Table 3.

Table 3.

Factors to Consider in the Design and Analysis of Case-Crossover Studies

Factor	Yes	No	Uncertain
Did the exposure or hazards periods of risk make reasonable biologic and/or clinical sense?	□	□	□
Have the investigators made attempts to independently validate the exposure factor(s) of interest?	□	□	□
Have an adequate number of incident cases been selected and are only persons with an initial disease event being studied?	□	□	□
Was effect modification potentially taking place in various pre-defined subgroups?	□	□	□
Was there potential for recall bias of the key exposure factor of interest?	□	□	□
Were patients interviewed at an appropriate time in their course of disease and setting to give as accurate a set of responses as possible?	□	□	□
Have differences in absolute from relative risk been clearly distinguished?	□	□	□
Were the study findings clearly presented and adequately interpreted?	□	□	□
Were the study strengths and limitations clearly acknowledged?	□	□	□

Randomized Controlled Trials

One of the major differences between the longitudinal study and the randomized controlled trial is the element of randomization. Random assignment of a sufficiently large study sample to “exposure” or treatment status and placebo or usual care attempts to ensure that measured as well as unmeasured confounding variables are similarly distributed between the trial’s respective comparison groups. This is accomplished if the randomization schema is carried out properly and the study sample size is large enough. Study staff, as well as patients, may be “blinded” to the intervention as well as outcome status of study participants, thereby diminishing the likelihood of several forms of bias taking place through use of this design.

Several factors need to be considered in the design and analysis of the “gold standard” randomized controlled trial (Table 4). There are a number of seminal articles that have been written about clinical trials in the past as well as entire journals devoted to this topic^15–19.

Table 4.

Factors to Consider in the Design and Analysis of Randomized Controlled Trials

Factor	Yes	No	Uncertain
Have sufficient data been generated from observational studies to justify conducting a randomized controlled trial?	□	□	□
Have the investigators sufficiently described the trial’s inclusion and exclusion criteria and how consideration of these criteria might affect the generalizability of the trial’s findings?	□	□	□
Was the randomization of study subjects to treatment or exposure status done appropriately?	□	□	□
Were potentially confounding factors adequately measured and were they equally distributed between experimental and control groups?	□	□	□
Was the trial single, double, or triple blinded?	□	□	□
Did the trial utilize a run-in phase which may affect the questions being asked?	□	□	□
Were the treatment regimens adequately described and are they generalizable to actual clinical practice?	□	□	□
Was adherence to the treatment under study assessed and adequately described?	□	□	□
Did an adequate placebo exist to the active therapy being tested or was the new therapeutic approach being compared with standard medical practice?	□	□	□
If competing interventions were being used as either adjunctive or standard therapy, were these interventions similarly distributed between the respective comparison groups and/or have they been controlled for analytically?	□	□	□
Were the data analyzed according to the principles of intention to treat?	□	□	□
Was the sample size large enough to adequately examine differences in the principal study outcome(s) between intervention and comparison groups?	□	□	□
Was follow-up of the respective study groups adequate and of sufficient duration?	□	□	□
Were the study strengths and limitations clearly acknowledged?	□	□	□

Conclusion

In-depth discussions of the strengths and limitations of observational studies and randomized controlled trials have been described elsewhere in textbooks and published articles. In addition to the checklists provided, there are other guidelines that have been published to consider in reviewing the validity and completeness of observational studies^20,21. These guidelines represent the consensus of a broad methodological community and have been based on accumulating evidence that has undergone cycles of iterative improvement. In contrast, our checklists might be viewed as more “ad hoc” and based on the opinion of a small group of individuals. The intent of our broad overview is to highlight areas of methodological and analytic concern that one needs to consider in conducting and critically reviewing the results of observational studies and randomized trials.

In addition to consideration of the checklists provided in the accompanying tables, which have been modified from a previous editorial on this topic²², in deciding whether to utilize the results of a particular study as pertaining to one’s clinical practice, readers should pay particular attention to the scientific, clinical, and biological rationale and importance of the question(s) under study, measurement of key predictor and outcome variables, the study’s inherent strengths and limitations, quality control measures used in the collection of data, and data analytic approaches used. These overarching themes pertain to all study designs. Careful attention to the concerns we have raised will hopefully assist investigators in planning their research and facilitate a more systematic and high-quality peer review of manuscripts that use the study designs we have discussed.