Table 1.

Advantages, disadvantages, and important pitfalls in using quasi-experimental designs in healthcare epidemiology research.

Advantages	Notes
Less expensive and time consuming than RCTs or Cluster Randomized Trials	Do not need to randomize groups
Pragmatic	Include patients that are often excluded in RCTs, tests effectiveness more than efficacy, may have good external validity
Can retrospectively analyze policy changes	Even if policy implementation is out of your control
Meets some requirements of causality	Quasi-experimental studies meet some requirements for causality including temporality, strength of association and dose response2
Designs can be strengthened with control groups, multiple measures over time and cross-overs	Not gold standard to establish causation but can be next level below RCT if well-designed
Disadvantages	Notes
Retrospective data is often incomplete or difficult to obtain	Need processes to assess availability, accuracy and completeness during baseline phase before implementation
Not randomized	Nonrandomized designs tend to overestimate effect size3 Does not meet all requirements to determine causality Lack of internal validity
Potential pitfalls	Notes
Selection Bias	When group receiving the intervention differs from the baseline group.2
Maturation Bias	Maturation bias can occur when natural changes over the passage of time may influence the study outcome.1 Examples include seasonality, fatigue, aging, maturity or boredom.2
Hawthorne Effect	Could bias quasi-experimental studies in which baseline rates are collected retrospectively and intervention rates are collected prospectively, because the intervention group could be more likely to improve when they are aware of being observed.3
Historical Bias	Historical bias is a threat when other events occur during the study period that may have an effect on the outcome.2
Regression to the Mean	Regression to the mean is a statistical phenomenon in which extreme measures tend to naturally revert back to normal.2
Instrumentation Bias	Instrumentation bias occurs when a measuring instrument changes over time (e.g. improved sensitivity of laboratory tests) or when data are collected differently before and after an intervention.2
Ascertainment Bias	Systematic error or deviation in the identification or measurement of outcomes.
Reporting Bias	Reporting bias is especially prevalent in retrospective quasi-experimental studies, in which researchers only publish quasi-experimental studies with positive findings and do not publish null or negative findings.
Need advanced statistical analysis when using more complex designs	With time series designs, should use interrupted time series analysis, not just single measurements before and after a response to an outbreak. Should account for intracluster correlation in power calculations

Note: RCT, randomized controlled trial.