Research Methods in Healthcare Epidemiology and Antimicrobial Stewardship: Randomized Controlled Trials

Abstract

Randomized controlled trials (RCT) produce the strongest level of clinical evidence when comparing interventions. RCTs are technically difficult, costly, and require specific considerations including the use of patient- and cluster-level randomization and outcome selection. In this methods paper, we focus on key considerations for RCT methods in healthcare epidemiology and antimicrobial stewardship (HE&AS) research, including the need for cluster randomization, conduct at multiple sites, behavior modification interventions, and difficulty with identifying appropriate outcomes. We review key RCTs in HE&AS with a focus on advantages and disadvantages of methods used. A checklist is provided to aid in the development of RCTs in HE&AS.

BACKGROUND

Randomized controlled trials (RCTs) are the gold standard of research methods in clinical scientific disciplines.¹ These trials attempt to establish causality between an intervention and outcome. Although they are the best clinical design to reduce bias and confounding, RCTs are laborious and expensive.

In a traditional RCT, patient-level randomization occurs when participants are assigned to an intervention group or a control group. In healthcare epidemiology and antimicrobial stewardship (HE&AS) research, cluster randomization, in which groups such as clinics or hospitals are randomized, is often necessary. With either method, the randomization step is the key step to decreasing bias and better establishing causality. General information on RCTs is available from texts and reviews,^2–4 Consolidated Standards of Reporting Trials (CONSORT) statements,⁵ and a Journal of the American Medical Association Users’ Guide to the Medical Literature.⁶ In this paper, we focus on important and unique aspects of RCTs in HE&AS.

ADVANTAGES AND DISADVANTAGES

Randomized controlled trials have several methodological advantages (Table 1). By clearly identifying the study population, randomizing participants or groups to reduce bias, and conducting well-accepted statistical analyses, RCTs can optimally compare 2 or more approaches. Despite the numerous advantages to RCTs, several notable disadvantages are particularly relevant to HE&AS research. For example, RCTs are time and resource intensive and require extensive regulatory oversight.⁷ Finally, a RCT may not be necessary for interventions with dramatic and/or rapid effects, such as the central-line–associated bloodstream infection (CLABSI) bundle⁸ or may not be ethical if the variable is the indisputable “standard of care” (eg, hand hygiene compared to no hand hygiene).

TABLE 1.

Advantages, Disadvantages, and Potential Pitfalls of Using Randomized Controlled Trials (RCT) in Healthcare Epidemiology and Antimicrobial Stewardship (HE&AS) Research

Advantages	Notes
Gold standard for determining causality	Leads to highest level of evidence in standard grading schemes, though individual trials typically need to be reproduced
Randomization is the best method to reduce bias and confounding	Blinding further reduces potential bias Cluster randomization improves feasibility of many HE&AS interventions
Study population can be clearly identified and described	Population may be limited to healthier individuals and not generalizable
Generally accepted statistical approaches	Intention to treat analyses are well accepted and essential for causal inference
Disadvantages	Notes
Challenging to conduct	Often performed by larger, more experienced groups
Time and resource intensive	Typically require large grant or funding source
May not be required to set standard of care for all interventions	Many practices become standard without strong evidence
May be unethical for certain interventions or comparator groups	Simple interventions in the infection control setting may not be ethical to test
Potential Pitfalls	Tips and Solutions
Outcomes may not be clinically relevant	Focus on definite outcomes over surrogate outcomes Choose outcomes relevant to patients
Completeness of data (attrition bias important for ITT)	Determine realistic follow-up plan
Changes in protocol after study initiation (or lack of documentation of protocol)	Finalize protocol prior to initiation Review CONSORT guidelines to know what will be “expected” Upload documents to clinicaltrials.gov
Insufficient power	Select relatively common outcomes Perform power calculations
Problems implementing intervention	Perform pilot study to understand intervention and develop tools for implementation
Problems with data collection	Identify data sources prior to initiating study. Perform pilot study or pilot data collection Choose outcomes that are easy to collect
Potential statistical complexity	Include a biostatistician on the research team
Subject to standard research concerns
Confounding	Need effective randomization
Applicability	Patient selection important
Generalizability/external validity	Trial conditions do not often reflect real world. Focus is on setting of study. Provides data on feasibility and cost of intervention being studied.

PITFALLS AND TIPS

Some pitfalls in RCTs can be limited through careful planning and documentation of plans prior to initiation. Pilot studies are often employed to both estimate the potential effect of an intervention and to develop tools for implementation. Determining the appropriate outcome is essential in a RCT. Outcomes must be sufficiently common to detect a difference between experimental and control groups, easy and objective to ascertain, and relevant to patients and clinicians. These demands are often in opposition. For example, a RCT would need to enroll 21,000 surgical patients for adequate power to detect a 50% decrease from a baseline surgical site infection (SSI) rate of 2 per 100 procedures. Instead, healthcare epidemiology researchers could choose common outcomes that can be automated, such as hospital-onset bacteremia, instead of rare and/or labor intensive outcomes, such as NHSN-defined CLABSIs. The choice of outcomes in antimicrobial stewardship research is particularly challenging; thus, novel metrics have recently been proposed.⁹

Internal validity is essential in determining outcomes, but the best trials achieve both internal and external validity (or generalizability).² Patient-level RCTs often use patients with fewer comorbidities because of either specific inclusion/exclusion criteria or requirement for informed consent (as well as the difficulty of using legally authorized representatives). The outcomes of these patients might not be generalizable to the overall population of patients with the condition or exposure being studied. Healthcare epidemiology studies using cluster randomization, especially randomization that includes all patients in a unit or hospital, overcome some issues of generalizability.

Many interventions in healthcare epidemiology rely on behavioral changes, eg, use of gowns and gloves or hand hygiene. Behavioral change is difficult to study. Thus, the feasibility of making even small changes to clinical behavior and being able to demonstrate its efficacy in a clinical trial is important to consider, particularly if the standard practice is relatively well followed.

The use of a standard protocol and analysis plan are critical for RCTs. Because of the strength of recommendations that can come from RCTs, a high standard is placed on objective documentation of a protocol and statistical analysis plan prior to initiation of the trial (and uploaded to http://clinicaltrials.gov). Although the initial documentation requirements are burdensome, a well-constructed protocol and analysis plan developed in advance and adhered to over the course of the study provide the highest level of evidence and potential impact.

Statistical analysis of RCTs can be complex. A cornerstone of clinical trials is use of the “intention to treat” (ITT) analysis in which participants or clusters are analyzed according to their allocation (ie, intervention or control) regardless of adherence to the intervention. Such an approach is subject to loss of data, misclassification bias, and “lost to follow-up” (attrition bias). Cluster randomization leads to additional statistical complexities in power calculation and analyses such as the need to account for intracluster correlation and lack of independence among individuals within a cluster.¹⁰ Nevertheless, ITT maintains study randomization and therefore is essential for causal inference. In contrast, the “per-protocol” (PP) analysis, which involves only participants or clusters with documented completion of the study protocol, may help determine the attributable impact of an intervention, but PP analysis is subject to loss of power and selection bias.

While the design of RCTs reduces the risk of bias and confounding, these limitations still must be considered when designing a study. If allocation of the intervention is not random, then the benefit of an RCT is lost. Likewise, many forms of bias, including inadequate implementation of the intervention or inadequate collection of outcome data, generally bias RCTs toward the null hypothesis. Using a crossover design in a cluster trial limits the problems of clustering. Likewise, for a cluster trial of a “minimal risk” intervention, obtaining a waiver of informed consent is important for feasibility and generalizability.

PUBLISHED RCTS IN HEALTHCARE EPIDEMIOLOGY AND ANTIMICROBIAL STEWARDSHIP

Trials using patient-level randomization generally involve interventions aimed at improving outcomes in the same patient receiving the intervention. When the primary outcome is dependent on an event in an individual participant that can be modified through intervention on that patient, patient-level randomization is appropriate. For example, patient-level trials have evaluated masks to prevent influenza in healthcare workers,¹¹ chlorhexidine/alcohol surgical scrub for SSI,¹² a chlorhexidine sponge for CLABSI,¹³ and antimicrobial catheters for catheter-associated urinary tract infection (CAUTI).¹⁴ Others have explored copper surfaces or a bundle of interventions in long-term care.^15,16

Trials using cluster-level randomization are more common in HE&AS research. Most interventions in these trials are designed to prevent infection and transmission, and the patient-level assumption that one participant’s outcome is not influenced by another participant’s outcome is violated. Several RCTs have used intensive care unit (ICU) as the unit of randomization. For example, 4 trials explored the use of chlorhexidine bathing in ICUs to decrease the incidence of ICU-related infections;^17–20 2 trials evaluated strategies for screening and isolation of patients with multidrug-resistant pathogens;^21,22 and 1 trial evaluated the use of universal contact precautions to decrease the transmission of methicillin-resistant Staphylococcus aureus (MRSA) and vancomycin-resistant Enterococcus (VRE).²³ In 2 trials, individual nursing home was the unit of randomization.^24,25 In 2 other trials, outpatient clinic was the unit of randomization used to evaluate stewardship interventions.^26,27 In 2 studies, hospital ward (ie, ICU or and acute care ward) was the unit of randomization used to evaluate the impact of hand hygiene improvements²⁸ and hydrogen peroxide vapor decontamination.²⁹ Finally, in 1 trial, the entire hospital was randomized to evaluate multi-dimensional infection-prevention interventions in resource-limited settings.³⁰

All trials have limitations, and the RCTs cited above are not exceptions. In fact, most large RCTs have included accompanying editorials and letters to the editor outlining these limitations. In addition, most authors document the limitations they observed during the conduct and analysis of the trial being reported. Limitations of HE&AS RCTs can be loosely placed into 3 groups: protocol adherence, outcomes, and generalizability.

Protocol adherence is important because high protocol adherence strengthens the ability to demonstrate a cause–effect relationship between the intervention and the outcome. Protocol adherence, however, can be difficult to measure, depending on the intervention. Several recently published studies failed to monitor protocol adherence.^11,20,21 The RCT by Milstone et al¹⁹ underscores the importance of monitoring adherence. In this trial, chlorhexidine bathing did not lead to a reduction in ICU-related infections in the ITT analysis. Protocol adherence was 64%, however, because 2 participating units required documentation of informed consent and had low participation. Ultimately, the PP analysis demonstrated a significant reduction in ICU-related infections following chlorhexidine bathing.

Outcome limitations are related to inadequate statistical power, ascertainment bias, definitions, plausibility, comparator groups, and multiple interventions performed at the same time. Studies that focus on infrequent outcomes have to enroll many patients to have sufficient power to demonstrate a difference. For example, 2 recently published “negative” studies had limited power related to the low frequency of outcomes.^20,24 Additionally, outcomes must be measured at points that capture the impact of the intervention. For example, Bellini et al²⁴ found that a MRSA decolonization regimen was ineffective 12 months after the intervention but conceivably could have been effective earlier.

HE&AS RCTs often take advantage of previously established data collection strategies, such as surveillance data by infection preventionists using NHSN definitions. While this approach simplifies data collection, it potentially introduces “surveillance bias,” a form of ascertainment bias, to the study.³¹ This point is underscored by the difference between rates of SSI reported to NHSN. For example, rates of colon surgery SSIs are <5% according to NHSN surveillance, but when individual patients are tracked, this rate is >10%.³² Other RCTs have used “proxy” outcomes or definitions to maximize statistical power or to improve measurability of outcomes. Harris et al²³ used rates of acquisition of MRSA or VRE in their RCTs of universal gloves and gowns in ICUs because they had insufficient power to identify differences in rates of infection. Pickard et al¹⁴ used antibiotic therapy as a proxy for CAUTI to improve identification and measurability of outcomes. Finally, some RCTs have used groups of outcomes (ie, any bloodstream infection), but some specific components of these outcome groupings may be more or less relevant (eg, coagulase-negative Staphylococci).^17,18

RCTs can be limited by differences between the experimental and control groups that the randomization process did not or could not adjust for. For example, Darouiche et al¹² studied the impact of CHG-alcohol preoperative skin preparation against povidone-iodine for the reduction of SSI. Because alcohol is a highly effective skin antiseptic, it is difficult to determine whether the improved rate of SSI observed with CHG-alcohol was related to CHG, alcohol, or both. Randomization is not perfect; thus, other studies have unintentional differences among study groups despite appropriate randomization. Some recent studies had high baseline rates in clusters randomized to the intervention;^18,23,33 thus, it is conceivable that at least a portion of the changes observed in these groups was related to regression to the mean.

RCTs in HE&AS interventions have commonly used multiple simultaneous interventions.^{15,18,21,25,26} This approach precludes interpretation of the impact of individual components of the intervention. For example, Huang et al. demonstrated that universal CHG bathing and mupirocin decolonization in the ICU leads to lower rates of bloodstream infection.¹⁸ It is unclear, however, whether the impact was primarily due to a single component of the intervention or only to the combination of components.

Finally, generalizability is important to consider for all trials. Limitations related to generalizability are related to patient population and intervention. Some RCTs are performed in settings that are not representative of other practice settings. For example, stewardship interventions that rely on electronic reminders may be difficult to replicate on a different electronic health record.²⁶ Baseline epidemiology of infection is important to consider when evaluating an RCT. For example, studies to reduce SSIs performed in settings where MRSA is not prevalent are difficult to generalize to settings where MRSA is the leading cause of SSI.³³ Some interventions may be performed in specific patient populations (eg, postoperative patients); therefore, it is difficult to determine whether the intervention (eg, antimicrobial urinary catheter) will be effective in other patient groups.¹⁴ Finally, some interventions that require significant time or capital outlay are logistically challenging to execute outside of controlled study settings.^16,29

MAJOR TAKE-HOME POINTS

Researchers embarking on an RCT in healthcare epidemiology or antimicrobial stewardship must consider multiple issues (Table 2, outlined in a checklist). All RCTs require careful planning prior to initiation. This pre-study planning must include the development of a study protocol with explicit study questions, randomization scheme, patient/cluster-unit randomization strategies, hypotheses, and outcomes. A biostatistician should compute power calculations and assist with protocol development. CONSORT statements for clinical trials should be reviewed and referenced during protocol development.^5,34

TABLE 2.

Checklist of Key Considerations When Developing a Randomized Controlled Trial to Evaluate an Intervention in Healthcare Epidemiology or Antimicrobial Stewardship

Question	Notes
1. What is the research question you hope to answer?	Establish the study hypothesis with primary and secondary outcomes prior to initiation of the study.
2. What is the intervention to be evaluated and is it clinically relevant?	Clearly define the specific intervention to be given. Determine how the intervention will be delivered.
3. How will the study be conducted?	Develop, use, and update a study protocol. Clearly outline how participants will be randomized. Review CONSORT guidelines prior to development of study protocol to guide protocol development. The protocol should include basic components such as study population, definitions for interventions and outcomes, inclusion and exclusion criteria, statistical analysis plan, and timetable.
4. Is the study ethical and safe?	Obtain IRB approval prior to initiation of the study. If examining cluster-level interventions, consider requesting IRB approval of waiver of informed consent.
5. What type of participants or groups will be studied?	Clearly define and identify the type of individual subjects (for subject-level randomization) or groups (for cluster-level randomization) that will be randomized. Specify inclusion and exclusion criteria.
6. How many participants/clusters will be required?	Perform power calculations. Power calculations for most RCTs will need to be performed or validated by a statistician.
7. What outcomes will be evaluated?	Determine primary and secondary outcomes prior to initiation of the study. Try to evaluate outcomes that are meaningful, generalizable, and reflect true improvement in care.
8. How to ensure that the intervention does not cause more harm than good?	Measure all outcomes including adverse events (e.g., mortality, length of stay, etc.)
9. How will the results be analyzed?	Develop a statistical analysis plan prior to collection of outcomes data. We strongly recommend involving a qualified statistician in all phases of the study.
10. Will the study have external validity?	Register the study in ClinicalTrials.gov (https://clinicaltrials.gov). Many journals and grant agencies require trials to be registered. Choose a study population with broad generalizability.
11. How will the results be reported?	Follow the recommendations from CONSORT.

CONCLUSIONS

RCTs in healthcare epidemiology and antimicrobial stewardship research are essential to advancing these fields and generating evidence-based changes in clinical practice. Unique challenges, including the need for cluster randomization, conduct at multiple sites, behavior modification interventions, and difficulty with identifying appropriate outcomes, make RCTs in these fields technically difficult and costly. Nevertheless, the rigorously conducted RCTs outlined above have helped move the field forward in recent years. The specific challenges of RCTs in healthcare epidemiology and antimicrobial stewardship research outlined in this report should be addressed in the design of future studies.