Randomized controlled trials in resuscitation

Highlights

• •Clinical trials are vital to the advancement of resuscitation science.
• •The out-of-hospital setting is a complex and uncontrolled environment. Planning clinical trials in this setting should include someone with specific expertise in this field.
• •Cardiac arrest patients are heterogeneous and require careful consideration for inclusion in clinical trials.
• •Novel trial designs and analyses may improve trial efficiency and should be considered for trials in resuscitation science.

Abstract

Randomized controlled trials (RCTs) are a gold standard in research and crucial to our understanding of resuscitation science. Many trials in resuscitation have had neutral findings, questioning which treatments are effective in cardiac resuscitation. While it is possible than many interventions do not improve patient outcomes, it is also possible that the large proportion of neutral findings are partially due to design limitations. RCTs can be challenging to implement, and require extensive resources, time, and funding. In addition, conducting RCTs in the out-of-hospital setting provides unique challenges that must be considered for a successful trial. This article will outline many important aspects of conducting trials in resuscitation in the out-of-hospital setting including patient and outcome selection, trial design, and statistical analysis.

Introduction

Randomized controlled trials (RCTs) are considered a gold standard in research and play a crucial role in advancing resuscitation science. Multiple large RCTs have shaped our current understanding of best practices in the management of out-of-hospital cardiac arrest (OHCA).1, 2, 3, 4, 5 The majority of cardiac arrest guideline recommendations, however, are based on low level evidence (e.g. observational studies, animal data) resulting in “weak” recommendations with high degree of uncertainty around the true effect of cardiac arrest interventions.6, 7

Interestingly, most clinical trials in OHCA resuscitation have produced neutral or negative results, leading to questions about what interventions are effective in cardiac resuscitation. While it may be that many cardiac arrest interventions do not improve patient outcomes (admittedly many are untested in a rigorous scientific fashion), it also begs the question as to whether there are issues in the design and/or implementation of clinical trials that are prompting these neutral findings. For instance, many of the large clinical trials from the Resuscitation Outcomes Consortium (ROC) collaborative from which we base much of our understanding of cardiac resuscitation had neutral results, including continuous chest compressions in the ROC-CCC trial,² and use of antiarrhythmic medication in the ROC-ALPS trial.¹

There are many challenges to conducting a successful RCT. They require extensive resources, have stringent regulatory requirements, and require significant time and funding to complete. RCTs require substantial oversight to ensure compliance with randomization schedules and treatment, and prolonged follow-up to collect patient-important outcomes. In addition to the generic challenges of conducting a high-quality RCT, there are many context-specific challenges related to conducting a trial in the out-of-hospital setting. It is important to maintain high scientific rigour, while also appreciating the complexities and uniqueness of this setting. The out-of-hospital environment often increases the demand for resources and oversight to ensure appropriate conduct of clinical research. First, participation in clinical research is often unfamiliar to many prehospital practitioners, with few having formal training in research methodology. Additional time and training may be required to ensure buy-in and appropriate preparation of clinicians. Second, to enrol the required number of patients in most clinical trials, it may be necessary to train multiple paramedic services which may include hundreds or thousands of paramedics in the study protocol. In addition, there are often additional providers (e.g. firefighters, medical first responders) who may be involved in patient care and require familiarization with the study. This can create logistical challenges trying to standardize the intervention across thousands of different providers from different organizations. Last, the out-of-hospital setting is an uncontrolled environment to conduct clinical research. There are a myriad of operational challenges in this environment including, control of bystanders and other personnel at the scene, patient access and extrication, and the unpredictable situations that paramedics encounter each day that are difficult to control and account for in clinical research.

This paper will outline important aspects of clinical trials in resuscitation such as patient and outcome selection, as well as some alternatives to traditional research designs that may improve the successful implementation and efficiency of resuscitation trials in the out-of-hospital setting.

Features of well-designed randomized controlled trials

Randomized controlled trials maintain their place atop the hierarchy of evidence pyramid much to their ability to control for bias. Proper patient randomization and blinding of those involved in the trial to the allocated intervention helps to ensure a balance of both observed and unobserved characteristics across comparator groups and to reduce selection bias of patients into the trial; two major sources of error in non-randomized clinical research.⁸ To ensure proper randomization and avoid patient selection bias, all members of the research team and clinicians responsible for care should be unaware of the randomization assignment until after a patient has been enrolled in the study. This is accomplished through a process called allocation concealment – where randomization occurs separately from the person recruiting patients. This is often done by using external agencies to determine the randomization schedule with computer generated random numbers.⁹

A second related concept is that of blinding. Ideally, all those involved in a clinical trial should be blinded to the treatment received by the patient. This is especially important for the treating clinician, the patient (rarely a concern in resuscitation trials), and the outcome assessors. While this fundamental concept is key to proper conduct of an RCT, pragmatically, this is often not possible in resuscitation trials. Aside from drug trials, where placebo administration is often included in the control arm, many other interventions are known to the treating clinicians (e.g. mechanical CPR devices, use of advanced airways, etc.). While blinding may not always be practical, it is still possible to protect against biases through proper study design and evaluation.¹⁰

Effectiveness vs. Efficacy trials

Clinical trials often focus on the “effectiveness” or “efficacy” of an intervention, each one requiring a different approach to trial development and implementation. Efficacy, or explanatory, trials are aimed at determining the true impact of an intervention under ideal conditions and focus on ensuring high internal validity of a study.¹¹ Effectiveness, or pragmatic trials, attempt to examine the benefit of an intervention under real-world conditions, and typically focus less on attempting to control all other aspects of the trial and instead try to maximize the generalizability of the results.¹¹ Efficacy and effectiveness exist on a continuum and are not a binary approach to clinical trials. To this extent, many resuscitation trials sit on the effectiveness (real-world implementation) end of the continuum, which has important implications for interpretation of the trial results. One of the strengths of effectiveness trials is in helping to interpret the results of the study within the context of real-world application. This helps to maximize the generalizability of the study findings. While it is important to determine the impact of an intervention in the setting in which it will be utilized, pragmatic trials are at risk of missing the effect of an intervention by incorporating the complexity and unpredictability of real-world settings.

Selecting the right patient population

One of the difficulties in resuscitation trials is attempting to select the right patient population. Cardiac arrest patients are heterogeneous with many underlying etiology, comorbidities, and treatment-related factors that make each resuscitation unique. For example, patients who suffer a sudden cardiac arrest due to a cardiac etiology, such as coronary artery disease, are significantly different than patients who have a cardiac arrest secondary to opioid toxicity, or to patients who have a cardiac arrest at the end of life after cancer or other palliative disease. Yet despite these differences, we often combine these patients into a single cohort during cardiac arrest research and expect them all to respond the same to our study intervention. This attempt to generalize research findings across an entire population of cardiac arrest patients may in fact hinder progress in clinical trials by minimizing the potential impact of trial interventions. Consideration of ways to be more selective with patient enrolment in resuscitation trials may help to identify the true effect of different interventions and reduce the probability of missing important findings.

There are a large proportion of cardiac arrest patients who are unlikely to benefit from many cardiac arrest interventions (e.g., injury severity is too severe, or too mild). In general, patient outcomes from cardiac arrest are poor, and there are subgroups of patients whose survival is approaching futility (<1% chance of survival). For example, patients with unwitnessed cardiac arrest with asystole as the presenting rhythm have less than 1% chance of survival to hospital discharge.¹² Including patients in a clinical trial who are unlikely to benefit from the intervention will reduce the overall effect of the intervention, increasing sample size requirements and/or increasing the probability of a null finding. In contrary, the incidence of OHCA is relatively rare, so including as many patients as possible is important when trying to enroll enough patients to meet a required sample size. This creates an important trade-off between enrolling patients who are likely to benefit from the intervention, but also ensuring there are enough patients to meet pre-determined sample size estimations.

There is a significant body of work that has examined prognostic factors for intra-arrest and post-arrest patients, including a number of risk stratification scores that may help to identify subgroups of patients for enrolment in clinical trials. Many risk stratification scores use easily accessible, commonly collected data points from standard clinical care. The out-of-hospital cardiac arrest (OHCA) score is the most widely studied risk stratification score for cardiac arrest patients. It uses initial cardiac rhythm, no flow interval, low flow interval, serum creatinine, arterial lactate and has been shown to have good predictive performance in comatose post-cardiac arrest patients, with an area under the curve (AUC) of 0.82 on derivation.¹³ There are a number of other scores that use similar variables that have all shown good to excellent predictive performance.14, 15 These risk stratification scores may be used to better select patients for enrolment in RCTs or provide post-trial subgroup analyses based on injury severity, or risk stratification. While these specific risk stratification scores cannot be used for the intra-arrest period there are many factors that could be used to properly select patients such as initial cardiac rhythm, witness status, and EMS response time. There may be a combination of variables that can be utilized in the prehospital setting, or for intra-arrest trials that could help to identify appropriate patients as well.¹⁶

Choosing important outcomes

Important outcomes in resuscitation science are a moving target. What defines an important trial outcome has evolved over time from return of spontaneous circulation to survival to hospital discharge, to neurological outcome at hospital discharge, to long-term quality of life outcomes.¹⁷ While improvement in long-term cognitive function and quality of life are no doubt the ultimate goals of any resuscitation, it may not always be the most practical outcome to study in resuscitation research. While some argue these are the only outcomes that matter, it may not be realistic to suggest that all important interventions applied by paramedics, or pre-paramedic arrival must result in neurological outcome post-discharge from hospital. There are many aspects of care, both prehospital, in-hospital, and post-discharge that must seamlessly happen to ensure positive long-term outcomes for patients who suffer a cardiac arrest.¹⁸ Focusing solely on long-term neurological survival may overlook small but important improvements in upstream outcomes such as return of spontaneous circulation (ROSC) and survival to hospital admission. The impact of a prehospital intervention is likely best judged by rates of ROSC and survival to hospital admission. Once in-hospital, there is considerable variation of in-hospital care that can confound the results of out-of-hospital interventions. In-hospital interventions such as targeted temperature management (TTM)18, 19 and percutaneous coronary intervention (PCI) are associated with improved patient outcomes18, 20 and have been shown to vary substantially between institutions.21, 22 Additionally it has been shown that there are substantial differences in outcomes across hospitals for patients admitted after OHCA.22, 23

Planning a study to improve long-term cognitive function requires a large number of patients and extended follow-up time which becomes susceptible to loss of patients over time, impacting the feasibility of the research. In addition, many trials in resuscitation over-estimate the potential effect size of the intervention resulting in studies that are under-powered to detect small but important differences in outcomes; this is especially true when using mortality as an outcome. This bias towards the null has been shown to occur in other areas of critical illness research as well, with trials over-estimating control group mortality and estimating unrealistic treatment effects.²⁴ This results in trials that have non-statistically significant, yet clinically relevant, differences in primary outcomes interpreted as no difference with the intervention.

Utilization of composite outcomes can help to increase power by increasing the number of “events” that occur in the study. This type of approach is common in cardiovascular research with use of composite outcomes such as major adverse cardiovascular events, or MACE.²⁵ When used appropriately, composite outcomes can increase the statistical efficiency of a trial, reduce the required sample size, trials are less costly, and the results of promising new treatments may be available earlier. Composite outcomes, however, are often hard to interpret. Each component endpoint in the composite score should be of equal importance to patients and practitioners and occur with similar frequency. Additionally, the intervention may impact different individual components differently (e.g. reduce biomarker levels, but increase death).26, 27

Internal pilot studies

Many researchers conduct pilot studies to ensure feasibility of the trial prior to implementation of a full scale RCT. This step can be essential to ensuring proper implementation and trial success. One option that is rarely considered is to conduct an internal pilot study. Internal pilot studies improve the efficiency of randomized trials by aiming to include patients from the pilot study as part of the full RCT. By including these patients, internal pilots enhance the efficiency of RCTs, prevent waste of valuable resources, and avoid recruitment of additional participants into a trial.28, 29 Conducting a pilot also allows for minor changes to be made to the trial protocol prior to continuing recruitment for the full RCT without impacting the validity of the trial as long as the changes are not substantial. The decision to move forward from the pilot study to the full RCT and to include the participants from the pilot study is then made based on the recruitment rate from each site, the feasibility of paramedics to adhere to the study protocol, and whether or not any changes to the protocol were required based on the initial recruitment. The pilot study also allows an opportunity to re-evaluate pre-determined sample size estimations using baseline survival from the control group, and to evaluate the data linkage strategy to obtain patient outcomes. This is an attractive option for prehospital resuscitation research as there are often limitations in patient enrolment, and timelines for recruitment of patients so including early patients in the pilot study can ensure efficiency of recruited patients. In addition, ensuring all patients enrolled in a trial are able to contribute data to the final results adheres to the ethical principles of clinical research.

Alternative research designs

Patient-level randomization with blinding of clinicians, researchers, and where applicable the patients, continues to be the gold standard for randomized trials, however, this may be difficult to achieve in the out-of-hospital setting. Difficulty in adhering to a study protocol can lead to poor patient enrolment, variability in application of the intervention (or non-compliance), cross-over (or contamination) of patients from one group to the other, or loss of follow-up. Each of these may bias the results of a clinical trial. Alternative trial designs such as cluster, stepped-wedge, and factorial trials may be used to help with patient enrolment and study adherence, while improve the efficiency of clinical trials in the out-of-hospital setting.

Cluster randomized trials are an attractive alternative to traditional parallel group randomized trials where randomization occurs at a unit rather than the individual patient. In some situations, patient-level randomization may not be practical or feasible, and specifically in the out-of-hospital setting, it may be more difficult to enrol patients at the individual level due to limited resources and added complexities in the clinical setting. Incomplete or non-compliance with randomization can significantly impact trial results, causing group imbalances or reducing the difference in effect between trial groups. In cluster randomization, the level of randomization occurs within a specific unit where all patients within that unit receive identical treatment. For trials in the prehospital setting, this unit of randomization can be at the service level, base level, or ambulance level. These natural units can improve trial compliance and reduce contamination by helping to reduce cognitive workload on practitioners in these complex settings.30, 31 The recently published DOSE VF trial was a cluster randomized trial comparing alternative defibrillation strategies for refractory ventricular fibrillation.²⁹ Randomization occurred at the level of paramedic service, where each service involved was randomized to the different arms of the study at 6-month intervals. The overall compliance with protocol was exceptionally high in this trial at approximately 90% with very little cross-over between groups. While cluster randomization can be an attractive alternative design, it does have some drawbacks compared to traditional individual-level randomization. Cluster RCTs can be prone to non-compliance, especially around periods of crossover and require careful planning and monitoring to ensure seamless crossover at the service level.³² Cluster RCTs can also reduce efficiency by having similarities within clusters and has the potential for imbalances in baseline factors to occur. Including a cross-over as part of the cluster RCT design can help to reduce the potential for imbalances to occur.30, 31

Stepped-wedge trials are a special design of cluster randomized controlled trials. In a stepped-wedge trial, all ‘clusters’ start in the control phase of the trial. Clusters are then randomized to crossover to the intervention arm at different, evenly spaced, time intervals (e.g. every 6 months) until all clusters are in the intervention arm.33, 34 Stepped-wedge trials can be a useful design to evaluate the implementation of an intervention into practice, especially in situations where it would be difficult to then remove the intervention. This design allows for a gradual rollout of the intervention, making it logistically feasible for implementation in real-world settings. This can be especially useful when the intervention requires time for training, adaptation, or infrastructure development.33, 34 Scales et al. (2016) used a stepped-wedge design to evaluate the implementation of a multifaceted quality intervention to improve neuroprognostication in the ICU.³⁵ Clusters were set at the individual ICU level and sites were randomized to switch to the intervention arm at 5-month intervals and were able to show improved rates of appropriate neuroprognostication in participating sites with the intervention.³⁵ While a practical implementation design, stepped-wedge trials can also be challenging. Stepped-wedge trials can be time-consuming and resource-intensive, as they involve multiple time periods and data collection points. Some clusters may have to wait for an extended period before receiving the intervention. This delay can be a source of frustration and may affect the willingness of participants to remain in the study.

Factorial designs allow researchers to investigate multiple interventions at the same time, as well as the interaction of the effect between different interventions. The standard for factorial designs is two interventions, each with two levels or a 2 × 2 factorial design, where patients can be randomized to receive either intervention, no intervention, or both interventions.36, 37 The importance of this design is that it allows evaluation of multiple therapies at once, eliminating the confounding associated with running multiple trials at the same time, or inefficiencies and delays in conducting trials in parallel. Factorial designs will also enable the trial to be completed with smaller sample size requirements then two independent parallel group studies, assuming that there is no interaction between the different interventions being studied.36, 38 This type of study design is likely underutilized and could provide significant benefit to resuscitation trials. There are a few recent well conducted trials in resuscitation science that have used a factorial design including COMACARE39, 40 and BOX trials.41, 42 The BOX trial was a 2x2 factorial design examining both oxygenation and blood pressure targets in comatose post-cardiac arrest patients. In this study the authors did not adjust for the factorial design as they did not expect an interaction between the two interventions. While more efficient, factorial design RCTs are increasingly complex and logistically challenging with the implementation of multiple interventions within the same trial. As well there can be challenges in interpretation if an interaction is observed between the two interventions.43, 44

Alternative analytical plans

Too often in clinical research, the results of years of work in developing and conducting a clinical trial boil down into a single p-value to indicate whether an intervention is positive or neutral; sometimes even going as far as to say that if a p-value is 0.04 we should accept the intervention as beneficial, however if the p-value is 0.06 the intervention is unhelpful. However, there are many known problems with use and interpretation of p-values.45, 46 They are often a reflection of sample size, have an arbitrary cut off for ‘statistical significance’, and provide no information on clinical relevance.45, 46 The recent EXACT trial, which was stopped early due to the COVID-19 pandemic, examined titration of post-arrest oxygenation and found titrated oxygen to 90 to 94% had an odds ratio for survival to discharge of 0.68 (95% CI: 0.46 to 1.00; p-value 0.05). While the trial concluded the results did not support the use of titrating oxygen to a saturation of 90% to 94%, it is likely that this is actually harmful but failed to reach statistical significance.⁴⁷ Bayesian statistics provide an alternative to traditional frequentist methods based on Bayes’ Theorem that calculate the probability of a treatment effect based on the available data instead of dichotomizing results based on arbitrary statistical significance.⁴⁸ Although a full understanding is well beyond the scope of this brief overview, Bayesian statistics utilizes previous information (prior probability distribution) to calculate a probability of the effect of intervention, the results of the trial are then used to update the prior distribution and calculate the posterior probability of an effect by using the data collected during the trial. The study results are then reported as the probability of an intervention being beneficial, given all available information.⁴⁸ The Paramedic-2 trial conducted a pre-planned Bayesian sensitivity analysis of the trial results.⁴ The authors were able to calculate the probability of a risk difference in 30-day survival of > 0 (0.99), >1% (0.37) and > 2% (0.002) between epinephrine and placebo, suggesting a small benefit for epinephrine.⁴ These results provide more information to quantify the potential effect of epinephrine beyond examining for statistical significance. Re-analysis of trial results using Bayesian statistics is becoming common in resuscitation and critical care; however to fully utilize the Bayesian methods, this should be part of the pre-planned study analysis.49, 50

Implementation challenges

Each prehospital setting presents unique challenges in the implementation of clinical research. Some of these challenges include a lack of familiarity with clinical research, conflicting service priorities, lack of buy-in from frontline practitioners, multitude of regulators and other partners involved, logistics of carrying out a trial intervention, and the ability to obtain complete data capture.⁵¹ Additionally, there is often unfamiliarity from researchers with the prehospital environment, making it difficult to plan and account for threats to trial implementation. Conducting trials in the out-of-hospital setting requires persistence, passion, and determination. Regardless of the local prehospital system, there are many individuals that may need to be involved to various degrees in the planning and implementation of prehospital research. These can include, emergency medical dispatchers, frontline paramedics and fire departments, medical oversight bodies, unions, government agencies, hospitals. All of these important organizations may have a role in delivery of prehospital medicine and require at a minimum to be informed of any ongoing research. Training is a key component to the success of any trial. Depending on the service, there could be thousands of different frontline providers to train in a study protocol. It is vital to the success of a study to ensure that training is not initiated in a study until all of the administration work has been completed and the trial is ready to start. Early training, followed by a delay before implementation can kill a trial prior to it starting. The second key to implementation is to involve frontline paramedics early in the development of your study. Creating collaboration with paramedics, in the form of steering committee members, operations committees, co-investigators or collaborators on studies or other opportunities can be instrumental in developing internal support at a paramedic service for a study. In addition, paramedic consultation can provide critical information on the logistics of a study such as where and how equipment can be used and stored, safety aspects for paramedics, restrictions on equipment size, obtaining consent and a variety of other factors related to trial success. Without this input even the best planned clinical trial may not be successful when implemented in the out-of-hospital setting.

Conclusion

Randomized controlled trials maintain an important role in advancement of resuscitation science. There are many challenges that are unique to the out-of-hospital environment that must be considered to conduct effective research in this setting. Clinical trials often require a unique approach that can be adapted for the complex environment that this research takes place. Failure to account for these complexities may impact the success of even the most well thought out clinical trial.

CRediT authorship contribution statement

Ian R. Drennan: Writing – original draft, Conceptualization. Shelley L. McLeod: Writing – review & editing, Conceptualization. Sheldon Cheskes: Writing – review & editing, Conceptualization.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Ian Drennan and Sheldon Cheskes are on the editorial board for Resuscitation.