Sample size and clinical trials in pediatric resuscitation

Ideally, high quality randomized controlled trials would be the foundation of every clinical decision. We especially want high quality evidence for all of our decisions in resuscitative care. Every day, in prehospital medicine, emergency medicine, and critical care, we are flooded with new research. It is understandable if we assume that our foundation of evidence is solid. The reality, however, is that the foundation of resuscitation science is more shifting sands than bedrock. There is a disheartening lack of high‐quality evidence for the vast majority of resuscitative decisions. In the 2020 update to the American Heart Association's Guidelines for Cardiopulmonary Resuscitation, only 32% of 491 individual recommendations were based on moderate or high quality evidence, and only 11% of the recommendations were based on randomized trials. ¹

The reasons for this lack of evidence should be obvious. Resuscitation research is expensive, in both money and time. Randomized controlled trials in resuscitation science typically require hundreds of thousands of dollars (at least) and hundreds of hours to execute. The best clinical trials require single‐center, observational studies and infrastructure building, followed by multicenter collaboration. Application for external funding is usually required, with funding rates depressingly low. Moreover, resuscitation clinical trials often produce negative or unimpressively positive results, meaning we need to complete many trials to get 1 that will positively influence care.

In pediatric resuscitation research, one factor has led to an even greater lack of high quality evidence—compared with adults, critical illness and injury is mercifully less common in children. Although tens of thousands of children require resuscitation each year, for single centers in the United States, there are rarely enough patients with any form of critical illness to conduct adequately powered clinical trials. ² , ³ This is a universal challenge in pediatric settings—prehospital, emergency department, and critical care—high quality research is prohibitively expensive because it cannot be done at 1 or 2 centers over a relatively short time period. Federal funding and multicenter designs are almost always required.

Compounding the sample size problem is that many, if not most, of the effect sizes studied in resuscitation research are small. Survival is perhaps the most common outcome in resuscitate research; however, it is uncommon for even a high quality study to show more than a modest absolute survival benefit for a given therapy. Survival and other binary outcomes, therefore, often require large sample sizes to detect more than a few percentage points difference in treatment groups. Binary outcomes also are descriptively limited, failing to capture the nuance of recovery from critical illness or injury, including functional and neurologic outcomes.

In an innovative study in this month's JACEP Open, Cho et al ⁴ studied intensive care unit (ICU)‐free days as an alternative to survival as the main outcome measure. Using data from one of the most well‐known pediatric resuscitation trials, the Pediatric Airway Management Project (PAMP), the investigators studied how the results of the PAMP study changed with ICU‐free days as the main outcome measure.

In the original study, investigators conducted a controlled clinical trial comparing survival and neurological outcomes between pediatric patients treated in the prehospital setting with bag‐valve‐mask (BVM) ventilation versus endotracheal intubation (ETI). The PAMP study was powered to detect an increase in survival to hospital discharge in infants in cardiopulmonary arrest, from 5% to 10% in the ETI group (effect size, 5%). The investigators estimated 400 patients would be needed per group, or 800 total. The PAMP study ultimately enrolled 830 subjects over 3.5 years.

Cho et al defined ICU‐free days as 30 minus the number of days in the ICU, with zero assigned to patients who died. This is a standard approach in studies using ICU‐free days. The results in Cho et al are fundamentally similar to the original PAMP study—the authors again found no overall statistical difference between BMV and ETI, with the median ICU‐free days for the BVM group 0 (interquartile range [IQR] = 0, 0) and for ETI 0 (IQR = 0, 0; P = 0.219). The original study found no significant difference in survival between the BVM group and the ETI group (odds ratio [OR] = 0.82; 95% confidence interval [CI] = 0.61, 1.11). Subgroup analyses were also generally similar, with both the PAMP study and Cho et al finding better outcomes for BVM with foreign body aspiration, child maltreatment, and respiratory arrest.

The “free days” concept is used commonly in the critical care literature, with 30 and 28 the standard minuends. In a simple PubMed search of the term “ICU‐free days,” in 2020 there were 33 English‐language publications with “free days” as a study outcome, including 3 randomized controlled trials (search date March 3, 2021). What is innovative about the study by Cho et al is use of data from a previous multicenter trial and its methodologic focus.

We used the results from Cho et al and the original PAMP study to graph the relative sample size differences needed for 2 potential outcome measures, survival versus ICU‐free days, based on a range of minimum difference between the BMV and ETI groups (Figure 1). We assumed that the median ICU‐free days and survival in the BMV group were 0 days and 5%, respectively. The Figure 1 shows how many patients would be needed to detect a range of absolute differences between the treatment (intubation) and control groups (BMV).

FIGURE 1.

Sample size comparison between ICU‐free days and difference in survival for clinical trials in resuscitation

A primary advantage of a continuous outcome measure, or quasi‐continuous like ICU‐free days, is the potential for increased precision and, therefore, increased sensitivity to differences between study groups. As Figure 1 shows, for studies that need to be powered for relatively small treatment effects, say an absolute survival benefit of less than 5%, ICU‐free days might be a good alternative to decrease the required sample size. This assumes that the potential treatment effect would not be expected to be larger than about 4 ICU‐free days. Beyond approximately 4 ICU‐free days and 5% survival, the differences become negligible.

An additional advantage of ICU‐free days is that, by combining mortality and neurologic outcomes, a single measure can capture more of the clinically important treatment effects. The literature on neurologic outcomes after cardiac arrest in infants and children is limited by variation in metrics used. ICU‐free days could be an ideal main outcome measure, as it captures both survival and functional/neurologic outcomes and has the potential for smaller sample sizes.

ICU‐free days as a main outcome measure are not without flaws. Both patients who die and those admitted to the ICU for longer than 30 days receive a score of 0, which is not especially intuitive. Cho et al suggest a solution to this problem (ie, assigning −1 to the patients who die). The potential sample size advantages of continuous or quasi‐continuous outcome measures is also not absolute—both within and between group variation have greater potential to vary, potentially leading to overlapping confidence intervals and limiting or eliminating the advantage for sensitivity. Finally, the dichotomy is somewhat false—a range of study outcomes can and should be included in any resuscitation trial. The choice between outcomes is only relevant to the sample size/power calculation. The selection of the main outcome should be rooted in the study's theoretical model and not based solely on sample size issues. Where the theoretical model supports both survival and ICU‐free days as outcomes, then investigators should perform a range of sample size calculations with both outcome measures.

We applaud Cho et al for their work—getting even 1 more study out of a costly pediatric trial is a worthy effort. Although not a novel outcome measure, we believe ICU‐free days should have an even more prominent place in resuscitation and prehospital research. Investigators should strongly consider using ICU‐days to limit the high costs of this critically needed work.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

AUTHOR CONTRIBUTIONS

All authors contributed to the conception and writing of this editorial.