Primary Outcome Measures in Pediatric Septic Shock Trials: A Systematic Review

Abstract

Objective

To evaluate all published pediatric randomized controlled trials (RCTs) of patients with septic shock from any cause to examine the outcome measures used, the strengths and limitations of these measurements and whether the trial outcomes met feasibility criteria.

Data source

We used a previously published database of pediatric critical care RCTs (PICUtrials.net) derived from searches of MEDLINE, EMBASE, LILACS and CENTRAL.

Study selection

We included RCTs of interventions to children admitted to a pediatric intensive care unit (PICU) with septic or dengue hemorrhagic shock which were published in English.

Data extraction

Study characteristics and outcomes were retrieved by two independent reviewers with disagreement being resolved by a third reviewer. We defined feasibility as 1) recruitment of at least 90% of the targeted sample size and agreement of the observed outcome rate in the control group with the rate used for the sample size calculation to within 10% or 2) finding of a statistically significant difference in an interim or final analysis.

Data synthesis

Nineteen of 321 identified articles were selected for review. Fourteen of 19 studies (74%) provided an a priori definition of their primary outcome measure in their methods section. Mortality rate was the most commonly reported primary outcome (8/14, 57%), followed by duration of shock (4/14, 29%) followed by organ failure (1/14, 7%). Only 3 of 19 included trials met feasibility criteria.

Conclusions

Our review found that use of mortality alone as a primary outcome in pediatric septic shock trials was associated with significant limitations and that long-term patient centered outcomes were not utilized in this setting. Composite outcomes incorporating mortality and long-term outcomes should be explored for use in future pediatric septic shock trials.

Background

Clinical research is necessary to ensure progress and improvement in the care of children with septic shock. However, clinical trials in this patient population are complex and face numerous challenges including inaccurate sample size estimates and difficulties with recruitment (1). The sample size, feasibility and relevance of findings of a clinical trial are affected by the choice of primary outcome measure, thereby making this decision a critical one.

In recent years, outcomes relevant to patients and families have become a priority in clinical research. Pediatric intensive care unit (PICU) clinicians, researchers and families consider survival to be an important outcome following critical illness (2;3). However, mortality may not always be a feasible primary outcome measure for pediatric septic shock trials due to variable mortality rates and smaller overall numbers of patients (4–7). Consequently, investigators have employed surrogate primary outcome measures such as time to vasoactive free hemodynamic stability (8) and organ dysfunction (9) to determine the effectiveness of interventions for pediatric septic shock. However, several researchers have highlighted the limitations of such outcome measures and suggested that they may not be important to patients and their families or clinically meaningful (10;11). There is presently no consensus on the most appropriate and feasible outcome measure to use for interventional trials in pediatric septic shock.

The objective of this study was to evaluate all published pediatric randomized controlled trials (RCTs) whose primary focus was patients with septic shock from any cause in order to determine the outcome measures used, the potential limitations of these measurements and if the trial outcomes met pre-defined feasibility criteria. This information will be valuable for the design and choice of primary outcome measures for future interventional pediatric shock trials.

Methods

Trial eligibility

Studies were eligible for inclusion if they satisfied the following criteria: 1) were RCTs found in the PICUtrials.net database, 2) examined prospectively administered intervention(s) to children with a primary diagnosis of septic shock from any cause or dengue hemorrhagic shock who were admitted to a pediatric intensive care unit (PICU) and 3) were published in English. We included trials of all phases and of patients with dengue hemorrhagic shock in order to broaden the scope of outcome measures reviewed. Combined adult and pediatric trials were included if the pediatric data was reported separately. We considered a unit to be an intensive care unit if the authors described it as such and it had the capacity to provide mechanical ventilation. We excluded trials that 1) enrolled exclusively post-operative cardiac surgery patients, 2) enrolled exclusively neonatal patients, 3) were only published as abstracts or 4) did not evaluate at least one of the following as a primary or secondary outcome: mortality, duration of vasoactive-inotropic support, length of mechanical ventilation, organ dysfunction, PICU length of stay, hospital length of stay, or functional status or quality of life. We excluded sub-studies and secondary analyses of included RCTs.

Study selection

We used a previously published database of pediatric critical care RCTs (PICUtrials.net)(1), from inception to January 4^th, 2016, to identify eligible studies. Briefly, this database uses comprehensive search strategies of MEDLINE, EMBASE, LILACS and CENTRAL (since their respective inceptions) to identify pediatric critical care RCTs. For the current study, we screened all published RCTs in the PICUtrials.net database as of January 4^th, 2016 selecting those that included children with septic shock or dengue hemorrhagic shock. Two reviewers performed level one screening of titles following by level two screening of abstracts to determine eligibility for data extraction. Disagreements were resolved by consensus or using a third independent reviewer. Key statistical concepts were verified by a biostatistician.

Data extraction

Eight reviewers, working in pairs independently and in duplicate extracted data from all included trials into an electronic data capture tool, Research Electronic Data Capture (REDCap) (12) which is hosted at the Children's Hospital of Eastern Ontario Clinical Research Unit. We resolved disagreements using an independent third reviewer. A priori definitions were established for type of study intervention, type of control, classification of primary outcome measure and classification of reason for stopping the study early. Countries were classified as developing versus developed as per the United Nations definitions (13).

Outcomes

The primary objective of this study was to describe the primary and secondary outcome measures reported in the included pediatric septic shock RCTs. A primary outcome measure was defined as a single or composite outcome measure that was clearly identified in the methods section of the article and was used for the sample size calculation. The secondary objectives were to describe the potential limitations of these outcome measures and to determine the feasibility of the chosen primary outcome measure. We defined feasibility as 1) recruitment of at least 90% of the targeted sample size and agreement of the observed outcome rate in the control group with the rate used for the sample size calculation to within 10% or 2) finding of a statistically significant difference in an interim or final analysis. The feasibility of trials that did not report a target sample size was recorded as indeterminate. We recorded potential barriers to feasibility including sample size, sources of funding, number of participating centres, consent rates, recruitment rates and study duration.

Data analysis

Search results were reported according to PRISMA recommendations (14). Data analysis was performed using SPSS version 23 (IBM Corporation, New York). Summary statistics and data from eligible studies were described in text, tables and figures. Measures of dispersion were standardized by converting means to medians when necessary (15). Comparison of medians was performing using non-parametric tests for independent groups in SPSS and comparison of mean was done using student's t-test. Statistical significance was defined as a p-value less than 0.05.

Results

Of 321 trials identified, 19 satisfied our pre-defined inclusion and exclusion criteria. The PRISMA diagram of included studies is shown in Figure 1. Initial screening of paper article titles excluded 297 articles, while further screening of the article abstracts excluded an additional five. Nineteen of the 321 trials in the database met our inclusion criteria. Trials were published between 1992 and 2015.

Figure 1.

Flow diagram for included trials.

Patient characteristics for included trials

The characteristics of included trials are shown in Table 1. The median lower age for included studies was 1 year (IQR 1, 3; range 1, 6) and the median upper age for included studies was 16 years (IQR 12, 18; range 12, 19). Of the 19 trials, nine were conducted in developing countries, nine in developed countries and in one, the geographic location of the participating centres was not unequivocally discernable (16).

Table 1.

Population characteristics of included trials.

Study	Patient population	Developing world	Age range	PRISMa (mean ± s.d.)
Ventura, 2015 (23)	Septic shock	Yes	1 month to 15 years	27 ± 6
Long, 2013 (30)	Septic shock	No	28 days – 16 years	Not reported
Chopra, 2011 (24)	Septic shock	Yes	2 – 12 years	11 ± 5
Choong, 2009 (8)	Vasodilatory shock	No	1 month – 17 years	13 ± 8
De Oliveira, 2009 (31)	Septic shock	Yes	>1 month to 19 years	8 ± 5
Valoor, 2009 (26)	Septic shock	Yes	2 months to 12 years	25 ± 42
Yildizdas, 2008 (27)	Septic shock	Yes	Not reported	28 ± 10
Santhanam, 2008 (29)	Septic shock	Yes	1 month to 12 years	14 ± 10
Nadel, 2007 (16)	Septic shock	unclear	38 weeks post conception	15 ± 8
Briassoulis, 2005 (20)	Septic shock	No	Not reported	12 ± 2
Upadhyay, 2005 (19)	Septic shock	Yes	1 month – 12 years	7 ± 7
Cifra, 2003 (25)	Dengue hemorrhagic shock	Yes	Not reported	Not reported
De Kleijn, 2003 (9)	Meningococcal septic shock	No	1 month – 18 years	20 ± 16
Levin, 2000 (18)	Meningococcal septic shock	No	12 weeks – 18 years	15 ± 9
Derkx, 1999 (17)	Meningococcal septic shock	No	3 months – 18 years	Not reported
Reeves, 1999 (32)	Septic shock	No	Not reported	Not reported
Dung, 1999 (28)	Dengue hemorrhagic shock	No	5 months to 15 years	Not reported
Barton, 1996 (21)	Septic shock	No	6 months to 18 years	Not reported
J5 Study Group, 1992 (22)	Purpuric shock	No	Not reported	Not reported

^aPRISM scores that were reported as medians with interquartile ranges were converted to means as per Wan et al ().

Study characteristics for included trials

The median number of patients randomized was 60 (IQR 40, 121, range 27, 496). The majority of trials (10/19, 52.6%) included more than one center with the median number of participating centers being two (IQR 1, 7; range 1, 104). The median number of participating countries was one (IQR 1, 2; range 1, 18). The median duration for enrollment in included studies was 21 months (IQR 12, 32; range 1, 62). The study interventions were classified as drug-related in nine trials (47%) (9;16–23), practice-related in seven (37%) (8;24–29) and device-related in three (15%) (30–32). The control group was a placebo in nine trials (47%) (9;16–23) and usual care in 8 trials (42%) (8;24–29). In the remaining two trials (11%) (31;32) it was unclear whether or not the control group represented usual care.

Primary outcome measures

Fourteen of 19 studies (74%) provided an a priori definition of their primary outcome measure in their methods section (see Table 2). Mortality rate was the most commonly reported primary outcome (8/14, 57%). Mortality was reported at the following time points – 28 days (5 trials); 14 days (n=1), hospital (n=1); not specified (n=1). Six of these studies were underpowered to find a difference based on their observed versus stated expected mortality rate (8;16–18;22;29). Three of the 14 studies demonstrated a statistically significant difference in their primary outcome between study groups (23;25;31). None of the studies in which mortality was a primary outcome provided data on cause of death, reflecting the common intent to measure all-cause mortality. The next most commonly reported primary outcome was duration of shock (n = 3) with the remaining primary outcomes being described in Table 2. The five trials that did not define an a priori primary outcome measure reported on multiple physiologic (21;24;27) and biochemical (9;20) outcomes as their collective endpoints.

Table 2.

Reported primary outcome measures.

Trial	Primary outcome definition	Expected outcome rate		Observed outcome rate		Target sample size	Attained sample size	Percentage of sample size attained	Statistical significance reached	Feasible
		Control group	Intervention group	Control group	Intervention group
Ventura, 2015 (23)	28 day mortality	25.0%	10.0%	20.6%	14.2%	152	120	80	yes	yes
Long, 2013 (30)	28 day mortality	50.0%	37.5%	17.0%	40.0%	524	48	9.2	no	no
Choong, 2009 (8)	Time to vasoactive-free hemodynamic stability	No baseline estimate provided, targeted hazard ratio of 3		47.1 hours	49.7 hours	69	69	100	no	n/a
De Oliveira, 2009 (31)	28 day mortality	57.3%	37.3%	39.2%	11.8%	268	102	38	yes	yes
Valoor, 2009 (26)	Time to shock reversal	Not reported	Not reported	70 hours	49.5 hours	n/a	40	n/a	no	n/a
Santhanam, 2008 (29)	Hospital mortality	50%	25%	17.8%	17.6%	160	160	100	no	no
Nadel, 2007 (16)	Composite Time to Complete Organ Failure Resolution Score	7.74 days	6.56 days	6.0 days	6.0 days	600	477	79.5	no	no
Upadhyay, 2005 (19)	Quantitation of plasma volume	Not calculated	Not calculated	52.5 ml/kg	52.2 ml/kg	n/a	60	n/a	no	n/a
Cifra, 2003 (25)	Duration of control of shock	Not reported	Not reported	31.3 hours	65.5 hours	n/a	27	n/a	yes	n/a
Levin, 2000 (18)	28 day mortality	25%	10%	9.9%	7.4%	400a	395	98.8	no	no
Derkx, 1999 (17)	28 day mortality	30%	15%	27%	18.5%	270	269	99.6	no	yes
Dung, 1999 (28)	Duration of shock	Not reported	Not reported	n/ac	n/ac	n/a	50	n/a	no	n/a
Reeves, 1999 (32)	14 day mortality	Not reportedb	Not reportedb	60%	66.7%	n/a	8	n/a	no	n/a
J5 Study Group, 1992 (22)	Mortality (timing unclear)	50%	25%	35%	25%	86	73	84.9	no	no

^aThe sample size calculation resulted in a sample size of 200 patients which was increased to 400 at the request of the data safety monitoring board to maintain the power to detect an odds ratio of 3 in the mortality difference between the two groups.

^bThis was a combined adult and pediatric study where a combined sample size was reported and the data was reported by age but was not analyzed separately. Post-hoc analysis showed that there was no difference between the mortality rates between the pediatric patients of the two groups.

^cThis study compared four different fluids, two of which were standard of care. They did not find a difference between in any of the pairwise comparisons.

Other outcome measures reported in trials

Secondary outcomes assessed in three or more trials are shown in Figure 2 and the variable definitions used for these outcome measures are listed in Table 3. Mortality was a secondary outcome in 10 of 19 trials (53%) (8;16;19;20;24–28) five of which did not find a statistically significant difference (8;16;19;26;27), four of which did not report a measure of statistical significance (9;20;24;25) and one in which there were no deaths (28).

Figure 2.

Reported secondary outcome measures in pediatric septic shock trials.

Table 3.

Outcome definitions in included trials.

Outcome	Definitiona
Mortality rate	28-day
	14-day
	30-day
	60-day
	PICU discharge
	Hospital discharge
	Unclear
Vasoactive agent use	Time on vasopressors
	Vasopressor free days
	Vasoactive free days 30 days after randomization
Mechanical ventilation	Length of mechanical ventilation
	Mechanical ventilation free days
Organ dysfunction	Organ failure free days
	Number of organs failing day 7
	SOFA score adapted for pediatrics
	SOFA score
	Number of organs failing
	Change in PELOD score

^aListed in order of how commonly they were reported.

Measures of time on vasopressors (8;19;23;24;29;31), duration of mechanical ventilation as either length of mechanical ventilation or ventilator free days (8;18;20;23;27;29;31), PICU length of stay (8;9;18;20;24;25;27;31) and organ dysfunction (8;16;23;26;30–32) were the next most common outcome measures reported. A statistically significant difference in time on vasopressors (31), ventilator free days (31) and in PICU length of stay (27) were found in one trial each respectively. None of the studies reporting PICU length of stay stated whether they accounted for censoring from death. A measure of organ dysfunction was reported (8;16;23;26;30–32) in seven trials, two of which found a statistically significant difference between groups (23;31).

Eleven trials reported on adverse events with none of them demonstrating a statistically significant difference between groups (9;16–18;20;21;23;24;26;29;32). One trial stated that there were no adverse events but did not define them (31). No trials reported on any health related quality of life (HRQL) or economic outcomes. One trial reported a statistically significant improvement in functional status (Pediatric Overall Performance Category) in the intervention group at six months as a secondary outcome (18).

Trial feasibility

Factors affecting the feasibility of the included trials are shown in Table 4. Of the 14 studies which reported a clearly defined primary outcome measure, nine studies reported an a priori sample size calculation. Of these nine trials, only one trial met the feasibility definition of recruiting 90% of the targeted sample size and having agreement between the observed and expected outcome rate (17) and two studies met the definition of finding a statistically significant result in the interim analysis or final analysis (23;31). Therefore, three of 19 trials (17;23;31) met our defined feasibility criteria. The feasibility of six trials was indeterminate (8;19;25;26;28;32).

Table 4.

Factors affecting feasibility^a.

Factorb	Feasible studies (n = 3)	Infeasible studies (n = 11)	P value
Source of funding - % industryc	66.7	9.1	0.038
Participating centres – median (IQR) - n	2 (2, 14)	1 (1, 5)	0.75
Sample size – median (IQR) - n	120 (111, 194)	60 (38, 106)	0.02
Study duration – median (IQR) - months	49 (35, 52)	21 (13, 27)	0.74
Recruitment rate – median - no. patients per month	2.2 (0.2, 2.6)	0.5 (0.3, 12.3)	0.78

^aBased on the fourteen studies in which a primary outcome measure was clearly defined.

^bConsent rates were not included in the table as consent data was only provided for two of the 14 studies in which the primary outcome was reported.

^cChoong et al had a small grant from industry but was primarily funded by granting agencies.

The mean recruitment rate was 4.5 patients per site per month (range 0.3 to 27). The mean recruitment rate in developing countries was 8.9 ± 8.7 patients per site per month (n = 7) versus 0.6 ± 0.4 patients per site per month (n = 7) in developed countries (p = 0.027). Six of the 19 trials (32%) reported ending prematurely (16;22;23;30–32). Two trials were terminated due to poor recruitment (16;30), two trials due to the observed mortality rates being lower than anticipated (23;32), one trial due to a lower than anticipated disease rate (22) and one trial due to the findings of the interim analysis (31). The median study duration was 21 months (IQR 12, 32). Approximately 60% of trials included all randomized patients in the final analysis (11/19, 58%) (9;19;21;22;24;25;27;28;30–32) with the remaining eight trials having a median loss of patients from randomization to analysis of 5% (IQR 0.7, 7.5). In these eight trials, the number of patients randomized was not equivalent to the number analyzed. Four of these trials (8;16;18;23) stated that they used an intent to treat analysis but only three of the eight trials (8;17;18) actually included all patients who received the intervention in the final analysis. The funding source was identified in 63% (12/19) of trials (8;9;16–19;22;25;29–32) with 26% each (5/19) being funded by industry (9;16–18;31) and foundations/granting agencies (8;22;25;28;29) respectively and 11% (2/19) (19;30) being reported as unfunded. The consent rate was 86% ± 11% (mean ± 1 SD) in the three trials (16%) in which it was reported or from which the rate was calculable.

Discussion

The primary objective of most pediatric critical care trials, including those studying septic shock, is to test whether interventions reduce the morbidity and mortality caused by critical illness. Whether the trial conclusively determines the intervention to be of value or not is highly dependent upon the outcome measure selected to inform the sample size (primary outcome) (33;34). We therefore reviewed all pediatric septic and dengue shock trials published since 1986 to examine their reported outcome measures and trial feasibility with the goal of informing the design of future septic shock trials.

Mortality rate was the most commonly reported outcome measure among the studies reviewed and was the primary outcome measure for the three studies that met feasibility criteria (17;23;31). Evaluation of these three trials identified that they were all conducted in the developing world with observed mortality rates between 20 and 40%. Secondly, these trials expected and achieved large relative reductions in mortality rates of between 35% and 60%, which may not be realistic targets for other trials. Mortality may be feasible as an outcome measure in the developing world due to its higher incidence and low cost of measurement.

However, we found several significant limitations in the use of mortality as a primary outcome measure. The first is that mortality is commonly over-estimated when studies are designed and sample sizes determined (18;22;23;29–31). Secondly, the expected decrease in mortality is often overly optimistic such that even when a trend toward a mortality is observed, it may fail to reach significance (17;22). Finally, most studies appear to include small numbers of centers and short recruitment periods which therefore limit the ability to detect a difference in mortality given the large sample sizes required. One could speculate that the latter is due to inadequate funding as 47% of trials did not report on source of funding or stated that they were unfunded.

It is important consider other potential limitations of mortality as a primary outcome in addition to those described in this study. First, not all sepsis interventions should be expected to impact mortality. Second, focusing on mortality inherently implies that interventions that do not decrease mortality are not clinically important and as such does not account for important outcomes such as HRQL and long-term morbidity. Furthermore, the use of short-term mortality as a primary outcome does not account for the fact that some sepsis deaths, especially in children with underlying chronic diseases, occur after PICU discharge (35;36).

Another category of potential primary outcome measures include physiologic markers such as time to reversal of shock (8;25;26;28) and new or progressive multiple organ dysfunction (MODS) (16;37). A recent study showed that 30% of pediatric septic shock patients developed new or progressive MODS (6) and two of the seven trials in which organ dysfunction was measured found a difference between groups suggesting its potential value as an outcome measure. However, its importance to patients and their families as well as correlation with longer-term outcomes has not been demonstrated. In fact, several pediatric critical care RCTs have demonstrated improvements in physiologic outcomes without significant improvements in clinical outcomes or worse clinical outcomes (38;39). Therefore, use of these measures alone as primary outcomes is not ideal. Duration of mechanical ventilation, PICU length of stay and hospital length of stay were frequently measured but were not used as primary outcome measures. As a result, none of the studies were adequately powered to find a statistically significant difference in PICU length of stay but seven of eight studies in which it was measured reported a decrease. However, length of stay may differ between units (due to practice variation, bed availability etc.) (38) necessitating the development and validation of readiness for discharge tools in order to standardize PICU length of stay prior to its use as a primary outcome measure. In addition, the trials reviewed used multiple measures of vasopressor use and organ dysfunction and did not specify whether PICU length of stay accounted for censoring from death making comparisons between trials difficult. Standardization of these measures would be useful prior to further pediatric sepsis trials.

There were several noteworthy observations regarding trial feasibility within the limitations of the small number of included trials. The first is that industry funded trials were more likely to be deemed feasible than those that were funded by granting agencies or foundations. The reasons for this are unclear but may include improved resources, more realistic study designs and/or stricter oversight of industry sponsored trials. It was also noted that the feasible trials were significantly larger than the rest of the trials perhaps suggesting that the infeasible trials were underpowered. Recruitment rates were significantly higher in developing countries than in the developed world. It would therefore be tempting to try to include developing and developed countries together in pediatric septic shock trials in order to improve enrolment efficiency. However, differences in access to care, availability of resources (40;41) and underlying causes of septic shock in developed versus developing countries (5;42) could seriously affect the integrity of the study design. While it is very important to conduct pediatric septic shock trials in different parts of the world, the interventions to be studied along with the relevant outcome measures may differ significantly depending on the region, making it difficult to conduct combined studies.

Given the limitations of mortality and common PICU outcomes, the question arises as to what other options exist for measuring improvements in care for children with septic shock. While none of the trials in our study included HRQL measures and only one trial assessed functional outcomes, evidence supports the importance of evaluating long-term outcomes for children and families who have survived critical illness (43–46). While these outcomes are clinically relevant and more patient centered (47), there is presently no consensus on how best to measure functional status or HRQL, the timing of these measurements and how interventions in pediatric septic shock may impact functional recovery. These challenges do not negate the importance of assessing HRQL but may explain why none of the trials identified in this review have measured it. Importantly, HRQL and functional outcomes have been successfully measured in several observational septic shock studies (48;49) and non-septic shock related pediatric critical care RCTs (50) suggesting that careful planning of future clinical trials in pediatric septic shock could overcome many of the difficulties described and allow for reliable collection of such data.

There were several limitations to this study. We excluded trials in centers that did not have access to PICU services as the patient populations, interventions and outcomes in those areas have the potential to be significantly affected by the lack of resources (40;41). In addition, this review focused on outcomes of RCTs for interventions for septic shock and therefore included trials on patients for whom the standard interventions (i.e. fluids and antibiotics) were not sufficient to reverse the process. We also excluded purely neonatal trials due to their differential physiology and outcomes (51). Furthermore, we were unable to comment on age specific outcomes as no information was provided on these within the reviewed trials. Finally, it is possible, especially in developing countries, that the reported deaths were not all directly attributable to septic shock. However, the majority of critical care trials that evaluate mortality as the primary outcome evaluate all-cause mortality, as it is very difficult to attribute death specifically to sepsis in most cases.

Conclusion

Mortality is an important and commonly reported outcome measure in pediatric septic shock trials. However, mortality rates are highly variable and therefore mortality alone may not be appropriate as a primary outcome in all settings. Given the high incidence of PICU-acquired morbidities, measuring functional status and HRQL may be appropriate in future trials. We suggest that future pediatric septic shock trials should consider a composite outcome capturing both mortality and longer term functional status, as this may better reflect the spectrum of health burden associated with septic shock.