Abstract
Objective
PERSEVERE, a pediatric sepsis risk model, uses biomarkers to estimate baseline mortality risk for pediatric septic shock. It is unknown how PERSEVERE performs within distinct septic shock phenotypes. We tested PERSEVERE in children with septic shock and thrombocytopenia-associated multiple organ failure (TAMOF), and in those without new onset thrombocytopenia but with multiple organ failure (MOF).
Design
PERSEVERE-based mortality risk was generated for each study subject (n = 660). A priori, we determined that if PERSEVERE did not perform well in both the TAMOF and MOF cohorts, we would revise PERSEVERE to incorporate admission platelet counts.
Setting
Multiple pediatric intensive care units in the United States.
Interventions
Standard care.
Measurements and Main Results
PERSEVERE performed well in the TAMOF cohort (AUC 0.84 [95% CI: 0.77 – 0.90]), but less well in the MOF cohort (AUC 0.71; [0.61 – 0.80]). PERSEVERE was revised using 424 subjects previously reported in the derivation phase. PERSEVERE-II had an AUC of 0.89 (0.85 – 0.93) and performed equally well across TAMOF and MOF cohorts. PERSEVERE-II performed well when tested in 236 newly enrolled subjects. Sample size calculations for a clinical trial testing the efficacy of plasma exchange for children with septic shock and TAMOF indicated PERSEVERE-II-based stratification could substantially reduce the number of patients necessary, when compared to no stratification.
Conclusions
Testing PERSEVERE in the context of septic shock phenotypes prompted a revision incorporating platelet count. PERSEVERE-II performs well upon testing, independent of TAMOF or MOF status. PERSEVERE-II could potentially serve as a prognostic enrichment tool.
Keywords: sepsis, biomarkers, organ failure, stratification, enrichment, trial simulation
INTRODUCTION
A recently published roadmap for future research in the field of sepsis encourages incorporation of biomarkers and enrichment strategies for clinical trials [1]. Septic shock is highly heterogeneous, which translates to complex and challenging clinical decision making [1, 2]. Several programs of research are attempting to directly improve clinical care by providing tools to differentiate patients with septic shock on the basis of mortality risk, and on the basis of their clinical phenotype [3–5]. While differentiating between septic shock phenotypes might guide selection of treatments specific to the phenotypic characteristics, tools that inform clinicians in real time about mortality risk both within and across septic shock phenotypes can help determine who needs aggressive and potentially higher risk treatments and who does not.
The Pediatric Sepsis Biomarker Risk Model (PERSEVERE) estimates baseline 28-day mortality risk for children with septic shock [6, 7]. PERSEVERE was derived using Classification and Regression Tree (CART) methodology, and the model incorporates a panel of biomarkers and age. The PERSEVERE biomarkers were selected objectively, using discovery oriented transcriptomic studies [7, 8]. PERSEVERE performs well when tested in a heterogeneous septic shock cohort [9], but it is unknown how PERSEVERE performs when applied to distinct clinical phenotypes of septic shock.
Thrombocytopenia-associated multiple organ failure (TAMOF) has been proposed as an important clinical phenotype of septic shock, with high mortality that is potentially modifiable by plasma exchange [10–12]. TAMOF is defined by new onset multiple organ failure with new onset thrombocytopenia. The mechanistic link between thrombocytopenia and organ failure is thought to involve a form of microangiopathy analogous to thrombotic thrombocytopenic purpura (TTP), including decreased levels of ADAMTS-13 (A Disintegrin And Metalloprotease with ThromboSpondin motifs) and increased von Willebrand factor activity [12, 13]. ADAMTS-13 regulates microvascular thrombosis by cleaving large and ultra-large thrombogenic von Willebrand factor multimers into smaller, less thrombogenic forms. Preliminary experience suggests plasma exchange restores ADAMTS-13 levels and restores organ function in children with TAMOF [10], although an appropriately powered study has yet to be conducted.
Whether PERSEVERE is useful for risk stratifying patients with TAMOF is unknown. In the current study we tested the performance of PERSEVERE in children with septic shock and TAMOF, in those without new onset thrombocytopenia but with multiple organ failure (MOF), and in those without MOF. Since PERSEVERE has potential as a prognostic enrichment tool, we then estimated the sample sizes required to conduct a clinical trial testing the efficacy of plasma exchange in children with TAMOF, with and without PERSEVERE-based stratification.
METHODS
Study Subjects and Data Collection
The study cohort included 660 subjects with septic shock. There were 424 subjects with available platelet data previously reported in the derivation and validation of PERSEVERE [6, 9]. An additional 236 new subjects newly enrolled since the derivation and validation of PERSEVERE were also included. No subjects received plasma exchange for TAMOF.
The protocol for collection and use of biological specimens and clinical data was approved by the Institutional Review Boards of each of the 18 participating institutions. Children ≤ 18 years of age admitted to the pediatric intensive care unit (PICU) and meeting pediatric-specific consensus criteria for septic shock were eligible for enrollment [14, 15]. There were no exclusion criteria, other than the inability to obtain informed consent, which was obtained from parents or legal guardians prior to any data or sample collection.
Serum samples were obtained within 24 hours of first meeting the criteria for septic shock in the PICU, which was typically at presentation to the PICU. Clinical and laboratory data were collected daily while in the PICU. Organ failure data were tracked up to day 7 of septic shock using previously published criteria [14]. Mortality was tracked for 28 days after enrollment. Complicated course was defined as the persistence of two or more organ failures at day seven of septic shock or 28-day mortality [4]. Illness severity was estimated using PRISM scores [16].
Definition of TAMOF and MOF
TAMOF was defined as new onset thrombocytopenia (platelet count < 100,000/μL) and two or more organ failures [11, 12]. MOF was defined as new onset of two or more organ failures but with either platelet counts ≥ 100,000/μL or with a known pre-existing condition resulting in thrombocytopenia. All other subjects were classified in the No MOF group.
PERSEVERE Biomarkers
PERSEVERE includes C-C chemokine ligand 3 (CCL3), interleukin 8 (IL8), heat shock protein 70 kDa 1B (HSPA1B), granzyme B (GZMB), and matrix metallopeptidase 8 (MMP8) [6]. Serum concentrations of these biomarkers were measured using a multi-plex magnetic bead platform (MILLIPLEX™ MAP) designed for this project by the EMD Millipore Corporation (Billerica, MA). Biomarker concentrations were measured in a Luminex® 100/200 System (Luminex Corporation, Austin, TX), according the manufacturers’ specifications. Assay performance data were previously published [6].
Statistical Analysis
Initially, data are described using medians, interquartile ranges, frequencies, and percentages. Comparisons between groups used the Mann-Whitney U-test, Chi-square, or Fisher’s Exact tests as appropriate. Descriptive statistics and comparisons used SigmaStat Software (Systat Software, Inc., San Jose, CA).
Each study subject was assigned a 28 day mortality probability using the previously published PERSEVERE model [6]. PERSEVERE performance is reported using diagnostic test statistics with 95% confidence intervals computed using the score method as implemented by the VassarStats Website for Statistical Computation [17].
A priori, we determined that if PERSEVERE did not fit equally well in patients with TAMOF as in patients without TAMOF, we would revise PERSEVERE to incorporate variables associated with thrombocytopenia. Specifically, we considered the presence of malignancy, immune suppression, bone marrow transplantation, and new onset thrombocytopenia as dichotomous predictor variables. Continuous predictor variables included the PERSEVERE biomarkers, age, and admission platelet count. To revise PERSEVERE, we used CART methodology (Salford Predictive Modeler v6.6, Salford Systems, San Diego, CA) [18, 19]. The primary outcome variable was 28-day mortality. Weighting of cases and the addition of cost for misclassification were not used in the modeling procedures. The code and data used to generate the model is available from the authors. The revised tree (PERSEVERE-II) was derived using the original 424 subjects used to develop PERSEVERE, and was subsequently internally validated in the 236 newly enrolled subjects.
Areas under the receiver operating characteristic curves (AUC) were compared using the method of Hanley and McNeil for non-independent samples [20]. The net reclassification improvement (NRI) was used to estimate the incremental predictive ability of PERSEVERE-II compared to PERSEVERE [21]. The NRI was computed using the R-package Hmisc [22].
Finally, we estimated the sample size needed for conducting a clinical trial comparing plasma exchange to no plasma exchange in children with TAMOF. We considered using PERSEVERE-II as an enrichment strategy to target patients most likely to benefit from plasma exchange. We assumed independence of study arms would be tested using a continuity corrected Chi-square test with power (1-β) set to 0.8 and α=0.05. Calculations were conducted using nQuery Advisor v 7.0 (Statistical Solutions Ltd., Cork, Ireland).
RESULTS
PERSEVERE performance in subjects with TAMOF and MOF
Table 1 shows the demographics and clinical characteristics of the TAMOF, MOF and No MOF cohorts. Subjects with TAMOF (n = 209) had a higher mortality, a higher rate of complicated course, and a higher median PRISM score when compared to the MOF (n = 290) and No MOF (n = 161) subjects. A lower proportion of subjects with TAMOF had no causative pathogen isolated, and a lower proportion had comorbidities when compared to the MOF and No MOF subjects. The subjects with No MOF were older than the TAMOF and MOF subjects. No other differences were noted.
Table 1.
Demographics and clinical characteristics of the study cohorts.
| TAMOF | MOF | No MOF | |
|---|---|---|---|
| N (%) | 209 (32) | 290 (44) | 161 (24) |
|
| |||
| Median Age, Years (IQR) | 2.1 (0.7 – 6.5) | 3.0 (1.0 – 7.1) | 7.0 (2.6 – 13.0)1 |
| Males, # (%) | 122 (58) | 158 (54) | 83 (52) |
|
| |||
| 28-day mortality, # (%) | 47 (22)2 | 39 (13)3 | 0 (0) |
| Complicated course, # (%) | 99 (47)2 | 87 (30)3 | 0 (0) |
| Median PRISM score (IQR) | 17 (11 – 24)2 | 11 (7 – 17)3 | 9 (5 – 12) |
|
| |||
| # with gram negative bacteria (%) | 60 (29) | 61 (21) | 32 (20) |
| # with gram positive bacteria (%) | 57 (27) | 56 (19) | 29 (18) |
| # with other pathogen isolated (%) | 18 (9) | 25 (9) | 13 (8) |
| # with no pathogen identified (%) | 74 (35)2 | 148 (51) | 87 (54) |
| # with comorbidity (%) | 79 (38)3 | 148 (51) | 76 (47) |
| # with malignancy (%) | 0 (0)2 | 53 (18) | 28 (17) |
| # with immune suppression (%) | 12 (6)2 | 67 (23) | 34 (21) |
| # with bone marrow transplantation (%) | 0 (0)2 | 25 (9) | 13 (8) |
p < 0.05 vs. TAMOF and MOF.
p < 0.05 vs. MOF and No MOF.
p < 0.05 vs. MOF.
Table 2 shows the test characteristics of PERSEVERE for estimating the probability of 28-day mortality in all subjects and for subjects in the TAMOF, MOF, and no MOF groups. PERSEVERE had very good performance in the TAMOF cohort, with an AUC of 0.84 (95% CI: 0.77 – 0.90), a sensitivity of 91% (95% CI: 79 – 97), and a negative likelihood ratio of 0.1 (95% CI: 0.01 – 0.3). In contrast, PERSEVERE performed less well in the MOF group, with an AUC of just 0.71 (0.61 – 0.80).
Table 2.
PERSEVERE test characteristics across the study cohorts.
| ALL SUBJECTS | TAMOF | MOF | NO MOF | |
|---|---|---|---|---|
| N | 660 | 209 | 290 | 161 |
|
| ||||
| False Positive | 151 | 57 | 77 | 17 |
| True Positive | 72 | 43 | 29 | 0 |
| True Negative | 424 | 105 | 175 | 144 |
| False Negative | 13 | 4 | 9 | 0 |
|
| ||||
| Sensitivity | 85 (75 – 91) | 91 (79 – 97) | 76 (59 – 88) | -- |
| Specificity | 74 (70 – 77) | 65 (57 – 72) | 69 (58 – 69) | -- |
|
| ||||
| PPV | 32 (26 – 39) | 43 (33 – 53) | 27 (19 – 37) | -- |
| NPV | 97 (95 – 98) | 96 (90 – 99) | 95 (91 – 98) | -- |
|
| ||||
| +LR | 3.2 (2.7 – 3.8) | 2.6 (2.1 – 3.3) | 2.5 (1.9 – 3.2) | -- |
| −LR | 0.2 (0.1 – 0.3) | 0.1 (0.05 – 0.3) | 0.3 (0.2 – 0.6) | -- |
|
| ||||
| AUC | 0.80 (0.75 – 0.86) | 0.84 (0.77 – 0.90) | 0.71 (0.61 – 0.80) | -- |
Derivation of PERSEVERE-II
Because PERSEVERE did not perform equally well in the TAMOF group and the MOF group, we proceeded to revise PERSEVERE taking into account variables associated with thrombocytopenia. Figure 1 shows PERSEVERE-II. The top node of the decision tree, the root node, provides the total number of subjects as well as the number and proportion of survivors and nonsurvivors. Subjects in the root node are subsequently allocated to daughter nodes based on the results of binary recursive partitioning. Each daughter node provides the criterion for deciding subsequent partitions, along with the number and proportion of survivors and nonsurvivors. Terminal nodes reflect the final assignment of risk to an individual case. We have annotated the tree to number the terminal nodes; the numbers appear in bold above each terminal node in the tree. All five PERSEVERE biomarkers (CCL3, IL8, GZMB, HSPA1B, and MMP8) contributed to the predictive capacity of PERSEVERE-II. Admission platelet count was found to augment predictive accuracy, but age and comorbidity burden did not.
Figure 1. The PERSEVERE-II classification tree.
The classification tree includes C-C chemokine ligand 3 (CCL3), interleukin 8 (IL8), heat shock protein 70 kDa 1B (HSPA1B), granzyme B (GZMB), matrix metallopeptidase 8 (MMP8), and admission platelet counts. The biomarker concentrations are shown in ng/ml and platelet counts are shown in number/μl. The root node provides the total number of patients in the derivation cohort, and the number of survivors and non-survivors, with the respective rates. Each daughter node provides the respective decision rule criterion and the number of survivors and non-survivors, with the respective rates. Terminal nodes (TN) TN1, TN 2, TN 5, TN8, and TN9 are low risk terminal nodes (≤1.9% risk of death). TN4, TN6, TN7, and TN10 are intermediate risk terminal nodes (16.7% to 33.3% risk of death. TN3 and TN11 are high risk terminal nodes (≥44.4% risk of death).
PERSEVERE-II had five low risk terminal nodes (≤1.9% risk of death; nodes 1, 2, 5, 8, and 9), four intermediate risk terminal nodes (16.7% to 33.3% risk of death; nodes 4, 6, 7, and 10), and two high risk terminal nodes (≥44.4% risk of death; nodes 3 and 11). Among the 273 subjects classified as low risk, two (0.7%) died by 28 days. Among the 151 subjects classified as intermediate or high risk, 47 (31.1%) died by 28 days. Table 3 shows the test characteristics of PERSEVERE-II in the derivation cohort.
Table 3.
PERSEVERE-II test characteristics in the derivation and test cohorts.
| ALL SUBJECTS | TAMOF | MOF | NO MOF | |
|---|---|---|---|---|
| Derivation cohort, N=424 | ||||
| N | 424 | 159 | 155 | 110 |
|
| ||||
| False Positive | 104 | 57 | 26 | 21 |
| True Positive | 47 | 34 | 13 | -- |
| True Negative | 271 | 66 | 116 | 89 |
| False Negative | 2 | 2 | 0 | -- |
|
| ||||
| Sensitivity | 96 (85 – 99) | 94 (80 – 99) | 100 (72 – 100) | -- |
| Specificity | 72 (67 – 77) | 54 (44 – 63) | 82 (74 – 87) | -- |
|
| ||||
| PPV | 36 (31 – 40) | 37 (28 – 48) | 33 (20 – 50) | -- |
| NPV | 99 (97 – 100) | 97 (89 – 99) | 100 (96 – 100) | -- |
|
| ||||
| +LR | 3.5 (2.9 – 4.1) | 2.0 (1.7 – 2.5) | 5.5 (3.9 – 7.7) | -- |
| −LR | 0.06 (0.01 – 0.2) | 0.1 (0.03 – 0.4) | -- | -- |
|
| ||||
| AUC | 0.89 (0.85 – 0.93) | 0.82 (0.75 – 0.88) | 0.93 (0.88 – 0.97) | -- |
|
| ||||
| Test cohort, N=236 | ||||
| N | 236 | 50 | 135 | 51 |
|
| ||||
| False Positive | 61 | 17 | 36 | 8 |
| True Positive | 34 | 11 | 23 | -- |
| True Negative | 138 | 22 | 73 | 43 |
| False Negative | 3 | 0 | 3 | -- |
|
| ||||
| Sensitivity | 92 (77 – 98) | 100 (68 – 100) | 88 (69 – 97) | -- |
| Specificity | 69 (62 – 76) | 56 (40 – 72) | 67 (57 – 76) | -- |
|
| ||||
| PPV | 36 (26 – 46) | 39 (22 – 59) | 39 (27 – 53) | -- |
| NPV | 98 (93 – 99) | 100 (82 – 100) | 96 (88 – 99) | -- |
|
| ||||
| +LR | 3.0 (2.4 – 3.8) | 2.3 (1.6 – 3.3) | 2.7 (2.0 – 3.6) | -- |
| −LR | 0.1 (0.04 – 0.3) | -- | 0.2 (0.1 – 0.5) | -- |
|
| ||||
| AUC | 0.84 (0.78 – 0.90) | 0.92 (0.84 – 1.00) | 0.79 (0.71 – 0.88) | -- |
Testing PERSEVERE-II
We tested the performance of PERSEVERE-II using the 236 subjects newly enrolled since the initial derivation of PERSEVERE. Supplementary Figure 1 shows how the 236 test subjects were classified. Among the 141 test subjects classified as low risk, three (2.1%) died by 28 days. Among the 95 test subjects classified as intermediate or high risk, 34 (35.8%) died by 28 days. Table 3 shows the test characteristics of PERSEVERE-II in the test cohort.
Comparison of PERSEVERE and PERSEVERE-II
Supplementary Figure 2 shows the PERSEVERE and PERSEVERE-II receiver operating characteristic curves for all subjects. For estimating the risk of 28-day mortality, the AUC for PERSEVERE-II (0.87; 95% CI 0.84 – 0.90) was superior to that of PERSEVERE (0.80; 95% CI 0.75 – 0.86; p = 0.031). When risk stratifying using PERSEVERE-II compared to PERSEVERE, the NRI was 0.50 (95% CI: 0.28 – 0.72; p < 0.0001), indicating improved classification of subjects. Admission platelet count alone had an AUC of 0.73 (95% C.I. 0.68 – 0.78) for estimating the risk of 28-day mortality.
Designing a clinical trial to test the efficacy of plasma exchange using PERSEVERE-II as a prognostic enrichment strategy
One practical application of PERSEVERE is as a prognostic enrichment tool to inform patient selection for clinical trials. For example, PERSEVERE-II could allow for exclusion of patients having a low baseline mortality probability with standard care and who would unlikely benefit from an experimental intervention. Exclusion of such patients could enrich the study population with patients having a higher baseline mortality probability, and consequently decrease the number of patients required for an interventional trial. As an example, one recent trial used the approach of restricting enrollment to patients with three or more organ failures as a means of enriching the study population with more severely ill patients (clinicaltrials.gov; NCT00118664).
In our study, 108 of the TAMOF subjects had at least three organ failures and could therefore have theoretically met the enrollment criteria for NCT00118664. We used data from these 108 subjects enrolled in our study to estimate the sample size required to conduct a clinical trial testing the efficacy of plasma exchange in children with TAMOF, with and without PERSEVERE-based stratification.
Among these 108 subjects there were 41 deaths (38% mortality), and PERSEVERE-II had an AUC of 0.82 (95% CI: 0.74 – 0.90) for estimating 28-day mortality. In comparison, PRISM had an AUC of 0.60 (95% CI: 0.49 – 0.72) for estimating 28-day mortality in these subjects. PERSEVERE-II correctly predicted 28-day survival for 32 subjects (true negatives) and incorrectly predicted 28-day survival for 2 subjects (false negatives). Using PERSEVERE-II-based stratification, these true and false negative subjects would be excluded from a clinical trial, leaving 39 true positive and 35 false positive subjects for inclusion. Among the true and false positive subjects, the 28-day mortality was 53%.
We calculated the number of un-stratified patients (38% mortality) and PERSEVERE-II-stratified patients (53% mortality) required in a trial randomizing patients with TAMOF and at least 3 organ failures to standard care or plasma exchange. We used a range of assumptions for the relative mortality reduction attributable to plasma exchange (10% to 50%), and we set power (1-β) to 0.8 and α to 0.05. Table 4 shows the results. In each scenario, PERSEVERE-II-based stratification reduced the number of subjects needed in each study arm by between 39% and 44%.
Table 4.
Number of un-stratified and PERSEVERE-II-based stratified subjects required in each study arm in a trial of plasma exchange for TAMOF (assuming a continuity corrected chi-square test to compare two equally sized groups with power (1-β) set to 0.8 and α=0.05).
| Projected relative reduction in mortality with plasma exchange | No. of un-stratified patients required in each study arm (% absolute mortality reduction) | No. of stratified patients required in each study arm (% absolute mortality reduction) |
|---|---|---|
| 10% | 2,559 (3.8) | 1,434 (5.3) |
| 20% | 637 (7.6) | 366 (10.6) |
| 30% | 281 (11.4) | 165 (15.9) |
| 40% | 156 (15.2) | 94 (21.2) |
| 50% | 98 (19.0) | 60 (26.5) |
DISCUSSION
Given the clinical and research interest in TAMOF as a distinct clinical phenotype of septic shock, we tested the performance of PERSEVERE in children with septic shock and TAMOF and found it had acceptable performance. In contrast, PERSEVERE did not perform well in children with septic shock and MOF in the absence of new onset thrombocytopenia. We revised PERSEVERE to have broader applicability across septic shock phenotypes. PERSEVERE-II incorporates admission platelet count into the risk stratification, and performs well in both the TAMOF and MOF phenotypes. This feature of PERSEVERE-II is biologically plausible because new onset thrombocytopenia is the primary determinant when distinguishing TAMOF from MOF [11, 12]. PERSEVERE-II continued to perform well when tested in a separate cohort. Our data demonstrate that PERSEVERE-II outperforms PERSEVERE, as measured by comparisons of the receiver operating characteristic curves and the NRI.
The current study supports the concept that TAMOF is a distinct clinical phenotype of septic shock [11]. Patients with septic shock and TAMOF had significantly higher rates of mortality and complicated course compared to patients with MOF in the absence of new onset thrombocytopenia. Thus, the design of therapies specifically targeting the TAMOF phenotype seems warranted.
Plasma exchange may be a potential therapy for TAMOF, but it has not been tested in a randomized clinical trial [1, 11]. The American Society of Apheresis classifies the use of plasma exchange for “sepsis with multiple organ failure” as being supported with only level III evidence, and they provide a grade 2B recommendation [23]. This states that the role of plasma exchange in this condition is not truly established, and that the supporting evidence is considered weak and of moderate quality. Because plasma exchange carries more than minimal risk given the need for large caliber central venous access, exposure to blood products, and need for an extracorporeal circuit, a rational strategy for using plasma exchange as a therapeutic strategy for TAMOF should optimize the risk to benefit ratio.
A potential strategy for optimizing the risk to benefit ratio when employing higher risk experimental therapies is to stratify patients based on baseline mortality risk. This concept is known as prognostic enrichment [24]. By using a tool to estimate outcome risk, prognostic enrichment selects patients with a greater event rate. Because sample size for an event-based study is inversely proportional to effect size and directly proportional to event rate, prognostic enrichment can allow for a smaller sample size. Importantly, prognostic enrichment does not affect relative risk reduction, but will increase the absolute effect size of an experimental therapy.
A recent publication opined that severity scores such as Acute Physiology and Chronic Health Evaluation (APACHE) and PRISM should not be used as entry criteria for clinical trials and provided several reasons in support [25]. The publication further called for the development of biomarker-based stratification strategies as a means to enhance selection criteria for clinical trials. The utility of PERSEVERE as an enrichment strategy is consistent with this recommendation.
We tested the concept of prognostic enrichment in the design of a trial of plasma exchange for patients meeting TAMOF criteria, based on entry criteria from a recent observational study. We demonstrate that when eligible patients are stratified for baseline mortality risk using PERESEVERE-II, the sample size needed to demonstrate efficacy is substantially reduced when compared to no stratification. The simulation demonstrates how almost one third of patients with septic shock and TAMOF could theoretically be spared exposure to plasma exchange based on a reliable estimation of low baseline mortality risk with standard care.
We note three main limitations of our study. First, in the absence of pre-PICU admission platelet count data we considered thrombocytopenia to be of new onset when the study subject had an admission platelet count < 100,000/μl and did not have a comorbidity associated with thrombocytopenia. Conversely, in subjects with a comorbidity associated with thrombocytopenia, any thrombocytopenia event was considered related to the comorbidity, rather than being of new onset. This could have led to misclassification of some subjects. Second, the test cohort used for internal validation was a convenience sample representing subjects newly enrolled since the derivation and validation of PERSEVERE. We are in the process of enrolling a prospective cohort in which to validate the performance of PERSEVERE-II. Finally, because PERSEVERE is designed to assign a baseline mortality probability, we only considered admission platelet counts in the modeling process. This precludes analysis of how temporal changes in platelet counts are associated with changing risk. We have developed a temporal version of PERSEVERE and will pursue the opportunity to consider how changes in platelet counts reflect changing risk over time.
In conclusion, testing the accuracy of PERSEVERE in the context of organ failure phenotypes of septic shock prompted a revision of PERSEVERE incorporating admission platelet count information. PERSEVERE-II performs well upon testing, independent of TAMOF or MOF status. Tools such as PERSEVERE-II have the potential to provide prognostic enrichment for a trial of plasma exchange in children with septic shock and TAMOF.
Supplementary Material
The test cohort subjects (n = 236) were classified according to PERSEVERE-II without any modifications. The same conventions apply to the decision tree as described for Figure 1.
The PERSEVERE curve is shown in gray and the PERSEVERE-II curve is shown in black. For estimating the risk of 28-day mortality, the AUC for PERSEVERE-II (0.87; 95% CI 0.84 – 0.90) was superior to that of PERSEVERE (0.80; 95% CI 0.75 – 0.86; p = 0.031).
Acknowledgments
FUNDING SOURCE
Supported by National Institutes of Health Grants RO1GM064619, RO1GM099773, and R01GM108025. Supported in part by an Institutional Clinical and Translational Science Award, NIH/NCRR 8UL1 TR000077.
We thank the following clinical research coordinators for enrolling patients at the various study sites: Debra Spear, Jenny Bush, Mary Ann De Liberto, Trisha Williams, Amber Hughes, Michelle Goldsworthy, Christi Rider, Mary Ellen Riodan, Tiffany Patterson, Ofelia Vargas, Monica Weber, Lauren Hoadley, Heather Anthony, Lisa Steele, Angela Doucette, Katherine Woods, and Claudia Rodriquez Paez.
Footnotes
AUTHOR COMPETING INTERESTS
Dr. Wong and the Cincinnati Children’s Hospital Research Foundation have submitted a provisional patent application for PERSEVERE.
Dr. Lindsell is named as a co-inventor in the above patent application.
The other authors have no competing interests to report.
AUTHOR CONTRIBUTIONS
Hector R. Wong: Conceived and developed the study, obtained funding for the study, directly took part in the analyses, and wrote the manuscript.
Natalie Z. Cvijanovich, Nick Anas, Geoffrey L. Allen, Neal J. Thomas, Michael T. Bigham, Scott L. Weiss, Julie Fitzgerald, Paul A. Checchia, Keith Meyer, Michael Quasney, Mark Hall, Rainer Gedeit, Robert J. Freishtat, Jeffrey Nowak, Shekhar S. Raj, and Shira Gertz: Enrolled patients, provided biological samples and clinical data for the database, and edited the manuscript.
Kelli Howard and Erin Frank: Maintained the clinical database and coordinated all inter-institutional research activity.
Kelli Harmon: Maintained the biological repository and processed all biological samples.
Patrick Lahni: Conducted all biomarker assays.
Kimberly Hart: Assisted with statistical analysis.
Trung Nguyen: Assisted in the concept and design of the study.
Christopher J. Lindsell: Developed the study, assisted with analysis, and edited the manuscript.
Copyright form disclosures: Dr. Wong disclosed other support (Provisional patent applications for the PERSEVERE biomarkers) and received support for article research from the National Institutes of Health (NIH). His institution received funding from the NIH (RO1GM064619), NIH (RO1GM099773), and NIH (R01GM108025). Dr. Cvijanovich received support for article research from the NIH and received funding from REACH Air Medical Services (assistant medical director) Her institution received funding from the Cincinnati Children’s Hospital Medical Center (via NIH funding), Boston Children’s Hospital (via NIAID #1RO1A1084011-01), and Boston Children’s Hospital (via NHLBI 5U01HL107681-04). Dr. Allen disclosed other support (The overall study has been supported by an NIH grant and his institution has been compensated a fixed dollar amount for each patient enrolled in the study) and received support for article research from the NIH. His institution received funding from Cincinnati Children’s Hospital. Dr. Thomas received funding from CareFusion, Therabron, and the FDA. Dr. Weiss received funding from Thermo-Fisher Scientific. Dr. Fitzgerald received support for article research from the NIH. His institution received funding from the NIH. Dr. Checchia received support for article research from the NIH. Dr. Meyer received support for article research from the NIH. His institution received funding from the NIH. Dr. Quasney received support for article research from the NIH. Dr. Freishtat received support for article research from the NIH. Dr. Raj received support for article research from the NIH and disclosed other support (NIH funding was for Dr. Hector Wong. Through his funding he paid out institution for participating and enrolling the patient). His institution received funding from the Cincinnati Children’s Hospital. Dr. Gertz received support for article research from the NIH and received funding from Hackensack University Medical Center. Her institution received funding from GENOMIC ANALYSIS OF PEDIATRIC SYSTEMIC INFLAMMATORY RESPONSE SYNDROME/PERSEVERE/SEPTIC SHOCK (Genomics) NIH 5R01GM099773-02. Dr. Howard received support for article research from the NIH. Her institution received funding from the NIH. Dr. Lahni received support for article research from the NIH. His institution received funding from the NIH. Dr. Frank received support for article research from the NIH. Dr. Hart received support for article research from the NIH. Her institution received funding from the NIH. Dr. Nguyen received support for article research from the NIH. His institution received funding from the NIH NIGMS. Dr. Lindsell received support for article research from the NIH. His institution received funding from the NIH and other (Dr. Lindsell is named as co-inventor and has a patent describing PERSEVERE). The remaining authors have disclosed that they do not have any potential conflicts of interest.
References
- 1.Cohen J, Vincent JL, Adhikari NK, Machado FR, Angus DC, Calandra T, Jaton K, Giulieri S, Delaloye J, Opal S, et al. Sepsis: a roadmap for future research. Lancet Infect Dis. 2015;15(5):581–614. doi: 10.1016/S1473-3099(15)70112-X. [DOI] [PubMed] [Google Scholar]
- 2.Hanna W, Wong HR. Pediatric sepsis: challenges and adjunctive therapies. Crit Care Clin. 2013;29(2):203–222. doi: 10.1016/j.ccc.2012.11.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Knox DB, Lanspa MJ, Kuttler KG, Brewer SC, Brown SM. Phenotypic clusters within sepsis-associated multiple organ dysfunction syndrome. Intensive Care Med. 2015;41(5):814–822. doi: 10.1007/s00134-015-3764-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Wong HR, Cvijanovich NZ, Anas N, Allen GL, Thomas NJ, Bigham MT, Weiss SL, Fitzgerald J, Checchia PA, Meyer K, et al. Developing a clinically feasible personalized medicine approach to pediatric septic shock. Am J Respir Crit Care Med. 2015;191(3):309–315. doi: 10.1164/rccm.201410-1864OC. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Wong HR, Lindsell CJ, Pettila V, Meyer NJ, Thair SA, Karlsson S, Russell JA, Fjell CD, Boyd JH, Ruokonen E, et al. A multibiomarker-based outcome risk stratification model for adult septic shock. Crit Care Med. 2014;42(4):781–789. doi: 10.1097/CCM.0000000000000106. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Wong HR, Salisbury S, Xiao Q, Cvijanovich NZ, Hall M, Allen GL, Thomas NJ, Freishtat RJ, Anas N, Meyer K, et al. The pediatric sepsis biomarker risk model. Crit Care. 2012;16(5):R174. doi: 10.1186/cc11652. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Alder MN, Lindsell CJ, Wong HR. The pediatric sepsis biomarker risk model: potential implications for sepsis therapy and biology. Expert Rev Anti Infect Ther. 2014;12(7):809–816. doi: 10.1586/14787210.2014.912131. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Kaplan JM, Wong HR. Biomarker discovery and development in pediatric critical care medicine. Pediatr Crit Care Med. 2011;12(2):165–173. doi: 10.1097/PCC.0b013e3181e28876. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Wong HR, Weiss SL, Giuliano JS, Jr, Wainwright MS, Cvijanovich NZ, Thomas NJ, Allen GL, Anas N, Bigham MT, Hall M, et al. Testing the prognostic accuracy of the updated pediatric sepsis biomarker risk model. PLoS One. 2014;9(1):e86242. doi: 10.1371/journal.pone.0086242. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Nguyen TC, Han YY, Kiss JE, Hall MW, Hassett AC, Jaffe R, Orr RA, Janosky J, Carcillo JA. Intensive plasma exchange increases a disintegrin and metalloprotease with thrombospondin motifs-13 activity and reverses organ dysfunction in children with thrombocytopenia-associated multiple organ failure. Crit Care Med. 2008;36(10):2878–2887. doi: 10.1097/ccm.0b013e318186aa49. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Nguyen TC, Han YY, Fortenberry JD, Zhou Z, Cruz MA, Carcillo JA. Thrombocytopenia-associated multiple organ failure. In: Wheeler DS, Wong HR, Shanley TP, editors. Pediatric Critical Care Medicine: Basic Science and Clinical Practice. Vol. 3. New York: Springer; 2014. pp. 481–492. [Google Scholar]
- 12.Nguyen TC, Liu A, Liu L, Ball C, Choi H, May WS, Aboulfatova K, Bergeron AL, Dong JF. Acquired ADAMTS-13 deficiency in pediatric patients with severe sepsis. Haematologica. 2007;92(1):121–124. doi: 10.3324/haematol.10262. [DOI] [PubMed] [Google Scholar]
- 13.Nguyen TC, Carcillo JA. Understanding the role of von Willebrand factor and its cleaving protease ADAM TS13 in the pathophysiology of critical illness. Pediatr Crit Care Med. 2007;8(2):187–189. doi: 10.1097/01.CCM.0000257468.75474.D4. [DOI] [PubMed] [Google Scholar]
- 14.Goldstein B, Giroir B, Randolph A. International pediatric sepsis consensus conference: definitions for sepsis and organ dysfunction in pediatrics. Pediatr Crit Care Med. 2005;6(1):2–8. doi: 10.1097/01.PCC.0000149131.72248.E6. [DOI] [PubMed] [Google Scholar]
- 15.Wong HR, Shanley TP, Sakthivel B, Cvijanovich N, Lin R, Allen GL, Thomas NJ, Doctor A, Kalyanaraman M, Tofil NM, et al. Genome-level expression profiles in pediatric septic shock indicate a role for altered zinc homeostasis in poor outcome. Physiol Genomics. 2007;30(2):146–155. doi: 10.1152/physiolgenomics.00024.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Pollack MM, Patel KM, Ruttimann UE. The Pediatric Risk of Mortality III--Acute Physiology Score (PRISM III-APS): a method of assessing physiologic instability for pediatric intensive care unit patients. J Pediatr. 1997;131(4):575–581. doi: 10.1016/s0022-3476(97)70065-9. [DOI] [PubMed] [Google Scholar]
- 17. [Accessed September 1, 2015]; http://faculty.vassar.edu/lowry/VassarStats.html.
- 18.Che D, Liu Q, Rasheed K, Tao X. Decision tree and ensemble learning algorithms with their applications in bioinformatics. Adv Exp Med Biol. 2011;696:191–199. doi: 10.1007/978-1-4419-7046-6_19. [DOI] [PubMed] [Google Scholar]
- 19.Muller R, Mockel M. Logistic regression and CART in the analysis of multimarker studies. Clin Chim Acta. 2008;394(1–2):1–6. doi: 10.1016/j.cca.2008.04.007. [DOI] [PubMed] [Google Scholar]
- 20.Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983;148(3):839–843. doi: 10.1148/radiology.148.3.6878708. [DOI] [PubMed] [Google Scholar]
- 21.Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128–138. doi: 10.1097/EDE.0b013e3181c30fb2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. [Accessed September 1, 2015]; http://biostat.mc.vanderbilt.edu/wiki/Main/Hmisc.
- 23.Schwartz J, Winters JL, Padmanabhan A, Balogun RA, Delaney M, Linenberger ML, Szczepiorkowski ZM, Williams ME, Wu Y, Shaz BH. Guidelines on the use of therapeutic apheresis in clinical practice-evidence-based approach from the Writing Committee of the American Society for Apheresis: the sixth special issue. J Clin Apher. 2013;28(3):145–284. doi: 10.1002/jca.21276. [DOI] [PubMed] [Google Scholar]
- 24.Temple R. Enrichment of clinical study populations. Clin Pharmacol Ther. 2010;88(6):774–778. doi: 10.1038/clpt.2010.233. [DOI] [PubMed] [Google Scholar]
- 25.Vincent JL, Opal SM, Marshall JC. Ten reasons why we should NOT use severity scores as entry criteria for clinical trials or in our treatment decisions. Crit Care Med. 2010;38(1):283–287. doi: 10.1097/CCM.0b013e3181b785a2. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
The test cohort subjects (n = 236) were classified according to PERSEVERE-II without any modifications. The same conventions apply to the decision tree as described for Figure 1.
The PERSEVERE curve is shown in gray and the PERSEVERE-II curve is shown in black. For estimating the risk of 28-day mortality, the AUC for PERSEVERE-II (0.87; 95% CI 0.84 – 0.90) was superior to that of PERSEVERE (0.80; 95% CI 0.75 – 0.86; p = 0.031).