Internal evaluation of risk stratification tool using serial procalcitonin and clinical risk factors in pediatric febrile neutropenia: The non-interventional, single institution experience prior to clinical implementation

Abstract

Risk stratification of pediatric febrile neutropenia (FN) is an established concept, yet clinical risk tools misclassify nearly 5% of clinical standard-risk episodes with severe outcomes. the internal evaluation of a clinical risk tool before implementation has not been well-described. In this noninterventional cohort study, we evaluated a study decision rules (SDR) tool; a clinical risk tool with serial procalcitonin. the study standard-risk (SSR) group met clinical standard-risk criteria with two serial procalcitonin <0.4 ng/mL. the study high-risk (SHR) group met clinical high-risk criteria or clinical standard-risk with a procalcitonin ≥0.4 ng/mL. Descriptive and bivariate statistics compared the groups and outcomes. Clinical criteria alone identified 39.1% (238/608) standard-risk episodes; 5.9% (14/238) had severe events. prospectively using the SDR, the SHR group encompassed 76.6% (92/120) of episodes; severe events occurred in 20% (3/15) of standard-risk episodes included due to elevated procalcitonin ≥0.4 ng/mL. the SHR group had more blood stream infections [21.7% (20/92) vs. 0% (0/28); P = 0.007] and intensive care admissions [13% (12/92) vs. 3.6% (1/28); P = 0.158]. In conclusion, the SDR with serial procalcitonin aided in identifying severe events in clinical standard-risk episodes, but analysis was limited. Institutions may consider similar internal evaluation methodology before FN episode risk stratification.

Introduction

Febrile neutropenia (FN) is a frequent complication of myelosuppressive chemotherapy in pediatric cancer patients associated with blood stream infections and high mortality.^1,2 Heterogeneity between FN episodes was observed leading to the novel concept of risk stratification and descriptions of clinical factors associated with severe outcomes.³ Patients at standard or low-risk for severe outcomes can safely receive de-escalated treatment with oral antibiotics at home in either a step-down or upfront outpatient manner.^4–8 Guidelines from the Children’s Oncology Group (COG), Infectious Disease Society of America, and National Cancer Center Network recommend adoption and clinical incorporation of a validated clinical risk stratification tool to aid in the identification of low-risk and high-risk patients.^9–11 However, a clinical risk tool validated in a certain regional population may not have a similar performance in a different region and population, attributed to variations in supportive care and chemotherapy regimens, differences in local antibiogram, and subjective criterion required in some tools.^10,12,13 Current clinical risk tools validated in North America may misclassify 4–5% of standard-risk patients with severe events.^14–16

Given the difficulty in validating a clinical risk tool and misclassification incidence, serum inflammatory biomarkers allow for objective data which may help in identifying severe events.¹⁷ A review of over 8,300 pediatric FN episodes concluded that procalcitonin <0.5 ng/mL was the best performing serum biomarker early in the infection course; serial trending may further improve accuracy in screening for adverse events.^17–19 Few international groups have seen improved performance of clinical risk tool and success rates in step-down therapy are observed when inflammatory biomarkers are combined with a clinical risk factors, but a combined clinical risk tool with procalcitonin has not been evaluated.^19–21

Methods

Aim statement

The primary aim of this study was to describe our single institution experience of the noninterventional, internal evaluation of a clinical risk tool alone and in combination with serial procalcitonin as the study decision rules (SDR) tool prior to local implementation: an important, “required” step recommended by the International Fever and Neutropenia Guideline Panel and COG.¹⁰

Design and definitions

This cohort study utilized a composite retrospective and prospective design. We used our hospital electronic medical record (EPIC, MiChart) to generate a report of FN admissions for a 34-month window, then prospective nonintervention observation for a 12-month period. Included episodes met the standard definition of FN per Infectious Disease Society of America and National Comprehensive Cancer Network guidelines.^9,11 Episodes were excluded if they did not meet FN definition or those which occurred near stem cell engraftment to restrict the analytic group to the general oncology cohort. Episodes with incomplete serial procalcitonin data were excluded from the prospective cohort subgroup analysis.

A misclassification was defined as a severe adverse event in a standard-risk episode. Severe adverse events were defined as diagnosis of bacterial or fungal bloodstream infection (BSI), pediatric intensive care unit (PICU) admission, or infection related death. A bacterial or fungal bloodstream infection was defined using the Centers for Disease Control and Prevention’s National Healthcare Safety Network definition.²² This manuscript was prepared in accordance with the STROBE guidelines for cohort studies.

Risk stratification

Risk stratification was assigned based on initial presentation using a modified version of a clinical risk tool initially published by Alexander, et al. validated in similar populations and recommended by COG to assign episodes as clinical standard-risk or clinical high-risk.^10,14,21,23 Expanded high-risk clinical factors were added to reflect our institution’s local population and acuity similar to prior studies, such as age less than 1 year and diagnosis of Down Syndrome.¹⁶ The evaluated risk tool, termed study decision rules (SDR), is the combination of the modified clinical risk tool with two serial procalcitonin values, one at presentation and after one night of inpatient observation to aid in BSI screening¹⁹ [Table 1]. Serial procalcitonin was incorporated into routine lab assessments to avoid accessing the central venous line for extra occurrences. Additionally, we determined this lab schedule would be most feasible to incorporate into clinical practice, and this rationale was reflected in the internal assessment.

Table 1.

The study decision rules (SDR) tool combined the modified clinical risk tool recommended by COG with serial procalcitonin as an additional restrictive high-risk criterion to identify study standard-risk (SSR) episodes.

AGE ≥ 1 YEAR OLD AND NOT DOWN SYNDROME

SERIAL PROCALCITONIN <0.4 NG/ML

ANTICIPATED NEUTROPENIA OF < 7 DAYS, DEFINED AS: Initial outpatient presentation And No diagnosis of AML No diagnosis of non-Hodgkin lymphoma No HSCT <1 year No cellular therapy <6 months Leukemia in remission Leukemia not in induction No concern for relapse or disease progression with marrow involvement

NO SIGNIFICANT COMORBIDITY AT PRESENTATION, ABSENCE OF: Hypotension requiring ≥ 40 ml/kg fluid resuscitation ≤4 hours of initial presentation Tachypnea or sustained hypoxia (SPO2 < 92%) Altered mental status Concern for focal infection requiring iv antibiotics (tunnel infection, pneumonia, perirectal abscess, etc.) Bacteremia Mucositis requiring intravenous narcotics Refractory vomiting or moderate/severe abdominal pain Grade 2 organ dysfunction (acute kidney injury, jaundice, etc.)

NO OTHER REASON FOR INPATIENT OBSERVATION

NO SOCIAL CONCERN FOR MEDICAL ADHERENCE TO FOLLOW UP (LOCAL ED <30 MINUTES, ACCESS TO CELL PHONE)

ABILITY TO TOLERATE ORAL ANTIBIOTICS

Using the SDR prospectively, episodes were stratified in the study standard-risk (SSR) group if they met clinical standard-risk criteria and low serial procalcitonin <0.4 ng/mL; this group would be eligible for hypothetical early discharge on day 2 of illness. The episodes included in the study high-risk (SHR) group met clinical high-risk criteria or clinical standard-risk with any procalcitonin ≥0.4 ng/mL; this group would hypothetically receive standard-of-care treatment. In attempt to minimize stratification bias, risk stratification was performed at presentation prior to review of the outcome variables.

Statistical analysis

The retrospective analysis was performed to informally compare outcomes and the incidence of misclassifications at our institution to prior validation studies of the clinical risk tool. A prospective analysis was performed to evaluate outcomes of the SDR with the addition of serial procalcitonin [Figure 1]. Continuous demographics were summarized with medians and ranges, while categorical demographics were summarized with proportions. Differences between the retrospective FN and prospective FN groups were determined with a Wilcoxon Rank Sum test and chi-squared test of association for continuous and categorical characteristics, respectively. AUC values were computed from a logistic regression model. All analyses were done in the statistical software R, version 4.0.4, and statistical significance was defined as a p-value less than 0.05.

Figure 1.

Depicts analytical flow diagram according to STROBE guidelines.

*lndicates prospective febrile neutropenia episodes subjected to risk stratification by study decision rules (SDR)

Ethical and regulatory oversight

This study was reviewed and deemed exempt by the University of Michigan Institutional Review Board (IRB). Data and private health information were stored on IRB-approved secure hard-drives.

Results

Performance of the modified clinical risk tool without procalcitonin

The performance of the modified clinical risk tool using only the clinical risk criterion was analyzed in 608 FN episodes [Supplemental Table 1]. There were 460 FN episodes analyzed in the retrospective group and 148 FN episodes analyzed prospectively; 79.9% (486/608) presented with outpatient fevers. The modified clinical risk tool identified 60.9% (370/608) as clinical high-risk episodes encompassing most severe events; 19.7% (73/370) had a BSI and 19.7% required PICU admission (73/370). In 238 clinical standard-risk episodes, 4.6% (11/238) were diagnosed with a BSI (sensitivity = 86.9%; NPV = 95.4%; AUC = 0.63) and 1.3% (3/238) required PICU admission (sensitivity = 96.1%; NPV = 98.7%; AUC = 0.70) [Figure 2]. In the 13 clinical standard-risk episodes with severe outcomes, 4 met low-risk criteria after one night observation, 8 developed new high-risk clinical features, and 1 met high-risk criteria only due to positive blood culture growth.

Figure 2.

Icon array depicting febrile neutropenia episode outcomes based upon risk stratification by clinical risk criteria only. The left stacked grouping represents clinical high-risk episodes, while the right stacked grouping is clinical standard-risk episodes. In the clinical standard-risk episodes at our institution, 4.6% are diagnosed with blood stream infection and 1.3% with intensive care admission. Blue = pediatric intensive care admission. Red = blood stream infection. Gray = no severe outcome.

In the prospective group, there were significantly more clinical high-risk episodes (68.9% [102/148] vs. 58.3% [268/460]; P = 0.021). No differences were detected in BSI (15.5% [23/148] vs. 13.3% [61/460]; P = 0.485) or PICU admissions (14.2% [21/148] vs. 12% [55/460]; P = 0.475). The prospective group saw more non-AML oncologic diagnoses, but no other significant differences were observed in the demographics or type of CVL [Supplemental Table 2].

Prospective analysis of study decision rules (SDR) tool; modified clinical risk tool with procalcitonin

Prospective analysis of the SDR, the modified clinical risk tool with serial procalcitonin, was performed in 81% (120/148) of patients; median time from fever onset to the first and second procalcitonin draw was 2 and 16 hours, respectively. There were 76.6% (92/120) of episodes in the SHR group, and severe events occurred in 20% (3/15) of clinical standard-risk patients in this group only due to elevated procalcitonin ≥0.4 ng/mL. The SHR group had significantly more BSI events [21.7% (20/92) vs. 0% (0/28); P = 0.007; AUC = 0.76] and more PICU admissions [13% (12/92) vs. 3.6% (1/28); P = 0.158; AUC = 0.77] [Supplemental Table 3]. Additionally, the SHR group had significantly fewer short hospital stays <3 days [17.4% (16/92) vs. 60.7% (17/28); P = <0.001) and longer median hospital length-of-stay [7.5 days (2–129) vs. 3 days (2–25); P = <0.001). The SSR group had a single severe event in a patient diagnosed with Pneumocystis jiroveci pneumonia when transferred to the PICU on day 7 of admission. The number study standard-risk episodes needed-to-test (NNT) using serial procalcitonin to identify a severe event is 6.

The modified clinical risk tool alone would have misclassified 3 clinical standard-risk episodes diagnosed with BSI events, yet these all had elevated procalcitonin (0.26 ng/mL and 4.32 ng/mL; 0.35 ng/mL and 0.71 ng/mL; 0.43 ng/mL and 0.51 ng/mL). In all 3 cases, the patients developed interval new clinical high-risk criteria day 2 of the episode. A fourth CSR episode with elevated procalcitonin (21.2 ng/mL and 20.74 ng/mL) was read-mitted in 5 days due to a soft tissue abscess from Actinomyces requiring surgical intervention.

Risk stratification using only serial procalcitonin identified 50.8% (61/120) with procalcitonin ≥0.4 ng/mL. Three clinical high-risk episodes with BSI from methicillin resistant Staphylococcus aureus, Streptococcus mitis, and Rothia mucilaginosa had serial low procalcitonin <0.4 ng/mL. In total, there were 12 episodes (10%) with an initial procalcitonin <0.4 ng/mL resulting in a severe outcome: 10 BSI events and 2 PICU admissions.

Discussion

Febrile neutropenia guidelines recommend implementation of a validated clinical risk tool to identify low-risk patients, and this report describes the important stage of tool selection and internal evaluation necessary prior to clinical implementation [insert citation].¹⁰ The clinical risk tool identified 39.1% of patients as clinical standard-risk with a severe event rate close prior studies.^14,16 When selecting a clinical risk tool, institutions should consider a tool validated in a similar patient population before subjecting it to internal evaluation. It is not clear if we recorded fewer low-risk episodes in the prospective observation due to fluctuations in patient acuity, significantly less viral infections, or study design.

Serial procalcitonin ≥0.4 ng/mL seemed to improve the clinical risk tool discretion by aiding in the identification severe events in clinical standard-risk episodes. The clinical standard-risk patients included in the SHR group only due to the elevated procalcitonin had severe events occur in 20% (3/15), which represent prevented mis-classifications. Most severe events were appropriately observed in the SHR group as this group would continue to hypothetically receive standard-of-care treatment. Those patients eligible for hypothetical early discharge—the SSR group—encompassed 23% (28/120) of episodes and had a single severe event. Risk stratification using either the clinical risk tool or procalcitonin alone misclassified several standard-risk episodes with severe events. The AUC analysis of the various models showed the study decision rules (SDR) had improved discretion compared to clinical risk tool alone, but statistical analysis was limited and prevented further conclusions.

The implementation of risk stratification in FN patients represented a significant change in practice for our center. Even though the modified clinical risk tool alone performed close to the validation studies, our institution has increased confidence in implementing risk stratification using the combined study decision rules (SDR) as the additional serial procalcitonin data appears to aid in detecting severe events in clinical standard-risk episodes. Our institution will perform ongoing evaluation of the SDR during implementation as a prospective quality improvement project.

There were limitations to our study. First, although we attempted to control for bias, episodes were subjected to stratification bias due to the retrospective design. Second, due to the low incidence of severe events in the clinical standard-risk population, our statistical AUC analysis was limited in the comparison of various risk models. Future studies could consider a multicenter prospective study which would allow for sufficient power for the statistical analysis. Third, the study design was intended to evaluate risk stratification to yield hypothetical results, therefore our group will perform continued analysis of the tool after clinical implementation as the hypothetical results may not directly translate into clinical practice. The addition of serial procalcitonin may prove beneficial as an objective data point, but this effect may not be appreciated until clinical implementation.

Conclusion

Internal evaluation of a clinical risk tool is an important step recommended by COG to evaluate both performance in the local population and to provide foundation for safe implementation when changing clinical practice.¹⁰ The SDR performed well and improved discretion with addition of serial procalcitonin at our institution. This internal evaluation was not only an imperative step in determining the tool’s clinical discretion in our institution, but also a highly regarded necessary evaluation in changing our clinical practice. Institutions may consider a similar methodological approach in the internal evaluation of risk stratification at their institution prior to the clinical implementation.