Failure to Validate a Multivariable Clinical Prediction Model to Identify Pediatric Intensive Care Unit Patients at High Risk for Candidemia

Abstract

We attempted to validate a previously derived clinical prediction rule for candidemia in the pediatric intensive care unit. This multicenter case control study did not identify significant association of candidemia with most of the previously identified predictors. Additional study in larger cohorts with other predictor variables is needed.

Candida species are the most common cause of invasive fungal disease in children. These infections are associated with a significant increase in length of stay and hospital charges and have an estimated attributable mortality of 10% [1]. Candidemia disproportionately affects critically ill children. Most pediatric candidemia episodes occur in patients admitted to a pediatric intensive care unit (PICU) [1] with resultant higher mortality rates [2,3]. Establishing measures to identify PICU patients at higher risk for candidemia would allow for interventions such as antifungal prophylaxis or early initiation of empiric therapy to reduce the morbidity and mortality in high-risk children.

Clinical prediction models for identifying invasive candidiasis have been developed and validated in intensive care settings in adults [4] but not in children. In 2010, Zaoutis et al [5] developed a clinical prediction rule for candidemia among PICU patients using a case-control design with incidence density sampling. The single-center study identified central venous catheter (CVC), recent exposure to certain antimicrobial agents, recent exposure to parenteral nutrition (PN), and underlying malignancy as factors independently associated with candidemia. Presence of a CVC was the strongest risk factor (odds ratio [OR], 30.4; 95% confidence interval [CI], 7.7–119.5). Inclusion of these factors into a clinical prediction model raised the predictive probability for candidemia to 46% (95% CI, 19–75) [5]. This result was encouraging, because it suggested that this rule could be leveraged to identify PICU patients at highest risk for candidemia and thus direct interventions to reduce this risk. In this retrospective, multicenter study, we aimed to validate this previously developed clinical prediction rule [5] for candidemia among children in the PICU.

METHODS

Study Design, Patient Population, and Outcome

Because candidemia is a relatively rare event in the PICU, a case-control design using patients from 6 pediatric institutions was performed to estimate the association of candidate risk factors and outcome. This validation study was conducted using the same study design and methods as the derivation study [5]. Study sites included 5 medical centers in the United States and 1 in Greece: The Children's Hospital of Philadelphia, All Children's Hospital, University of Colorado, Children's Hospital of Los Angeles, Baylor College of Medicine, and University of Thessaloniki. One of the US centers (Site 1) was also the center from which the prediction model was derived. Cases included patients less than 19 years of age, admitted to a PICU, and diagnosed with candidemia between January 1, 2005 and December 31, 2009. In patients with multiple episodes of candidemia, only the first episode was considered. Candidemia was defined as having at least 1 positive blood culture for 1 or moreCandida species. Cases were identified retrospectively and matched when possible to 2 controls from patients at the same institution using incidence density sampling [6]. Controls were randomly selected without replacement from the roster of patients admitted to the PICU at the same institution as the case. A control patient had to have a PICU length of stay at least as long as the time the case was in the PICU before diagnosis of candidemia. Individuals who eventually became cases were eligible to serve as controls until they were diagnosed with candidemia.

Collection of Demographic Data and Predictor Variables

At each institution, data were abstracted for subjects from the medical record directly into a structured data collection form using Research Electronic Data Capture (REDCap). Data were abstracted for only the predictors of candidemia that were identified from the previously derived prediction rule [5]: malignancy, presence of any CVC within 1 week before candidemia onset, any PN exposure within 1 week of candidemia onset, vancomycin exposure on more than 3 days in the 2 weeks before candidemia onset, and exposure to any antibiotics with antianaerobic activity on more than 3 days in the 2 weeks before candidemia onset.

Statistical Analysis

The frequency of the predictor variables was reported for cases and controls by each study site. Logistic regression conditioned on the matched sets was performed first to investigate whether candidemia was associated with each predictor variable. Subsequently, a multivariate conditional logistic regression model including all 5 predictor variables was planned to establish the predicted probability of candidemia for this current validation study. Data analyses were performed using Stata statistical software, version 12.0 (College Station, TX).

RESULTS

During the study period, 97 children less than 19 years of age were diagnosed with candidemia. One case was dropped from the analysis because the selected controls did not have complete data for the full exposure window. Of the remaining 96 cases of candidemia, 94 were successfully matched to 2 controls and 2 cases were matched to only 1 control each for a study size of 286 patients (96 cases and 190 controls). Patients with candidemia had a median age of 4 years (range, 43 days–18 years) and 55% were male; controls were of similar age (median, 3 years; range, 15 days–18 years) and gender (53%). Median PICU length of stay was 23 days (interquartile range [IQR], 12–47.5) and was slightly longer for controls (controls: median 24, IQR 12–49; cases: median 20.5, IQR 12–43).

Substantial variation was noted across the 6 participating sites in exposure to each of the 5 predictor variables among cases and controls (Table1). In addition, there was variation in the frequency of exposure to the predictive variables at site 1 when comparing data from the 2 different time periods (Table1). Only presence of a CVC was associated with subsequent candidemia in univariate analysis (OR, 9.6; 95% CI, 3.4–27.3). Given the lack of association with candidemia for 4 of the 5 predictor variables in bivariate analyses, and because of the variation of frequency of these predictors across institutions, a multivariate predictive model was not performed.

Table 1.

Distribution of Clinical Predictors of Candidemia by Site Compared With Data From the Derived Prediction Rule

	n (%)
	Site 1 Prior Data [5]	Site 1	Site 2	Site 3	Site 4	Site 5	Site 6
	Cases = 101 Controls = 184	Cases = 28 Controls = 56	Cases = 19 Controls = 38	Cases = 8 Controls = 15	Cases = 13 Controls = 25	Cases = 8 Controls = 16	Cases = 20 Controls = 40
Malignancy
Case	17 (17)	1 (4)	4 (21)	0 (0)	3 (23)	2 (25)	4 (20)
Control	12 (7)	11 (20)	10 (26)	3 (20)	1 (4)	1 (6)	8 (20)
Predictors within 1 week of study entry
Presence of CVC
Case	93 (92)	26 (93)	19 (100)	8 (100)	9 (69)	8 (100)	19 (95)
Control	104 (57)	24 (43)	32 (84)	14 (93)	16 (64)	9 (56)	33 (83)
Receipt of PN*
Case	69 (68)	17 (61)	16 (84)	8 (100)	6 (46)	8 (100)	4 (21)
Control	61 (33)	7 (13)	15 (39)	13 (87)	19 (76)	3 (19)	5 (13)
Predictors within 15 days of study entry
Receipt of antianaerobic antibiotic(s)
Case	49 (49)	13 (46)	15 (79)	1 (13)	3 (23)	6 (75)	18 (90)
Control	62 (34)	16 (29)	19 (50)	4 (27)	4 (16)	2 (13)	34 (85)
Receipt of vancomycin
Case	59 (49)	16 (57)	11 (58)	3 (38)	5 (38)	5 (63)	3 (15)
Control	61 (33)	23 (41)	29 (76)	7 (47)	4 (16)	2 (13)	11 (28)

Abbreviations: CVC, central venous catheter; PN, parenteral nutrition.

*Includes both total and peripheral PN.

DISCUSSION

Previous derivation of a clinical prediction rule from a single institution resulted in a 46% predictive probability of candidemia for children admitted to the PICU [5]. This result represented an opportunity for application in both clinical and research efforts. For example, the prediction rule could assist clinicians in targeting high-risk patients for antifungal prophylaxis. Alternatively, a prediction rule could aid an epidemiologist investigating the impact of targeted antifungal prophylaxis or empiric therapy. However, only 1 of the predictors from the prior study was significantly associated with candidemia using a new set of PICU patients with candidemia and incidence density selected controls from 6 different centers, and thus we were unable to validate the rule.

A number of possible explanations exist for our failure to validate the prediction rule. First, the identified patient-level predictors from the derived model might truly vary across study sites and within the same hospital across different time epics. Two of the predictors, recent exposure to vancomycin and antianaerobic antibiotics, varied substantially across institutions. Variation in antibiotic exposures is not surprising because utilization has been shown to vary between hospitals even within homogeneous patient populations [7,8]. Presence of a CVC did not appear to vary across institutions, but at site 1, CVC placement decreased between the derived and validated time periods, a finding consistent with the recent efforts to reduce CVC-associated bloodstream infections. Second, institutions may have varied practices that result in differences in the frequency of a patient-level factor or how it is managed. It is interesting to note that there was variation in the frequency of underlying malignancy among cases and controls across institutions. This variation may be related to the hospital's supportive care measures for children with cancer, which could alter the number of children with malignancy that need intensive care at any given time. Finally, our failure to validate the clinical prediction rule may be a function of an insufficient sample size to establish stable estimates of predictors that would translate into reproducible results.

The challenge of validating a clinical prediction rule is not unique to pediatric candidemia. Clinical prediction rules for invasive candidiasis have been derived in multicenter adult intensive care unit (ICU) cohorts that were at least in part dependent on patient level predictors [4,9]. The initial derivation studies were able to leverage a composite of clinical factors to identify a subset of patients with an increased incidence of candidemia. However, in an attempt to validate these rules at a single center, Hermsen et al [10] discovered that a number of the clinical factors from the 2 derived models were not significantly associated with invasive candidiasis in their validation cohort. Instead, the authors adapted the derived rules to include factors that were more consistent with the experience of patients at their hospital.

CONCLUSIONS

Our findings do not mean that the pursuit for early identification of candidemia should be abandoned. It is possible that increasing the sample size and number of candidemia events as well as collecting more information on patterns of care would allow for more stable estimates of predictor variables and refinement of the clinical prediction rule. Furthermore, because there is marked heterogeneity in the ICU patient population, prediction rules specific to patient age or underlying condition may be warranted. In addition, in an effort to establish a generalizable prediction rule, it may be prudent to include predictors that are less vulnerable to variation in processes of care across institutions and time periods. Fungal biomarkers that can directly or indirectly identify invasive candidiasis may represent a key component to a prediction rule that accurately identifies patients with candidemia as early as possible. Notably, a prediction rule that relies in part on fungal biomarker results would have different implications regarding clinical management. For instance, a prediction model that was inclusive of fungal biomarkers would not be leveraged for targeted prophylaxis, but it could be useful in a diagnostic-guided early antifungal treatment approach.