Abstract
Background
Although community-acquired pneumonia (CAP) is one of the most common infections in children, no tools exist to risk stratify children with suspected CAP. We developed and validated a prediction model to risk stratify and inform hospitalization decisions in children with suspected CAP.
Methods
We performed a prospective cohort study of children aged 3 months to 18 years with suspected CAP in a pediatric emergency department. Primary outcome was disease severity, defined as mild (discharge home or hospitalization for <24 hours with no oxygen or intravenous [IV] fluids), moderate (hospitalization <24 hours with oxygen or IV fluids, or hospitalization >24 hours), or severe (intensive care unit stay for >24 hours, septic shock, vasoactive agents, positive-pressure ventilation, chest drainage, extracorporeal membrane oxygenation, or death). Ordinal logistic regression and bootstrapped backwards selection were used to derive and internally validate our model.
Results
Of 1128 children, 371 (32.9%) developed moderate disease and 48 (4.3%) severe disease. Severity models demonstrated excellent discrimination (optimism-corrected c-indices of 0.81) and outstanding calibration. Severity predictors in the final model included respiratory rate, systolic blood pressure, oxygenation, retractions, capillary refill, atelectasis or pneumonia on chest radiograph, and pleural effusion.
Conclusions
We derived and internally validated a score that accurately predicts disease severity in children with suspected CAP. Once externally validated, this score has potential to facilitate management decisions by providing individualized risk estimates that can be used in conjunction with clinical judgment to improve the care of children with suspected CAP.
Keywords: pneumonia, children, prognosis, clinical prediction models, risk stratification
In this prospective study of 1128 children with suspected CAP, we developed a well-calibrated severity score of 7 variables (respiratory rate, blood pressure, oxygenation, retractions, capillary refill, radiographic findings, pleural effusion) with excellent discriminatory ability for moderate or severe disease.
Community-acquired pneumonia (CAP) ranks among the top 5 most prevalent and costly reasons for pediatric hospitalization in the United States [1]. Hospitalization also poses substantial, and potentially unnecessary, burdens on children, families, and the healthcare system, making the decision to hospitalize a child “the most important decision in the management of CAP” [2]. Hospital admission rates for pediatric CAP vary widely by region and hospital, even after adjustment for illness severity and geographic differences, suggesting that hospitalization criteria are not consistent across providers or institutions [3, 4].
Accurately assessing and predicting disease severity is critical to decision-making, yet management decisions in children with CAP are currently based on nonspecific examination findings, radiographic images, and conventional laboratory markers that do not reliably assess disease risk [5]. Several validated severity scores exist for adults with CAP, which reduce hospitalizations and broad-spectrum antibiotic use in low-risk patients [6, 7]. National guidelines for childhood CAP extrapolate illness severity criteria from adult guidelines [8]. These criteria were not derived in children and, thus, do not account for the distinct features of pediatric CAP. More than 50% of children classified with severe disease warranting continuous monitoring or intensive care using these criteria were discharged home from the emergency department (ED) and did not return, demonstrating only fair ability to predict disease severity [9]. The lack of prognostic tools results in increased risk, burden, and cost for those hospitalized unnecessarily, and potential delays in care for those at higher risk [4]. As such, the Pediatric Infectious Diseases Society (PIDS)/Infectious Diseases Society of America (IDSA) pediatric pneumonia guideline emphasizes defining risk factors for objective clinical outcomes as a key area for future research [8].
The goal of this study was to develop an accurate and precise prognostic model to estimate the risk of severe disease in a prospective cohort of children evaluated in the ED for suspected CAP.
METHODS
Study Design and Population
This was a prospective cohort study of children aged 3 months to 18 years who presented to a single, urban, tertiary-care pediatric ED from July 2013 to December 2017 with “suspected CAP,” defined as having signs and symptoms of lower respiratory tract infection (LRTI) and receiving a chest radiograph (CXR) for clinical suspicion of pneumonia [10]. We excluded children hospitalized ≤14 days before the study ED visit, as well as those with histories of aspiration/aspiration pneumonia or immunocompromising or chronic medical conditions that predispose to severe or recurrent pneumonia (eg, immunodeficiency, chronic corticosteroid use, chronic lung disease, malignancy, sickle cell disease, congenital heart disease, tracheostomy-dependent patients, neuromuscular disorders) (see Supplementary Materials). The study was approved by the Cincinnati Children’s Hospital Medical Center Institutional Review Board. Written informed consent was obtained from legal guardians and assent was obtained from children ≥11 years old. Details of all study procedures have been described previously [5, 11].
Outcome Measures
The main goal of these prediction models was to facilitate disposition decision-making in the ED; thus, the primary outcome was a 3-tiered composite ordinal outcome focused on acute clinical outcomes in children with suspected CAP across the disease severity spectrum. Outcome measures were determined by medical record review and telephone calls with the parent or patient approximately 1 week after discharge from the ED or hospital.
The outcome, derived from prior literature and expert consensus, classified children as developing mild, moderate, or severe illness as follows [12, 13]:
1. Mild illness: Discharge home from the ED or hospitalization for <24 hours with no inpatient use of supplemental oxygen or intravenous (IV) hydration.
2. Moderate illness: Hospitalization lasting >24 hours, or hospitalization <24 hours with supplemental oxygen use or IV hydration.
3. Severe illness: Requiring intensive care unit (ICU) admission for >24 hours, having a diagnosis of severe sepsis/septic shock, receipt of vasoactive infusions, positive-pressure ventilation (ie, continuous positive airway pressure, bilevel positive airway pressure, or intubation with mechanical ventilation), chest drainage procedures for empyema, extracorporeal membrane oxygenation, or death.
To investigate variation and the complexities of disposition decision-making, we developed a secondary model including all hospitalized patients with moderate disease and all ICU admissions regardless of duration, along with the other severe criteria listed above, as having severe disease, to represent the clinical decision to hospitalize.
Predictor Variables
Predictors under consideration had to be clinically and biologically plausible, available at the bedside, and have postulated association with disease severity [14]. After developing an initial list by literature review and expert consensus [12, 13], we excluded binary predictors that were present <5% or >95% of the time (eg, cough, apnea), were overly subjective (eg, decreased oral intake at home), or displayed multiple collinearity. The final list of candidate predictor variables included demographics, illness symptoms, vital signs, physical examination findings, and radiographic features (Table 1). We did not include laboratory findings, as these are often not obtained nor routinely recommended in children with suspected CAP, and require venipuncture [8]. To evaluate both oxygen saturation and oxygen requirement, oxygenation was reported using the oxygen saturation (SpO2)/fraction of inspired oxygen (FiO2) ratio, which is a noninvasive proxy for partial pressure of oxygen/FiO2 (PF ratio) [15, 16]. A lower PF ratio represents decreased SpO2 or increased FiO2 need. Two radiologists independently categorized CXRs as having no atelectasis/pneumonia, favoring atelectasis, atelectasis vs pneumonia, or favoring pneumonia. These radiologists agreed on 68% of initial reviews. Disagreements were finalized by a consensus meeting of the radiologists and principal investigator [5]. All predictor variables were assessed at initial ED presentation prior to occurrence of the outcomes.
Table 1.
Cohort Characteristics and Disease Severity
| Variable | Overall (N = 1128) |
Mild (n = 709) |
Moderate (n = 371) |
Severe (n = 48) |
|---|---|---|---|---|
| Demographics/medical history | ||||
| Age, y, median (IQR) | 3.3 (1.5–7.1) | 3.4 (1.6–7.1) | 3.2 (1.3–6.7) | 3.2 (1.1–8.6) |
| Male sex | 615 (55) | 399 (56) | 193 (52) | 23 (48) |
| Race/ethnicity (n = 1122) | ||||
| Non-Hispanic white | 689 (61) | 412 (58) | 243 (65) | 34 (71) |
| Non-Hispanic black | 332 (29) | 234 (33) | 91 (25) | 7 (15) |
| Hispanic | 34 (3) | 13 (2) | 18 (5) | 3 (6) |
| Other | 67 (6) | 46 (6) | 17 (5) | 4 (8) |
| Smoke exposure | 475 (42) | 304 (43) | 151 (41) | 20 (42) |
| Past medical history | ||||
| Pneumonia | 250 (22) | 147 (21) | 89 (24) | 14 (29) |
| Prior pneumonia hospitalization | 100 (40) | 56 (38) | 37 (42) | 7 (50) |
| Asthma | 364 (32) | 232 (33) | 114 (31) | 18 (38) |
| Prematurity | 200 (18) | 120 (17) | 67 (18) | 13 (27) |
| Season | ||||
| Winter | 404 (36) | 248 (35) | 138 (37) | 18 (38) |
| Spring | 245 (22) | 151 (21) | 82 (22) | 12 (25) |
| Summer | 155 (14) | 106 (15) | 44 (12) | 5 (10) |
| Fall | 324 (29) | 204 (29) | 107 (29) | 13 (27) |
| Illness symptoms/history of present illness | ||||
| Symptom duration, d, median (IQR) | 2 (1–4) | 2 (1–4) | 3 (1–5) | 2 (1–4) |
| Fever at home | 996 (87) | 616 (87) | 327 (88) | 43 (90) |
| Fever duration, d, median (IQR) | 4 (2–7) | 5 (2–7) | 4 (2–7) | 3.5 (2–6) |
| Decreased oral intake | 702 (62) | 415 (59) | 257 (69) | 30 (62) |
| Difficulty breathing | 916 (81) | 547 (77) | 325 (88) | 44 (92) |
| Rapid breathing | 835 (74) | 483 (68) | 310 (84) | 42 (88) |
| Wheezing | 724 (64) | 446 (63) | 247 (67) | 31 (65) |
| Vomiting | 577 (51) | 346 (49) | 207 (56) | 24 (50) |
| Abdominal pain | 360 (32) | 233 (33) | 116 (31) | 11 (23) |
| Physical examination | ||||
| Temperature (n = 1127), median (IQR) | 37.8 (37.2–38.7) | 37.8 (37.1–38.6) | 38 (37.4–38.8) | 37.9 (37.4–39.3) |
| Respiratory rate, median (IQR) | 42 (30–55.2) | 39 (28–48) | 49 (37–60) | 52 (44–66) |
| Heart rate (n = 1127), median (IQR) | 152 (130–168) | 144 (124–162) | 160 (140–174) | 168 (149.5–186.5) |
| SBP (n = 1019), median (IQR) | 107 (98–116) | 110 (100–118) | 105 (96–114) | 99.5 (89.5–109.5) |
| PF ratioa (n = 1034), median (IQR) | 452.4 (350–461.9) | 457.1 (447.6–466.7) | 413.4 (276–452.4) | 339.3 (140–447.6) |
| Retractions (n = 1097) | 476 (43) | 217 (31) | 225 (62) | 34 (72) |
| Grunting (n = 1095) | 76 (7) | 25 (4) | 41 (11) | 10 (21) |
| Nasal flaring (n = 1091) | 124 (11) | 45 (7) | 68 (19) | 11 (23) |
| Crackles (n = 1095) | 342 (31) | 189 (27) | 138 (38) | 15 (32) |
| Rhonchi (n = 1096) | 385 (32) | 221 (32) | 143 (40) | 21 (45) |
| Wheezing (n = 1097) | 325 (30) | 187 (27) | 116 (32) | 22 (47) |
| Decreased breath sounds (n = 1096) | ||||
| Not decreased | 719 (66) | 490 (71) | 208 (58) | 21 (45) |
| Focally decreased | 256 (23) | 139 (20) | 100 (28) | 17 (36) |
| Diffusely decreased | 121 (11) | 59 (9) | 53 (15) | 9 (19) |
| Abnormal skin color (n = 1096) | 92 (8) | 30 (4) | 53 (15) | 9 (19) |
| Capillary refill ≥3 sec (n = 1084) | 144 (13) | 51 (8) | 78 (22) | 15 (33) |
| Chest radiography findings | ||||
| Radiologist interpretation | ||||
| No atelectasis or pneumonia | 383 (34) | 312 (44) | 62 (17) | 9 (19) |
| Favoring atelectasis | 493 (44) | 266 (38) | 206 (56) | 21 (44) |
| Atelectasis vs pneumonia | 50 (4) | 26 (4) | 21 (6) | 3 (6) |
| Favoring pneumonia | 202 (18) | 105 (15) | 82 (22) | 15 (31) |
| Radiographic infiltrate pattern | ||||
| None | 383 (34) | 312 (44) | 62 (17) | 9 (19) |
| Unilateral and single lobe | 351 (31) | 201 (28) | 134 (36) | 16 (33) |
| Unilateral and multifocal | 52 (5) | 24 (3) | 25 (7) | 3 (6) |
| Bilateral/multifocal | 342 (30) | 172 (24) | 150 (40) | 20 (42) |
| Hyperinflation | 276 (24) | 137 (19) | 125 (34) | 14 (29) |
| Pleural effusion | 83 (7) | 28 (4) | 45 (12) | 10 (21) |
| Airways (ie, peribronchial) involvement | 601 (53) | 343 (48) | 232 (63) | 26 (54) |
Data are presented as No. (%) unless otherwise indicated.
Abbreviations: IQR, interquartile range; PF, partial pressure of oxygen/fraction of inspired oxygen; SBP, systolic blood pressure.
aTo evaluate both oxygen saturation and oxygen requirement, oxygenation was reported using the oxygen saturation/fraction of inspired oxygen (FiO2) ratio, which is a noninvasive proxy for partial pressure of oxygen/FiO2 [15–17].
Statistical Analysis
Model Development
Prediction models were developed using ordinal logistic regression. Missing values were imputed via the chained equations algorithm and model-derived estimates pooled across multiply imputed data sets [17, 18]. Simple ordinal regression models were fit for each predictor to evaluate individual associations. A full model was then fit including all eligible predictors [14]. Continuous predictors were included under restricted cubic spline transformations to permit modeling of nonlinear associations [19]. To account for age-based differences in heart rate, respiratory rate, and systolic blood pressure, we transformed these variables to sample-based z scores in 4 age groups (0–1, 2–4, 5–9, ≥10 years; Supplementary Figure 1). Given the large number of potential predictors, we fit the full model under penalized likelihood, in which nonlinear terms were penalized 5 times more heavily than linear terms. We examined the relative importance of each predictor to the full model through partial χ 2 statistics, which represent the fraction of explainable variation in severity for each variable. We reported the effect of each individual predictor on severity through their estimated adjusted proportional odds ratio.
To allow for a more parsimonious model, we developed a reduced model through a bootstrap backwards selection approach. We created 1000 bootstrap datasets, fit the full model in each, and selected predictors via backwards selection using the Akaike information criterion (AIC). Variables appearing in at least 50% of all bootstrap dataset-fit models were included in a reduced model that was then fit to the original data set.
Model Performance and Internal Validation
We evaluated and compared the performance of the full and reduced models by AIC and the concordance index (c-index). The c-index evaluates the ability of the model to discriminate children who develop moderate or severe outcomes from those who do not (c-index of 1 means perfect discrimination). We graphically assessed calibration by plotting observed vs predicted probabilities of mild, moderate, and severe CAP. An intercept of zero and a calibration slope of 1 is considered perfect calibration.
We assessed the possible performance of our models in future populations through internal bootstrap validation, from which we calculated optimism-adjusted estimates of the c-index and the intercept and slope from graphs of observed vs predicted probabilities of moderate or severe disease. All analyses were conducted in the open-source R software environment with the add-on package “rms” [19, 20].
RESULTS
Study Population
The study population included 1128 children (Figure 1), of whom 709 (62.9%) developed mild illness, 371 (32.9%) moderate illness, and 48 (4.3%) severe illness. The median age was 3.3 years (interquartile range, 1.4–7.1); 622 (54%) were boys; and 697 (61%) were non-Hispanic white (Table 1).
Figure 1.
Study flow diagram. Abbreviations: CARPE DIEM, Catalyzing Ambulatory Research in Pneumonia Etiology and Diagnostic Innovations in Emergency Medicine; ED, emergency department.
Model Development
Full Model
When all potential predictors were included in a multivariable ordinal logistic regression model, the following factors were associated with more severe outcomes (Table 2, Supplementary Table 1): white race, prior pneumonia hospitalization, retractions, grunting, prolonged capillary refill time, elevated respiratory rate, decreased systolic blood pressure, decreased PF ratio, atelectasis or pneumonia on CXR, unilateral and multifocal CXR findings, pleural effusion, and hyperinflation. These variables accounted for 55% of model outcome variation (Supplementary Figure 2). Secondary analysis examining the decision to hospitalize yielded similar results (Supplementary Table 2).
Table 2.
Multivariable Proportional Odds Logistic Regression for Disease Severity (Full Model)
| Potential Predictor | Adjusted pOR (95% CI)a |
|---|---|
| Age, y | |
| 1 | 1.28 (.89–1.85) |
| 2 | 1.19 (.91–1.56) |
| 5 | Ref. |
| 10 | 1.04 (.82–1.32) |
| Male sex | 0.79 (.59–1.06) |
| Nonwhite race | 0.66 (.48–.92)c |
| Smoke exposure | 0.85 (.63–1.14) |
| Prior pneumonia | |
| None | Ref. |
| Not hospitalized | 1.17 (.77–1.78) |
| Hospitalized | 1.71 (1.03–2.84)c |
| Asthma | 0.90 (.64–1.27) |
| Prematurity | 1.15 (.79–1.66) |
| Season | |
| Winter | Ref. |
| Spring | 1.24 (.84–1.83) |
| Summer | 0.80 (.50–1.29) |
| Fall | 1.05 (.73–1.53) |
| Days of illness | |
| 1 | 0.97 (.77–1.22) |
| 3 | 0.99 (.88–1.10) |
| 5 | Ref. |
| 10 | 0.99 (.82–1.20) |
| Fever at home | 1.01 (.55–1.85) |
| Decreased oral intake | 1.21 (.89–1.65) |
| Difficulty breathing | 1.09 (.68–1.76) |
| Rapid breathing | 1.07 (.70–1.64) |
| Wheezing at home | 0.76 (.54–1.07) |
| Vomiting | 1.14 (.85–1.53) |
| Abdominal pain | 1.36 (.81–2.37) |
| Temperature, ºC | |
| 37 | Ref. |
| 38 | 0.95 (.71–1.28) |
| 39 | 0.93 (.61–1.41) |
| 40 | 0.96 (.56–1.66) |
| RR (50th vs 95th percentile) | 2.43 (1.50–3.93)c |
| HR (50th vs 95th percentile) | 1.61 (.88–2.96) |
| SBP (50th vs 5th percentile) | 2.08 (1.23–3.51)c |
| PF ratiob | |
| 450 | Ref. |
| 400 | 2.52 (1.89–3.38)c |
| 300 | 3.66 (2.38–5.61)c |
| 200 | 2.84 (1.87–4.31)c |
| Retractions | 1.95 (1.32–2.87)c |
| Grunting | 1.77 (1.05–2.98)c |
| Nasal flaring | 1.27 (.80–2.00) |
| Crackles | 0.91 (.66–1.27) |
| Rhonchi | 1.20 (.87–1.66) |
| Wheezing | 0.80 (.54–1.19) |
| Decreased breath sounds | |
| Not decreased | Ref. |
| Focally decreased | 1.18 (.82–1.69) |
| Diffusely decreased | 1.30 (.82–2.06) |
| Abnormal skin color | 1.16 (.70–1.90) |
| Capillary refill ≥3 sec | 1.99 (1.30–3.03)c |
| Chest radiography findings | |
| No atelectasis or pneumonia | Ref. |
| Favoring atelectasis | 1.90 (1.28–2.80)c |
| Atelectasis vs pneumonia | 2.25 (1.12–4.52)c |
| Favoring pneumonia | 2.53 (1.54–4.17)c |
| Laterality/lobar involvement | |
| None | Ref. |
| Unilateral and single lobe | 2.02 (1.26–3.15)c |
| Unilateral and multifocal | 2.51 (1.29–4.75)c |
| Bilateral/multifocal | 2.08 (1.42–3.05)c |
| Pleural effusion | 2.35 (1.37–4.04)c |
| Hyperinflation | 1.55 (1.08–2.23)c |
| Airways (ie, peribronchial) involvement | 0.85 (.60–1.22) |
| Model performance | |
| C-index | 0.84 (.818–.862) |
| C-index (bootstrapped) | 0.812 |
| AIC | 1423.8 |
| Observed/predicted intercept | –0.06 |
| Observed/predicted slope | 0.88 |
Abbreviations: AIC, Akaike information criterion; CI, confidence interval; HR, heart rate; PF, partial pressure of oxygen/fraction of inspired oxygen; pOR, proportional odds ratio; RR, respiratory rate; SBP, systolic blood pressure.
aProportional odds ratios are interpreted as the odds of developing moderate or severe disease compared to the referent group. For example, a child with radiographic pneumonia had 2.53 times the odds of developing moderate or severe disease compared with those with no atelectasis or pneumonia on chest radiography.
bTo evaluate both oxygen saturation and oxygen requirement, oxygenation was reported using the oxygen saturation/fraction of inspired oxygen (FiO2) ratio, which is a noninvasive proxy for partial pressure of oxygen/FiO2 [15–17].
c P < .05.
Reduced Model
The reduced model included the following 7 predictors: elevated respiratory rate, decreased systolic blood pressure, decreased PF ratio, retractions, prolonged capillary refill, atelectasis or pneumonia on CXR, and pleural effusion (Table 3). Given similar performance and fit as the full model, with improved calibration and parsimony, this reduced model is our final model for disease severity. Secondary analysis examining the decision to hospitalize yielded similar results, with the addition of elevated heart rate and grunting as predictors (Supplementary Table 3).
Table 3.
Final Reduced Clinical Prediction Model for Disease Severity
| Predictor | Adjusted pOR (95% CI)a |
|---|---|
| Respiratory rate (50th vs 95th percentile) | 2.53 (1.63–3.93) |
| Systolic blood pressure (50th vs 5th percentile) | 2.20 (1.34–3.61) |
| PF ratiob | |
| 450 | Ref. |
| 400 | 2.56 (1.94–3.38) |
| 300 | 3.33 (2.22–4.49) |
| 200 | 2.28 (1.59–3.28) |
| Retractions | 2.06 (1.48–2.87) |
| Capillary refill ≥3 sec | 2.47 (1.69–3.61) |
| Chest radiography findings | |
| No atelectasis or pneumonia | Ref. |
| Favoring atelectasis | 1.94 (1.36–2.75) |
| Atelectasis vs pneumonia | 2.33 (1.21–4.49) |
| Favoring pneumonia | 2.79 (1.77–4.41) |
| Pleural effusion | 2.34 (1.39–3.92) |
| Model performance | |
| C-index | 0.82 (.796–.844) |
| C-index (bootstrapped) | 0.81 |
| AIC | 1422.9 |
| Observed/predicted intercept | –0.01 |
| Observed/predicted slope | 0.93 |
Abbreviations: AIC, Akaike information criterion; CI, confidence interval; PF, partial pressure of oxygen/fraction of inspired oxygen; pOR, proportional odds ratio.
aProportional odds ratios are interpreted as the odds of developing moderate or severe disease compared to the referent group. For example, a child with radiographic pneumonia had 2.79 times the odds of developing moderate or severe disease compared with those with no atelectasis or pneumonia on chest radiography.
bTo evaluate both oxygen saturation and oxygen requirement, oxygenation was reported using the oxygen saturation/fraction of inspired oxygen (FiO2) ratio, which is a noninvasive proxy for partial pressure of oxygen/FiO2 [15–17].
Nomogram
The nomogram of the final reduced model is presented in Figure 2. Points from individual variables are added and a vertical line is drawn from the total points line at the bottom downward to determine the predicted probability of moderate/severe and severe disease (Supplementary Figure 3). The nomogram offers a visual representation of the severity score allowing for calculation along the entire range of predicted probabilities; for a tabular representation of our model, a score chart is provided in Supplementary Table 4.
Figure 2.
Pediatric community-acquired pneumonia severity score reduced-model nomogram for disease severity. To calculate total score and predicted probability of moderate or severe disease, points from individual variables are added and a vertical line is drawn from the total points line at the bottom downward to determine the predicted probability of moderate/severe and severe disease. Abbreviations: PF, partial pressure of oxygen/fraction of inspired oxygen; PNA, pneumonia; RR, respiratory rate; SBP, systolic blood pressure.
Model Performance
Both the full and reduced models accurately predicted the risk of moderate or severe CAP, with optimism-corrected, bootstrapped c-indices of 0.81. The c-indices of both bootstrapped validation models were similar to the original, dataset-derived indices. Calibration curves (Figure 3) demonstrate strong agreement between observed and expected risk. The decision to hospitalize models had c-indices of 0.87 for the full model and 0.85 for the reduced model, with excellent calibration. Estimated regression coefficients from these models correlated with the primary models with correlation coefficients of 0.93 and 0.95 for the full and reduced models, respectively.
Figure 3.
Calibration plot for pediatric community-acquired pneumonia severity score final reduced model. Curves illustrate the proportion of children who experienced a mild, moderate, or severe outcome (vertical axis) at each level of predicted risk of moderate or severe disease (horizontal axis). The dotted line indicates perfect calibration with a slope of 1. Abbreviation: PNA, pneumonia.
Supplementary Table 5 illustrates the performance of the reduced model at various cut-points of predicted probability. A predicted risk cutoff of ≤10%–20% identifies a group at low risk of developing moderate or severe disease, with a sensitivity of 91.4%–97.1% and negative likelihood ratio of 0.11–0.17. A total of 397 (35.2%) children fell below the 10%–20% threshold. Of the 34 patients with moderate and 2 with severe disease with ≤20% predicted probability of moderate or severe disease (ie, low risk), 3 received supplemental oxygen and 10 received IV hydration. The 2 with severe disease were considered severe due to ICU stays of 26 and 44 hours. No patient classified as low risk by our model required positive-pressure ventilation or vasoactive medications, developed sepsis, or died.
Discussion
We prospectively derived and internally validated a clinical prediction model that accurately risk stratifies children with suspected CAP for disease severity in the ED setting. Our models showed excellent discriminatory ability with c-indices of 0.81 or greater and outstanding calibration for discriminating children with mild disease (who do not require hospitalization) from those with moderate or severe disease (who do). To our knowledge, no severity clinical prediction models exist for children presenting to the ED with CAP in a well-resourced country. Once externally validated and refined in a multicenter setting, these models are poised to serve as an evidence-based adjunct to clinical judgment to improve management decisions for children presenting with suspected CAP.
Several prognostic models have been developed in low-middle-income countries including the Respiratory Index of Severity in Children (RISC) in South Africa and the Pediatric Etiology Research for Child Health (PERCH) scores [21, 22]. These scores were derived to predict mortality in countries where pneumonia mortality rates are high, and the major comorbidities include malnutrition, human immunodeficiency virus, and malaria, making these scores less applicable in higher-resourced countries. In the United States, 3 prognostic models were developed within a single cohort of approximately 2000 hospitalized children with CAP as part of the Etiology of Pneumonia in the Community (EPIC) study [23]. The 2 reduced models in the EPIC study consisted of 9–10 variables, many similar to variables in our models, with bootstrapped c-indices of 0.76–0.77. The patients in the EPIC study were already hospitalized, precluding extrapolation to the ED where the spectrum of disease severity is wider.
Our models differ from and add to prior models in several important ways. First, we conducted this prospective study in the ED, the care setting where disposition decisions are most frequently made, and therefore prognostic models are most relevant. Our bootstrapped c-indices suggest improved discrimination over prior models [22, 23]. Second, in our outcome definition, we included children discharged from the ED, as well as those hospitalized, making our cohort generalizable when predicting risk at the time of presentation for care. Third, we included children with clinically suspected CAP, regardless of CXR findings, an approach that mirrors both quotidian clinical practice and the PIDS/IDSA guideline recommendations, where the CXR acts as diagnostic adjunct to a clinical diagnosis [8]. Additionally, the presence of atelectasis and pneumonia on radiograph were both associated with increased severity, highlighting the prognostic utility of CXR in the ED management of suspected CAP, even in cases where the CXR favors atelectasis. Atelectasis has been found to be associated with an increase in severity in children with bronchiolitis [24]. Our results suggest a similar phenomenon in suspected CAP.
Our risk models included variables easily obtained during an ED visit. Other than vital signs, our final model consists of only 2 physical examination findings. Given the limited reliability of many respiratory examination findings, this maximizes the ability for our model to be replicated across multiple sites and providers [11]. We did not include laboratory testing, as many children with suspected CAP do not routinely require blood tests. In fact, the PIDS/IDSA guideline recommends against routine measurement of a complete blood count or acute phase reactants [8]. Prior prediction models for disease severity in hospitalized children with CAP found that white blood cell count was not associated with severity [23]. Studies in adults suggest that blood biomarkers, including procalcitonin, improve prognostic performance of prediction models for CAP [25, 26]. Further study is warranted to investigate if blood biomarkers similarly improve prognostic models in children.
Our results suggest that there are 3 disease severity risk groups—low, intermediate, and high—that were defined from our models. A predicted risk of ≤20% indicates low risk of developing moderate or severe disease, and therefore low likelihood of requiring hospitalization. Of the 36 children with moderate or severe disease and a predicted risk of ≤20%, none required invasive respiratory or cardiovascular support or died. With a specificity of 92.7% and a positive likelihood ratio of 6.51, a predicted risk of >60% indicates high likelihood of moderate or severe outcomes. As with all clinical prediction models, external validation, ideally in a multicenter setting, is required to understand the generalizability of our findings and the stability of these risk thresholds across populations.
In prediction model development, there is a risk of overfitting, or producing model results specific to the population in which it was derived, and therefore resulting in poorly generalizable models [14]. We minimized overfitting, however, through several established methods, including use of penalized models and optimism-corrected c-indices [14]. In addition, results remained stable in separate models examining the decision to hospitalize. Prospective external validation and model refinement will be important to confirm, and potentially improve, model performance in new populations and account for the addition of newly available factors, such as point-of-care biomarkers, that may enhance risk stratification.
Implementation of clinical prediction models can be challenging, as demonstrated by mixed results of implementing adult CAP severity scores [27, 28]. We developed a reduced model that includes substantially fewer variables than the full model, but with similar fit, discriminatory ability, and improved calibration. This reduced model consists of 7 variables that are readily available in the electronic health record or are standard factors ascertained during routine ED evaluation. We present a nomogram based on model regression coefficients as one easy means of model use, in addition to a score chart. Regression coefficients can also be directly used in an online calculator or clinical decision support tool. Given the features of our score, we anticipate that use of computerized tools, such as electronic clinical decision support, will be the ideal method of implementation. Such tools have been implemented successfully in the ED for various conditions, including minor head trauma, appendicitis, and adult CAP [29–31].
Our study has several limitations. First, we included a convenience sample from a single children’s hospital. Although our study, which spanned multiple respiratory seasons, was internally valid, external validation with a new cohort is required. Second, CAP is a heterogeneous diagnosis that can be challenging due to overlapping clinical features across different etiologies and respiratory conditions. As our intention was to develop a prognostic model that was practical for use in the ED where diagnostic evaluation and definitions vary, we developed models in children where symptoms and signs of LRTI were present and a radiograph was ordered because CAP was clinically suspected. Third, our outcome included objective markers of disease severity and interventions supporting need for hospitalization. Some factors, however, may be institution-specific, have a subjective component, or are the result of factors unrelated to severity. When we performed secondary analyses examining the decision to hospitalize, the results were similar. Finally, we did not include children with comorbidities, which may affect generalizability. We attempted to minimize the heterogeneity introduced by including children with different comorbidities, each of which may have a different pathophysiological basis for pneumonia.
In conclusion, we prospectively derived well-calibrated prediction models that accurately identify and discriminate moderate and severe outcomes in children with suspected CAP who present to the ED. Our reduced model includes 7 variables that are readily available at the bedside and can be integrated into electronic clinical decision support tools. After external validation and implementation, this model can facilitate management decisions by providing individualized risk estimates that can be used with clinical judgment to improve the care of children with suspected CAP.
Supplementary Data
Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.
Notes
Acknowledgments. The authors acknowledge Judd Jacobs and Jessi Lipscomb for their role in data management for the Catalyzing Ambulatory Research in Pneumonia Etiology and Diagnostic Innovations in Emergency Medicine (CARPE DIEM) study. Mantosh Rattan, MD and Eric Crotty, MD, from the Department of Radiology at Cincinnati Children’s Hospital Medical Center, reviewed and interpreted all chest radiographs. The authors are grateful to the entire research team and patient services staff in the Divisions of Emergency Medicine and Hospital Medicine at Cincinnati Children’s Hospital Medical Center for their assistance with study procedures. Finally, the authors are especially grateful to the patients and families who enrolled in the CARPE DIEM study.
Disclaimer. The funders did not have any role in the study design, data collection, statistical analysis, or manuscript preparation.
Financial support. This work was supported by the National Institute of Allergy and Infectious Diseases of the National Institutes of Health (NIH) (grant numbers K23AI121325 to T. A. F. and K01AI125413 to L. A.); the Gerber Foundation (to T. A. F.); the NIH National Center for Research Resources; and the Cincinnati Center for Clinical and Translational Science and Training (grant number 5KL2TR000078 to T. A. F.).
Potential conflicts of interest. The authors: No reported conflicts of interest. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest.
References
- 1. Keren R, Luan X, Localio R, et al. . Pediatric Research in Inpatient Settings (PRIS) Network . Prioritization of comparative effectiveness research topics in hospital pediatrics. Arch Pediatr Adolesc Med 2012; 166:1155–64. [DOI] [PubMed] [Google Scholar]
- 2. Harris M, Clark J, Coote N, et al. . British Thoracic Society guidelines for the management of community acquired pneumonia in children: update 2011. Thorax 2011; 66(Suppl 2): ii1–23. [DOI] [PubMed] [Google Scholar]
- 3. Gorton CP, Jones JL. Wide geographic variation between Pennsylvania counties in the population rates of hospital admissions for pneumonia among children with and without comorbid chronic conditions. Pediatrics 2006; 117:176–80. [DOI] [PubMed] [Google Scholar]
- 4. Florin TA, French B, Zorc JJ, Alpern ER, Shah SS. Variation in emergency department diagnostic testing and disposition outcomes in pneumonia. Pediatrics 2013; 132:237–44. [DOI] [PubMed] [Google Scholar]
- 5. Florin TA, Ambroggio L, Brokamp C, et al. . Biomarkers and disease severity in children with community-acquired pneumonia. Pediatrics 2020; 145:e20193728. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Fine MJ, Auble TE, Yealy DM, et al. . A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med 1997; 336:243–50. [DOI] [PubMed] [Google Scholar]
- 7. Lim WS, van der Eerden MM, Laing R, et al. . Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax 2003; 58:377–82. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Bradley JS, Byington CL, Shah SS, et al. . Pediatric Infectious Diseases Society and the Infectious Diseases Society of America . The management of community-acquired pneumonia in infants and children older than 3 months of age: clinical practice guidelines by the Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. Clin Infect Dis 2011; 53:e25–76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Florin TA, Brokamp C, Mantyla R, et al. . Validation of the Pediatric Infectious Diseases Society–Infectious Diseases Society of America Severity Criteria in Children With Community-Acquired Pneumonia. Clin Infect Dis 2018; 67:112–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Jain S, Williams DJ, Arnold SR, et al. . CDC EPIC Study Team . Community-acquired pneumonia requiring hospitalization among U.S. children. N Engl J Med 2015; 372:835–45. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Florin TA, Ambroggio L, Brokamp C, et al. . Reliability of examination findings in suspected community-acquired pneumonia. Pediatrics 2017; 140:e20170310. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Dean P, Florin TA. Factors associated with pneumonia severity in children: a systematic review. J Pediatric Infect Dis Soc 2018; 7:323–34. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Dean P, Schumacher D, Florin TA. Defining pneumonia severity in children: a delphi study [manuscript published online ahead of print 15 September 2020]. Pediatr Emerg Care 2020. doi: 10.1097/PEC.0000000000002088 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. New York: Springer, 2009. [Google Scholar]
- 15. Lobete C, Medina A, Rey C, Mayordomo-Colunga J, Concha A, Menéndez S. Correlation of oxygen saturation as measured by pulse oximetry/fraction of inspired oxygen ratio with PaO2/fraction of inspired oxygen ratio in a heterogeneous sample of critically ill children. J Crit Care 2013; 28:538.e1–7. [DOI] [PubMed] [Google Scholar]
- 16. Khemani RG, Patel NR, Bart RD 3rd, Newth CJL. Comparison of the pulse oximetric saturation/fraction of inspired oxygen ratio and the PaO2/fraction of inspired oxygen ratio in children. Chest 2009; 135:662–8. [DOI] [PubMed] [Google Scholar]
- 17. Van Buuren S, Brand JPL, Groothuis-Oudshoorn CGM, Rubin DB. Fully conditional specification in multivariate imputation. J Stat Comput Simul 2006; 76:1049–64. [Google Scholar]
- 18. van Buuren S, Groothuis-Oudshoorn K. MICE: multivariate imputation by chained equations in R. J Stat Softw 2011; 45:1–67. [Google Scholar]
- 19. Harrell FE Jr. Regression modeling strategies, with applications to linear models, logistic and ordinal regression, and survival analysis. 2nd ed. New York: Springer, 2015. [Google Scholar]
- 20. R Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing, 2020. [Google Scholar]
- 21. Reed C, Madhi SA, Klugman KP, et al. . Development of the Respiratory Index of Severity in Children (RISC) score among young children with respiratory infections in South Africa. PLoS One 2012; 7:e27793. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Gallagher KE, Knoll MD, Prosperi C, et al. . The predictive performance of a pneumonia severity score in human immunodeficiency virus-negative children presenting to hospital in 7 low- and middle-income countries. Clin Infect Dis 2020; 70:1050–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Williams DJ, Zhu Y, Grijalva CG, et al. . Predicting severe pneumonia outcomes in children. Pediatrics 2016; 138:e20161019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Shaw KN, Bell LM, Sherman NH. Outpatient assessment of infants with bronchiolitis. Am J Dis Child 1991; 145:151–5. [DOI] [PubMed] [Google Scholar]
- 25. Menéndez R, Martínez R, Reyes S, et al. . Biomarkers improve mortality prediction by prognostic scales in community-acquired pneumonia. Thorax 2009; 64:587–91. [DOI] [PubMed] [Google Scholar]
- 26. Torres A, Ramirez P, Montull B, Menéndez R. Biomarkers and community-acquired pneumonia: tailoring management with biological data. Semin Respir Crit Care Med 2012; 33:266–71. [DOI] [PubMed] [Google Scholar]
- 27. Aujesky D, McCausland JB, Whittle J, Obrosky DS, Yealy DM, Fine MJ. Reasons why emergency department providers do not rely on the pneumonia severity index to determine the initial site of treatment for patients with pneumonia. Clin Infect Dis 2009; 49:e100–8. [DOI] [PubMed] [Google Scholar]
- 28. Julián-Jiménez A, Palomo de los Reyes MJ, Parejo Miguez R, Laín-Terés N, Cuena-Boy R, Lozano-Ancín A. Improved management of community-acquired pneumonia in the emergency department. Arch Bronconeumol 2013; 49:230–40. [DOI] [PubMed] [Google Scholar]
- 29. Kharbanda AB, Madhok M, Krause E, et al. . Implementation of electronic clinical decision support for pediatric appendicitis. Pediatrics 2016; 137:e20151745. [DOI] [PubMed] [Google Scholar]
- 30. Jones BE, Collingridge DS, Vines CG, et al. . CDS in a learning health care system: identifying physicians’ reasons for rejection of best-practice recommendations in pneumonia through computerized clinical decision support. Appl Clin Inform 2019; 10:1–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Dayan PS, Ballard DW, Tham E, et al. . Use of traumatic brain injury prediction rules with clinical decision support. Pediatrics 2017; 139:e20162709. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.