Validation of the British Thoracic Society Severity Criteria for Pediatric Community-acquired Pneumonia

Abstract

Background

The British Thoracic Society (BTS) guideline for pediatric community-acquired pneumonia (CAP) outlines severity criteria to guide clinical decision-making. Our objective was to examine the predictive performance of the criteria on the need for hospitalization (NFH) and disposition.

Methods

This was a retrospective cohort study of children 3 months–18 years of age diagnosed with CAP in an urban, pediatric emergency department (ED) in the United States from September 2014 to August 2015. Children with chronic medical conditions, recent ED visits, and ED transfers were excluded. The main outcomes were interventions or diagnoses that necessitate hospitalization (ie, NFH) and disposition (eg, admit vs. discharge). Test characteristics, stratified by age, were calculated for each outcome.

Results

Of 518 eligible children, 56.6% (n = 293) were discharged from the ED with 372 children meeting at least 1 BTS criterion. Overall BTS criteria were specific but not sensitive for NFH nor for disposition. For children <1 year of age sensitive criteria included not feeding and temperature for NFH and tachycardia, cyanosis and not feeding for disposition. For children ≥1 year of age, tachycardia had a sensitivity of >0.60 for both outcomes. The areas under the receiver operator characteristic curves for predicting any BTS criteria was 0.57 for NFH and 0.84 for disposition.

Conclusions

The BTS CAP severity criteria had fair to excellent ability to predict NFH and disposition, respectively. Although specific, the low sensitivity and poor discriminatory ability for NFH of these criteria suggest a need for improved prognostic tools for children with CAP.

Community-acquired pneumonia (CAP) is diagnosed in 52 million children younger than 5 years each year in Europe; the incidence of CAP decreases with older age.^1,2 Although mortality has decreased in the developed world, significant morbidity continues to exist despite widespread vaccination and antibiotic therapy.² Challenges in predicting severe courses of CAP result in nonstandardized care and overutilization of resources, including hospitalization.^3,4

The British Thoracic Society (BTS) published a revised guideline for pediatric CAP in 2011. The guideline stated that the disposition decision was the “most important decision in the management of CAP.”² Despite the importance of this decision, there are currently no validated prediction rules and prior studies of severity criteria in national guidelines have found limited performance.⁵ In addition, while a clinician’s decision to admit is based on the illness severity, it is also influenced by other system, family and provider characteristics, including the provider’s interaction with the parents or caregivers. Social factors or parental anxiety, which are not commonly documented in the electronic health record (EHR), may influence a provider to admit a child they may not otherwise admit based on physiologic factors alone. Thus, diagnoses (eg, empyema) or interventions (eg, chest drainage procedures) that can, for the most part, only occur in the inpatient setting may be more objective indicators for the need for hospitalization (NFH), as opposed to the clinician’s decision to admit the child.⁵

The BTS guideline provides age-specific criteria for mild to moderate and severe disease (see Table, Supplemental Digital Content 1, http://links.lww.com/INF/D490).² The severity criteria, which include hypoxia, fever, tachypnea and tachycardia, are intended to inform decisions regarding hospital admission and potentially, admission to the intensive care unit (ICU). These criteria have not been validated, and therefore the objective of this study was to examine the performance of the BTS severity criteria in predicting the decision to admit and NFH in previously healthy children with pneumonia.

METHODS

Study Design and Population

Children, 3 months–18 years of age, who were diagnosed with CAP in the emergency department (ED) at Cincinnati Children’s Hospital Medical Center (CCHMC), located in Cincinnati, Ohio, United States between September 1, 2014 and August 31, 2015 were eligible for inclusion in this retrospective cohort study. This study was approved by the CCHMC Institutional Review Board.

A previously validated algorithm based on International Classification of Diseases-9-Clinical Modification (ICD-9 CM) codes (ie, 480.0–2, 480.8–9, 481, 482.0, 482.30–2, 482.41–2, 482.83, 482.89–90, 483.8, 484.3, 485, 486, 487.0; in the primary or any secondary position) indicating a diagnosis of pneumonia⁶ and a provider diagnosis of pneumonia ascertained by manual EHR review were used to define the cohort. Children with ICD-9-CM codes indicating a chronic complex condition (CCC),⁷ or those with documented chronic neuromuscular, cardiovascular or pulmonary disease, sickle cell disease, immunodeficiency/immunosuppression, malignancy, or genetic/metabolic disorders, as ascertained by manual EHR review were excluded. Children transferred to the CCHMC from another institution or with an ED visit or hospitalization 14 days before the study visit were excluded to limit inclusion of children with hospital-acquired infection.

The cohort was developed using a 2-stage process to minimize misclassification of diagnosis assignment. First, the EHR (Epic, Verona, WA) was queried using the validated ICD-9-CM codes for pneumonia in any diagnosis position to identify the initial cohort (n = 1113). These codes often do not differentiate bacterial versus viral pneumonia as in most cases of children diagnosed with pneumonia etiology is not known. From this cohort, after manual chart review children were excluded due to the documentation of a CCC (n = 318). During the second stage, the charts of the remaining children (n = 795) were manually reviewed to exclude children that did not have a provider diagnosis of pneumonia (n = 57) and to exclude any child that did not meet inclusion/exclusion criteria (n = 254). A child may have had more than 1 exclusion criteria thus a total of 595 children were excluded from the original Epic extraction, leaving an analytic dataset of 518 children (see Figure, Supplemental Digital Content 2, http://links.lww.com/INF/D491). Two trained abstractors (R.M. and B.D.) manually reviewed the charts of all eligible children using a coding manual to obtain the remaining data used in the analysis.

Data Collection

To minimize potential bias in retrospective chart reviews we followed previously established methods.⁸ Data were entered by the abstractors into a standardized case report form in Research Electronic Data Capture, a secure, web-based application that is IRB compliant.⁹ The abstractors had no knowledge of the objective of this study. After the initial training sessions, 5% of the charts were jointly reviewed to ensure consistency in data abstraction. Inconsistencies were addressed at weekly coding meetings.

The primary predictor variable was the BTS severity criteria stratified by age. Infants were defined as children less than 1 year and older children were ≥1 year of age (see Table, Supplemental Digital Content 1, http://links.lww.com/INF/D490). All criteria were assessed at the time of presentation in the ED. The BTS criteria do not specifically indicate the number of criteria that needed to be met to indicate severe disease; therefore, each individual criterion was assessed in relation to the outcomes. Cyanosis was considered present if either documented in the EHR or if the child was considered hypoxic (oxygen saturation <92% in the ED). Tachycardia was adjusted for age and temperature according to a previously published percentiles developed for hospitalized children.¹⁰ Respiratory rate, temperature and oxygen saturation were extracted from the EHR as the first recorded vital sign, minimum and maximum. These represent measurements objectively taken by clinicians and documented in the EHR. The maximum temperature and respiratory rate and the minimum oxygen saturation measure were used for the analysis. All other criteria were considered present on examination if the provider documented the criterion in the EHR for the child. A criterion that was specifically not documented in the EHR was therefore considered not present in the child during the ED examination.¹¹

There were 2 primary outcomes examined in this study. The first primary outcome, NFH, represents a more objective outcome of items directly related to the child’s physiologic state (Table 1). NFH was developed using a combination of published literature and expert opinion from clinicians in pediatrics–infectious diseases, hospital medicine, critical care and emergency medicine.^12,13 All components of the NFH outcome occurred in the inpatient setting (ie, after leaving the ED). Three main categories describe the components included in the NFH outcome: (1) Interventions directly linked to physiologic parameters (eg, supplemental oxygen therapy with documented hypoxia); (2) Interventions that could only occur in the inpatient setting (eg, chest drainage procedures); and (3) Interventions that indicated disease progression (eg, broadening of antibiotic therapy). The second primary outcome, was the decision by the provider to admit the child to the hospital, as assessed by the child’s actual disposition (ie, admitted or discharged).

TABLE 1.

NFH Criteria⁵

Interventions	Diagnosis
2+ IV fluid boluses in 4-hour period	Empyema
Continuous IV fluids ≥24 hours	Lung necrosis/abscess
Supplemental oxygen (O2 < <90%)	Pneumothorax
Narrow to broad spectrum antibiotics	Bronchopleural fistula
Respiratory support	Hemolytic-uremic syndrome
Vasoactive infusions	Sepsis
Extracorporeal membrane oxygenation (ECMO)	Death
Cardiopulmonary resuscitation (CPR)
Chest drainage procedure

Statistical Analysis

Continuous variables were described using mean and SD and categorical variables were described using count and percentages. Severity criteria and outcomes were treated as binary variables and the sensitivity, specificity and likelihood ratios were calculated based on two-by-two tables.¹⁴ Individual BTS criteria that were not met by any children in the cohort were omitted from further calculations. Moderate to large influence on pretest probability of each outcome was defined as a positive likelihood ratio of ≥5 or negative likelihood ratio of ≤0.2.¹⁵ Using the number of total severity criteria met by each child as a continuous predictor, we generated receiver-operator characteristic curves (ROC) for each outcome and the area under the curve (AUC) was calculated. Risk ratios and their corresponding confidence intervals for each outcome and predictor were calculated using Wald normal approximation.¹⁶ All statistical analysis was completed using R version 3.4.3.

RESULTS

Study Population

Of the 518 children included, the mean age was 57.7 months (SD: 49.5), 50.4% Caucasian, 57.7% had public insurance and 46.9% were female. Only 10% of children enrolled had a history of prior pneumonia and 22% had a history of asthma or documentation of prior wheezing history in the EHR. About half (56.6%, n = 293) of the eligible children were discharged from the ED with 2.1% (n = 11) revisiting the ED within 72 hours. Of the 225 hospitalized children, 76.9% (n = 173) had a length of stay of less than 24 hours and 11.6% (n = 26) children were admitted to the ICU during their stay.

Most children (71.8%, n = 372) met one or more BTS criterion. In addition, most children (63%) who met at least 1 criterion were discharged home from the ED (see Figure, Supplemental Digital Content 3, http://links.lww.com/INF/D492). Two or more criteria were present in 47% of children (n = 244), with 34% of these children being discharged home and 9% requiring admission to the ICU. In addition, of the 146 children who did not meet any BTS criteria, 18 (12%) were admitted to the hospital and 1 child was admitted to the ICU.

Need for Hospitalization

The severity criteria associated with increased the NFH for children ≥1 year of age was severe difficulty breathing. None of the severity criteria increased the risk for NFH in the younger children <1 year of age (Fig. 1). In addition, no criteria were predictive of NFH based on likelihood ratios (Table 2). The severity criteria poorly discriminated who needed hospitalization versus those who did not with an AUC of 0.57 (Fig. 2A).

FIGURE 1.

Risk ratios of BTS criteria categorized by age.

TABLE 2.

Test Characteristics for NFH by the Individual BTS Criteria

	NFH (n = 102), n (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	LR+	LR−
Children <1 year	n = 12
Temperature >38.5C	8 (66.7)	0.67	0.33	0.67	0.33	1	1
Respiratory rate >70	4 (33.3)	0.33	0.67	0.67	0.33	1	1
Moderate to severe recessions*	0	—	—	—	—	—	—
Nasal flaring	4 (33.3)	0.33	0.83	0.8	0.38	2	0.8
Cyanosis	12 (100)	1.0	—	0.67	—	1.0	—
Intermittent apnea*	0	—	—	—	—	—	—
Grunting	2 (16.7)	0.17	0.67	0.5	0.29	0.5	1.25
Not feeding	10 (83.3)	0.83		0.62		0.83
Cap refill ≥2 seconds	2 (16.7)	0.18	0.83	0.67	0.36	1.09	0.98
Tachycardia	9 (75.0)	0.75	0.17	0.64	0.25	0.90	1.50
Children ≥1 year	n = 90
Temperature >38.5C	34 (37.8)	0.38	0.76	0.74	0.41	1.61	0.81
Respiratory rate >50	52 (57.8)	0.58	0.53	0.68	0.42	1.23	0.8
Severe difficulty breathing	1 (1.1)	0.01	1	1	0.36		0.99
Nasal flaring	17 (18.9)	0.19	0.86	0.71	0.38	1.38	0.94
Cyanosis	61 (67.8)	0.68	0.22	0.60	0.28	0.86	1.49
Grunting	7 (7.8)	0.08	0.9	0.58	0.36	0.79	1.02
Dehydration signs	12 (13.3)	0.13	0.9	0.71	0.37	1.36	0.96
Cap refill ≥2 seconds	8 (8.9)	0.09	0.94	0.73	0.36	1.47	0.97
Tachycardia	76 (84.4)	0.84	0.24	0.66	0.46	1.10	0.66

^*We had zero patients with this finding therefore we did not compute characteristics for this finding.

FIGURE 2.

Discrimination (shown as AUC curves) of BTS criteria for NFH (A) and admission (B).

The most sensitive criterion for children <1 year of age in predicting NFH was “not feeding well” (0.83) and tachycardia (0.75). The most specific criteria included nasal flaring and capillary refill ≥2 seconds (Table 2). Only tachycardia for children ≥1 year of age had high sensitivity (0.84); however, multiple criteria including temperature >38.5°C, nasal flaring, grunting, dehydration signs, and capillary refill ≥2 seconds had a high specificity for predicting NFH (Table 2).

Disposition

Clinicians were more likely to decide to hospitalize children <1 year of age that had a respiratory rate >70 were grunting, or had a capillary refill ≥2 seconds (Fig. 1). However, older children had a statistically higher risk of being admitted to the hospital if they presented with any criteria except for a temperature >38.5°C (Fig. 1). The severity criteria had excellent ability, AUC of 0.84, to discriminate between children who were hospitalized versus not hospitalized (Fig. 2A).

In predicting the clinician’s decision to admit a child with CAP to the hospital, the most sensitive criteria for children <1 year of age was cyanosis (1.0) and not feeding (0.78) (Table 3). Several criteria had a specificity of >0.70 including respiratory rate, moderate-to-severe retractions, nasal flaring, grunting and capillary refill ≥2 seconds (Table 2). However, the positive likelihood ratio for nasal flaring, grunting and respiratory rate indicate higher odds of hospitalization in children <1 year of age. Only tachycardia for children ≥1 year of age had a sensitivity of >0.70 for predicting admission to the hospital; however all criteria except for temperature >38.5°C and tachycardia had high specificity (Table 2). Respiratory rate, nasal flaring, grunting and capillary refill had the highest positive likelihood ratios indicating higher odds of hospitalization in the ≥1 year of age group.

TABLE 3.

Test Characteristics for Admission by Individual BTS Criteria

	Discharged From ED (n = 293), n (%)	Admitted (n = 225), n (%)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	LR+	LR−
Children <1 year	n = 19	n = 27
Temperature >38.5C	9 (47.4)	16 (59.3)	0.59	0.53	0.64	0.48	1.25	0.77
Respiratory rate >70	1 (5.3)	7 (25.9)	0.26	0.95	0.87	0.47	4.93	0.78
Moderate to severe retractions	1 (5.3)	1 (3.7)	0.04	0.95	0.50	0.41	0.7	1.02
Nasal flaring	1 (5.3)	5 (18.5)	0.19	0.95	0.83	0.45	3.52	0.86
Cyanosis	19 (100)	27 (100)	1.00	0	0.59	—	1.00	—
Intermittent apnea*	0	0	—	—	—	—	—	—
Grunting	1 (5.3)	6 (22.2)	0.22	0.95	0.86	0.46	4.22	0.82
Not feeding	11 (57.9)	21 (77.8)	0.78	0.42	0.66	0.57	1.34	0.53
Cap refill ≥2 seconds	0	3 (11.1)	0.12	1.00	1.00	0.43		0.88
Tachycardia	9 (47.4)	8 (42.1)	0.7	0.58	0.70	0.58	1.67	0.51
Children ≥1 year	n = 274	n = 198
Temperature >38.5C	89 (32.5)	72 (36.4)	0.36	0.68	0.45	0.59	1.12	0.94
Respiratory rate > 50	32 (11.7)	105 (53.0)	0.53	0.88	0.77	0.72	4.52	0.53
Severe difficulty breathing	0	1 (0.5)	0.01	1.00	1.00	0.58		0.99
Nasal flaring	3 (1.1)	28 (14.1)	0.14	0.99	0.90	0.61	12.9	0.87
Cyanosis	7 (2.6)	114 (57.6)	0.01	1.00	1.00	0.58		0.99
Grunting	3 (1.1)	15 (7.6)	0.08	0.99	0.83	0.60	6.92	0.93
Dehydration signs	11 (4.0)	22 (11.1)	0.11	0.96	0.67	0.60	2.77	0.93
Cap refill ≥2 seconds	4 (1.5)	16 (8.1)	0.09	0.98	0.80	0.60	5.63	0.93
Tachycardia	108 (39.4)	162 (81.8)	0.82	0.61	0.60	0.82	2.08	0.30

^*We had zero patients with this finding therefore we did not compute characteristics for this finding.

DISCUSSION

The BTS CAP severity criteria have only poor ability to discriminate children who met NFH criteria compared with those who did not, and excellent ability to discriminate children who were admitted to the hospital versus those who were not. Most children in the study who presented to the ED met at least 1 BTS severity criterion, but more than half of these children were discharged home from the ED. The criteria generally have high specificity but low sensitivity in predicting NFH and disposition.

In pediatric CAP, age of the child has been associated with potential etiology of disease, but it has not been strongly associated with severity.^17–19 The severity criteria outlined by the BTS guideline specify different characteristics of severity based on age. The following characteristics are added to the severity criteria for children <1 year of age: moderate-to-severe recession, intermittent apnea, not feeding and respiratory rate of >70 breaths per minute. Although the prevalence of these characteristics was low in our <1 year of age study population, importantly except for respiratory rate, none of these characteristics were statistically associated with admission or NFH based on risk ratios. Interestingly, there was a significant association with admission among the shared severity criteria between the 2 age groups, specifically nasal flaring, grunting and tachycardia, for children ≥1 year of age but not for the younger children. Grunting and capillary refill ≥2 seconds were the only shared severity criteria that were significantly associated with admission in both age groups. Therefore, when applying the severity criteria to clinical practice it may be important to consider the low prevalence of criteria unique to the younger children and rather focus on common characteristics that are important indicators in both age groups.

To date there are no clear definitions of severe CAP, nor are there validated clinical prediction rules in children to predict morbidity associated with CAP. The BTS guideline addresses this gap by providing a list of criteria for clinicians to consider in their decision making. Our study suggests, however, that 63% of children who met at least one of these criteria were discharged home and 12% of children who did not meet any criteria were hospitalized. The BTS criteria thus may lead to admitting children in whom it is not necessary, while also discharging children in whom admission is warranted.

In addition, the BTS guideline on pediatric pneumonia states that the most important clinical decision in the management of CAP is to determine if a child can be treated as an outpatient or whether the child needs to be admitted.² Admission to the hospital can be a result of myriad factors not directly related to the medical status of the child, thus in our study we included an objective outcome, NFH, modified from the adult literature.^12,13 NFH is based on interventions that warrant hospitalization and are more directly linked to the physiological state of the child. Although 43.4% of children in the study were admitted, only 26.6% met criteria for NFH. As such the lack of association between the BTS criteria and NFH suggests that more specific and sensitive criteria are needed to predict severe CAP.

Studies that have used the BTS guideline to define severity have mixed results. Clark et al²⁰ defined severity as meeting any one of the criteria listed in the BTS guideline. Most children (89%) were hospitalized, with 5% of hospitalized children admitted to the ICU. Interestingly, 4% of children identified as having severe CAP were not admitted to the hospital compared with less than 1% of children in the mild and moderate severity groups not admitted to the hospital. The study concluded that although the guideline has expertly-derived criteria to define severity, the application of this guideline in practice is not well-defined. Another study investigated the association between severity and findings indicating pneumonia on chest radiography; no association was found using the BTS criteria, but 1 was found using any one of the WHO criteria (ie, stridor, fast breathing, chest wall indrawing and difficulty in breathing).^21,22

There are several limitations to our study. First, this was a retrospective cohort study dependent on medical chart review. Certain criteria, such as not feeding, are difficult to ascertain accurately if not precisely defined and documented. These criteria are typically deemed clinically important in the clinical decision to admit to the hospital thus are usually documented as reasons for admission. Additionally, documentation of a child’s ED visit may be done retrospectively by the provider and recall bias may be a concern depending on the provider’s disposition decision. The BTS guideline specifies criteria in which ICU admission is highly recommended. Criteria (eg, rising respiratory and pulse rate with clinical evidence of respiratory distress) was not available for this analysis, and the utility of these parameters on the accuracy of ICU admission based on these criteria could not be assessed. Second, infectious etiology is unknown in most cases of pneumonia thus it may be underdiagnosed in young children categorized as bronchiolitis, leading to a limited sample size in children <1 year of age. Third, the low prevalence of some individual criterion (eg, none with intermittent apnea) may have limited our ability to accurately estimate the sensitivity and positive likelihood ratio characteristics of that finding or may be rare in otherwise healthy children with pneumonia. Fourth, each of the BTS severity criteria were given equal weight in the statistical analysis as the BTS guideline suggested a global assessment of clinical severity and did not guide the reader to weight any criterion more significantly than any other. It is reasonable to believe that some criteria (eg, severe difficulty breathing and cyanosis) may be more important in the clinical decision to admit compared with the criteria of fever or signs of dehydration. Fifth, the BTS guideline was last updated in 2011 and according to BTS policy, guidelines greater than 5-year-old are archived. These guidelines however continue to be cited therefore indicating their use is still relevant to many providers and researchers.^23–25 Lastly, the BTS severity criteria were intended for the management of children with CAP in the United Kingdom and therefore may not be generalizable to children in the United States. However, the progression and management of CAP in children in the United Kingdom and the United States are similar when comparing components of national and international guidelines.^2,26,27

In conclusion, our results suggest utility of the BTS severity criteria in their current form for predicting the clinical decision to admit to the hospital but limited utility for predicting the NFH in otherwise healthy children diagnosed with CAP. Only severe difficulty breathing increased the risk of needing hospitalization among the older children and no criterion increased the risk of needing hospitalization in the younger children. The more criteria a child met in this study the more likely they were admitted to the hospital; however, it remains unclear from the guideline the absolute number of criteria that need to be met to categorize a child with severe CAP.