Association between Cerebrospinal Fluid Testing Guidelines and Clinical Outcomes among Febrile Infants Aged 29–56 Days

Abstract

OBJECTIVE

To describe the association between clinical outcomes and clinical practice guidelines (CPGs) recommending universal cerebrospinal fluid (CSF) testing in the emergency department for older febrile infants aged 29–56 days.

STUDY DESIGN

Using 2007–2013 administrative data from 32 U.S. children’s hospitals, we performed a difference-in-differences analysis comparing 7 hospitals with CPGs recommending universal CSF testing for older febrile infants (CPG group) with 25 hospitals without such CPGs (control group). We compared differences in clinical outcomes between these groups among older febrile infants with the corresponding differences among younger febrile infants aged 7–28 days. The primary outcome was the occurrence of an adverse event, defined as a delayed diagnosis of bacterial meningitis, mechanical ventilation, placement of a central venous catheter, extracorporeal membrane oxygenation, or in-hospital mortality. Analyses were adjusted for race/ethnicity, gender, median annual household income by zip code, primary insurance source, discharge season, and discharge year.

RESULTS

The proportion of older febrile infants undergoing CSF testing was higher (P<0.001) in the CPG group (64.8%) than the control group (47.8%). CPGs recommending universal CSF testing for older febrile infants were not associated with significant differences in adverse events (difference-in-differences: +0.31 percentage points, 95% confidence interval −0.18 to 0.85; p=0.22).

CONCLUSIONS

Hospital CPGs recommending universal CSF testing for febrile infants aged 29–56 days were not associated with significant differences in clinical outcomes.

INTRODUCTION

According to several clinical practice guidelines (CPGs) based on expert opinion, infants aged 0–28 days who present to the emergency department (ED) for evaluation of fever should undergo urine, blood, and cerebrospinal fluid (CSF) testing to facilitate prompt diagnosis of urinary tract infections, bacteremia/sepsis, and meningitis.^1–3 However, the management of older febrile infants aged 29–56 days in the ED has been debated in the literature for decades.^4–10 While there is general agreement that these infants should undergo urine and blood testing, no such consensus exists for CSF testing.^1,2 Universal CSF testing for older febrile infants could prevent missed or delayed diagnoses of bacterial meningitis, leading to better clinical outcomes. On the other hand, if providers can accurately identify which older febrile infants need CSF testing after considering clinical presentation and results from other laboratory testing, universal CSF testing could lead to unnecessary stress for families, a higher risk of procedural complications, and hospitalizations of otherwise low-risk infants following traumatic lumbar punctures.^11–13

Due to the lack of evidence supporting national guidelines on the management of older febrile infants, many U.S. children’s hospitals have implemented institution-specific CPGs to standardize clinical practice. Based on well-known but differing criteria to identify febrile infants at low-risk for serious bacterial infections, some U.S. children’s hospitals have adopted CPGs recommending universal CSF testing in the ED for febrile infants aged 29–56 days, while others have adopted CPGs recommending selective CSF testing after considering other factors.¹⁴ To date, no study has compared the clinical benefits of these approaches. While randomization would be ideal for causal inference, such an approach would be infeasible due to the practical and ethical difficulties of enrolling a large number of infants to potentially undergo an invasive procedure like lumbar puncture.

The objective of this study was to evaluate the association between clinical outcomes and hospital CPGs recommending universal CSF testing in the ED for older febrile infants aged 29–56 days. We used a strong quasi-experimental approach that exploited the variation in CSF testing recommendations among CPGs for older febrile infants at U.S. children’s hospitals. Specifically, we examined hospitals with and without CPGs recommending universal CSF testing and compared the differences in clinical outcomes between these hospital groups among older febrile infants to the corresponding differences among younger febrile infants.

METHODS

Study design

We compared 7 hospitals with CPGs recommending universal CSF testing for older febrile infants in the ED (CPG group) with 25 hospitals without such CPGs (control group). In the control group, 8 hospitals had CPGs recommending selective CSF testing for older febrile infants meeting specific criteria, while 17 did not have CPGs guiding management of older febrile infants (de facto selective CSF testing). We included the 17 hospitals without CPGs based on a previous study showing no changes in rates of CSF testing for febrile infants after implementation of a care process model that recommended selective CSF testing.¹⁵ This finding suggests that there is a similar CSF testing approach among hospitals without CPGs for older febrile infants and hospitals with CPGs explicitly recommending a selective CSF testing approach.

We used a difference-in-differences analysis to estimate differences in clinical outcomes between the CPG and control groups among older febrile infants that were not predicted by the corresponding differences among younger febrile infants. An important advantage of this approach is that it adjusted for differences in patient characteristics between the CPG and control groups that did not vary with age. For example, even if infants’ severity of illness at presentation differed systematically between the CPG and control groups, the effect of this confounder would be netted out by our comparisons of older versus younger infants, as long as the difference in illness severity was the same in both age groups.

Data source

Data for this study were obtained from the 2007–2013 Pediatric Health Information System (PHIS), an administrative database containing encounter-level information from 45 non-profit, tertiary U.S. children’s hospitals affiliated with the Children’s Hospital Association (Overland Park, Kansas, USA). Participating hospitals provide discharge data for inpatient, ED, and observation unit visits, including demographic information, International Classification of Diseases Ninth Revision (ICD-9) diagnosis codes, ICD-9 procedure codes, and charges for clinical services.¹⁶ Because the PHIS contains de-identified data, the Institutional Review Board of Boston Children’s Hospital deemed this study exempt from review.

Study sample

We defined older febrile infants as ages 29–56 days and younger febrile infants as ages 7–28 days. We excluded infants ages 0–6 days because of the unique clinical circumstances during the immediate perinatal period.⁵ Following other studies, we excluded 7 of the 45 PHIS hospitals with missing ED visit data and 1 hospital with emergency department and inpatient records that could not be linked.¹⁷ To assign the remaining hospitals to the CPG and control groups, we determined the presence, content, and implementation year of CPGs for older febrile infants based on a previously administered survey of ED medical directors at PHIS hospitals (89% response rate).¹⁴ Of the 37 hospitals in the survey, we excluded 4 due to survey non-response and 1 due to inaccurate discharge diagnosis information for ED visits, leaving 32 hospitals in the sample.

For hospitals implementing CPGs, we excluded data from years before the CPGs were implemented. We further excluded records with discharge diagnosis codes indicating a complex chronic condition¹⁸ (e.g., congenital heart disease) since febrile infants with these conditions often undergo non-standard evaluations in the ED.¹⁷ After these exclusions, there were 415,280 potentially eligible records for infants aged 7–56 who presented to the ED of the 32 hospitals.

Following previous research on the management of febrile infants, we restricted the sample to records with 1 of the following 4 fever-related codes in an admission or discharge diagnosis field: 780.6 (Fever and other physiologic disturbances of temperature regulation), 780.60 (Fever, unspecified), 780.61 (Fever presenting with conditions classified elsewhere), and 778.4 (Other disturbances of temperature regulation of infant).^3,17 To capture febrile infants with records containing infection-related codes but not fever-related codes, we also included records with an infection-related admission or discharge diagnosis code that predicted a complete sepsis evaluation (urine, blood, and CSF testing) for at least 50% of infants aged 7–28 days (Appendix 1; online and Appendix 2; online). This strategy was based on the assumption that complete sepsis evaluations are good proxies for fevers among infants aged 7–28 days. In support of this assumption, previous research indicates that most febrile infants ≤ 28 days old undergo these evaluations in PHIS hospital EDs.³ Complete sepsis evaluations are less likely to predict fevers among febrile infants aged 29–56 days, who less frequently undergo these evaluations.¹⁷ As such, we did not include these infants when screening diagnosis codes.

In sensitivity analyses, we used cutoffs of 25% or 75% instead of 50%, restricted the sample to records with any of the 4 fever-related diagnosis codes in an admission or discharge diagnosis field, and excluded infants who underwent no urine, blood, or CSF testing (since these infants may not have been truly febrile).

Study variables

The primary outcome was the occurrence of an adverse event, defined as any of the following: a delayed diagnosis of bacterial meningitis; in-hospital mortality (based on the PHIS disposition variable); placement of a central venous catheter (based on ICD-9 procedure codes); mechanical ventilation (based on charges or ICD-9 procedure codes); or extracorporeal membrane oxygenation (based on charges or ICD-9 procedure codes). We considered a delayed diagnosis of bacterial meningitis to have occurred when a record that did not contain a discharge diagnosis code indicating bacterial meningitis was followed by a revisit record within 3 days of discharge that did contain such a code. We included the 4 additional adverse events because this measure likely underestimated the true prevalence of delayed diagnoses of bacterial meningitis due to its reliance on diagnosis codes in administrative data. For example, a readmission for a seizure due to bacterial meningitis may include a discharge diagnosis code for seizure but not for bacterial meningitis, or a missed diagnosis of bacterial meningitis may lead to death without CSF testing ever being performed. We selected the 4 additional events as potential indicators of complications from bacterial meningitis that would expectedly be more prevalent among patients with delayed or missed diagnoses of bacterial meningitis. Although these complications could result from other conditions, we would not expect CSF testing to affect the diagnosis and treatment of other conditions. Therefore, our difference-in-differences estimates should reflect effects of universal CSF testing on adverse events secondary to delayed or missed diagnoses of bacterial meningitis only.

Statistical analysis

We used logistic regression to model the occurrence of an adverse event as a function of age group (older versus younger febrile infant) and its interaction with CPG group status (CPG versus control). We included an indicator for each hospital (omitting a reference hospital) to control for hospital-specific factors common to younger and older febrile infants, such as geographic location and the case mix of infants served by the hospitals. Covariates were the patient’s race/ethnicity, gender, primary insurance payer, median annual household income by zip code of residence, season of discharge, and discharge year. We used robust variance estimators to account for clustering at the hospital level.¹⁹ To improve interpretability, we retransformed regression estimates to probabilities using simulation. Further details on the regression models and simulation procedure are available online (Appendix 3; online).

The coefficient for the interaction between age group and CPG group was the difference-in-differences estimate, or the mean difference between the CPG and control groups among older febrile infants that was not predicted by the corresponding difference among younger febrile infants and not explained by any differences in observed covariates that varied with age group. A positive difference-in-differences estimate would suggest that CPGs recommending universal CSF testing for older febrile infants were associated with increased adverse events, whereas a negative difference-in-differences estimate would suggest that these CPGs were associated with decreased adverse events.

Our analyses relied on the assumption that differences in adverse events between the CPG and control groups would have been the same among older and younger febrile infants in the absence of differences in CSF testing recommendations for older febrile infants. We performed several tests of this assumption. First, we compared differences in observed patient characteristics between the CPG and control groups among older febrile infants with the corresponding differences among younger febrile infants. The existence of differences that varied with age would suggest potential bias from differences in unobserved characteristics present in older febrile infants but not in younger febrile infants. Second, we assessed whether differences between the CPG and control group varied with age for management decisions other than CSF testing, including urine testing, blood testing, parenteral antibiotic use, and hospitalization. Additional analyses assessing the validity of the study’s underlying assumption are described online (Appendix 4; online).

Estimates could be biased if the probability of return visits to a non-PHIS hospital differed between the CPG and control groups, since any adverse events associated with return visits to a non-PHIS hospital would not be captured by our dataset. This potential bias would be more likely to exist if the practice of selective CSF testing in the control group led to fewer hospitalizations and therefore a higher chance of return visits, and if PHIS and non-PHIS hospitals frequently shared patients in the same market. To test for this bias, we compared the proportions of older febrile infants who were hospitalized following the initial ED visit between the CPG and control groups. Furthermore, in a sensitivity analysis, we excluded 14 hospitals located in metropolitan statistical areas with at least 1 other general children’s hospital (details are available in Appendix 5; online).²⁰

In a final sensitivity analysis, we excluded the 17 hospitals without CPGs for older febrile infants from the control group. We performed all analyses using SAS 9.4, Stata 13.0, and R version 3.1.1. Two-sided p values < 0.05 were considered statistically significant.

RESULTS

Of 85,815 records meeting sample inclusion criteria, we excluded 6.0% due to missing covariate data, leaving 80,074 records in the main sample. When we included records with missing covariate data but did not adjust for these covariates in regressions, results did not change substantially.

The CPG group included 17,970 records (6,766 from younger febrile infants and 11,204 from older febrile infants), while the control group included 62,734 records (24,270 from younger febrile infants and 38,464 from older febrile infants). Among younger febrile infants, CPG and control groups had similar proportions of females and infants of Asian or other race/ethnicity; different proportions of infants of Hispanic, black, and white race/ethnicity; different proportions of infants in all but 1 category of annual median household income by zip code; and different proportions of infants in each category of primary insurance payer (Table 1). Differences between the CPG and control groups were similar among older and younger febrile infants except in 2 categories of race/ethnicity (Hispanic and black) and primary insurance payer (public insurance and self-pay/other). Although statistically significant, the age-related differences between group differences in these characteristics were small (< 2.1 percentage points).

Table 1.

Age-related group differences in demographic characteristics

	Older febrile infants (n = 49,668)		Younger febrile infants (n = 31,036)		Age-related group differences
Demographic characteristic (%)	CPG (n = 11,204)	Control (n = 38,464)	CPG (n = 6,766)	Control (n = 24,270)	CPG – control: older febrile infantsa	CPG – control: younger febrile infantsa	Difference-in-differencesb
GENDER
Female	44.6	44.5	44.7	44.3	0.1	0.4	−0.3
RACE/ETHNICITY
Hispanic	23.7	29.8	21.6	29.7	−6.1*	−8.1*	2.0*
Black (non-Hispanic or ethnicity unknown)	24.5	22.8	26.7	23.0	1.7*	3.7*	−2.0*
Asian (non-Hispanic or ethnicity unknown)	2.5	1.9	2.4	2.2	0.6*	0.2	0.4
Other (non-Hispanic or ethnicity unknown)	6.9	6.9	7.4	7.0	0.0	0.4	−0.4
White (non-Hispanic or ethnicity unknown)	42.4	38.6	42.1	38.2	3.8*	3.9*	−0.1
MEDIAN ANNUAL HOUSEHOLD INCOME BY ZIP CODE
$0 to $30,000	26.5	21.3	25.6	20.9	5.2*	4.7*	0.5
$30,001 to $50,000	52.1	54.6	53.8	54.8	−2.5*	−1.0	−1.5
$50,001 to $70,000	16.3	18.3	15.6	18.2	−2.0*	−2.6*	0.6
> $70,000	5.1	5.8	5.0	6.1	−0.7*	−1.1*	0.4
PRIMARY PAYER
Private insurance	37.7	23.1	38.1	23.7	14.6*	14.4*	0.2
Public insurance	51.5	69.0	52.5	67.9	−17.5*	−15.4*	−2.1*
Self-pay or other	10.9	7.9	9.4	8.5	3.0*	0.9*	2.1*

Abbreviations: CPG, clinical practice guideline

^*p< 0.05

^ap value was derived from a chi squared test.

^bThis column refers to the difference between age-related group differences and equals (CPG – control among older febrile infants) – (CPG – control among younger febrile infants). We fitted logistic regression models modeling each characteristic as a function of the indicator of age group (older vs. younger febrile infant), the indicator for CPG group (CPG vs. control), and their interaction. The p value was derived from the hypothesis test that the coefficient of the interaction term equaled zero.

Febrile infant management profiles by age for the CPG and control groups are displayed in Figure 1. Age-related group differences in management decisions are displayed in Table 2. The proportion of younger febrile infants undergoing CSF testing was slightly higher (p<0.001) in the CPG group (67.9%) than in the control group (64.7%). The proportion of older febrile infants undergoing CSF testing was much higher (p<0.001) in the CPG group (64.8%) than in the control group (47.8%).

Figure 1. Hospital management profile by age.

Circles represent urine testing, squares represent blood testing, and triangles represent cerebrospinal fluid testing. A) CPG group; B) Control group.

Table 2.

Age-related group differences in management decisions

	Older febrile infants		Younger febrile infants		Age-related group differences
Management decisiona (%)	CPG	Control	CPG	Control	CPG – control: older febrile infantsb	CPG – control: younger febrile infantsb	Difference-in-differencesc
Urine testing	76.0	72.1	71.0	70.2	3.9*	0.8	3.1*
Blood testing	78.5	75.2	76.3	75.8	3.3*	0.5	2.8*
CSF testing	64.8	47.8	67.9	64.7	17.0*	3.2*	13.8*
Parenteral antibiotic	58.8	52.7	72.6	72.2	6.1*	0.4	5.7*
Hospitalization	52.5	52.5	77.6	78.4	0.0	−0.8	0.8

Abbreviations: CPG, clinical practice guideline; CSF, cerebrospinal fluid

^*p< 0.05

^aFor definitions of management decisions, see Appendix 1.

^bp value was derived from a chi squared test.

^cThis column refers to the difference between age-related group differences and equals (CPG – control among older febrile infants) – (CPG – control among younger febrile infants). We fitted logistic regression models modeling each characteristic as a function of the indicator of age group (older vs. younger febrile infant), the indicator for CPG group (CPG vs. control), and their interaction. The p value was derived from the hypothesis test that the coefficient of the interaction term equaled zero.

The proportion of older febrile infants hospitalized following the initial ED visit was similar (p=1.00) for the CPG (52.5%) and control (52.5%) groups. For hospitalization, differences between the CPG and control groups were similar among older and younger febrile infants. For urine testing, blood testing, and parenteral antibiotic use, the differences between age-related group differences were statistically significant but small (< 5.7 percentage points) compared with the corresponding difference for CSF testing (13.8 percentage points) (Table 2).

Age profiles for adverse events are displayed in Figure 2. Unadjusted means for adverse events in the four comparison groups are shown in Table 3. The unadjusted mean percentage of younger febrile infants experiencing any adverse event was 2.93% in the CPG group and 2.95% in the control group (difference: −0.02%), while the unadjusted means for older febrile infants were 2.11% in the CPG group and 1.78% in the control group (difference: 0.33%), resulting in an unadjusted difference-in-differences estimate of 0.35% (95% CI: −0.32, 0.95, p=0.29) (Table 3). This estimate was similar in adjusted analyses (difference-in-differences estimate: 0.31%; 95% CI: −0.18 to 0.85; p=0.22).

Figure 2. Adverse events by age in the CPG and control groups.

Squares represent the CPG group and circles represent the control group.

Table 3.

Unadjusted means for adverse events among older and younger febrile infants in the CPG and control groups

	Unadjusted means (%)				Difference-in-differences estimates
	Older febrile infants		Younger febrile infants		Unadjusted			Adjusted
	CPG	Control	CPG	Control	Estimatea	95% CIa	P Value	Estimatea	95% CIa	P Value
Any adverse event	2.11	1.78	2.93	2.95	0.35	−0.32, 0.95	0.29	0.31	−0.18, 0.85	0.22

Abbreviations: CPG, clinical practice guideline; CI, confidence interval

^aEstimates and 95% confidence intervals were multiplied by 100 and represent absolute differences in percentage points.

In sensitivity analyses, results did not substantively differ from the main analysis when we used alternative sample identification strategies, excluded PHIS hospitals with nearby competitors, or excluded hospitals without CPGs for older febrile infants (Table 4; online).

DISCUSSION

In this study of 32 tertiary U.S. children’s hospitals, hospital CPGs recommending universal CSF testing for older febrile infants were not associated with significant differences in adverse events. Although CSF testing confers important clinical benefits for certain older febrile infants, our findings do not support a clear clinical benefit of CPGs recommending CSF testing for all older febrile infants.

The lack of a significant association between CPGs recommending universal CSF testing for older febrile infants and adverse events suggests that providers in the control group accurately determined which older febrile infants were at high-risk for bacterial meningitis after considering clinical and laboratory factors. In support of this possibility, previous research demonstrated low rates of serious bacterial infections among older febrile infants classified as low-risk by the Rochester protocol, which recommends universal urine and blood testing but not universal CSF testing for this population.^6,7 Diagnostic evaluations for febrile infants, particularly lumbar punctures, can be extremely stressful to infants and their families.¹¹ The lack of association between universal CSF testing and adverse events suggests that many families of older febrile infants could be spared the stress of CSF testing without harm.

Our findings have implications for clinical guidelines and low-value care. The majority of clinical services have both low-value and high-value applications.^21,22 As noted by the Institute of Medicine, a well-evaluated, reliable CPG can reduce low-value applications of clinical services.²³ Our study suggests that CPGs may also encourage low-value applications of services without increasing their high-value applications. As such, it is important that organizations developing national CPGs rely on the most rigorous and recent evidence on the intended and unintended effects of CPG implementation.¹⁵

Although we used a quasi-experimental design to control for unobserved differences between the CPG and control groups that did not vary with age, we could not control for unobserved differences that did vary with age. This potential source of bias would be more likely, however, if there were differences in observed characteristics between the CPG and control groups among older febrile infants that were not predicted by the corresponding differences among younger febrile infants. While these differences existed for a few characteristics we examined, they were small and paled in comparison to the differences for CSF testing. In addition, results adjusted for observed patient characteristics were similar to the unadjusted results, suggesting that further adjustment for unmeasured patient characteristics would not likely change our results substantially.

Our study has other limitations. First, we lacked sufficient statistical power to detect small differences in adverse events. However, the lower bound of the 95% confidence interval for our adjusted difference-in-differences estimate (−0.18%) suggests that a reduction in adverse events of 10% or greater from CPGs recommending universal CSF testing for older febrile infants was extremely unlikely (calculation based on a mean rate of adverse events of 1.78% among older febrile infants in the control group). Second, estimates could be biased if the likelihood of return visits to a non-PHIS hospital differed between older febrile infants in the CPG and control groups. However, older febrile infants in these groups were equally likely to be hospitalized following the initial ED visit, and results of a sensitivity analysis excluding hospitals with nearby competitors were not substantively different from the main results. Third, we may have underestimated the true number of adverse events due to our reliance on administrative data, though we would expect this potential bias to affect the CPG and control groups equally. Fourth, we relied on diagnosis codes to identify infants with fevers. While this strategy may have led to the inclusion of infants without fevers into our sample, results were substantively unchanged in sensitivity analyses using alternative sample identification strategies. Finally, our sample was derived from large, tertiary pediatric hospitals. As such, our findings may not generalize to other types of hospitals and primary care settings.

CONCLUSION

In this study of U.S. children’s hospitals, CPGs recommending universal CSF testing for older febrile infants were not associated with differences in adverse events. These CPGs may encourage applications of CSF testing that are not associated with clinical benefits.

LIST OF ABBREVIATIONS

CI

confidence interval

CSF

cerebrospinal fluid

CPG

clinical practice guideline

ECMO

extracorporeal membrane oxygenation

ED

emergency department