Abstract
Objective
Broadly, to create a bidirectional communication link between public health surveillance and clinical practice. Specifically, to measure the impact of integrating public health surveillance data into an existing clinical prediction rule. We incorporate data about recent local trends in meningitis epidemiology into a prediction model differentiating aseptic from bacterial meningitis.
Design and Measurements
Retrospective analysis of a cohort of all 696 children with meningitis admitted to a large urban pediatric hospital from 1992 to 2000. We modified a published bacterial meningitis score by adding a new epidemiological context adjustor variable. We examined 540 possible rules for this adjustor, varying both the number of aseptic meningitis cases that needed to be seen, and the recent time window in which they were seen. We performed sensitivity analyses with each of 540 possibilities in order to identify the optimal rule—namely, the one that included the most cases of aseptic meningitis without missing additional cases of bacterial meningitis, as compared with the published prediction model. We used bootstrap methods to validate this new score.
Results
The optimal rule was found to be: “at least four cases of aseptic meningitis in the previous 10 days.” The epidemiological context adjustor based on surveillance of recent cases of meningitis allowed the correct identification of an additional 47 cases (7%) of aseptic meningitis without missing any additional cases of bacterial meningitis. The epidemiological context adjustor was validated, showing significance in 84% of 1,000 bootstrap samples.
Conclusion
Epidemiological contextual information can improve the performance of a clinical prediction rule. We provide a methodological framework for leveraging regional surveillance data to improve medical decision-making.
Introduction
Healthcare “situational awareness”—real-time information about the local, regional and national incidence of disease—has the potential to improve clinical decision-making by extending clinicians’ knowledge beyond what can be gleaned from patient history, physical examination, clinician experience, and the medical literature. One tool used by clinicians to make diagnoses is the clinical prediction rule which typically takes as inputs, patient history, physical exam, and readily available diagnostic test results to calculate the probability of a disease state and to support medical decision-making. 1,2 This approach is static and does not factor in community-level disease trends when estimating disease likelihood. For infectious diseases which are person-to-person transmissible, an individual’s likelihood of infection is dependent on the local disease incidence. 3 A clinician’s knowledge and cognitive interpretation of disease incidence is dependent on observed trends or unusual cases, 4 or through recent personal clinical experience. 5 Ideally, clinicians should be prepared to respond clinically to changes in environmental context. To date, no published prediction rules have incorporated epidemiological factors from regional, real-time surveillance data. Yet the increasing uses of biosurveillance technologies will make the necessary regional data widely available.
We propose a method for adjusting prediction rules for epidemiological context, and evaluate it using a prediction rule for bacterial meningitis among pediatric patients. Pediatric patients who present to the emergency department with cerebrospinal fluid (CSF) pleocytosis present a diagnostic and management challenge to the clinician trying to distinguish between aseptic and bacterial meningitis. Most patients with CSF pleocytosis have aseptic, rather than bacterial meningitis. Bacterial meningitis is very rare, but the consequences can be devastating, because it causes severe morbidity and mortality. 6 While CSF culture is the definitive test to exclude bacterial meningitis, the result may take days, which means the clinician must make a decision about treatment and patient disposition based on incomplete data. Clinicians often take the conservative approach of admitting patients to the hospital for intravenous antibiotics for what ultimately proves to be self-limited cases of aseptic meningitis. This conservative approach can lead to overuse of antibiotics, as well as the risks of hospitalization, including nosocomial infections, allergic reactions to therapies, and unnecessary use of inpatient beds. 7–9 Accurately distinguishing bacterial from aseptic meningitis at the time of patient presentation could save significant health care resources, reduce hospital admission rates for viral disease, and decrease antibiotic resistance by decreasing the empirical use of antibiotics. 10,11 Previously published prediction rules have combined patient specific factors, such as historical, physical exam, and laboratory features to help identify patients at low-risk for bacterial meningitis. 12–19
We sought to examine the effect of a new surveillance data-derived variable that adds real-time epidemiological context to a published meningitis prediction rule 12 to determine whether and to what extent, data about recent meningitis epidemiology could improve the performance of the existing prediction model in differentiating aseptic from bacterial meningitis.
Materials and Methods
We performed a retrospective analysis of a dataset used to derive a published multivariate logistic regression model. 12 The dataset consisted of 696 patients aged 29 days to 19 years admitted to a tertiary care pediatric hospital from 1992–2000 with a final diagnosis of meningitis. One hundred and twenty five (18%) patients had bacterial meningitis, and 571 (82%) had aseptic meningitis. Details of the patient population, definitions, exclusion criteria, data collection, and statistical methods have already been published. 12 The Children’s Hospital Boston’s institutional review board approved the study. Bacterial meningitis was defined as positive CSF culture, or CSF pleocytosis (>7 WBC per mm) with positive blood culture or positive CSF latex agglutination for Streptococcus pneumoniae, Neisseria meningitidis, Haemophilus influenzae, or Streptococcus agalactiae (group B Streptococcus). Aseptic meningitis was defined as a CSF pleocytosis with negative cultures and negative latex agglutination if performed. Using logistic regression and recursive partitioning, the bacterial meningitis score was derived and validated using a split set methodology. The following high-risk predictors were identified: positive CSF gram stain (two points), CSF protein >80 mg/dL (one point), CSF acute neutrophil count >1,000 cells/mm3 (one point), peripheral acute neutrophil count >10,000 cells/mm (one point), seizure at or prior to the time of presentation (one point) (▶). Patients with a total score of zero were considered very low risk for bacterial meningitis; score of one to six were considered not very low risk for bacterial meningitis. This prediction rule was recently validated in a multicenter study. 20
Table 1.
Table 1 The Bacterial Meningitis Score
| Predictor Variable | Points |
|---|---|
| Positive CSF gram stain | 2 |
| CSF absolute neutrophil count ≥1,000 cells/mm3 | 1 |
| Peripheral blood absolute neutrophil count ≥10,000 cells/mm3 | 1 |
| CSF protein ≥80 mg/dL | 1 |
| History of seizure prior to or at time of presentation | 1 |
Patients with a total score of zero were considered very low risk for bacterial meningitis, and one to six were considered not very low risk for bacterial meningitis. Score can range from 0–6.
We created a new variable called the epidemiological context adjustor. This adjustor is a binary variable that reflects whether there have been more than n cases of aseptic meningitis during the past d days. To find the optimal adjustor, we examined 540 different possible adjustor rules by varying both n (anywhere from 1–20 cases) and d (anywhere from 3–30 days). A three day minimum was chosen for d because exclusion of bacterial meningitis requires negative CSF and blood cultures which takes two to three days to reliably exclude bacterial growth. 21
We performed sensitivity analysis for all of the 540 possible rules. For each rule, we incorporated the adjustor variable into the bacterial meningitis score by decreasing the score by one point if the number of patients (n) admitted over the previous number of days (d) had been exceeded on the day that a patient (p) presented with meningitis. For example, if n=2 and d=6, then for a patient whose bacterial meningitis score was one, we would decrease the score from one to zero if two or more cases of aseptic meningitis had been admitted in the previous six days. We then calculated the sensitivity and specificity for bacterial meningitis at each threshold (n, d). Keeping sensitivity for bacterial meningitis high to avoid the catastrophic error of missing additional cases of bacterial meningitis, we sought to maximize the specificity for bacterial meningitis—that is to identify the threshold that would identify correctly the maximum number of aseptic meningitis cases.
After selecting the adjustor rule that resulted in the greatest increased identification of aseptic meningitis cases without missing any additional cases of bacterial meningitis, we used bootstrap methods for validation. One thousand bootstrap samples were selected, using p<0.05 in over half the samples for predictor retention. 22–27 We performed all statistical analyses using SAS (SAS, Cary, NC) and SPSS 13 (SPSS Inc, Chicago, IL).
Results
In ▶, decreasing the number of cases “n” along the X-axis, or increasing the duration (number of days “d”) along the Y-axis, decreases the sensitivity (seen on the Z-axis)—that is more cases of bacterial meningitis would be incorrectly identified as having aseptic meningitis. The ceiling of the figure represents a sensitivity of 98%, which is the performance of the published prediction model for correctly identifying patients with bacterial meningitis. The bottom left corner of ▶ shows that the worst sensitivity (93%) of the model is reached when there has been one case in the previous 30 days.
Figure 1.
Sensitivity for bacterial meningitis (ability of score to identify bacterial meningitis) using epidemiological context adjustor at different cut-offs.
In ▶, the floor of the figure represents the 72% of aseptic meningitis cases identified by the published prediction model. Decreasing the number of cases “n” along the X-axis, or increasing the duration (number of days “d”) along the Y-axis increases the percent of patients correctly identified with aseptic meningitis (specificity for bacterial meningitis, seen on the Z-axis). The top left corner of ▶ shows that the maximum specificity (89%) of the model occurs when there has been only one case in the previous 30 days.
Figure 2.
Specificity for bacterial meningitis (ability of score to identify aseptic meningitis) using epidemiological context adjustor at different cut-offs.
In order to identify additional cases of aseptic meningitis without missing any additional cases of bacterial meningitis, we set the sensitivity at 98% (equal to the published model) and examined specificity (range 72%–81%). We selected the cutoff that gave 81% specificity with the maximum 98% sensitivity. The greatest improvements in model performance were obtained when there were at least four cases of aseptic meningitis in the previous 10 days. This allowed for the greatest increase in identification of patients with aseptic meningitis without missing any additional cases of bacterial meningitis.
▶ shows the confusion matrices for the published model and for three of the 540 different combinations of numbers of cases and durations that we examined. Sample rule C (four cases in previous 10 days) shows the performance for the best combination of number of cases “n” and duration “d”.
Table 2.
Table 2 Confusion Matrices (2×2 Tables) for Original Rule and New Rules that Incorporate “Epidemiological Context Adjustor”
| Original Rule (no adjustment for epidemiological context) | |||
|---|---|---|---|
| Actual | Predicted |
||
| Aseptic | Bacterial | Total | |
| Aseptic | 384 | 151 | 535 |
| Bacterial | 2 | 117 | 119 |
| Total | 386 | 268 | 654 |
| Sensitivity 98%, Specificity 72%. | |||
| Sample Rule A (2 cases in previous 6 days) | |||
|---|---|---|---|
| Actual | Predicted |
||
| Aseptic | Bacterial | Total | |
| Aseptic | 433 | 102 | 535 |
| Bacterial | 3 | 116 | 119 |
| Total | 436 | 218 | 654 |
| Sensitivity 97%, Specificity 81%. | |||
| Sample Rule B (8 cases in 4 previous days) | |||
|---|---|---|---|
| Actual | Predicted |
||
| Aseptic | Bacterial | Total | |
| Aseptic | 384 | 151 | 535 |
| Bacterial | 2 | 117 | 119 |
| Total | 386 | 268 | 654 |
| Sensitivity 98%, Specificity 72%. | |||
| Sample Rule C (4 cases in previous 10 days) – Best Rule | |||
|---|---|---|---|
| Actual | Predicted |
||
| Aseptic | Bacterial | Total | |
| Aseptic | 431 | 104 | 535 |
| Bacterial | 2 | 117 | 119 |
| Total | 433 | 221 | 654 |
Sensitivity 98%, Specificity 81%.
▶ shows the test characteristics of the previously published bacterial meningitis score and our refined model, using the epidemiological context adjustor with three sample cut-offs (two cases in previous six days, eight cases in previous four days, and four cases in the previous 10 days). The best model resulted in an additional 47 patients (7%) with CSF pleocytosis previously deemed not very low risk for bacterial meningitis that could now be categorized as low risk. None of these patients had bacterial meningitis.
Table 3.
Table 3 Test Characteristics of the Prediction Rule Without 12 and With Epidemiological Context
| Nigrovic Published Prediction Model (%, 95% Confidence Interval) | Sample Rule “A” (%, 95% Confidence Interval) | Sample Rule “B” (%, 95% Confidence Interval) | Sample Rule “C” - Best Model (%, 95% Confidence Interval) | |
|---|---|---|---|---|
| Sensitivity | 98 (94–100) | 97 (93–99) | 98 (94–100) | 98 (94–100) |
| Specificity | 72 (68–76) | 81 (77–84) | 72 (68–76) | 81 (77–84) |
| Positive predictive value | 44 (38–50) | 53 (46–60) | 44 (38–50) | 53 (46–60) |
| Negative predictive value | 99 (98–100) | 99 (98–100) | 99 (98–100) | 100* (98–100) |
Negative predictive value of sample rule C is 431/433, which was rounded to 100 from 99.5%.
We validated the multivariate regression analysis with a bootstrap technique. The epidemiological context adjustor was significantly associated in 84% of the iterations (p<0.05).
Discussion
In this study, we demonstrate the important role that surveillance can play in adding epidemiological context to improve clinical prediction rules for infectious diseases. Adding information about current infectious diseases circulating in the community can change the predictive probability of a diagnostic prediction rule. Using an existing data set, this strategy worked well—we were able to correctly identify an additional 47 cases of aseptic meningitis (7%) without missing a single additional case of bacterial meningitis.
The widespread use of the highly effective conjugate vaccines for Haemophilus influenzae type b and Streptococcus pneumoniae vaccinations have made bacterial meningitis a rare childhood disease. 6,28 The addition of an epidemiological context adjustor to a validated clinical prediction rule could add more confidence to a clinician’s diagnosis of aseptic meningitis in a patient who presents with CSF pleocytosis. Many clinicians already knowingly or unknowingly incorporate this type of information into their clinical decision-making. Knowledge of recent cases of infectious diseases may have profound effects on how doctors approach their next patient. 29
Public health agencies now collect emergency department-based infectious disease surveillance data on many conditions. 30,31 The ability to incorporate real-time surveillance data into existing, validated prediction rules has the potential to improve diagnostic accuracy by shifting the prior probability of disease for a given patient, depending on current conditions. This application of Bayes theorem would work especially well for conditions that are communicable or seasonal. For example, a prediction rule that could incorporate the current burden of strep throat in the community with the Centor score 32 used to evaluate patients with pharyngitis, might help inform clinicians which patients should be tested and/or treated with antibiotics.
Our study has some limitations. First, all patient data come from the era before the widespread pediatric use of the pneumococcal vaccine, a time when bacterial meningitis was more common than currently. Despite this, we still did not miss any additional cases of bacterial meningitis. Second, we tested the addition of the epidemiological context adjustor using the same data set from which the bacterial meningitis score was derived and validated. We did not split our data into a derivation and validation set, because of the relatively small size of the data set. Rather, we chose to use the bootstrap validation method so that we could include all available data to derive and evaluate our final model. Ideally we would prefer to have had a second independent data set to validate our final model. In the absence of such data we examined fitted models on bootstrap samples of our data to describe the stability of our final model. Relying on previously published work as a standard, we choose 50% as representing a large enough proportion of statistically significant finding to support the significance of a variable in our model. 27 However, in our Results section, we state that this proportion actually exceeded 70% for all variables and was equal to 84% for the environmental context adjustor.
Our new model needs independent validation because prediction model performance may differ significantly when tested in an independent population. 33 However, the purpose of this analysis was to determine whether epidemiological context could have an impact on a prediction model. This study demonstrates the first step in establishing that contextual information can add value to prediction models. Future studies could use public health surveillance data to adjust the meningitis score to account for regional variation in meningitis season. These findings also highlight the opportunity to incorporate contextual infectious disease surveillance data when creating new prediction rules for diseases that have seasonal variation or are communicable.
Incorporating local disease incidence adds valuable context and can improve the ability to discriminate bacterial from aseptic meningitis. These findings demonstrate that epidemiological context can improve the performance of a clinical prediction rule, and provide a framework for leveraging regional surveillance data to improve the clinician’s decision-making.
Footnotes
This work was supported by grant 1 R01 LM007677-01 from the National Library of Medicine, National Institutes of Health, by National Research Service Award grant T32 HD40128-01 (Research Training in Pediatric Emergency Medicine), and grant P01 CD000260-01 from the Centers for Disease Control and Prevention.
The authors gratefully acknowledge Drs. Nathan Kuppermann and Richard Malley for their careful reading and critique of the manuscript, and for sharing of their expert knowledge of the dataset used in this investigation.
References
- 1.Laupacis A, Sekar N, Stiell IG. Clinical prediction rulesA review and suggested modifications of methodological standards. JAMA 1997;277:488-494. [PubMed] [Google Scholar]
- 2.Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions Ann Intern Med 2006;144:201-209. [DOI] [PubMed] [Google Scholar]
- 3.Parker AA, Staggs W, Dayan GH, Ortega-Sanchez IR, Rota PA, Lowe L, et al. Implications of a 2005 measles outbreak in Indiana for sustained elimination of measles in the United States N Engl J Med 2006;355:447-455. [DOI] [PubMed] [Google Scholar]
- 4.Kahneman D, Slovic P, Tversky A. Judgment under Uncertainty: Heuristics and Biases. New York: Cambridge University Press; 1982. [DOI] [PubMed]
- 5.Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them Acad Med 2003;78:775-780. [DOI] [PubMed] [Google Scholar]
- 6.Schuchat A, Robinson K, Wenger JD, Harrison LH, Farley M, Reingold AL, et al. Bacterial meningitis in the United States in 1995 N Engl J Med 1997;337:970-976. [DOI] [PubMed] [Google Scholar]
- 7.Hamilton MS, Jackson MA, Abel D. Clinical utility of polymerase chain reaction testing for enteroviral meningitis Pediatr Infect Dis J 1999;18:533-537. [DOI] [PubMed] [Google Scholar]
- 8.Marshall GS, Hauck MA, Buck G, Rabalais GP. Potential cost savings through rapid diagnosis of enteroviral meningitis Pediatr Infect Dis J 1997;16:1086-1087. [DOI] [PubMed] [Google Scholar]
- 9.Nigrovic LE, Chiang VW. Cost analysis of enteroviral polymerase chain reaction in infants with fever and cerebrospinal fluid pleocytosis Arch Pediatr Adolesc Med 2000;154:817-821. [DOI] [PubMed] [Google Scholar]
- 10.Robinson CC, Willis M, Meagher A, Gieseker KE, Rotbart H, Glode MP. Impact of rapid polymerase chain reaction results on management of pediatric patients with enteroviral meningitis Pediatr Infect Dis J 2002;21:283-286. [DOI] [PubMed] [Google Scholar]
- 11.Ramers C, Billman G, Hartin M, Ho S, Sawyer MH. Impact of a diagnostic cerebrospinal fluid enterovirus polymerase chain reaction test on patient management JAMA 2000;283:2680-2685. [DOI] [PubMed] [Google Scholar]
- 12.Nigrovic LE, Kuppermann N, Malley R. Development and validation of a multivariable predictive model to distinguish bacterial from aseptic meningitis in children in the post-Haemophilus influenzae era Pediatrics 2002;110:712-719. [DOI] [PubMed] [Google Scholar]
- 13.Oostenbrink R, Moll HA, Moons KG, Grobbee DE. Predictive model for childhood meningitis Pediatr Infect Dis J 2004;23:1070-1071. [DOI] [PubMed] [Google Scholar]
- 14.Oostenbrink R, Moons KG, Derksen-Lubsen AG, Grobbee DE, Moll HA. A diagnostic decision rule for management of children with meningeal signs Eur J Epidemiol 2004;19:109-116. [DOI] [PubMed] [Google Scholar]
- 15.Oostenbrink R, Moons KG, Donders AR, Grobbee DE, Moll HA. Prediction of bacterial meningitis in children with meningeal signs: reduction of lumbar punctures Acta Paediatr 2001;90:611-617. [PubMed] [Google Scholar]
- 16.Freedman SB, Marrocco A, Pirie J, Dick PT. Predictors of bacterial meningitis in the era after Haemophilus influenzae Arch Pediatr Adolesc Med 2001;155:1301-1306. [DOI] [PubMed] [Google Scholar]
- 17.Spanos A, Harrell Jr. FE, Durack DT. Differential diagnosis of acute meningitisAn analysis of the predictive value of initial observations. JAMA 1989;262:2700-2707. [PubMed] [Google Scholar]
- 18.Negrini B, Kelleher KJ, Wald ER. Cerebrospinal fluid findings in aseptic versus bacterial meningitis Pediatrics 2000;105:316-319. [DOI] [PubMed] [Google Scholar]
- 19.Bonsu BK, Harper MB. Differentiating acute bacterial meningitis from acute viral meningitis among children with cerebrospinal fluid pleocytosis: a multivariable regression model Pediatr Infect Dis J 2004;23:511-517. [DOI] [PubMed] [Google Scholar]
- 20.Nigrovic L, Kuppermann N, Macias C, Cannavino C, Moro-Sutherland D, Schremmer R, et al. Multicenter validation and refinement of a clinical decision rule for identifying children with cerebrospinal fluid pleocytosis at low risk of bacterial meningitis Acad Emerg Med 2006;13(suppl):S167-S168. [Google Scholar]
- 21.Poppert S, Essig A, Stoehr B, Steingruber A, Wirths B, Juretschko S, et al. Rapid diagnosis of bacterial meningitis by real-time PCR and fluorescence in situ hybridization J Clin Microbiol 2005;43:3390-3397. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Efron B, Gong G. A leisurely look at the bootstrap, the jackknife, and cross-validation Am Stat 1983;37:36-48. [Google Scholar]
- 23.Efron B, Tibshirani R. Statistical data analysis in the computer age Science 1991;253:390-395. [DOI] [PubMed] [Google Scholar]
- 24.Chen CH, George SL. The bootstrap and identification of prognostic factors via Cox’s proportional hazards regression model Stat Med 1985;4:39-46. [DOI] [PubMed] [Google Scholar]
- 25.Schumacher M, Hollander N, Sauerbrei W. Resampling and cross-validation techniques: a tool to reduce bias caused by model building? Stat Med 1997;16:2813-2827. [DOI] [PubMed] [Google Scholar]
- 26.Nishime EO, Cole CR, Blackstone EH, Pashkow FJ, Lauer MS. Heart rate recovery and treadmill exercise score as predictors of mortality in patients referred for exercise ECG JAMA 2000;284:1392-1398. [DOI] [PubMed] [Google Scholar]
- 27.Glaser N, Barnett P, McCaslin I, Nelson D, Trainor J, Louie J, et al. Risk factors for cerebral edema in children with diabetic ketoacidosis N Engl J Med 2001;344:264-269. [DOI] [PubMed] [Google Scholar]
- 28.Whitney CG, Farley MM, Hadler J, Harrison LH, Bennett NM, Lynfield R, et al. Decline in invasive pneumococcal disease after the introduction of protein-polysaccharide conjugate vaccine N Engl J Med 2003;348:1737-1746. [DOI] [PubMed] [Google Scholar]
- 29.Croskerry P. Achieving quality in clinical decision making: cognitive strategies and detection of bias Acad Emerg Med 2002;9:1184-1204. [DOI] [PubMed] [Google Scholar]
- 30.Mandl KD, Overhage JM, Wagner MM, Lober WB, Sebastiani P, Mostashari F, et al. Implementing syndromic surveillance: A practical guide informed by the early experience J Am Med Inform Assoc 2004;11:141-150. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Bradley CA, Rolka H, Walker D, Loonsk J. BioSense: implementation of a National Early Event Detection and Situational Awareness System MMWR Morb Mortal Wkly Rep 2005;54(suppl):11-19. [PubMed] [Google Scholar]
- 32.Ebell MH, Smith MA, Barry HC, Ives K, Carey M. The rational clinical examinationDoes this patient have strep throat?. JAMA 2000;284:2912-2918. [DOI] [PubMed] [Google Scholar]
- 33.Charlson ME, Ales KL, Simon R, MacKenzie CR. Why predictive indexes perform less well in validation studiesIs it magic or methods?. Arch Intern Med 1987;147:2155-2161. [PubMed] [Google Scholar]