Randomized Trial of a Clinical Decision Support System: Impact on the Management of Children with Fever without Apparent Source

Abstract

Objective

To assess compliance with a clinical decision support system (CDSS) for diagnostic management of children with fever without apparent source and to study the effects of application of the CDSS on time spent in the emergency department (ED) and number of laboratory tests.

Design

The CDSS was used by ED nursing staff to register children presenting with fever. The CDSS identified children that met inclusion criteria (1–36 months and fever without apparent source (FWS)) and provided patient-specific diagnostic management advice. Children at high risk for serious bacterial infection were randomized for the ‘intervention’ (n = 74) or the ‘control’ (n = 90) group. In the intervention group, the CDSS provided the advice to immediately order laboratory tests and in the control group the ED physician first assessed the children and then decided on ordering laboratory tests.

Results

Compliance with registration of febrile children was 50% (683/1,399). Adherence to the advice to order laboratory tests was 82% (61/74). Children in the intervention group had a median (25^th–75^th percentile) length of stay at the ED of 138 (104–181) minutes. The median length of stay at the ED in the control group was 123 (83–179) minutes. Laboratory tests were significantly more frequently ordered in the intervention group (82%) than in the control group (44%, p < 0.001, χ² test).

Conclusion

Implementation of a CDSS for diagnostic management of young children with fever without apparent source was successful regarding compliance and adherence to CDSS recommendations, but had unexpected effects on patient outcome in terms of ED length of stay and number of laboratory tests. The use of the current CDSS was discontinued.

Introduction

The management of young febrile children is an everyday challenge for emergency department (ED) physicians. Distinguishing children with mild viral disease from those with serious bacterial infection (SBI) is difficult as clinical presentation is often non-specific. ^1–3 Early identification of children at risk for SBI could support appropriate management in terms of diagnostic and therapeutic decisions.

For several diagnostic and therapeutic problems, guidelines or clinical prediction rules were developed. ^4–10 Implementation of guidelines may result in reduced diagnostic testing, improved documentation, more appropriate treatment, and a reduction of the time spent in the ED. ^5,6 However, the translation of guidelines and prediction rules into clinical practice is still a major challenge. Studies of the actual clinical impact of decision rules, i.e., whether or not the use of a decision rule improves clinical decisions or benefits patient care, are limited. Reilly found that decision rule impact analysis was performed in only 9 out of 109 studies in their review. This may be due to limited implementation strategies of decision rules in routine clinical practice. ¹¹ Clinical decision support systems (CDSS) may be able to integrate available knowledge (e.g., decision rules) into clinical practice. ¹² Key features for successful CDSS implementation have been described, but little is known about the effects of CDSS-utilization on patient outcomes. ^13–15

At the Erasmus Medical Center in Rotterdam, The Netherlands, a CDSS was developed for the diagnostic management of young children with fever without apparent source (FWS), based on previously derived and validated prediction rules. From July 2003, the computerized CDSS was used routinely by the ED nursing staff to register children presenting with fever. First, the CDSS automatically identified children with FWS, second, children at high risk for SBI were identified based on clinical characteristics, and third, a patient-specific diagnostic management advice was provided.

Our aims were to assess compliance with the system and to assess the effects of the CDSS on 1) time spent at the ED and 2) amount of performed diagnostic tests in children with FWS at high risk for serious bacterial infection. Our hypothesis was that initiation of diagnostic workup directly after ED nurse evaluation increased ED efficiency by reducing both time spent in the ED and number of diagnostic tests. ⁶

Methods

Study Population and Setting

The emergency department of the Sophia Children’s Hospital is visited by 9000 children per year. From July 1, 2003 until December 31, 2005, children aged 0–16 years presenting with fever at the emergency department (ED) of the Sophia Children’s Hospital in Rotterdam were routinely registered by ED nurses, using standardized ED forms. Additionally, ED nurses used OpenSDE; a structured data entry application that, for the purpose of this study, was tailored for data collection on febrile children (▶). ¹⁶ Patient characteristics (gender, age, reason of the visit, visit date), referral profile, duration of the febrile episode, symptoms (earache, sore throat, runny nose, coughing, vomiting, diarrhea), and observations and measures from physical examination (e.g., vital signs, temperature, presence of chest-wall retractions, clinical appearance, and meningeal irritation) were collected during or immediately after ED nurse evaluation, as the patient was waiting for the attending ED physician. Total registration time per patient was less than two minutes.

Figure 1.

CDSS screenshot. The advice is to order laboratory testing for this child. ☑ = present, ☒ = absent.

Patients with chronic comorbidity (e.g., malignancies, cystic fibrosis, severe psychomotor retardation) were excluded. Furthermore, we excluded acutely ill patients since they were treated immediately. This study was approved by the Institutional Review Board (IRB) of the Erasmus Medical Center: Informed consent was obtained (94%, n = 164) after discharge from the ED as an informed consent procedure prior to the intervention would have interfered with the outcome measures (see Measurements section), and because, according to the IRB-review, the intervention had no additional risks for the patient.

Prediction Rules

Two prediction rules were previously developed for children with fever without apparent source. ^17–19 The prediction rule generated patient-specific risk-scores for SBI and was developed using data of 381 patients between 1 and 36 months of age, who presented to the ED of either the Sophia Children’s Hospital in Rotterdam (1996–1998, 2000–2001) or the Juliana Children’s Hospital in The Hague (1998, 2000–2001) with fever without apparent source (FWS). FWS was defined as a body temperature of at least 38.0 degrees Celsius, for which no clear source was identified after evaluation by the GP or after the history was taken by the pediatrician. A ‘clinical model’ and a ‘clinical + lab model’ were derived with an area under the receiver operating characteristic curve (AUC) of 0.69 and 0.86, respectively. See ▶ for predictor variables. The AUC of the prediction model for the ‘GP-referred’ patients was 0.60 in the self-referred patients. Because of the poor discriminative ability, the prediction rule was inadequate to classify SBI in self-referred patients. Therefore, for self-referred patients, a separate prediction rule was developed based on data of 109 self-referred patients attending the ED of either the Sophia Children’s Hospital in Rotterdam (1996–1998) or the Juliana Children’s Hospital in The Hague (1998), with FWS. A ‘clinical’ and ‘clinical + lab’ model for self-referred patients were derived, with AUCs of 0.70 and 0.81, respectively. ¹⁸ See ▶ for predictor variables. Furthermore, in Europe a GP has a ‘gatekeeper’ function; the GP selects the patients who need hospital evaluation. Therefore the groups of GP-referred and self-referred patients differ considerably in their characteristics.

Appendix 1.

Appendix 1 Clinical Score Chart for Referred Children

Characteristic	Points to assign		Score
Duration of fever (days)	Days	Points
	½	0
	1	2
	1½	4
	2–2½	5
	3–3½	6
	4–4½	7
	5–6	8
	6½–8½	9
	≥ 9	10
History of vomiting	no = 0 / yes = 5 points
Ill clinical appearance	no = 0 / yes = 4 points
Chest-wall retractions + tachypnea	no = 0 / yes = 12 points
Poor peripheral circulation	no = 0 / yes = 7 points		+
			Clinical Score

Low risk-score (≤ 10 points) corresponds with ≤ 12% risk of serious bacterial infection.

High risk-score (> 10 points) corresponds with > 40% risk of serious bacterial infection.

Appendix 2.

Appendix 2 Clinical Score Chart for Self-referred Children

Characteristic	Points to assign		Score
Duration of fever (days)	For each day of fever 1 point (½ days are rounded up), maximum 10 points
Age	≤ 1 year	0
	> 1 year	−4
Temperature (°C)	< 38 ∗	0
	> 38	1 point for every 0.5°C
			Total + 3
			Clinical Score

^∗ If temperature at examination is < 38°C, then temperature in history must be ≥ 38°C. Low risk-score (< 7 points) corresponds with < 6% risk of serious bacterial infection. High risk-score (≥ 7 points) corresponds with ≥ 20% risk of serious bacterial infection.

Methods

A clinical decision support system (CDSS) for the diagnostic management of children attending the ED with fever without apparent source (FWS) was developed and implemented at the ED from July 1, 2003, two months prior to the start of the study. In this period, all ED nurses received standardized training in how to use the CDSS. Registration of the data took approximately 2 minutes per patient and did not alter workflow significantly.

FWS was defined as body temperature ≥38.0 degrees Celsius, and no apparent source found after evaluation by the ED nurse. ¹⁷ The following symptom or combinations of symptoms were considered an apparent source of the fever: neck stiffness, two or more specified upper-airway symptoms (earache, sore throat, rhinitis), coughing, and at least one upper-airway symptom, or the combination of vomiting and diarrhea. All others were classified as FWS, and were automatically identified based on collected data. In each individual case, the CDSS calculated a clinical risk-score for SBI (see ▶ and ▶ for score-charts). Based on a prior defined cutoff point, children were classified according to the likelihood of having either a low or a high risk for SBI (see ▶ and ▶). ¹⁷ When data items necessary to calculate the risk-score (i.e., predictors for the presence of SBI) were missing, the CDSS provided a reminder during patient data registration.

Children with a high risk-score were eligible for early initiation of diagnostic workup: all children with a high risk-score were randomized to determine whether the advice ‘order laboratory tests’ was shown to the user (▶).

Patient selection, patients with high and low risk scores, and the randomization of high risk-patients are graphically shown in ▶. Randomization was based on a computer algorithm, sampling a number between 1 and 1000 in each case. High risk patients were assigned to the ‘intervention group’ (i.e., order laboratory tests) when an even number and to the ‘control group’ when an odd number was sampled. The laboratory tests to be ordered were based on prior consensus and consisted of complete blood count (CBC) and C-reactive protein (CRP). ¹⁷ All patients were evaluated by ED physicians.

Figure 2.

Patient selection and randomization. FWS = Fever Without apparent Source. CDSS = Clinical Decision Support System.

Measurements

Based on the time of arrival at the ED and the time of departure from the ED as registered in the ED nursing record, total ED time was calculated as the difference between time of arrival and time of departure in minutes. Data on all performed laboratory tests and additional diagnostic tests (e.g., Roentgenograms) were collected for each patient from the computer-based hospital information system.

All final diagnoses were classified as either serious bacterial infection (SBI) or non-SBI. SBI was defined as culture or radiographically proven bacterial infection (e.g., meningitis, sepsis, bacteremia, pneumonia, urinary tract infection, bacterial gastroenteritis, osteomyelitis, or ethmoiditis). Detailed descriptions of outcome diagnoses had been published earlier. ^17,20 Assessment of the outcome measures was blinded for risk-scores and CDSS recommendations. The relation between high and low clinical risk score and the outcome measure SBI was presented.

Statistical Analysis

First, we calculated compliance with CDSS registration of febrile patients as the percentage of all febrile patients who attended the ED with fever. Secondly, we assessed the adherence with CDSS recommendations by calculating the percentage of cases in which laboratory tests were ordered as advised.

In the children identified with FWS and having a high SBI risk-score, we compared the ‘intervention group’ to the ‘control group’ regarding total time spent at the ED and the frequency of diagnostic testing. A Mann-Whitney U-test was used to quantify the difference in total ED time between the two groups, and a Chi-square test to assess the difference in frequency of diagnostic testing. Both an intention to treat analysis (all children in the intervention group analyzed) and per protocol analysis (only the children in the intervention group in whom laboratory tests were actually ordered analyzed) were performed. In a sub-analysis we only analyzed children who had laboratory tests ordered (intervention and control groups). Furthermore, the frequency of SBI was compared between the children with a low and children with a high risk-score, using the Chi-square test. The area under the receiver operating characteristic curve (AUC) of the prediction rule was calculated as measure of discriminative ability.

General characteristics, laboratory test results, risk-scores and incidence of SBI were compared between the intervention and the control group, using a Chi-square test for categorical and the Mann-Whitney U-test for continuous variables. A p-value lower than 0.05 was considered statistically significant. We used SPSS software (version 12.0, SPSS Inc, Chicago, IL) for the statistical analyses.

Results

From September 1, 2003 to December 31, 2005, 1,774 children with fever, aged 1–36 months, attended the ED of the Sophia’s Children’s Hospital in Rotterdam. Of those, 375 children were excluded because of chronic comorbidity. Compliance with CDSS registration of febrile children was 49%: 683 of the 1,399 eligible patients were registered. Of the 683 registered patients, 390 (57%) were identified with fever without apparent source (FWS). A total of 172 children had a high risk-score (i.e., 218 had a low risk-score) and were thus randomized (▶). In 30 patients, total ED time could not be calculated due to missing time of arrival at or departure from the ED (16 in the intervention group, 14 in the control group). Informed consent was obtained in 164 patients (95%).

▶ shows the characteristics of children with a low risk- and high risk-score, the latter stratified for intervention and control group. Both high risk groups had similar characteristics.

Table 1.

Table 1 General Characteristics of Children with Fever Without Apparent Source, Stratified for Risk-score and Randomization

Characteristics	Low Risk (n = 218)	High Risk (n = 164)
		Intervention (n = 74)	Control (n = 90)
Patient history
Male gender ∗	132 (61)	44 (59)	46 (51)
Age (years)†	1.3 (0.7–2.0)	1.0 (0.7–1.6)	0.9 (0.6–1.4)
Duration of fever (days)†	1.0 (1.0–2.0)	2.5 (1.0–4.0)	3.0 (1.8–6.0)
History of vomiting ∗	57 (26)	34 (46)	46 (51)
Physical examination
Temperature (° Celsius)†	39.0 (38.4–39.7)	39.5 (39.0–40.0)	39.4 (38.9–40.0)
Ill clinical appearance ∗	111 (51)	53 (72)	61 (68)
Poor peripheral circulation ∗	22 (10)	5 (7)	13 (14)
Chest-wall retractions ± tachypnea ∗	6 (3)	10 (14)	7 (8)
Clinical risk-score
Score†	5 (3–6)	11 (9–14)	11 (9–14)
Final diagnosis
SBI ∗	21 (10)	10 (14)	16 (18)

^∗ Absolute number (%);

^† Median (25^th and 75^th percentile).

In ▶, the median time spent in the ED is shown for both the intervention and the control group. The total ED time was not significantly different between the intervention and the control group. Also in the per protocol analysis no significant difference was found.

Table 2.

Table 2 Time Spent in the ED for Intervention and Control Group (High-risk Patients)

Analysis	Total ED-time (minutes)				p-value
	Intervention	(n)	Control	(n)
Intention to treat	138 (104–181)	58	123 (83–179)	76	0.16
Per protocol	140 (116–184)	52	123 (83–179) ∗	76	0.06
Lab tests ordered	140 (116–184)	52	160 (115–213)	33	0.43

^∗ By definition identical to intention to treat value.

Numbers represent median (25^th and 75^th percentile).

In a subgroup-analysis (‘lab tests ordered’), we found that median ED time for children in the intervention-group with laboratory tests performed was 140 minutes compared to 160 minutes for children in the control group with laboratory tests performed, but this was not significant (p = 0.43).

A significant difference in the number of laboratory tests was found between the intervention and the control group. In 82% (61 out of 74) of the cases in which the ED nurse received the advice to order laboratory tests, the tests were actually ordered. In the control group, laboratory tests were ordered in 44% (40 out of 90) of the patients, at the discretion of the ED physician.

▶ shows a cross-tabulation of the presence of an SBI and the clinical risk-score of all 390 patients. SBI was found slightly more often among the children with a high risk-score (15%) than among the children with a low risk-score (10%, p = 0.10). Furthermore, the ability to predict SBI based on high or low risk-score was disappointing in this ED population, as indicated by an AUC of 0.56 (95%CI 0.48–0.65).

Table 3.

Table 3 Presence of a Serious Bacterial Infection (SBI)

	Clinical Score
	Low Score	High Score
SBI present	21 (10)	26 (15)	47
SBI absent	197 (90)	146 (85)	343
Total	218	172	390

Numbers represent absolute numbers (percentage within risk stratum).

p-value: χ² test = 0.10.

Discussion

In this study we implemented a clinical decision support system (CDSS) for the diagnostic management of young children with fever without apparent source, who are at risk for serious bacterial infection (SBI). Compliance with registration of febrile children in the CDSS was moderate, with 49% of the children being registered by ED nursing staff. Adherence with the advice to order laboratory tests was good: in 82% of the cases in which the advice to order laboratory tests was given, the tests were actually ordered. The clinical decision rule, predicting whether children are at low or high risk for SBI, had a lower discriminative ability (AUC 0.56 (0.48–065)) than expected based on the validation study. ¹⁹ The children in whom laboratory tests were ordered immediately after nurse evaluation spent no shorter time in the ED than the children in whom laboratory tests were ordered at the discretion of the attending physician.

In a recent review, four CDSS features were identified that were closely correlated with the ability of a CDSS to improve patient care. First, the decision support should be part of the routine workflow. Second, a computer system should be used to provide decision support. Third, an explicit patient-specific recommendation should be given rather than a probability, and fourth, decision support should be delivered at the time and location of decision making. In this study most of these features were included: children presenting with fever were registered in a computer system that automatically provided an actionable recommendation on diagnostic management at time and location of the nurse’s decision. However, registration of the children was not yet part of the routine workflow and was performed voluntarily by ED nursing staff. This might account for the fact that only 49% of the febrile children were registered. Furthermore, ED crowding and time constraints may have accounted for the moderate registration rate, as it was an extra task. Furthermore, the ED nurses were aware that the decision rule only included children with fever without source. The registration rate for children with fever with an apparent source might therefore be low. The percentage of children with FWS was 57% (390/683) in our study. In the literature however, incidences of 10–20% of FWS in general pediatric ED populations are described. ^21,22 Therefore, the actual registration rate of children with FWS may have been relatively high. Although we did not define a minimum compliance and adherence rate prior to the study, we conclude that implementation of the CDSS with regard to application of the system and adherence to the advice was successful. For example, in a 2005 study by Brehaut et al. the compliance with a well known clinical decision rule, the Ottawa ankle rule, was only 40%, while 90% reported using the rule most of the time. ²³

In contrast to our expectations, the children in the intervention group spent no shorter time in the ED than the children in the control group. The difference of 20 minutes in the ‘per protocol analysis’ was, however, statistically not significant. Explanations for the prolonged ED time in the intervention group include the large difference in the frequency in which laboratory tests were ordered in the intervention (82%) and control group (44%). However, this difference also indicates that the amount of diagnostic tests significantly increased when the CDSS was used, which was in contrast with the expected decrease. The subgroup-analysis in which only the children who had laboratory tests performed were compared, revealed that children whose tests had been ordered according to the CDSS recommendation had a median ED time that was 20 minutes less than children whose tests had been ordered at the discretion of the physician (140 vs. 160 minutes). This indicates that the CDSS may potentially be effective, when children at high risk for SBI are more accurately identified. It has already been recognized that the effectiveness of a CDSS depends for a major part on the strength of the knowledge base that is used. Sim et al. stated that “a CDSS can only be as effective as the strength of the underlying evidence base; the effectiveness of CDSS will be limited by any deficiencies in the quality or relevance of the research evidence.” ²⁴ The incidence of SBI in the high-risk group (15%) was only slightly higher compared to the low-risk group (10%). The prediction rules that were integrated in the CDSS need adjustment such that a better discriminative ability is achieved than the current version of the rule. This requires further study on the relationships between patient characteristics and the presence of SBI, and further multivariable analyses to update the prediction model. ²⁵

Some other aspects of this study need to be addressed. The clinical condition of the children who were registered in the CDSS by ED nurses may have been better than of the children who were not registered. We are, however, not able to compare these groups as information on whether the non-CDSS-registered children had FWS according to the ED nurse; this cannot accurately be determined in hindsight. Secondly, all children were evaluated by the attending physician. When laboratory tests were already ordered by the ED nurse, the physician was automatically aware that the child had a high risk-score for SBI. This may have affected subsequent diagnostic testing.

Although the system was successful regarding compliance and adherence to CDSS recommendations, the unexpected effects on patient outcomes shows that the decision rules have to be revised. Further study is needed to assess whether new prediction models should be developed or that the number of false ‘high-risk’ classifications should be reduced by increasing the risk-score ‘threshold’, or both.

Conclusion

Implementation of a CDSS for the diagnostic management of young children with fever without apparent source was successful regarding compliance and adherence to CDSS recommendations, but had unexpected effects on patient outcome in terms of ED length of stay and amount of performed laboratory tests. The use of the current CDSS was discontinued. This study stresses the importance of including patient outcomes in the evaluation of clinical decision support systems.