Identifying serious bacterial illness in children: is it time to show the red card to NICE’s red flags?

THE TRAFFIC LIGHT SYSTEM

This review analyses current evidence for the diagnostic accuracy of the Traffic Light System (TLS) in primary care, its implications for GPs, and its alternatives. Two linked studies^1,2 of the performance of the National Institute for Health and Care Excellence’s TLS concluded that it was unfit for purpose. TLS aims ‘to improve clinical assessment and help healthcare professionals diagnose serious illness among young children [under 5] who present with fever’,³ a notoriously difficult assessment since fever in children is common while serious bacterial infection (SBI) is rare. The TLS groups clinical features into three categories with corresponding recommendations: red (serious illness likely, hospital admission recommended), green (serious illness unlikely, manage at home), and amber (intermediate risk, judgement required). Clark et al² reported that the TLS was insufficiently accurate to identify infants in primary care with SBI and to exclude those without. Arguably these findings suggest abandoning the TLS. In preparing this review, all references from both studies were retrieved. Backward and forward searches from the references were cascaded until no new references were found.

WHAT CLARK AND BLYTH FOUND

Blyth et al¹ and Clark et al² mapped data from the previous prospective cohort DUTY study⁴ of over 6700 children presenting with constitutional or urinary tract infection symptoms to GPs, walk-in centres, and emergency departments (EDs) to TLS items. A third of under-5s had at least one red feature.¹ The accuracy of the TLS, using hospital-diagnosed SBI as the reference standard, was calculated for two thresholds: any red feature and either red or amber feature.² Any red feature had a sensitivity of 58.8% (95% confidence interval [CI] = 32.9% to 81.6%) and a specificity of 68.5% (CI = 67.4% to 69.6%). Using the lower threshold improved sensitivity to 100% (CI = 80.5% to 100) but debased specificity to 5.7% (CI = 5.2% to 6.3%). Such performance for a national guideline is disappointing. The authors could not map all features to TLS, which may explain the poor performance, but other explanations exist.

THE EVIDENCE MAY NOT BE RELEVANT TO PRIMARY CARE

Many TLS features were first tested in secondary care, where prevalence is higher.^2,5–9 Diagnostic accuracy calculated in secondary care cannot automatically apply to primary care because the populations differ, as demonstrated by the Wells thromboembolism scores.¹⁰ Furthermore, while the TLS targets under-5s, the studies on which it is based included some that restricted the age range to infants less than 6 months and others that included older children.

THE TLS ITEMS MAY LACK ACCURACY IN ANY SETTING

Likelihood ratios (LRs) are useful in comparing tests with one another. A rule of thumb for a ‘good’ test is a positive LR, LR+ ≥5, and a negative LR, LR–≤0.2. The poor performance of the TLS even in the high-prevalence populations of EDs, where LR+ lies between 1.09 to 1.99,^6,7 suggests that the test in its present form lacks merit anywhere. Yet some individual TLS items may be strong indicators of SBI, although the aggregate performs poorly. Several studies in EDs have reported high performance of vital or pathognomic signs: decreased consciousness, poor circulation, petechial rash, meningism, tachypnoea, tachycardia, temperature ≥39.5°C, and O₂ saturations <94%.^8,11,12 Practitioner-perceived ‘ill appearance’ or ‘instinct’ has been reported to be a good predictor in primary care with an odds ratio (OR) of 62,¹³ LR+ 7.3 (5.3‒10.1) and LR–0.2 (0.1–0.8)⁵ as well as in paediatric emergency care with ORs of 2.63 (0.99–7.00),¹¹ 1.62 (1.12‒2.33)¹⁴ and 1.90 (1.04–3.68).⁸

ALTERNATIVES TO TLS

Two primary care alternatives to TLS are the Dutch College of General Practitioners guideline Children with Fever (NHG)¹⁵ and the decision tree of Van den Bruel et al.¹³ NHG is a narrative description of symptoms and signs to look for. Both NHG and TLS were tested against a previous prospective cohort study.¹⁶ The results for LR+ were TLS 1.87 (1.25–2.78) and NHG 1.04 (0.97‒1.12), and for LR–TLS 0.60 (0.34‒1.06) and NHG 0.39 (0.25‒6.22). NHG also recommends dip testing the urine in the under-2s, a bedside test that improves accuracy for urinary tract infections (UTIs).⁷

Van den Bruel¹³ derived a ‘decision tree’ fr+om a prospective study of nearly 4000 children aged 0–15.9 years in primary care. The tree features questions in sequential steps: a positive answer at any step is taken to indicate a high risk of SBI. The 6-step tree had high accuracy: LR+ 8.4 (7.6–9.4), and LR–0.04 (0.01–0.2). Such impressive results suggest it is reliable. A model derived in one sample (derivation group) should be tested in another sample (validation group) because observational studies may inadequately control for unknown confounders.¹⁷ Two validation studies of the decision tree, Verbakel¹⁶ and van Ierland,¹⁸ used referral as a surrogate, which cannot be assumed to be an accurate estimate of the incidence of SBI. The third, Verbakel et al,⁵ used the more objective outcome of hospital diagnosis. The results for LR+ and LR–were: 6.13 (5.34‒7.03) and 0.11 (0.04–0.33);¹⁶ 1.7 (1.6‒1.8) and 0.7 (0.6‒0.7);¹⁸ and LR+ 7.3 (5.3‒8.1) and LR–0.2 (0.1‒0.8).⁵ However, the study using diagnosis as outcome⁵ combined two nodes to create a new node, which means it is not an exact replication of the original study.

DISCUSSION

This analysis updates previous reviews. Verbakel et al’s¹⁶ retrospective evaluation of TLS and NHG used the surrogate outcome, referral, on a small sample, approximately 500.¹⁹ Thompson et al’s²⁰ review had only the original Van den Bruel study.¹³ This current analysis includes the larger data set of Clark et al with the more accurate outcome of SBI and includes further studies of the decision tree.

The red items of the TLS have been found to have poor sensitivity and specificity for SBI in EDs and primary care. Clarke may have underestimated TLS’s performance since not all features were mapped. However, the underestimation would have to be huge to raise the diagnostic accuracy to an acceptable level. Sensitivity could be improved, to 100%, by treating red and amber items equally but only with undesirable consequences: an unrealistic rate of referral, 94% of children presenting with fever. The Dutch NHG did not have superior accuracy. In practice, GPs do not follow the TLS guidance even on red items. In the DUTY study, only 43 (6.2%) children with a red flag were referred for same-day assessment.¹ Studies elsewhere report that practitioners refer fewer children than guidelines recommend. In a Netherlands primary out-of-hours service, only 19% of 3424 children with a positive indication were referred to the ED.¹⁸ Ambulatory care paediatricians in the US diverged from guidelines in 36% of cases.¹¹

So primary care practitioners’ decisions to refer patients for same-day assessment depend on more than the presence of clinical features specified in guidelines. Divergence from guidelines is not unreasonable: a tool that rules out SBI when neither red nor amber features are present is of little use when it applies to only 5.7% of cases. Can more research find a solution? ‘Instinct’ appears useful but we do not know what it constitutes nor how big a part it plays in GPs’ decisions. Perhaps qualitative studies of GP instinct may provide understanding? The decision tree has been the best performing tool but further replication studies are desirable. However, it returned a high proportion of positives, 23%, in primary care in the most recent study.⁵ Overall only 11.8% were admitted, indicating that practitioners use judgement not present in the tool.

Future research might move from listing individual features or their combinations towards the development of a prediction rule that weights features to calculate a predictive value, as do the Wells score and QFracture. To be accurate. it must use objective rather than surrogate outcomes. To address the problem correctly, the sample should consist of children under 5. To be relevant to primary care, such research must be developed in primary care itself.

Provenance

Freely submitted; externally peer reviewed.