Abstract
Objective
To externally validate four commonly used rules in computed tomography (CT) for minor head injury.
Design
Prospective, multicentre cohort study.
Setting
Three university and six non-university hospitals in the Netherlands.
Participants
Consecutive adult patients aged 16 years and over who presented with minor head injury at the emergency department with a Glasgow coma scale score of 13-15 between March 2015 and December 2016.
Main outcome measures
The primary outcome was any intracranial traumatic finding on CT; the secondary outcome was a potential neurosurgical lesion on CT, which was defined as an intracranial traumatic finding on CT that could lead to a neurosurgical intervention or death. The sensitivity, specificity, and clinical usefulness (defined as net proportional benefit, a weighted sum of true positive classifications) of the four CT decision rules. The rules included the CT in head injury patients (CHIP) rule, New Orleans criteria (NOC), Canadian CT head rule (CCHR), and National Institute for Health and Care Excellence (NICE) guideline for head injury.
Results
For the primary analysis, only six centres that included patients with and without CT were selected. Of 4557 eligible patients who presented with minor head injury, 3742 (82%) received a CT scan; 384 (8%) had a intracranial traumatic finding on CT, and 74 (2%) had a potential neurosurgical lesion. The sensitivity for any intracranial traumatic finding on CT ranged from 73% (NICE) to 99% (NOC); specificity ranged from 4% (NOC) to 61% (NICE). Sensitivity for a potential neurosurgical lesion ranged between 85% (NICE) and 100% (NOC); specificity from 4% (NOC) to 59% (NICE). Clinical usefulness depended on thresholds for performing CT scanning: the NOC rule was preferable at a low threshold, the NICE rule was preferable at a higher threshold, whereas the CHIP rule was preferable for an intermediate threshold.
Conclusions
Application of the CHIP, NOC, CCHR, or NICE decision rules can lead to a wide variation in CT scanning among patients with minor head injury, resulting in many unnecessary CT scans and some missed intracranial traumatic findings. Until an existing decision rule has been updated, any of the four rules can be used for patients presenting minor head injuries at the emergency department. Use of the CHIP rule is recommended because it leads to a substantial reduction in CT scans while missing few potential neurosurgical lesions.
Introduction
Minor head injury or mild traumatic brain injury is a common injury increasingly seen in emergency departments.1 2 Possible causes for this increase are ageing of the population and increased awareness of the potential intracranial complications of minor head injury among general practitioners and paramedics.3 4 Although the risk of intracranial complications after minor head injury is low, the consequences are important because these patients need close observation and sometimes even neurosurgical intervention.5 Several clinical decision rules exist that aim to identify those patients with minor head injuries who are at high risk for intracranial complications and need computed tomography (CT) of the head. Examples of frequently used decision rules are: the New Orleans criteria (NOC), Canadian CT head rule (CCHR), and the National Institute for Health and Care Excellence (NICE) guideline for head injury (appendix 1).6 7 8
The purpose of these rules is to detect all relevant intracranial traumatic lesions while minimising the number of unnecessary CT scans. Relevant lesions are those that need neurosurgical intervention or prolonged clinical observation because of a risk of neurological deterioration. Although the number of patients that present at the emergency departments with minor head injury has increased substantially, the overall incidence of disease specific mortality after head injury has remained fairly stable.9 An increased number of patients leads to more CT scans, longer waiting times at the emergency department, burden for the patients, radiation risks, and higher costs.10 The need for reliable CT decision rules for minor head injury to reduce unnecessary CT scans is therefore even more apparent.
Two of the decision rules were developed for patients who had had blunt trauma to the head, had a Glasgow coma scale score of 13-15 at presentation, and had experienced loss of consciousness or post-traumatic amnesia.6 7 However, these two rules could not be applied to patients who had not experienced loss of consciousness or post-traumatic amnesia.11 12 Therefore, a new decision rule was developed, the CT in head injury patients (CHIP) rule, which includes patients with and without loss of consciousness or post-traumatic amnesia.13 The potential reduction of CT scans by use of the CHIP rule was estimated at 23% compared with the scanning of all patients.13
The NOC, CCHR, and NICE guidelines were externally validated in previous studies, but there has been no external validation of the CHIP rule, even though this is necessary to determine whether the rule is generally applicable.14 15 16 17 18 19 20 21 Our aim was to perform an external validation of frequently used CT decision rules for minor head injury (CHIP, NOC, CCHR, and NICE) and compare their performance in a multicentre study in the Netherlands in university and non-university hospitals.
Methods
Study design
We conducted a prospective, multicentre cohort study between March 2015 and December 2016 in the Netherlands. Three university emergency departments (all level 1 trauma centres) and six non-university emergency departments (trauma level 1 (two centres), trauma level 2 (two centres), and trauma level 3 (two centres)) participated in this study. The emergency departments were all situated at an urban location. Institutional ethics and research board approval was obtained and informed consent was waived.
Inclusion criteria were age 16 years and over, presentation within 24 h after blunt trauma to the head, and a Glasgow coma scale score of 13-15 at presentation at the emergency department. Patients with and without loss of consciousness or post-traumatic amnesia were included. We excluded all patients with a Glasgow coma scale score of less than 13, patients younger than 16 years, transferred from other hospitals, or with any contraindication for CT.
Definition of risk factors
Clinical data concerning risk factors for intracranial complications used in the CCHR, NOC, NICE, and CHIP decision rules were collected.6 7 8 13 These clinical risk factors were:
Age
History of coagulopathy
Use of anticoagulants
Dangerous trauma mechanism (pedestrian/cyclist v vehicle, ejected from vehicle, fall from elevation (>1 m or 5 stairs), or an equivalent mechanism)
Fall from any elevation
Loss of consciousness reported by patient or witness
Retrograde amnesia
Post-traumatic amnesia
Headache
Vomiting
Intoxication with drugs or alcohol (history or suggestive findings on examination)
Post-traumatic seizure
Glasgow coma scale score on presentation
Significant injury above clavicles
Suspected open or depressed skull fracture
Contusion of skull
Clinical signs of skull base fracture (eg, raccoon eyes, battle sign, haemotympanum, cerebrospinal fluid otorrhea, cerebrospinal fluid rhinorrhea, palpable discontinuity, or bleeding from ear)
Neurological deficit (paresis, dysphasia, or other such as cranial nerve damage including diplopia, changes in sensibility, asymmetrical reflexes or pathological reflexes, coordination problems and ataxia)
Deterioration in Glasgow coma scale 1 h after presentation.
Main outcome measures
The primary outcome was any (intra)cranial traumatic finding on CT, defined as a subdural haematoma, epidural haematoma, subarachnoid haemorrhage, cerebral lesions (haemorrhagic contusion, non-haemorrhagic contusion, diffuse axonal injury), intraventricular haemorrhage, and skull fracture. The secondary outcome was any potential neurosurgical lesion, which was defined as an intracranial traumatic finding on CT that could lead to a neurosurgical intervention or death. Examples of potential neurosurgical lesions are an epidural haematoma, large acute subdural haematoma (mass), large contusion(s) (mass), depressed skull fracture, and any lesion with a midline shift or herniation. To compare our findings with previous studies, we also assessed the performance of decision rules for detecting neurosurgical interventions. All outcome measures were chosen a priori.
Study procedures
During patient inclusion in the study, neurologists (in training) and emergency physicians (in training) followed their local guideline for CT scanning in patients with minor head injury. Most participating centres used the same national guideline based on the CHIP rule, two centres followed a slightly adapted guideline (appendix 2).
Eligible patients were consecutively included by trained researcher physicians, who did not personally interview the patients. Clinical data were collected before diagnostic tests as far as possible by using forms the clinicians could fill in for each patient. The head CT scans were performed according to a routine trauma protocol at each hospital. The scans were interpreted by (neuro)radiologists who were aware of the patient’s history and clinical findings, but they were not aware of the actual score of the CT decision rules.
The clinical risk factors were collected by taking the patient’s history or information from a witness or family member. Characteristics such as injury severity score were also collected. All patients’ details about hospital admission, neurosurgical intervention, and moment of discharge were collected. If the patient was scanned, details about CT findings were recorded. The electronic health records were reviewed 30 days after the injury to assess follow-up information about a neurosurgical intervention. All data were entered by researcher physicians in the case report forms of the web based data management system OpenClinica (LCC, version 3.12.2).
Data management
After patient inclusion and data entering, two authors (KAF and CLvdB) checked the database for correct patient inclusion and completeness of data using IBM statistical package for social sciences (SPSS) version 21. Missing data were assumed to be missing at random; so to avoid bias, missing data were imputed on the basis of all the risk factors mentioned above, using multiple imputation (n=5) with the “multivariate imputation by chained equations” function in R, version 3.3.2 (R foundation for statistical computing).
Data analysis
The study population was described in terms of demographic characteristics, risk factors, admission to the hospital, and neurosurgical intervention. In patients with a CT scan, we also evaluated any intracranial traumatic findings and potential neurosurgical lesions on CT. Continuous variables were described as mean and interquartile range, categorical variables as frequencies and percentages.
The diagnostic performance of the CHIP, NOC, CCHR, and NICE decision rules for detecting intracranial traumatic findings and potential neurosurgical lesions were compared. Because the NOC and CCHR rules were developed in a specific patient population, we performed the analysis in our entire study population, as well as in a subset of the study population (based on the inclusion/exclusion criteria of the development studies of the NOC and CCHR; referred to as original NOC and original CCHR), and in our entire study population with adjustment of the rules. In the adjusted rules, the exclusion criteria of the NOC and CCHR rules were added as additional risk factors (referred to as adjusted NOC and adjusted CCHR). For the NOC rule, a Glasgow coma scale score of 13 or 14 and presence of neurological deficit were added. Finally, for the CCHR rule, use of anticoagulation, post-traumatic seizure, and presence of neurological deficit were added. All patients who had a risk factor according to the NOC or CCHR rules scored positive on these rules, indicating that they needed a CT scan.
The sensitivity, specificity, and proportion of patients needing a CT scan (with 95% confidence intervals) were assessed for each of the four decision rules. Sensitivity was calculated by dividing the number of patients in whom the outcome measure was present and the decision rule was positive, by the total number of patients in whom the outcome measure was present. Specificity was calculated by dividing the number of patients in whom the outcome measure was absent and the decision rule was negative, by the total number of patients in whom the outcome measure was absent. The Cochran’s Q test was used to directly compare the sensitivities and specificities between the four decision rules, but it should be noted that results of this test do not automatically imply that any one rule is better than the other.22 The proportion of patients needing a CT scan was calculated by dividing the number of patients in whom the decision rule was positive by the total number of patients. Confidence intervals were calculated by a bootstrapping method in R, which analyses the performance for each rule 500 times and derived the confidence intervals from the results.
In patients without a CT scan, the outcomes could not be observed. In these patients, the expected outcomes (any intracranial traumatic finding and potential neurosurgical lesion) were imputed on the basis of their risk factors with multiple imputation, in order to avoid selection bias and thus yield unbiased estimates of sensitivity and specificity.23 This imputation was possible for patients from six of the nine centres, because the other three centres had not included patients without a CT scan. The patients with and without CT scans (with imputed outcomes) from these six centres were used for the primary analysis. In addition, we analysed all patients with a CT scan from all the centres in a secondary (sensitivity) analysis, which in theory would lead to an overestimation of sensitivity and underestimation of specificity of all the rules.
In this decision problem, avoiding false negatives was more important than avoiding false positives: a false negative result leads to not performing a CT scan and thus potentially misses a lesion, whereas a false positive result leads to performing an unnecessary CT scan. The decision rule should identify all patients with potential neurosurgical lesions and most with intracranial traumatic findings, because of the severe clinical consequences (intracranial surgery, neurological sequelae, death).
Net proportional benefit has been proposed to incorporate such weighting in calculation of clinical usefulness of decision rules.24 25 For each rule, we expressed the net proportional benefit using the formula: (true positives/total number) − weight×(false positives/total number). Over a range of different weights, the net proportional benefit was calculated and compared with the scanning of all patients. The weight in this formula expresses the ratio of harmful consequences due to a false positive divided by the harmful consequences of a false negative, and it is equivalent to the odds of a lesion above which one would perform a CT scan. At a low threshold for performing CT, we would avoid false negatives of the decision rule (that is, maximise true positives) at the cost of performing many CT scans: if the threshold is 1%, this level implies performing 100 CT scans to avoid one missed lesion. At a higher threshold for performing CT, we would avoid false positives of the decision rule: if the threshold is 10%, this level implies performing 10 CT scans to avoid one missed lesion. We considered an intermediate range of thresholds (4-6% for any traumatic finding and 0.5%-1% for potential neurosurgical lesion) acceptable from a clinical point of view.24 26 Net proportional benefit expresses the true positives, and the decision rule with the highest net benefit at the intermediate thresholds has the highest clinical value.24 All statistical analyses were performed with R software, version 3.3.2 (R foundation for statistical computing, Vienna, Austria).
Patient involvement
No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for design or implementation of the study. No patients were asked to advise on interpretation or writing up of results. There are plans to disseminate the results of the research to the relevant patient community.
Results
Between March 2015 and December 2016, 5839 consecutive patients with minor head injury were entered in the database in the participating centres (fig 1). After checking the inclusion and exclusion criteria, 322 patients were excluded from the study (Glasgow coma scale score <13, age <16 years, or no blunt head injury). In three of the nine centres, only patients with a CT scan were included (n=960). The remaining six centres included patients with and without a CT scan (n=4557).
Fig 1.
Study flow diagram. *Six centres=one university centre (trauma level 1) and five non-university centres (trauma levels 1 (two centres), 2 (one), 3 (two)), including patients with and without CT scans; three centres=two university centres (both trauma level 1) and one non-university centre (trauma level 2), including only patients with a CT scan. CT=computed tomography
For the primary analysis, 4557 patients from six centres were included; 3742 (82.1%) received a CT scan and 815 (17.9%) did not. Compared with patients who received a CT scan, more patients without a scan had a Glasgow coma scale score of 15 (n=3109 (83.1%) v n=805 (98.8%)), and fewer patients experienced loss of consciousness (n=1136 (30.3%) v n=56 (6.8%)) or post-traumatic amnesia (n=1075 (28.7%) v n=29 (3.5%); table 1). Some data were unknown to the including physician, which was most frequently the case for retrograde amnesia (n=675, 14.8%), loss of consciousness (n=651, 14.3%), post-traumatic amnesia (n=502, 11%), and headache (n=630, 13.8%; table 1).
Table 1.
Baseline characteristics of 4557 study patients from six centres*
| Characteristic | All patients (n=4557) | Missing | Patients with CT (n=3742) | Patients without CT (n=815) |
|---|---|---|---|---|
| Age (years; mean (range)) | 53.1 (16-101) | — | 56.9 (16-101) | 35.7 (16-96) |
| Male sex | 2656 (58.3) | — | 2145 (57.3) | 511 (62.7) |
| Glasgow coma scale score at presentation | ||||
| 13 | 143 (3.1) | — | 141 (3.8) | 2 (0.2) |
| 14 | 500 (11.0) | 492 (13.1) | 8 (1.0) | |
| 15 | 3914 (85.9) | 3109 (83.1) | 805 (98.8) | |
| Use of anticoagulation | ||||
| None | 4045 (88.8) | 29 (0.6) | 3233 (86.4) | 812 (99.6) |
| Coumarin | 418 (9.2) | 418 (11.2) | — | |
| Direct oral anticoagulants | 54 (1.2) | 53 (1.4) | 1 (0.1) | |
| Use of thrombocyte aggregation inhibitors | 615 (13.5) | 33 (0.7) | 577 (15.4) | 38 (4.7) |
| Bleeding disorder | 44 (1) | 33 (0.7) | 41 (1.1) | 3 (0.4) |
| Mechanism of injury | ||||
| Pedestrian in road traffic accident | 64 (1.4) | 47(1.0) | 57 (1.5) | 7 (0.9) |
| Cyclist in road traffic accident | 162 (3.6) | 152 (4.1) | 10 (1.2) | |
| Fall from height | 574 (12.6) | 532 (14.2) | 42 (5.2) | |
| Other† | 3710 (81.4) | 2955 (79.0) | 755 (92.6) | |
| Ejected from vehicle | 150 (3.3) | 56 (1.2) | 135 (3.6) | 15 (1.8) |
| Loss of consciousness | ||||
| None | 2714 (59.6) | 651 (14.3) | 1968 (52.6) | 746 (91.5) |
| ≤15 min | 1160 (25.5) | 1105 (29.5) | 55 (6.7) | |
| >15 min | 32 (0.7) | 31 (0.8) | 1 (0.1) | |
| Retrograde amnesia | ||||
| None | 3425 (75.2) | 675 (14.8) | 2637 (70.5) | 788 (96.7) |
| ≤30 min | 312 (6.8) | 303 (8.1) | 9 (1.1) | |
| >30 min | 145 (3.2) | 144 (3.8) | 1 (0.1) | |
| Post-traumatic amnesia | ||||
| None | 2951 (64.8) | 502 (11) | 2185 (58.4) | 766 (94.0) |
| ≤2 h | 976 (21.4) | 948 (25.3) | 28 (3.4) | |
| 2-4 h | 69 (1.5) | 68 (1.8) | 1 (0.1) | |
| >4 h | 59 (1.3) | 59 (1.6) | — | |
| Intoxication with drugs or alcohol‡ | 1031 (22.6) | 85 (1.9) | 922 (24.6) | 109 (13.4) |
| Post-traumatic seizure | 36 (0.8) | 68 (1.5) | 33 (0.9) | 3 (0.4) |
| Headache | 1410 (30.9) | 630 (13.8) | 1208 (32.3) | 202 (24.8) |
| Vomiting | ||||
| Once | 158 (3.5) | 50 (1.1) | 148 (4.0) | 10 (1.2) |
| Twice or more | 144 (3.2) | 142 (3.8) | 2 (0.2) | |
| Deterioration in Glasgow coma scale (1 h after presentation) | ||||
| 1 point | 38 (0.8) | 23 (0.5) | 38 (1.0) | — |
| ≥2 points | 12 (0.3) | 12 (0.3) | — | |
| Neurological deficit | 130 (2.9) | 141 (3.1) | 128 (3.4) | 2 (0.2) |
| Signs of skull base fracture | 144 (3.2) | 25 (0.5) | 139 (3.7) | 5 (0.6) |
| Visible injury of the head | 2564 (56.3) | 19 (0.4) | 2208 (59) | 356 (43.7) |
| Visible injury of the face | 1631 (35.8) | 22 (0.5) | 1315 (35.1) | 316 (38.8) |
| Suspicion of open fracture | 11 (0.2) | 40 (0.9) | 11 (0.3) | — |
| Injury severity score (mean (range)) | 6.5 (0-75) | — | 7.1 (0-75) | 3.5 (0-29) |
Data are number (%) of patients unless stated otherwise. CT=computed tomography.
These centres refer to those on the left hand side of figure 1, for the primary analysis.
Includes patients with mild head injury such as a bumped head against an object.
History or suggestive findings on examination (eg, nystagmus, abnormal walking).
In 384 patients (8.4%), CT showed an intracranial traumatic finding, mostly consisting of traumatic subarachnoid haemorrhages (n=182, 4.0%) and skull fractures (n=150, 3.3%; table 2). Of 74 (1.6%) patients with a potential neurosurgical lesion, 18 (0.4%) underwent a neurosurgical intervention for head injury within 30 days after the injury.
Table 2.
Traumatic CT findings in 3742 patients with a CT scan from six centres*
| CT finding | No (%)† |
|---|---|
| Total | 384 (8.4) |
| Skull fracture | 150 (3.3) |
| Depressed fracture | 19 (0.5) |
| Linear fracture | 66 1.4) |
| Skull base fracture | 68 (1.5) |
| Subarachnoid haemorrhage | 182 (4.0) |
| Contusion | |
| Small | 115 (2.5) |
| Large (mass) | 10 (0.2) |
| Subdural haematoma | |
| Small | 126 (2.8) |
| Large (mass) | 22 (0.5) |
| Epidural haematoma | |
| Small | 30 (0.7) |
| Large (mass) | 5 (0.1) |
| Suspicion of diffuse axonal injury on CT | 13 (0.3) |
| Basal cisterns compressed or obliterated | 11 (0.2) |
| CT shift | |
| 0-4 mm | 16 (0.4) |
| ≥5 mm | 9 (0.2) |
CT=computed tomography.
These centres refer to those on the left hand side of figure 1, for the primary analysis.
Some patients had more than one CT finding.
In 116 of 3742 patients without loss of consciousness and in 117 of 3742 patients without post-traumatic amnesia, an intracranial traumatic finding was found (table 3). In total, 20 patients without loss of consciousness had a potential neurosurgical lesion and four patients underwent a neurosurgical intervention. In patients without post-traumatic amnesia, 14 had a potential neurosurgical lesion and three underwent a neurosurgical intervention.
Table 3.
Baseline characteristics of 3742 patients with a CT scan from six centres*, according to status of CT findings
| Characteristic | Patients with normal CT findings (n=3358) | Patients with abnormal CT findings (n=384) | All patients with a CT scan (n=3742) |
|---|---|---|---|
| Age (years; mean (range)) | 56.6 (16-101) | 59.1 (17-98) | 56.9 (16-101) |
| Male sex | 1901 (56.6) | 244 (63.5) | 2145 (57.3) |
| Glasgow coma scale score at presentation | |||
| 13 | 94 (2.8) | 47 (12.2) | 141 (3.8) |
| 14 | 401 (11.9) | 91 (23.7) | 492 (13.1) |
| 15 | 2863 (85.3) | 246 (64.1) | 3109 (83.1) |
| Use of anticoagulation | |||
| None | 2886 (85.9) | 347 (90.4) | 3233 (86.4) |
| Coumarin | 387 (11.5) | 31 (8.1) | 418 (11.2) |
| Direct oral anticoagulants | 50 (1.5) | 3 (0.8) | 53 (1.4) |
| Use of thrombocyte aggregation inhibitors | 502 (15.0) | 75 (19.5) | 577 (15.4) |
| Bleeding disorder | 39 (1.2) | 2 (0.5) | 41 (1.1) |
| Mechanism of injury | |||
| Pedestrian in road traffic accident | 48 (1.4) | 9 (2.3) | 57 (1.5) |
| Cyclist in road traffic accident | 127 (3.8) | 25 (6.5) | 152 (4.1) |
| Fall from height | 451 (13.4) | 81 (21.1) | 532 (14.2) |
| Other† | 2691 (80.1) | 264 (68.8) | 2955 (79) |
| Ejected from vehicle | 120 (3.6) | 15 (3.9) | 135 (3.6) |
| Loss of consciousness | |||
| None | 1852 (55.2) | 116 (30.2) | 1968 (52.6) |
| ≤15 min | 943 (28.1) | 162 (42.2) | 1105 (29.5) |
| >15 min | 21 (0.6) | 10 (2.6) | 31 (0.8) |
| Retrograde amnesia | |||
| None | 2443 (72.8) | 194 (50.5) | 2637 (70.5) |
| ≤30 min | 251 (7.5) | 52 (13.5) | 303 (8.1) |
| >30 min | 102 (3.0) | 42 (10.9) | 144 (3.8) |
| Post-traumatic amnesia | |||
| None | 2068 (61.6) | 117 (30.5) | 2185 (58.4) |
| ≤2 h | 776 (23.1) | 172 (44.8) | 948 (25.3) |
| 2-4 h | 54 (1.6) | 14 (3.6) | 68 (1.8) |
| >4 h | 38 (1.1) | 21 (5.5) | 59 (1.6) |
| Intoxication with drugs or alcohol‡ | 836 (24.9) | 86 (22.4) | 922 (24.6) |
| Post-traumatic seizure | 26 (0.8) | 7 (1.8) | 33 (0.9) |
| Headache | 1086 (32.3) | 122 (31.8) | 1208 (32.3) |
| Vomiting | |||
| Once | 131 (3.9) | 17 (4.4) | 148 (4.0) |
| Twice or more | 119 (3.5) | 23 (6.0) | 142 (3.8) |
| Deterioration in Glasgow coma scale (1 h after presentation) | |||
| 1 point | 33 (1.0) | 5 (1.3) | 38 (1.0) |
| ≥2 points | 6 (0.2) | 6 (1.6) | 12 (0.3) |
| Neurological deficit | 100 (3.0) | 28 (7.3) | 128 (3.4) |
| Signs of skull base fracture | 89 (2.7) | 50 (13.0) | 139 (3.7) |
| Visible injury of the head | 1945 (57.9) | 263 (68.5) | 2208 (59) |
| Visible injury of the face | 1181 (35.2) | 134 (34.9) | 1315 (35.1) |
| Suspicion of open fracture | 6 (0.2) | 5 (1.3) | 11 (0.3) |
| Injury severity score (mean (range)) | 6.2 (0-54) | 15.2 (1-75) | 7.1 (0-75) |
Data are number (%) of patients unless stated otherwise. CT=computed tomography.
These centres refer to those on the left hand side of figure 1, for the primary analysis.
Includes patients with mild head injury such as a bumped head against an object.
History or suggestive findings on examination (eg, nystagmus, abnormal walking).
In a subgroup analysis of the 3914 patients with a Glasgow coma scale score of 15, more than half the patients (n=2465, 63%) had no loss of consciousness and no post-traumatic amnesia. Ninety three (3.8%) patients had any intracranial traumatic finding, seven (0.3%) had a potential neurosurgical lesion, and one underwent a neurosurgical intervention.
Of all 4557 patients, 1511 (33.2%) were admitted to the hospital for head injury and other reasons. Of the admitted patients, 226 (5.0%) were admitted for two nights or longer because of head injury; 52 (1.1%) had neurological deterioration during admission, and six (0.1%) were intubated for longer than 24 h. Eleven (0.2%) patients died as a result of head injury, and 21 (0.5%) died as a result of a different illness or trauma.
Performance of the decision rules
After imputation of outcomes in patients without a CT scan, 23 of 815 patients had any intracranial traumatic finding and no patient had a potential neurosurgical lesion. None of these 815 patients without a CT scan had undergone a neurosurgical intervention in 30 days after injury. The sensitivity for identifying patients with any intracranial traumatic finding on CT ranged from 72.5% for the NICE criteria to 98.8% for the NOC rule (table 4; appendix 3).
Table 4.
Performance of the four decision rules* used for CT in 4557 patients with minor head injury presenting at six centres†
| CT decision rule | Positive outcome (No) | Negative outcome (No) | Sensitivity (%; 95% CI) | Specificity (%; 95% CI) |
Positive likelihood ratio (95% CI) | Negative likelihood ratio (95% CI) |
|---|---|---|---|---|---|---|
| CHIP (n=4557) | ||||||
| Any traumatic finding on CT | — | — | 94.1 (91.5 to 96.3) | 21.6 (20.4 to 22.9) | 1.20 (1.16 to 1.23) | 0.27 (0.17 to 0.40) |
| CHIP, positive | 383 | 3253 | — | — | — | — |
| CHIP, negative | 24 | 897 | — | — | — | — |
| Potential neurosurgical lesion | — | — | 97.3 (93.1 to 100) | 20.5 (19.4 to 21.7) | 1.22 (1.17 to 1.26) | 0.13 (0 to 0.34) |
| CHIP, positive | 72 | 3564 | — | — | — | — |
| CHIP, negative | 2 | 919 | — | — | — | — |
| NICE (n=4557) | ||||||
| Any traumatic finding on CT | — | — | 72.5 (67.8 to 77.2) | 60.9 (59.3 to 62.5) | 1.85 (1.72 to 2.0) | 0.45 (0.37 to 0.53) |
| NICE, positive | 295 | 1624 | — | — | — | — |
| NICE, negative | 112 | 2526 | — | — | — | — |
| Potential neurosurgical lesion | — | — | 85.1 (76.4 to 92.9) | 58.6 (57.1 to 60.1) | 2.06 (1.84 to 2.27) | 0.25 (0.12 to 0.40) |
| NICE, positive | 63 | 1856 | — | — | — | — |
| NICE, negative | 11 | 2627 | — | — | — | — |
| NOC (n=4557) | ||||||
| Any traumatic finding on CT | — | — | 98.8 (97.6 to 99.8) | 4.4 (3.8 to 5.1) | 1.03 (1.02 to 1.05) | 0.28 (0.06 to 0.53) |
| NOC, positive | 402 | 3966 | — | — | — | — |
| NOC, negative | 5 | 184 | — | — | — | — |
| Potential neurosurgical lesion | — | — | 100 (100 to 100) | 4.2 (3.6 to 4.8) | 1.04 (1.04 to 1.05) | 0 (0 to 0) |
| NOC, positive | 74 | 4294 | — | — | — | — |
| NOC, negative | 0 | 189 | — | — | — | — |
| CCHR (n=4557) | ||||||
| Any traumatic finding on CT | — | — | 80.3 (76.1 to 84.2) | 44.2 (42.7 to 45.9) | 1.44 (1.35 to 1.52) | 0.44 (0.36 to 0.55) |
| CCHR, positive | 327 | 2314 | — | — | — | — |
| CCHR, negative | 80 | 1836 | — | — | — | — |
| Potential neurosurgical lesion | — | — | 87.8 (79.7 to 94.9) | 42.5 (41.0 to 44.1) | 1.53 (1.40 to 1.66) | 0.29 (0.12 to 0.47) |
| CCHR, positive | 65 | 2576 | — | — | — | — |
| CCHR, negative | 9 | 1907 | — | — | — | — |
CT=computed tomography.
CHIP=CT in head injury patient rule; NICE=National Institute for Health and Care Excellence guideline for head injury; NOC=New Orleans criteria; CCHR=Canadian CT head rule.
These centres refer to those on the left hand side of figure 1, for the primary analysis.
The sensitivity for identifying patients with potential neurosurgical lesions was 100% for NOC, the NICE criteria had the lowest sensitivity (85.1%) for identifying potential neurosurgical lesions (table 4). The NICE criteria would have missed 11 of 74 patients with potential neurosurgical lesions (appendix 4). The CHIP criteria would have missed two patients with potential neurosurgical lesions, who both had a small epidural haematoma, which did not need neurosurgical treatment. Of these two missed patients, one had surgery to repair a depressed skull fracture (appendix 4).
The specificity for identifying any intracranial traumatic finding was lowest for the NOC rule (4.4%) and highest for the NICE criteria (60.9%). The specificity for potential neurosurgical lesions ranged from 4.2% (NOC) to 58.6% (NICE criteria). The sensitivity and specificity differed significantly between all the rules (Cochran’s Q P<0.001). Sensitivity and specificity for the original CCHR and NOC groups were slightly different from the adjusted versions (see the methods section for definition of the original and adjusted groups; appendix 5). For the outcome of neurosurgical intervention, the NOC rule had the highest sensitivity (100%) and the NICE criteria the highest specificity (58.1%; appendix 6a).
Clinical usefulness
The decision curve of the NOC rule was almost identical to CT scanning all patients in both study outcomes (fig 2). When using a low threshold for performing CT (to avoid false negatives of the decision rule), we found that the NOC rule and the scanning of all patients had the highest net proportional benefit. When using a high threshold for performing CT (to avoid false positives), we found that the NICE criteria had the highest net proportional benefit (fig 2). Over a narrow range of intermediate thresholds, the CHIP criteria had the highest net proportional benefit (0.038-0.054 for intracranial traumatic findings and 0.008-0.012 for potential neurosurgical lesions). For the neurosurgical intervention outcome, the differences in net proportional benefit were small (appendix 6b).
Fig 2.
Decision curves for study outcomes showing net proportional benefit per CT decision rule. CT=computed tomography; CHIP=CT in head injury patient rule; NICE=National Institute for Health and Care Excellence guideline for head injury; NOC=New Orleans criteria; CCHR=Canadian CT head rule; scan all=scanning of all patients; scan none=scanning no patients. For each rule, the net proportional benefit was calculated with the formula: (true positives/total number) − weight×(false positives/total number)
Proportion of patients needing CT
According to the different decision rules, the proportion of the study population needing CT was 95.9% (95% confidence interval 95.3% to 96.5%) with the NOC rule, 79.8% (78.6% to 80.9%) with the CHIP criteria, 58.0% (56.4% to 59.4%) with the CCHR rule, and 42.1% (40.6% to 43.6%) with the NICE criteria. To increase the sensitivity of the CHIP criteria to the level of the NOC rule, 733 more CT scans would have been needed to identify 19 more patients with intracranial traumatic findings and identify two more patients with a potential neurosurgical lesion.
Secondary (sensitivity) analysis in all patients receiving CT scans
In all included centres, 4702 patients received a CT scan (fig 1). Most of these patients had a Glasgow coma scale score of 15 at presentation (n=3798; 80.8%), 1511 (32.1%) experienced loss of consciousness, and 1480 (31.5%) had post-traumatic amnesia (appendix 7a). We found that 528 (11.2%) patients had an intracranial traumatic finding on CT (appendix 7b). Although the sensitivity of all rules was higher and the specificity lower, their ordering was the same. The NOC rule had the highest sensitivity (99.1%) and lowest specificity (3.1%) for any intracranial traumatic finding, whereas the NICE guideline had the highest specificity (50.3%) and lowest sensitivity (77.5%; appendix 7c). Net proportional benefit analysis showed the same pattern as in the primary analysis (appendix 7c).
Discussion
Principal findings
In this large, multicentre, external validation study of CT decision rules for minor head injury patients, the NOC rule had the highest sensitivity and was the only rule with a 100% sensitivity for potential neurosurgical lesions. Nevertheless, the high sensitivity of the NOC rule comes at the cost of an extremely low specificity, with a consequence that nearly all patients would need a CT scan. The NICE guideline had the highest specificity and the lowest proportion of patients who needed a CT scan, but at the cost of a low sensitivity. The sensitivity of the CHIP criteria was high (97% for potential neurosurgical lesions) with an acceptable specificity and a substantial reduction in the proportion requiring CT. The sensitivity for the identification of patients with any intracranial traumatic finding on CT was less than 100% for all decision rules.
Which decision rule is the best for the situation depends on several factors. It depends not only on its characteristics but also on how many CT scans the physician is willing to perform to identify one patient with an intracranial traumatic finding or potential neurosurgical lesion. Because a potential neurosurgical lesion could have serious consequences, such as a neurosurgical intervention or even death, most professionals would agree that the sensitivity of the decision rule should be 100%.27 However, it is less easy to agree on the desired sensitivity for finding any intracranial traumatic lesion, because not all small intracranial traumatic findings have clinical consequences. If a CT decision rule gives a false positive result, the patient receives an unnecessary CT scan and will be discharged after spending a few hours in the emergency department. If the rule gives a false negative result, the patient will be discharged without a CT scan and an intracranial traumatic finding will be missed. If this intracranial traumatic finding was a potential neurosurgical lesion and adequate treatment was omitted or was given too late, this missed scan could have serious consequences.27
The net proportional benefit analysis might help in finding the best decision rule for different thresholds, but interpretation of the curves can be challenging.24 If a low threshold is chosen, the best rule to use in order to identify all patients with any lesion is the NOC rule, but this choice would imply that practically all patients undergo CT. At a high threshold, use of the NICE criteria avoids unnecessary scans and has the highest net proportional benefit, but important lesions might be missed. For the outcome of potential neurosurgical lesions, a very low net proportional benefit threshold and 100% sensitivity is desired. For intermediate thresholds, use of the CHIP criteria makes a trade-off between avoiding missed lesions and achieving a substantial reduction in CT scans of 21%. For the outcome of intracranial traumatic findings, the threshold can be higher, because it is not necessary that all findings are identified. From a societal perspective, not only clinical usefulness but also cost effectiveness is important. A cost effectiveness study showed that a prediction rule needs a sensitivity of at least 97% for identifying potential neurosurgical lesions in order to be cost effective, otherwise performing CT in all patients with minor head injury is more cost effective.26 In our study, only the NOC and CHIP rules fulfilled this criterion.
Comparison with other studies
Several other studies have validated and compared the sensitivity and specificity of CT decision rules for adult patients with minor head injury, but only the NOC, CCHR and NICE decision rules have been externally validated.13 14 15 16 17 28 Our study adds the CHIP rule to externally validated decision rules and compares it head-to-head with the other rules. Validation studies vary in design and in outcome measures (eg, clinically significant findings on CT are not uniformly defined), and are therefore difficult to compare. In addition, the case mix of our study is different from previous validation studies because we included all patients with blunt traumatic minor head injury, including those without risk factors. Our study is in line with earlier findings that the NOC rule has a high sensitivity but leads to a high scan rate, whereas the CCHR rule and NICE guideline can reduce the number of CT scans substantially, but at the cost of a lower sensitivity. However, the potential reduction in CT scans has not been proved in clinical practice yet. In terms of sensitivity and specificity, the CHIP rule lies between the NOC and CCHR rules.
All the decision rules in this study have been designed for an emergency department population. Although only the NICE and CHIP criteria have been designed to apply to all patients with minor head injury, in daily practice the NOC and CCHR rules probably apply to these patients as well. Therefore, we also investigated adjusted versions of the NOC and CCHR rules, which are applicable to all patients with minor head injury. The sensitivity and specificity of these two adjusted rules were comparable to the sensitivity and specificity of their original versions.
Our study population had a mean age of 53.1 years; by comparison, patients in the development studies for the NOC, CCHR, and CHIP rules had a mean age of 36-41 years. This difference is probably indicative of ageing of the population, as well as other factors such as changes in referral patterns or the increasing incidence of fall accidents.9 The percentage of patients with any intracranial traumatic finding (8.4%) was comparable with most other studies (6.9-12.1%).6 7 13 The percentage of patients who underwent a neurosurgical intervention within 30 days after injury in our study (0.4%) was low compared with most other studies (0.4%-1.5%). This difference might be because the indication for neurosurgery not only depends on clinical factors, but also differs from country to country and from neurosurgeon to neurosurgeon and could have changed over time.29 We therefore believe that instead of actual neurosurgical interventions, it is better to use “potential neurosurgical lesions” as an outcome measure. The confidence intervals for neurosurgical intervention were wide (sensitivity 71-100%) because of the low prevalence of this outcome.
Patients with minor head injury presenting at the emergency department not only reflect the ageing of the population but also the result of the decision rules themselves. In the Netherlands, use of anticoagulants (coumarines or direct oral anticoagulants) is considered a risk factor for intracranial complications and a reason for referral to the emergency department in both the ambulance and general practitioner protocols.30 The percentage of patients using anticoagulants in our study was higher than in the CHIP rule development cohort (9.2% v 6.9%).15
Limitations of the study
A limitation of our study was that not all consecutive patients with minor head injury were scanned. Following the guidelines for CT scanning at the participating centres resulted in patients with 0-1 minor criteria who did not undergo a CT scan. Therefore, patients who did not receive a CT scan but had intracranial traumatic findings (that is, those with false negative results) could have been missed. To detect this patient subgroup and precisely estimate their relative frequency among unscreened patients would need many thousands of individuals, which was not feasible. Missing patients without a CT scan could have led to a slight overestimation of the sensitivity and an underestimation of the specificity. We therefore performed the primary analysis on data from six centres which also collected data for patients without a CT scan. For all the rules, the new calculated sensitivities were a little lower and the specificities higher, as expected. The fact that most centres in our study used CT guidelines based on the CHIP rule could have introduced a bias in favour of the CHIP rule, owing to possible missed lesions (because the patient was not scanned according to the local guideline) that would have been detected by the other rules. However, by imputing the outcomes of the patients without a CT scan, we were able to keep this bias to a minimum.
Because most physicians used the CHIP rule on a regular basis, they were more likely to apply it correctly. However, many risk factors are the same for all rules and the validation was performed based on the scored risk factors, not on the physicians’ judgment of a rule being positive or negative. In addition, in our centres, it is clinical practice to assess not only risk factors from the CHIP rule, but also other risk factors such as headache and retrograde amnesia. In our study, it was unclear how quickly patients proceeded to CT and whether lesions appeared after this time. However, af Geijerstam and colleagues concluded in a literature review that the risk of a patient developing an intracranial lesion after an early normal CT scan is very low.31
Another limitation was the possibility that we missed patients undergoing a neurosurgical intervention in a different hospital. However, because the participating centres were all the primary neurosurgery centres of the area, this potential bias is highly unlikely. Furthermore, because we used potential neurosurgical lesions as a secondary outcome instead of neurosurgical intervention, our main findings would not have been affected. In the development studies of the four decision rules, potential neurosurgical lesions were not used as an outcome measure.
Conclusions and policy implications
Application of the CHIP, NOC, CCHR, or NICE decision rules leads to a wide variation in CT scanning among patients with minor head injury, resulting in unnecessary CT scans and missed intracranial traumatic findings. Only the NOC rule did not miss potential neurosurgical lesions, but this was at the cost of having to scan nearly all patients. Although the NICE guideline had the highest reduction of CT scans (58%), missing 15% of patients with potential neurosurgical lesions would be unacceptable to most physicians in the emergency department, because it would mean that for every 200 patients not be scanned according to the NICE criteria, one patient would turn out to have a potential neurosurgical lesion.
Of the four investigated rules, the CHIP rule performed the best with an acceptable sensitivity of 97% for potential neurosurgical lesions according to previous cost effectiveness analysis, the highest net proportional benefit at intermediate thresholds, and a substantial reduction of CT scans of 21% compared with the scanning of all patients. Updating an existing decision rule might increase the sensitivity and specificity for detecting potential neurosurgical lesions. Until this update is conducted, it is justified to use any of the four rules for patients with minor head injury presenting at the emergency department. We recommend use of the CHIP rule because it leads to a substantial reduction of CT scans and misses very few potential neurosurgical lesions.
What is already known on this topic
Several decision rules in computed tomography (CT) have been developed to identify patients with a higher risk of intracranial complications after sustaining a minor head injury
The New Orleans criteria have a high sensitivity but would lead to a high scan rate, whereas criteria from the Canadian CT head rule and the National Institute for Health and Care Excellence guidance for head injury reduce the number of CT scans substantially but with a lower sensitivity
Commonly encountered risk factors such as loss of consciousness and anticoagulation in these patients have not been addressed with current decision rules; the CT in head injury patients (CHIP) rule includes these risk factors
What this study adds
This external validation study directly compared the CHIP rule with the other three frequently used and validated rules for CT scanning of patients presenting with minor head injury
The CHIP rule showed an acceptable sensitivity for potential neurosurgical lesions, the highest net proportional benefit at intermediate thresholds of performing CT scans, and a substantial reduction of scans compared with the scanning of all patients
Web extra.
Extra material supplied by authors
Web appendix: Supplementary materials
Contributors: KAF and CLvdB contributed equally to the study. KAF, CLvdB, JvdN, BJ, EdJ, HFdB, OS, MIG, CES, KEJ, HFV, ER, MB, GGS, KJ, and DWJD conceived the study. KAF, CLvdB, KJ, and DWJD designed the study. KAF performed the data analysis. KAF and CLvdB performed the literature search and drafted the manuscript. HFL, JvdN, BJ, EJ, HFdB, OS, PP, SP, MIG, CES, KEJ, HFV, MGMH, ER, MB, GGS, EWS, KJ, and DWJD performed a critical review of the manuscript. DWJD is the guarantor. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding: This study did not receive any specific funding. CLvdB received a personal grant from the St Jacobus Foundation, which is a non-profit organisation that supports research; the funder played no role in the design, analysis, or presentation of the study.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no specific support from any organisation for the submitted work; MGMH and DWJD were the principal investigators of the CHIP (CT in head injury patients) development study; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Institutional ethics and research board approval was obtained, and informed consent was waived.
Data sharing: Anonymised patient level data are available on reasonable request from the corresponding author at k.foks@erasmusmc.nl.
The lead authors affirm that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.
References
- 1. Maas AIR, Menon DK, Adelson PD, et al. InTBIR Participants and Investigators Traumatic brain injury: integrated approaches to improve prevention, clinical care, and research. Lancet Neurol 2017;16:987-1048. 10.1016/S1474-4422(17)30371-X [DOI] [PubMed] [Google Scholar]
- 2. Brazinova A, Rehorcikova V, Taylor MS, et al. Epidemiology of traumatic brain injury in Europe: A Living Systematic Review. J Neurotrauma 2016. 10.1089/neu.2015.4126 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Peeters W, van den Brande R, Polinder S, et al. Epidemiology of traumatic brain injury in Europe. Acta Neurochir (Wien) 2015;157:1683-96. 10.1007/s00701-015-2512-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Roozenbeek B, Maas AI, Menon DK. Changing patterns in the epidemiology of traumatic brain injury. Nat Rev Neurol 2013;9:231-6. 10.1038/nrneurol.2013.22 [DOI] [PubMed] [Google Scholar]
- 5. af Geijerstam JL, Britton M. Mild head injury - mortality and complication rate: meta-analysis of findings in a systematic literature review. Acta Neurochir (Wien) 2003;145:843-50, discussion 850. 10.1007/s00701-003-0115-1 [DOI] [PubMed] [Google Scholar]
- 6. Haydel MJ, Preston CA, Mills TJ, Luber S, Blaudeau E, DeBlieux PM. Indications for computed tomography in patients with minor head injury. N Engl J Med 2000;343:100-5. 10.1056/NEJM200007133430204 [DOI] [PubMed] [Google Scholar]
- 7. Stiell IG, Wells GA, Vandemheen K, et al. The Canadian CT Head Rule for patients with minor head injury. Lancet 2001;357:1391-6. 10.1016/S0140-6736(00)04561-X 10.1016/S0140-6736(00)04561-X [DOI] [PubMed] [Google Scholar]
- 8.National Clinical Guideline C. National Clinical Guidance Centre. (2014). CG 176 Head Injury Triage, assessment, investigation and early management of head injury in children, young people and adults. National Institute for Health and Care Excellence 2014. [PubMed]
- 9. Van den Brand CL, Karger LB, Nijman ST, Hunink MG, Patka P, Jellema K. Traumatic brain injury in the Netherlands, trends in emergency department visits, hospitalization and mortality between 1998 and 2012. Eur J Emerg Med 2017. 10.1097/MEJ.0000000000000457 [DOI] [PubMed] [Google Scholar]
- 10. Brenner DJ, Hall EJ. Computed tomography--an increasing source of radiation exposure. N Engl J Med 2007;357:2277-84. 10.1056/NEJMra072149 [DOI] [PubMed] [Google Scholar]
- 11. Smits M, Hunink MG, Nederkoorn PJ, et al. A history of loss of consciousness or post-traumatic amnesia in minor head injury: “conditio sine qua non” or one of the risk factors? J Neurol Neurosurg Psychiatry 2007;78:1359-64. 10.1136/jnnp.2007.117143 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Dunning J, Stratford-Smith P, Lecky F, et al. Emergency Medicine Research Group A meta-analysis of clinical correlates that predict significant intracranial injury in adults with minor head trauma. J Neurotrauma 2004;21:877-85. 10.1089/0897715041526122 [DOI] [PubMed] [Google Scholar]
- 13. Smits M, Dippel DW, Steyerberg EW, et al. Predicting intracranial traumatic findings on computed tomography in patients with minor head injury: the CHIP prediction rule. Ann Intern Med 2007;146:397-405. 10.7326/0003-4819-146-6-200703200-00004 [DOI] [PubMed] [Google Scholar]
- 14. Stiell IG, Clement CM, Rowe BH, et al. Comparison of the Canadian CT Head Rule and the New Orleans Criteria in patients with minor head injury. JAMA 2005;294:1511-8. 10.1001/jama.294.12.1511 [DOI] [PubMed] [Google Scholar]
- 15. Smits M, Dippel DW, de Haan GG, et al. External validation of the Canadian CT Head Rule and the New Orleans Criteria for CT scanning in patients with minor head injury. JAMA 2005;294:1519-25. 10.1001/jama.294.12.1519 [DOI] [PubMed] [Google Scholar]
- 16. Easter JS, Haukoos JS, Meehan WP, Novack V, Edlow JA. Will neuroimaging reveal a severe intracranial injury in this adult with minor head trauma?: the rational clinical examination systematic review. JAMA 2015;314:2672-81. 10.1001/jama.2015.16316 [DOI] [PubMed] [Google Scholar]
- 17. Harnan SE, Pickering A, Pandor A, Goodacre SW. Clinical decision rules for adults with minor head injury: a systematic review. J Trauma 2011;71:245-51. 10.1097/TA.0b013e31820d090f [DOI] [PubMed] [Google Scholar]
- 18. Moons KG, Kengne AP, Grobbee DE, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012;98:691-8. 10.1136/heartjnl-2011-301247 [DOI] [PubMed] [Google Scholar]
- 19. Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: validating a prognostic model. BMJ 2009;338:b605. 10.1136/bmj.b605 [DOI] [PubMed] [Google Scholar]
- 20. Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. Springer, 2009. 10.1007/978-0-387-77244-8 . [DOI] [Google Scholar]
- 21. Steyerberg EW, Harrell FE., Jr Prediction models need appropriate internal, internal-external, and external validation. J Clin Epidemiol 2016;69:245-7. 10.1016/j.jclinepi.2015.04.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Leard Statistics. Cochran’s Q test using SPSS Statistics. 2018. https://statistics.laerd.com/spss-tutorials/cochrans-q-test-in-spss-statistics.php
- 23. Sullivan TR, Lee KJ, Ryan P, Salter AB. Multiple imputation for handling missing outcome data when estimating the relative risk. BMC Med Res Methodol 2017;17:134. 10.1186/s12874-017-0414-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ 2016;352:i6. 10.1136/bmj.i6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Hunink M, Wittenberg MWE, Drummond M, Pliskin J, Wong J. Decision making in health and medicine: integrating evidence and values. Cambridge University Press, 2014. 10.1017/CBO9781139506779 . [DOI] [Google Scholar]
- 26. Smits M, Dippel DW, Nederkoorn PJ, et al. Minor head injury: CT-based strategies for management--a cost-effectiveness analysis. Radiology 2010;254:532-40. 10.1148/radiol.2541081672 [DOI] [PubMed] [Google Scholar]
- 27. Marincowitz C, Lecky FE, Townend W, Borakati A, Fabbri A, Sheldon TA. The risk of deterioration in GCS13-15 patients with traumatic brain injury identified by computed tomography imaging: a systematic review and meta-analysis. J Neurotrauma 2018;35:703-18. 10.1089/neu.2017.5259 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Bouida W, Marghli S, Souissi S, et al. Prediction value of the Canadian CT head rule and the New Orleans criteria for positive head CT scan and acute neurosurgical procedures in minor head trauma: a multicenter external validation study. Ann Emerg Med 2013;61:521-7. 10.1016/j.annemergmed.2012.07.016 [DOI] [PubMed] [Google Scholar]
- 29. van Essen TA, de Ruiter GC, Kho KH, Peul WC. Neurosurgical treatment variation of traumatic brain injury: evaluation of acute subdural hematoma management in Belgium and The Netherlands. J Neurotrauma 2017;34:881-9. 10.1089/neu.2016.4495 [DOI] [PubMed] [Google Scholar]
- 30. Batchelor JS, Grayson A. A meta-analysis to determine the effect of anticoagulation on mortality in patients with blunt head trauma. Br J Neurosurg 2012;26:525-30. 10.3109/02688697.2011.650736 [DOI] [PubMed] [Google Scholar]
- 31. af Geijerstam JL, Britton M. Mild head injury: reliability of early computed tomographic findings in triage for admission. Emerg Med J 2005;22:103-7. 10.1136/emj.2004.015396 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Web appendix: Supplementary materials