Development and validation of a new score for predicting functional outcome of neurocritically ill patients: The INCNS score

Summary

Aims

To develop and validate a novel score for prediction of 3‐month functional outcome in neurocritically ill patients.

Methods

The development of the novel score was based on two widely used scores for general critical illnesses (Acute Physiology and Chronic Health Evaluation II, APACHE II; Simplified Acute Physiology Score II, SAPS II) and consideration of the characteristics of neurocritical illness. Data from consecutive patients admitted to neurological ICU (N‐ICU) between January 2013 and June 2016 were used for the validation. The modified Rankin Scale (mRS) was used to evaluate 3‐month functional outcomes. APACHE II scores, SAPS II scores, and our novel scores at 24 hours and 72 hours in N‐ICU were obtained. We compared the prognostic performance of our score with APACHE II and SAPS II.

Results

We developed a 44‐point scoring system named the INCNS score, and it includes 19 items which were categorized into five parts: inflammation (I), nutrition (N), consciousness (C), neurological function (N), and systemic function (S). We validated the INCNS score with a cohort of 941 N‐ICU patients. The 72‐hours INCNS score achieved an area under the receiver operating characteristic curve (AUC) of 0.828 (95% CI: 0.802‐0.854), and the 24‐hours INCNS score achieved an AUC of 0.788 (95% CI: 0.759‐0.817). The INCNS score exhibited significantly better discriminative and prognostic performance than APACHE II and SAPS II at both 24 hours and 72 hours in N‐ICU.

Conclusion

We developed an INCNS score with superior predictive power for functional outcome of neurocritically ill patients.

1. INTRODUCTION

Neurocritical illness, defined as life‐threatening neurological and neurosurgical illness, is a catastrophic event with high mortality and prolonged functional dependence.1, 2, 3 An accurate and specialized prognostic score for neurocritical illness will provide clinicians a useful overview to summarize clinical findings, help stratify patients in critical care researches, and facilitate more effective communication with patients and their families about goals of care, choices of treatments, and plans for transitions.

To date, there are several prognostic scores for patients in general intensive care unit (ICU), such as the Acute Physiology and Chronic Health Evaluation II (APACHE II)4 and Simplified Acute Physiology Score II (SAPS II).5 Meanwhile, some scales were also developed for specific critical illnesses, such as the Ventilator‐Associated Pneumonia PIRO Score,6 Obstetric Early Warning Score,7 Oncological Pediatric Risk of Mortality Score (O‐PRISM),8 the Edinburgh Cardiac Surgery Score,9 Initiation of Renal Replacement Therapy (IRRIV) Score10 and so on. However, so far there is no prognostic score for patients with neurocritical illness who have characteristics distinct from that of patients with other critical illnesses. And the lack of assessment of neurological deficits in those risk adjustment models biases the severity evaluation and outcome prediction in patients with neurocritical illnesses.

Glasgow coma scale (GCS)11 and the Full Outline of UnResponsiveness (FOUR) score12 are common methods for bedside assessment of impairment of conscious level,13 and are widely used in neurological intensive care unit (N‐ICU). Although these two scales are strongly associated with severity and outcome which are assessed by other indices,14, 15, 16 the assessment of consciousness per se is unable to make comprehensive predictive statements.

Given the paucity of prognostic scores in neurocritical illness, we aimed to develop a novel score for prediction of 3‐month functional outcome in neurocritically ill patients, and compare it with APACHE II and SAPS II in a cohort of neurocritically ill patients.

2. METHODS

2.1. Hypothesis generation

We hypothesized that a specialized score for neurocritical illness would be the combination of systemic condition evaluation and neurological function assessment. Since the variables in APACHE II and SAPS II were derived from expert consensus and large cohorts of general critical illness4, 5 and have been validated all over the world,17, 18, 19, 20, 21 we believed that the integration and modification of these two scores would provide a reliable and reasonable basis for evaluating systemic condition accurately. Measurements concerning different aspects of neurological deficits and being routinely used in N‐ICU were also considered. The process of the generation of the novel score named INCNS is depicted in Figure 1.

Figure 1.

Flow chart for the variable generation of the INCNS score. ALB albumin, APACHE II Acute Physiology and Chronic Health Evaluation II, BUN blood urea nitrogen, MAP mean artery pressure, N‐ICU neurological intensive care unit, SAPS II Simplified Acute Physiology Score II, SBP systolic blood pressure, SCr serum creatinine, WBC white blood cell

2.2. Variable selection

On the basis of 22 different variables in APACHE II and SAPS II, reductions and additions were made in order to form a prognostic score for neurocritical illnesses with good accuracy, availability, and ease of use. The reduction procedure followed three principles: (a) simplify congeneric items (systolic blood pressure was retained, and mean arterial pressure was removed; serum creatinine was retained, and blood urea nitrogen and urine volume were removed; co‐morbidities were removed because their severities could be partly mirrored by the remaining variables); (b) suspend the variables which have not been fully proven to exert direct impacts on functional outcomes of neurocritical illnesses (hematocrit was removed; reasons for ICU admission were also removed, because some patients who don’t need emergency surgery might have much poorer outcomes than those who need, such as the patients with acute intoxication or cardiac arrest for instance); (c) reduce the bias caused by common missing values: blood gas analysis (pH, PaO2) or serum HCO₃ is not necessary for each neurocritically ill patient. The addition procedure followed two principles: (a) incorporate some routine neurologic examinations in N‐ICU (papillary light reflex, corneal reflex, muscle strength test, consiousness test, and water swallow test); (b) include biomarkers which are associated with functional outcomes of common neurocritical illnesses (blood glucose and serum albumin). Previous studies proved that abnormal blood glucose and serum albumin levels were independently related to the mortality and morbidity in some common neurologic diseases, such as stroke22, 23, 24, 25, 26, 27, 28, 29 and status epilepticus.30, 31 After the above reduction and addition procedures, 19 variables were finally determined. In order to make those variables easier to understand and remember, we grouped them into five categories: inflammation, nutrition, consciousness, neurological function and systemic condition (INCNS for short).

2.3. Definition of the variables

White blood cell, serum albumin, random blood glucose, serum sodium, serum potassium, serum creatinine, and total bilirubin are routine labs. Temperature, respiration, heart rate, and systolic blood pressure are vital signs. Consciousness is evaluated from two aspects: (a) arousal, assessed by eye opening test (spontaneous, to verbal command, to pain, and no response); (b) awareness, assessed by behavioral tests (correct or confused behavioral responses to question or command) and tests for non‐reflex movements (visual pursuit or non‐contingent behaviors). The assessment of arousal status is derived from GCS which is included in APACHE II and SAPS II. The behavioral tests for awareness are routine bedside examinations in N‐ICU. The examiner may ask a question about the patient’s name or command the patient to move eyeballs and/or hands, if appropriate. A clear and accurate answer was considered correct, and a clearly intentional communicative response, such as head nods/shakes and thumbs up, was considered as confused. For the patients who are unable to communicate, tests for non‐reflex movements were performed. Non‐reflex movements include visual pursuit, orientation to pain or non‐contingent behavior (eg smiling to the presence of a family member and not to others).32 Neurologic function is evaluated from six aspects: (a) pupillary light reflex; (b) corneal reflex; (c) verbal response; (d) motor response; (e) swallowing function; (f) respiration. The pupillary light reflex and corneal reflex are brainstem reflexes that test mesencephalon and pons function. The assessments of verbal and motor function are derived from GCS which is included in APACHE II and SAPS II. For the intubated patients, their verbal function is scored as no verbal response. The muscle strength test is also used to assess the motor function. Either the muscle strength test or motor response to painful stimulus is performed in each patient. Patients who are unable to cooperate with muscle strength test choose the motor response to painful stimulus test. To be noted, unlike GCS score that records the best motor response from any limb, the worst motor response from any limb is recorded in our score. Swallowing function is assessed by the 30 mL water swallow test. In APACHE II the respiratory function is measured only by the respiratory rate no matter the patient is ventilated or non‐ventilated. However, the respiratory function of the ventilated patients usually is much poorer than those who don’t need mechanical ventilation. Therefore, we classified the respiratory function into four degrees: (a) not intubated, respiratory rate of 12 to 24; (b) not intubated, respiratory rate ≤ 11 or ≥25; (c) breathes above ventilator rate; (d) breathes at ventilator rate or apnea.

2.4. Weight assignment

In order to make this score easy to calculate and use, the weighing system was based on a simple scale of 0 to 3 (like APACHE II) instead of a sophisticated scale of irregular numbers (like SPAS II) generated by statistic calculation. 0 denotes normal, and a higher score means a more serious condition. Considering that brainstem function tests play a pivotal role in outcome prediction, we assigned higher scores to the slow and absent responses of papillary light reflex and corneal reflex.

2.5. Population

This analysis was based on a prospective database of all the patients admitted to N‐ICU in Xijing Hospital, a tertiary academic medical care institution with 3200 beds in Xi’an, China. Consecutive patients who have stayed in N‐ICU for longer than 72 hours between January 2013 and June 2016 were used for the validation of the INCNS score. The ethics committee of the Xijing hospital reviewed and approved the study. All the procedures were carried out in agreement with Chinese laws and the Helsinki declaration relative to patients’ rights.

2.6. Data collection

The data concerning all the variables included in APACHE II, SAPS II, and INCNS score were reviewed. The most deranged reading during each patient’s initial 24 hours and 72 hours in N‐ICU were recorded and used in the validation analysis. All the neurologic examinations involved in these three scores are daily routines in N‐ICU, and were performed by the neurologist who was in charge of each patient.

2.7. Definition of outcomes

The modified Rankin Scale (mRS), a seven‐point scale ranging from 0 (no symptoms) to 6 (death), was used to evaluate the functional outcome 3 months after the discharge from N‐ICU, by a trained research assistant who was blinded to clinical data. A mRS score < 3 was considered favorable (independence), and a mRS score ≥ 3 was considered unfavorable (dependence or death).

2.8. Statistical analysis

Continuous variables were expressed as mean ± SD or median (interquartile range, IQR), and categorical variables were expressed as percentages. Continuous variables were analyzed using Student’s t test for normally distributed variables, and Mann ‐ Whitney U test for non‐normally distributed variables. Categorical variables were analyzed using χ ² test analysis, and Fisher’s exact tests, when appropriate. All the above tests were two‐sided, and a P value < 0.05 was considered statistically significant.

The receiver operating characteristics curve (ROC) analysis was used to determine the predictive power of APACHE II, SAPS II, GCS, FOUR and INCNS score. Sensitivity, specificity, positive (PPV), negative predictive value (NPV), number of correctly classified (CC) patients and the maximum accuracy determined by cut‐off values (Youden Index) were calculated for INCNS, APACHE II, and SAPS II. Mc Nemar’s test was used to compare sensitivity, specificity and CC.33 A modification of Wald tests was used to examine the significance of difference of PPV and NPV.34 A P value <0.0167 for multiple comparison tests on performances was considered statistically significant.35 The probability values of predicting poor prognosis of INCNS score were derived from the logistic regression model. Statistical analyses were performed using SPSS 22.0 and Matlab 2012a.

3. RESULTS

The INCNS score was presented in Table 1.

Table 1.

Scoresheet of the INCNS score

	Variable	Points
		0	1	2	3
Inflammation	WBC (109/L)	4 ~ 10	2.9 ~ 3.9, 10.1 ~ 25.0	≤2.8, ≥25.1	—
	Temperature (axillary, °C)	36 ~ 38.4	≤35.9, 38.5 ~ 40	≥40.1	—
Nutrition	Albumin (g/L)	≥35	25 ~ 34.9	≤24.9	—
Consciousness	Arousal	Spontaneous eye opening	Eye opening to verbal command	Eye opening to pain	None
	Awareness	Correct response to question or commanda	confused response to question or commanda	Non‐reflex movementsb	None
Neurologic function	Pupillary light reflex	Bilateral sensitive	—	Unilateral slow/absent	Bilateral slow/absent
	Corneal reflex	Bilateral sensitive	—	Unilateral slow/absent	Bilateral slow/absent
	Verbal response	Accurate speech	Confused/inappropriate speech	Incomprehensible speech/none	—
	Motor responsec	Unilateral/bilateral muscle strength scores ≥ 4	Unilateral/bilateral muscle strength scores of 2‐3	Unilateral muscle strength scores ≤ 1	Bilateral muscle strength scores ≤ 1
		Obeying to command	Localizing to/withdrawal from pain	Flexing/extending to pain	None
	Swallowing function	Water swallow test I‐II	Water swallow test III‐IV/unable to assess	—	—
	Respiration	Not intubated, 12 ~ 24	Not intubated, ≤11/≥25	Breathes above ventilator rate	Breathes at ventilator rate/ apnea
Systemic condition	Age (y)	≤44	45 ~ 64	65 ~ 74	≥75
	Heart rate	60 ~ 100	40 ~ 59, 101 ~ 149	≤39, ≥ 150	—
	SBP (mm Hg)	90 ~ 140	70 ~ 89, 141 ~ 199	≤69, ≥ 200	—
	Blood glucose (mmol/L)	3.9 ~ 11.1	2.2 ~ 3.8, 11.2 ~ 19.3	≤2.1, ≥ 19.4	—
	Serum sodium (mmol/L)	130 ~ 150	120 ~ 129, 151 ~ 159	≤119, ≥ 160	—
	Serum potassium (mmol/L)	3.5 ~ 5.5	2.5 ~ 3.4, 5.6 ~ 6.9	≤2.4, ≥ 7.0	—
	Serum creatinine (μmol/L)	44 ~ 132	≤43, 133 ~ 171	≥172	—
	Total bilirubin (μmol/L)	≤34.1	34.2 ~ 102.5	≥102.6	—

SBP, systolic blood pressure; WBC, white blood cell.

^aThe examiner may ask a question about the patient’s name or command the patient to move eyeballs and/or hands, if appropriate.

^bInclude evidence of visual pursuit or non‐contingent behaviors.

^cEither the muscle strength test or motor response to painful stimulus is performed in each patient.

3.1. Validation cohort

From January 2013 and June 2016, a total of 1037 patients were admitted to our N‐ICU and had stayed for longer than 3 days. Ninety‐six patients were lost to follow‐up, and 941 patients were enrolled for the analysis. The median age for the cohort was 52 (IQR, 37‐64) years, and 59.9% were male. The median hospital length of stay was 15 (IQR 9‐23) days, and the median length of N‐ICU stay was 10 (IQR 6‐16) days. At 3 months post N‐ICU discharge, 384 (40.8%) patients attained favorable outcomes (mRS 0‐2), while 557 (59.2%) patients had unfavorable outcomes (mRS 3‐6), among whom 232 (24.7%) died. Baseline characteristics of patients and the worst values of all variables during initial 24‐hours and 72‐hours in N‐ICU were presented in Tables S1 and S2 respectively. The etiology distribution in this cohort was provided in Table S3.

3.2. Predictive performance of INCNS score

The 24‐hours INCNS scoring system yielded an area under the receiver operating characteristic curve (AUC) of 0.788 (95% CI, 0.759‐0.817), and the 72‐hours INCNS score yielded an AUC of 0.828 (95% CI, 0.802‐0.854) (Figure 2, Table S4). The cut‐off value determined by the maximum sum of sensitivity and specificity was nine for both 24‐hours and 72‐hours INCNS scores. Using the cutoff score, the 72‐hours INCNS score predicted 3‐month unfavorable outcomes with a predictive accuracy of 75.5% (sensitivity: 75.0%, specificity: 76.0%, PPV: 82.0%, NPV: 67.7%, CC: 75.5%), and the 24‐hours INCNS score had a predictive accuracy of 72.9% (sensitivity: 73.8%, specificity: 71.9%, PPV: 79.2%, NPV: 65.4%, CC: 73%) (Figure 3, Table S4). Figure 4 showed that the probability of an unfavorable outcome of neurocritically ill patients increased as the 72‐hours INCNS score rose.

Figure 2.

Comparisons of area under the receiver operating characteristic curve (AUC) for INCNS, Acute Physiology and Chronic Health Evaluation II (APACHE II) and Simplified Acute Physiology Score II (SAPS II) to discriminate the 3‐mo functional outcome in neurocritically ill patients. A, ROC at 24 h: the P value for the comparison of AUC between INCNS and APACHE II is 0.0011, between INCNS and SAPS II is 0.0117. B, ROC at 72 h: the P value for the comparison of AUC between INCNS and APACHE II is <0.0001, between INCNS and SAPS II is also <0.0001. Level of significance corrected for multiple testing P < 0.0167

Figure 3.

Comparisons of sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and number of correctly classified (CC) patients between INCNS, Acute Physiology and Chronic Health Evaluation II (APACHE II) and Simplified Acute Physiology Score II (SAPS II) to identify the predictive performance of 3‐mo functional outcome in neurocritically ill patients. Sensitivity: *P = 0.000000000000000036, **P = 0.0000043, ***P = 0.000022; specificity: *P = 0.0081, **P = 0.0041; NPV: *P = 0.00000000013, **P = 0.000020, ***P = 0.013; CC: *P = 0.00028, **P = 0.015. Level of significance corrected for multiple testing P < 0.0167

Figure 4.

Points and their corresponding predicted estimates of the 3‐mo unfavorable outcome of neurological intensive care unit (N‐ICU) patients based on the 72‐h INCNS score

3.3. Comparison of INCNS, APACHE II and SAPS II score

Overall performances of INCNS, APACHE II and SAPS II were presented in Figure 2, Figure 3 and Table S4. According to the results of ROC analyses, the 72‐hours INCNS scoring system had the greatest discriminative power and highest predictive accuracy, followed by the 24‐hours INCNS, 72‐hours SAPS II, 24‐hours SAPS II, 72‐hours APACHE II and 24‐hours APACHE II scoring systems. At both 24 hours and 72 hours in N‐ICU, the INCNS score had significantly greater discriminative power than APACHE II (24 hours: P = 0.0011; 72 hours: P < 0.0001) and SAPS II (24 hours: P = 0.0117; 72 hours: P < 0.0001). The sensitivity, NPV, and CC of the INCNS score were also significantly higher than those of APACHE II and SAPS II at both 24 hours and 72 hours (Figure 3, Table S4). Moreover, the INCNS score showed much better discriminative performance compared with GCS (24 hours: P < 0.0001; 72 hours: P < 0.0001) and FOUR (24 hours: P < 0.0001; 72 hours: P < 0.0001) at both 24 hours and 72 hours as well (Figure S1).

4. DISCUSSION

Based on two widely used scores for critical illness (APACHE II and SAPS II) and consideration of the characteristics of neurocritical illness, we developed a novel clinically useful yet statistically accurate and valid score for predicting functional outcome in neurocritically ill patients. We named it INCNS, which is an acronym for inflammation, nutrition, consciousness, neurological function and systemic function. Data from 941 neurocritically ill patients validated that the INCNS score had significantly greater predictive ability than APACHE II and SAPS II.

In this study, multivariate logistic regression was not used to identify predictor variables in the INCNS score, because the limited sample size and population diversity would bias the conclusions of the regression model. The major measurements of the INCNS score were derived from APACHE II and SAPS II, which have been validated in abundant cohorts worldwide.17, 18, 19, 20, 21 This provided a robust support for the INCNS score to be used reliably in different regions where different distributions of neurological illness, age, race may be encountered.

The verbal and motor response tests in INCNS were also derived from GCS, however, we made several modifications to it for the purpose of evaluating verbal and motor ability rather than merely consciousness level. For intubated patients, we believed that the use of intubation would inevitably affect their language, therefore they were scored as no response in the verbal response test in INCNS score. Besides, since the motor response to painful stimulus test in GCS was designed for comatose patients, we combined muscle strength test with the motor response to painful stimulus test to evaluate the motor function. Only one of these two tests was performed for each patient according to his ability, and the examiner may choose the appropriate one. Moreover, the scoring method in the motor response test was also modified. The worst motor response from any limb rather than the best one is recorded in our score, because the purpose of the INCNS score is to predict 3‐month functional outcome, and the disability of any limb will definitely impair motor skills.

Arousal and awareness are essential parts of consciousness. Arousal is the psychological and physiological state of being awoken, and awareness is the ability to perceive or to be cognizant of the self‐existence and events. In INCNS score, we used the eye opening test, behavioral tests and tests for non‐reflex movements to evaluate the levels of arousal and awareness. The INCNS score also includes the assessments of brainstem reflexes and swallowing function. The pupillary light reflex and corneal reflex are routine neurologic examinations in N‐ICU and convenient approaches for assessing the function of brainstem, which plays a pivotal role in maintaining many basic functions, such as consciousness, breathing, heart rate, and sleeping. As a common result of a variety of neurological diseases (cerebral infarction, intracerebral hemorrhage, traumatic brain injury, myasthenia gravis, dementia, etc),36 dysphagia is associated with some unfavorable outcomes, including pneumonia, reintubation, in‐hospital mortality and prolonged hospital length of stay.36, 37, 38, 39 Moreover, the difficulty in swallowing will also affect the quality of life in long term. Therefore, the water swallow test was adopted to assess the swallowing function. All the neurologic examinations used in INCNS score are common and routine tests, and have been proven to possess excellent interrater reliability.40, 41, 42

APACHE II and SAPS II were established based on the worst values of the measurements during the first 24 hours in general ICU.4, 5 However, some studies suggested that a severity score (APACHE II or the mortality probability models) that performed at 72 hours in ICU worked much better than the one performed on admission, 24 hours or 48 hours.43, 44 In this study, predictive performances of both 24‐hours and 72‐hours INCNS scores were examined in 941 N‐ICU patients, and they were compared with the performances of APACHE II and SAPS II. Results from our studies showed that INCNS score had significantly stronger predictive power in discriminative power, sensitivity, NPV, and CC than APACHE II and SAPS II at both 24 hours and 72 hours in N‐ICU, however, APACHE II score had a greater specificity with respect to the other two scores. Specificity quantified the proportion of patients with actual favorable outcome that were correctly identified as such. Therefore, compared to APACHE II score, INCNS score and SAPS II score had a higher rate of false positives, which indicated that more people with favorable outcome were predicted wrongly as the ones with unfavorable outcome. Our study also confirmed that the INCNS score performed at 72 hours achieved better accuracy than the one performed at 24 hours, which might be partially because the progressive worsening usually extends beyond 24 hours after the admission to ICU.

Several weaknesses of the current study should be noted. First, the selection of predictor variables was based on previous studies, clinical experience, and usability considerations, instead of statistical calculation. To some extent, it may be arbitrary. However, the drawbacks of a limited derivation cohort (eg in number and diversity of population) could be avoided in this way. Second, weight assignment was based on a simple scale of 0 to 3 instead of numbers generated by statistic analysis. Although it made INCNS easier to calculate, it may compromise the overall accuracy. Third, with 19 items, the INCNS score is a little unwieldy. However, INCNS score already has less total scores and items than APACHE II and SAPS II, and all the items in INCNS score are routine labs and neurologic examinations in N‐ICU. Thus, rating the INCNS score won’t bring extra work. Moreover, patients receiving sedative agents or neuromuscular function blockers were not excluded. The validation results in those patients may be biased by the overestimation of their consciousness impairment. However, this is inevitable in the real world. Future studies are required to further examine the performance of INCNS score in more diverse populations.

5. CONCLUSIONS

We established a novel risk prediction score, named INCNS, to predict the 3‐month functional outcome of neurocritically ill patients. The INCNS score integrates measurements from five aspects: inflammation (I), nutrition (N), consciousness (C), neurologic function (N) and systemic condition (S). Data from a cohort of 941 neurocritically ill patients showed that INCNS score had significantly stronger predictive power than APACHE II and SAPS II at both 24 hours and 72 hours in N‐ICU.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

Supporting information

 

Gao Q, Yuan F, Yang X‐A, et al. Development and validation of a new score for predicting functional outcome of neurocritically ill patients: The INCNS score. CNS Neurosci Ther. 2020;26:21–29. 10.1111/cns.13134