Full Outline of UnResponsiveness score versus Glasgow Coma Scale in critically ill patients with altered sensorium: A comparison of inter-observer variability and outcomes

Abstract

Background and Aims:

Glasgow Coma scale (GCS), the most widely used tool for evaluation of the level of consciousness has various limitations. The Full Outline of UnResponsiveness (FOUR) score is a possible alternative. The present study was designed to examine the inter-rater reliability and outcome predictability of these scores in the Intensive Care Unit (ICU).

Methods:

The GCS and FOUR scores of 111 adult patients with altered sensorium, admitted to the ICU, were assessed as early as possible after admission by the Senior Resident (SR), Junior Resident (JR) and Staff Nurse (SN) of ICU. The outcomes measured survival and modified Rankin Scale (mRS) and Glasgow outcome scale (GOS) of the patients at discharge.

Results:

The inter-observer agreement was measured using the kappa ('k') statistic. For GCS it was higher (k = 0.472 to 0.555) than FOUR score (k = 0.352 to 0.448). A higher 'k' score in either score was recorded between SR and JR. Linear regression analysis showed no significant association of either score with the duration of ICU stay or mechanical ventilation. Survival in ICU was correlated with both GCS and FOUR scores on logistic regression. GOS and mRS were correlated with either GCS or FOUR scores on ordinal regression.

Conclusion:

The inter-observer agreement with FOUR score was not superior to GCS in this study, possibly due to lack of familiarity with the FOUR score. Both the scores were statistically correlated with the rate of survival.

INTRODUCTION

Coma represents a derangement in cerebral function due to various pathophysiological processes such as traumatic, toxic, metabolic, vascular, neoplastic, infective or seizure-induced; which ultimately presents a decreased arousal and awareness.[1] A simple assessment scale for evaluation of coma facilitates communication between healthcare providers and can predict morbidity and mortality. The Glasgow Coma Scale (GCS)[2,3] and Full Outline of UnResponsiveness (FOUR) score[4] are among the various scales developed for assessing patients with altered consciousness. GCS is being widely used not only by neurologists, neurosurgeons but also by other physicians as a standardised coma scale. The FOUR score which assess four components - eye response, motor response, brainstem reflexes and respiration pattern; was developed to overcome the shortcomings of GCS. The initial validation study of the FOUR score by Wijdicks et al,[4] revealed equivalent inter-rater reliability between the FOUR score and the GCS. Kramer et al.[5] studied the FOUR score ratings among nurses treating 907 critically ill adult patients in seven ICU's within five hospitals across United States and reported an almost perfect inter-rater agreement.

At present, studies on the inter-rater agreement of commonly used coma scores, between doctors and nurses or among doctors at various levels of training, are lacking. In addition, most of the available studies originate from developed countries where medical and nursing education, training and practice differ significantly from our country. In view of these factors, we designed this study with the primary aim to compare the inter-observer agreement of GCS and FOUR scores among resident trainee doctors at various levels of training and nurses in a heterogeneous patient population admitted to the Intensive Care Unit (ICU). The secondary objective was to compare the outcome predictability of both these scores.

METHODS

After obtaining approval from the Institutional Ethics Committee (Reference NK/335/MD/10189-90 dated 01.03.2013), we conducted a prospective observational study from January to December 2013 in the multidisciplinary ICU of a tertiary care teaching hospital, to compare the inter-observer variability of the FOUR score and GCS, and their association with resultant outcomes. The study was registered with the Clinical Trial Registry-India (CTRI 2017/10/010140). The study was carried out in conformity with the 2013 Declaration of Helsinki. Around 111 non-consecutive adult patients with altered sensorium were included in the study, after procuring necessary consent from a family caregiver of the patient. The causative factors of altered sensorium include traumatic head injury, stroke, non-traumatic intracranial haemorrhage, central nervous system infections, metabolic encephalopathy, status epilepticus, post-anoxic-ischemic encephalopathy, brain tumor and other conditions [Table 1]. Patients younger than 18 years of age were excluded from the study.

Table 1.

Demographic characteristics of the study population

Parameter	Values (%)
Male	80 (72.0)
Female	31 (27.9)
Diagnosis
Head Injury	51 (45.9)
Sepsis	20 (18.0)
ARDS	9 (08.1)
CVA	7 (06.3)
Poisoning	6 (05.4)
Hepatic Failure	4 (03.6)
Post-cardiac arrest care	4 (03.6)
Uraemia	3 (02.7)
Meningitis	3 (02.7)
Others*	4 (03.6)

(Percentages don’t add up to hundred as figures were rounded to first decimal). *Others included two cases of diabetic keto-acidosis, one case of snake bite and one of congestive cardiac failure

At the beginning of the study, all family caregivers of the participants or their next kin were briefed regarding the use of GCS and FOUR scores. Patients were tested once during their stay in the ICU, as early as possible after admission, with the patient off sedatives or neuromuscular junction blockers. Each patient was rated on both scales by three different raters - one senior resident (SR) and one junior resident (JR) doctor each from the Anesthesia Department; and one staff nurse (SN) from the ICU. The raters were not aware of ratings of the others. The SR and JR respectively represented doctors certified with a postgraduate qualification and those undergoing three year postgraduate medical training in anaesthesiology as per an already existent structured residency program.

For coma scale scoring, the raters were provided with a form of descriptive and diagrammatic instructions. In intubated patients, the rating for the verbal domain of GCS was defined to be one. Illustrative definitions of GCS score and FOUR score were placed in multiple places near the ICU. In addition, periodic verbal instructions regarding GCS and FOUR scores were also provided to the resident doctors and nurses of the ICU by consultant anaesthesiologist and intensivist of the ICU. Patient demographic data, diagnosis, duration of altered sensorium, vital signs, associated comorbid illness and any surgical procedures performed were recorded. We studied the following outcomes: Survival in the ICU, duration of mechanical ventilation, duration of ICU stay, modified Rankin Scale (mRS)[6] and Glasgow Outcome Scale (GOS)[7] at the time of discharge from the ICU.

The data collected in the prescribed proforma was entered into a Microsoft Excel spreadsheet and exported to and analyzed with R statistical software version 2.15.2[8] and R-Commander 1.9-2.[9] The analysis of the data collected was done using the kappa statistic('k'). Kappa is a measure of agreement, standardised to lie on a minus one to one scale, where one is a perfect agreement, zero is exactly what would be expected by chance, and negative values indicate agreement less than chance, i.e., potential systematic disagreement between the observers (The kappa statistic value and the extent of agreement are as:-kappa value <0: Less than chance agreement; 0.01–0.20: slight agreement; 0.21–0.40: fair agreement; 0.41–0.60: moderate agreement; 0.61–0.80: substantial agreement and 0.81–0.99: almost perfect agreement). A logistic regression analysis was performed between FOUR score and GCS with survival rate, a linear regression analysis between FOUR score and GCS with duration of mechanical ventilation and duration of ICU stay and an ordinal regression analysis between FOUR score and GCS with mRS and GOS at the time of discharge from ICU.

RESULTS

The baseline characteristics of the study population are shown in Tables 1 and 2.

Table 2.

Demographic and outcome characteristics of the study population

Parameter	Values
Age (Mean±SD)	40.1±17.6
Number of patients on mechanical ventilation	110 (99.1%)
Duration of mechanical ventilation in days (Mean±SD)	9.7±9.5
Duration of ICU stay in days (Mean±SD)	10.7±9.7
Outcome
Survival [n(%)]	73 (65.7)
Mortality [n(%)]	38 (34.2)

The 'k' values obtained for GCS and FOUR scores are shown in Table 3.

Table 3.

Table depicting the ‘k’ values obtained for GCS and FOUR scores across matched comparison of scorers

Coma score	Component scores	Senior Resident versus Junior Resident	Senior Resident versus Staff Nurse	Junior Resident versus Staff Nurse
GCS	Eye opening	0.721	0.608	0.579
	Motor response	0.605	0.689	0.605
	Verbal respose	1.000	0.744	0.744
	Total score	0.555	0.533	0.472
FOUR score	Eye response	0.633	0.501	0.548
	Motor response	0.603	0.680	0.648
	Brain stem reflexes	0.641	0.733	0.630
	Respiration	0.696	0.598	0.566
	Total score	0.448	0.419	0.352

The 'k' value of total GCS ranged from 0.472-0.555 with a higher agreement noted between SR and JR. The agreement was higher between SR and JR for component scores of eye opening and verbal response. Perfect agreement with 'k' = 1 was obtained between SR and JR for verbal response in GCS. The GCS component score agreement was 65.8% for eye opening, 58.6% for motor response and 98.2% for a verbal response.

The 'k' value of total FOUR score ranged from 0.352-0.448, with a higher agreement noted between SR and JR. The agreement was higher between SR and JR for component scores of eye response and respiration. The agreement was higher between SR and SN for component scores of motor responses and brainstem reflexes. The component score agreement was 57.7% for eye movement, 60.4% for motor response, 76.6% for brainstem reflexes and 81.1% for respiration.

The complete GCS and FOUR score rated by the SR were used for outcome analysis. Linear regression analysis showed no significant association of length of ICU stay with either GCS (P = 0.375, R²= 0.007) or FOUR score (P = 0.798, R²= 0.001). Further analysis showed no significant association between GCS and duration of mechanical ventilation (P = 0.177, R²= 0.017) and FOUR score and duration of mechanical ventilation (P = 0.474, R²= 0.005).

Regression analysis was used for estimating the relationship between outcome variables to either score. Logistic regression analysis showed significant association between GCS and ICU survival (P = <0.001, AIC (Akaike information criterion): 109.47) and FOUR score and ICU survival (P = <0.001, AIC: 105.46). A higher GCS and FOUR score were associated with a higher probability of survival and vice versa.

Ordinal regression analysis showed a significant association between GCS and FOUR scores and GOS and mRS at ICU discharge (P = <0.001). Higher mRS scores indicate higher disability and the highest possible score of six in the scale stands for mortality. Higher GCS and FOUR scores were associated with a higher probability of lower mRS with a better outcome. Lower GCS and FOUR scores were associated with higher probability of higher mRS indicating more disability and in some cases mortality.

Higher GOS indicates a better outcome with low disability. The lowest possible score of one in GOS indicates mortality. Higher GCS and FOUR scores were associated with a higher probability of higher GOS with a favorable outcome and low disability. Lower GCS and FOUR scores were associated with lower GOS signifying more disability and in some cases mortality [Figures 1–4].

Figure 1.

Relation of Senior Resident (SR) scored total GCS on outcome (Glasgow Outcome Score)

Figure 4.

Relation of Senior Resident (SR) scored total FOUR score on outcome (modified Rankin Score)

Figure 2.

Relation of Senior Resident (SR) scored total FOUR score on outcome (Glasgow Outcome Score)

Figure 3.

Relation of Senior Resident (SR) scored total GCS on outcome (modified Rankin Score)

DISCUSSION

GCS has been validated as a useful tool for outcome prediction post intracranial heamorrhage,[10] subarachnoid heamorrhage (SAH),[11] poisonings,[12] neurodegenerative diseases,[13] drowning,[14] cardiac arrest,[15] prediction of death in palliative care[16] and tuberculous meningitis.[17] GCS has also been incorporated into many other scoring systems such as Revised Trauma Score (RTS),[18] the APACHE II,[19] the Simplified Acute Physiology Score (SAPS), SAPSII,[20] the Circulation, Respiration, Abdomen, Motor, Speech scale (CRAMS),[21] the Traumatic Injury Scoring System (TRISS),[22] and A Severity Characterization of Trauma (ASCOT) scale[23] which provides ample testimony to the wide acceptance and ease of use of GCS.

The FOUR score was developed in the context of multiple demerits of GCS. Sub-optimal degrees of inter-rater agreement among inexperienced staff, especially in patients with moderate scores, ('intermediate' levels of consciousness) is widely reported with GCS.[24,25] The verbal sub-score assessment in intubated and aphasic patients remains a constrain in GCS.[26] GCS has limited efficacy in children prior to the acquisition of language (age <3 years). Withdrawal from pain, which can be wrongly interpreted as a flexion response, can confound GCS.[27]

This study compared the inter-observer variability of GCS and FOUR score among anaesthesia resident doctors at various levels of postgraduate medical training and ICU nurses; including a wide diagnostic category of patients medical, surgical and trauma. The SR represented more qualified and experienced observer in this study. The 'k' value of total GCS corresponded to a moderate agreement in comparison to higher agreement between SR and JR and a lower agreement between JR and SN. Among the component scores of GCS, higher agreement was observed between SR and JR except for motor response where the agreement was more between SR and SN. Agreement ranging from substantial to perfect was seen for a verbal response, which we attribute to most of the patients being intubated for mechanical ventilation in the ICU and thus being easy to score correctly [Table 3].

The 'k' value of total FOUR score was in the range of fair to moderate agreement. However, there was a higher agreement between SR and JR. Similarly, among the components of FOUR score, there was a higher agreement between SR and JR for eye response and respiration, whereas SR and SN recorded a higher agreement for motor response and brainstem reflexes.

The paired 'k' scores for eye opening component of GCS were in the range of substantial agreement (SR ~ JR: 0.721, SR ~ SN: 0.608 and JR ~ SN: 0.579). The paired 'k' scores for eye response component of FOUR score were in the range of moderate agreement (SR ~ JR: 0.633, SR ~ SN: 0.501 and JR ~ SN: 0.548). The eye responses in a majority of cases were distributed towards the lower side of both the scales i.e., most of the patients had no eye opening or there was eye response only to pain (E1/E2 in GCS and E0/E1 in FOUR score). This skewing of eye response to one side of the scale enhanced its inter-observer agreement. Addition of extra sub-score in the eye response component of FOUR score increased the total responses to five instead of four with GCS but this failed to show any added clinical advantage.

The paired 'k' scores for motor component of GCS and FOUR scores were in the range of substantial agreement (SR ~ JR: 0.605, SR ~ SN: 0.689 and JR ~ SN: 0.605; SR ~ JR: 0.603, SR ~ SN: 0.680 and JR ~ SN: 0.648; respectively). The motor responses were evenly distributed for both the scales with a majority of patients localizing pain or obeying commands i.e., M5/M6 in GCS and M4/M3 in FOUR score. Even with a deletion of withdrawal response in the motor component of FOUR score which can be confounded as flexion response in GCS, the observers in our study rated the motor component equally well. Addition of specific motor responses into the M4 sub-score of FOUR score such as 'shows thumbs-up, fist or peace sign' was moderately accepted by the observers. Our study population was unfamiliar with the above specific commands due to cultural differences.

The brainstem reflexes were distributed in a majority of patients towards the higher side of the scale with most of them having intact pupillary and corneal reflexes. The substantial agreement of brainstem reflexes of FOUR score among all the observers in our study is remarkable and prompts incorporation of these components individually for coma evaluation in future.

The substantial agreement of the respiration component of FOUR score should be interpreted with caution, as a comprehensive evaluation of breathing parameters could not be satisfactorily tested. Most of our patients were intubated for mechanical ventilation at admission; thus the assessment of the score was made easier. Intubation for mechanical ventilation restricts the respiration component to either RO/R1 sub-scores based on an assessment of ventilator waveforms. However, perfect inter-rater agreement was not achieved mostly due to some of our nursing personnel being unable to interpret the ventilator graphics.

Wijdicks et al.[4] prospectively studied the FOUR score in 120 intensive care unit patients and compared it with the GCS score using neuroscience nurses, neurology residents and neurointensivist and found excellent inter-rater reliability with the FOUR score ('k' = 0.82). The agreement among raters was similar to the GCS score ('k' = 0.82). A further study[28] by the same author states that the FOUR score provides a greater neurological detail than the GCS, recognizes a locked-in syndrome, and is superior to the GCS due to the availability of brainstem reflexes, breathing patterns, thereby having ability to recognize different stages of brainstem herniation. However, the same raters scored all patients in their study which could have confounded the results as raters gain more experience as the study progresses.

Iyer et al.[29] found excellent inter-rater agreement with the FOUR score and GCS; and similar results for subcomponent scores of each. The authors state that, study inclusion of only a limited number of trained scorers (physicians and nurses) could have contributed to this exceptionally high inter-rater agreement. Further study was conducted by Akavipat et al.[30] in a small group of exclusively neurosurgical patients.

In contrast to earlier studies from the Mayo Clinic and elsewhere, GCS fared better in terms of the inter-observer agreement compared to the FOUR score in our study[4,28,29,30] which may be attributed to many factors. The study population of this study was different from previous studies. The scoring assessments were done by resident doctors of varied experience and nurses. Experienced observers had a better inter-rater agreement than those with less experience, both with GCS and FOUR scorings in our study.

The 'k' scores in our study were higher in comparisons involving resident doctors than those involving nurses. Nurses performed lowest in our study with regard to the inter-rater agreement. Lacunae in the current nursing education with multitudes of training programs and curricula cannot be ignored and may be partially responsible to this. A streamlined course consisting of a unified and comprehensive syllabus, replacing different level courses is a consideration here. Unfamiliarity with the new clinical scoring system also needs to be considered.

Moreover, in a majority of previous studies, the scorings were done by a selected number of observers who assessed the patients throughout the study period. This was not in concordance with the real-life scenario, as the scoring need to be done by residents and nurses on duty in the ICU during patient admission. Therefore, in our study scoring was done by a large number of resident doctors and nurses. In our ICU resident posting changes monthly and nursing staff duty shifts change every six hours. Even though it was ensured that scorers are well-informed regarding the new scoring system, this volume of scorers might have considerably reduced the inter-rater agreement. This methodology is in concordance with a real-life scenario and regular working pattern of ICUs here, rather than a hypothetical situation, where the same small group observers examined the patients throughout the study.

Neither of the scores was significantly associated with the duration of mechanical ventilation or the duration of ICU stay. This may be attributed to the patients with extremes of scores having shorter duration of these parameters. In short, patients with higher GCS and FOUR scores had a faster recovery with shorter duration of mechanical ventilation and shorter ICU stay whereas severely ill patients with low admission GCS and FOUR scores scored poor with respect to ICU survival, thereby having lower duration of mechanical ventilation and ICU stay.

In our study, both the GCS and FOUR scores were significantly associated with survival i.e., most of the patients with poor admission GCS and FOUR scores sustained mortality. Patients with lower coma scales at ICU admission were severely ill and had more frequent organ failure rates, leading to higher mortality. All patients in our study were assessed for mRS and GOS at ICU discharge. The GCS and FOUR scores were significantly associated with mRS and GOS at ICU discharge i.e., patients with higher GCS and FOUR scores had better recovery profiles.

Despite limitations, the GCS remains the standard coma scale in our setting. However, the specific advantages of FOUR score such as the omission of verbal response, the examination of brainstem reflexes and grading breathing patterns, cannot be ignored. Hence, there is potential for considering selected components of FOUR score in coma evaluation. Although the FOUR score provides more neurological detail than the GCS it cannot replace the GCS; a score with a long tradition and validation in the ICU.

The methodological strengths of this study are as follows: (a) Inclusion of varied causes of altered sensorium both medical and surgical; (b) Inclusion of nurses along with resident doctors with different levels of medical training and experience as raters; (c) Study being done in concordance with the real-life scenario with scoring done by doctors and nurses on duty in ICU rather than selecting specific groups of observers who rate patients throughout the study; (d) Multiple outcome analysis done in terms of survival, duration of mechanical ventilation, duration of ICU stay, mRS and GOS at ICU discharge.

The raters in our study were not blinded to the diagnosis of the case; hence bias judgments could have affected the clinical assessment. This is important when medical, surgical and trauma patients are involved in the study sample. Multiple observers were involved in the study which added to the unfamiliarity of the new scoring system. The GOS and mRS in our study were assessed at ICU discharge for methodological convenience, rather than for long term follow-ups. Many ICU patients may undergo rehabilitation after discharge which thereby affects the long-term clinical and neurologic outcome. Long term outcome analysis with detailed neurologic examination along with mRS and GOS scores need to be incorporated in further studies.

CONCLUSION

In terms of inter-rater agreement, GCS continues to remain the gold standard for assessing sensorium in our setting. Both GCS and FOUR score can be used for predicting ICU survival. The duration of mechanical ventilation and duration of ICU stay cannot be predicted from ICU admission GCS and FOUR scores. Lower admission GCS and FOUR scores were associated with lower chances of survival and unfavourable mRS and GOS scores at ICU discharge.

Declaration of patient consent

The authors certify that they have obtained all appropriate patient consent forms. In the form, the patient(s) and/or their surrogates has/have given his/her/their consent for his/her/their images and other clinical information to be reported in the journal. The patients/surrogates understand that their names and initials will not be published and due efforts will be made to conceal their identity, but anonymity cannot be guaranteed.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.