Abstract
Objective
The aim of this report is to describe the sensitivity and specificity of Coma Recovery Scale-Revised (CRS-R) total scores in detecting conscious awareness.
Design
Data were retrospectively extracted from the medical records of patients enrolled in a specialized Disorders of Consciousness (DoC) program. Sensitivity and specificity analyses were completed using CRS-R-derived diagnoses of Minimally Conscious State (MCS) or Emerged from Minimal Conscious State (EMCS) as the reference standard for conscious awareness and the total CRS-R score as the “test criterion”. A receiver operating curve (ROC) was constructed to demonstrate the optimal CRS-R total cut-off score for maximizing sensitivity and specificity.
Setting
Specialized DoC program
Participants
252 patients enrolled in the DoC program (157 male; mean age = 49 years; mean time from injury =48 days; traumatic etiology=127, non-traumatic etiology=125; diagnosis of coma or VS=72, diagnosis of MCS or EMCS=182)
Interventions
Not applicable.
Main Outcome Measure(s)
Sensitivity and specificity of CRS-R total scores in detecting conscious awareness
Results
A CRS-R total score of 10 or higher yielded a sensitivity of 0.78 for correct identification of patients in MCS or EMCS, and specificity of 1.00 for correct identification of patients who did not meet criteria for either of these diagnoses (i.e., were diagnosed with VS or coma). The area under the curve (AUC) in the ROC analysis is 0.98.
Conclusion(s)
A total CRS-R score of 10 or higher provides strong evidence of conscious awareness but resulted in a false negative diagnostic error in 22% of patients who demonstrated conscious awareness based on CRS-R diagnostic criteria . A cut-off score of 8 provides the best balance between sensitivity and specificity, accurately classifying 93% of cases. The “optimal” total score cut-off will vary depending on the user's objective.
Keywords: Disorder of Consciousness, Brain Injury, Sensitivity, Specificity
Introduction
Patients emerging from coma after severe brain injury often transition through states of altered consciousness, including the vegetative (VS) and minimally conscious states (MCS). In VS, there is recovery of eye-opening but no behavioral evidence of self or environmental awareness [1]. MCS is characterized by clearly discernible but inconsistent behavioral signs of conscious awareness [2]. Distinguishing MCS from VS during the early stages of recovery is critically important as there is strong evidence that functional outcome is significantly more favorable for patients in MCS relative to those in VS, particularly following traumatic brain injury [3]. Prior research suggests that when diagnosis is made based on clinical consensus of the medical team, approximately 40% of patients diagnosed with VS actually retain conscious awareness [4]. These findings point to the need for more accurate diagnostic procedures.
The Coma Recovery Scale-Revised (CRS-R) is a standardized neurobehavioral assessment measure comprised of six subscales designed to assess arousal level, audition, language comprehension, visuoperception, motor function, oromotor capacity, expressive speech, and yesno communication in patients with disorders of consciousness (DoC) [5]. The current diagnostic criteria for coma, VS and MCS are embedded in the CRS-R to provide clinicians with a reliable and valid means of establishing a differential diagnosis. Although the CRS-R total score (TS) has been used for prognostic purposes [6], its diagnostic utility has not been investigated. This represents an important evidentiary gap given that prior studies have relied on the CRS-R TS when tracking functional recovery [7], investigating the relationship between behavioral and physiological markers of consciousness [8], and determining the effectiveness of treatment interventions [9]. The primary aim of this report is to investigate the sensitivity and specificity of CRS-R TS in detecting conscious awareness. Results are expected to provide additional empirical support and interpretive guidance for use of the CRS-R total score in clinical practice and research.
Methods
Following IRB approval to conduct retrospective medical record review , the research team from the medical records of patients enrolled in a specialized DoC program. Eligible patients were admitted to either an inpatient rehabilitation facility or a long-term acute care hospital, both of which rely on a standardized assessment protocol. Between 05/04/2011 and 09/29/2014, obtained at the time of admission into the program were collected for 252 patients (157 male; mean age=49 years [SD=19.7]; mean time from injury=48 days [SD =53]; traumatic etiology=127; mean CRS-R total score=11.27 [SD=5.83]; median=11; coma/VS=72 (mean=4.64 [SD=2.07]), MCS=138 (mean=12.21 [SD=3.67]), or EMCS=44 (mean=18.89 [SD=3.41]). Non-traumatic etiologies included hypoxia/anoxia, aneurysm, tumors and hemorrhage.
Sensitivity and specificity analyses were completed using CRS-R-derived diagnoses of MCS/EMCS (as denoted by an asterisk or cross, respectively, on the facesheet of the scale) as the reference standard for conscious awareness and the total CRS-R score as the “test criterion”. In this context, sensitivity (or true positive rate) represents the proportion of patients who retain conscious awareness (i.e., have a diagnosis of MCS/EMCS) and are correctly identified by the CRS-R TS. Specificity (or true negative rate) is the proportion of patients who are unconscious (i.e., do not have a diagnosis of MCS/EMCS) and are correctly identified as not in MCS /EMCS by the CRS-R TS (see Supplementary Figure S1 for method used to calculate sensitivity, specificity and accuracy). A receiver operating curve (ROC) was constructed to demonstrate the optimal CRS-R total cut-off score for maximizing sensitivity and specificity for detection of conscious awareness. All analyses were conducted using the [R] statistical software package [10] and Statistical Package for the Social Sciences (SPSS) Version 20.0.
Results
A CRS-R TS of 10 or higher yielded a sensitivity of 0.78 for correct identification of patients in MCS/EMCS, and specificity of 1.00 for correct identification of patients who did not meet criteria for either of these diagnoses (i.e., were diagnosed with VS or coma). Thus, all patients who obtained a CRS-R TS of 10 or higher demonstrated conscious awareness (i.e., met existing diagnostic guidelines for MCS/EMCS). A cut score of 10 or higher also misclassified as unconscious 22% of those who actually retained conscious awareness.
Table 1 indicates that the optimal combination of sensitivity and specificity is obtained with a CRS-R cut-off score of 8. TS of 8 yields a true positive rate of 93%, true negative rate of 96% and diagnostic accuracy rate of 93%.
Table 1.
Sensitivity, specificity, and accuracy rates for detection of conscious awareness at CRS-R total score cut-offs between 7-11.
| CRS-R Total Score Cut-off | 7 | 8 | 9 | 10 | 11 |
|---|---|---|---|---|---|
| Sensitivity | 0.97 | 0.93 | 0.88 | 0.78 | 0.73 |
| Specificity | 0.80 | 0.963 | 0.97 | 1 | 1 |
| Accuracy | 0.921 | 0.937 | 0.905 | 0.841 | 0.802 |
As shown in figure 1, the area under the curve (AUC) in the ROC analysis is 0.98 (Figure 1). AUC's between 0.9-1.0 are considered to have “excellent” diagnostic accuracy, based on standard classification guidelines. This finding suggests that CRS-R TSs provide “excellent” accuracy in differentiating individuals who meet diagnostic criteria for conscious awareness from those who do not.
Figure 1.
Receiver Operating Curve indicating sensitivity and specificity of CRS-R total scores highlighting the optimal cut-off score of 8. This score yields a sensitivity of 93% and specificity of 96%. The area under the curve is 0.98, indicating that the CRS-R total score accurately differentiates patients who meet diagnostic criteria for conscious awareness from those who do not.
Discussion
Our findings indicate that a total CRS-R score of 10 or higher provides strong evidence of conscious awareness as defined by existing diagnostic criteria [5]. All patients who received a CRS-R total score in this range were in MCS or had emerged from MCS. On the other hand, a cut-off score of 10 resulted in a false negative diagnostic error (ie, unconscious state) in 22% of cases who actually retained conscious awareness. Thus, approximately one in five patients who demonstrate conscious awareness score below 10. A cut-off score of 8 provides the best balance between sensitivity and specificity (or true positive and true negative rates, respectively), accurately classifying 93% of cases. The “optimal” TS cut-off will vary depending on the user's objective. For example, an investigator wishing to conduct a treatment study focusing exclusively on patients who retain conscious awareness would be advised to adopt a cut-off score of 10 as this will guarantee ascertainment of a homogeneous sample of participants in MCS/EMCS. On the other hand, a clinician involved in differential diagnostic assessment might adopt a cut-off score of 8, which offers the best odds of concurrently avoiding false positive and negative errors. Clinicians should rely on the full subscale profile when available to ensure the most accurate diagnosis. When relying on TS only, clinicians should be aware that the further the TS falls below 10, the lower the likelihood that the patient retains conscious awareness.
Study Limitations
The findings from this study should be viewed in the context of several limitations. First, our results may not be generalizable to all patients with DoC. Our sample was comprised of inpatients undergoing rehabilitation during the post-acute phase of recovery, most of whom were in MCS or had emerged from MCS. This may not reflect the larger pool of patients with DoC who are more acutely injured and are receiving care in the intensive care setting. Future studies will need to replicate these results in a larger sample of patients with more variable lengths of time since injury receiving treatment across a variety of healthcare settings.
Conclusion
Accurate detection of conscious awareness using the CRS-R is best achieved through analysis of the full CRS-R performance profile, which includes all six subscale scores. In the absence of a full CRS-R profile, the total CRS-R score can be interpreted as a marker of conscious awareness with confidence when the score is 10 or higher. CRS-R users should be aware of the advantages and limitations of selecting different TS cut-offs. This report provides data that may help guide decision-making for both clinicians and researchers who wish to use the CRS-R for diagnostic purposes.
Supplementary Material
Acknowledgements
The contents of this report were developed under a grant from the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR grant number 90DP0039-01-00). NIDILRR is a Center within the Administration for Community Living (ACL), Department of Health and Human Services (HHS). Additional support was provided by The James S. McDonnell Foundation, Harvard Catalyst | The Harvard Clinical and Translational Science Center (National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health Award UL1 TR001102) and financial contributions from Harvard University and its affiliated academic healthcare centers. The content is solely the responsibility of the authors and does not necessarily represent the official views of NIDILRR, ACL, HHS, NIH, the James S. McDonnell Foundation, Harvard Catalyst, or Harvard University and its affiliated academic healthcare centers, and you should not assume endorsement by the Federal Government.
Abbreviations
- CRS-R
Coma Recovery Scale-Revised
- DOC
Disorders of Consciousness
- VS
Vegetative State
- MCS
Minimally Conscious State
- EMCS
Emerged from Minimally Conscious State
- TS
Total Score
- SPSS
Statistical Package for the Social Sciences
- ROC
Receiver Operating Curve
- AUC
Area Under the Curve
References
- 1.Medical Aspects of the Persistent Vegetative State. New England Journal of Medicine. 1994;330(21):1499–1508. doi: 10.1056/NEJM199405263302107. [DOI] [PubMed] [Google Scholar]
- 2.Giacino JT, et al. The minimally conscious state: definition and diagnostic criteria. Neurology. 2002;58(3):349–53. doi: 10.1212/wnl.58.3.349. [DOI] [PubMed] [Google Scholar]
- 3.Lammi MH, et al. The minimally conscious state and recovery potential: a follow-up study 2 to 5 years after traumatic brain injury. Arch Phys Med Rehabil. 2005;86(4):746–54. doi: 10.1016/j.apmr.2004.11.004. [DOI] [PubMed] [Google Scholar]
- 4.Schnakers C, et al. Diagnostic accuracy of the vegetative and minimally conscious state: clinical consensus versus standardized neurobehavioral assessment. BMC Neurol. 2009;9:35. doi: 10.1186/1471-2377-9-35. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Giacino JT, Kalmar K, Whyte J. The JFK Coma Recovery Scale-Revised: measurement characteristics and diagnostic utility. Arch Phys Med Rehabil. 2004;85(12):2020–9. doi: 10.1016/j.apmr.2004.02.033. [DOI] [PubMed] [Google Scholar]
- 6.Giacino JT, et al. Monitoring rate of recovery to predict outcome in minimally responsive patients. Arch Phys Med Rehabil. 1991;72(11):897–901. doi: 10.1016/0003-9993(91)90008-7. [DOI] [PubMed] [Google Scholar]
- 7.Bagnato S, et al. EEG predictors of outcome in patients with disorders of consciousness admitted for intensive rehabilitation. Clin Neurophysiol. 2015;126(5):959–66. doi: 10.1016/j.clinph.2014.08.005. [DOI] [PubMed] [Google Scholar]
- 8.Rodriguez Moreno D, et al. A network approach to assessing cognition in disorders of consciousness. Neurology. 2010;75(21):1871–8. doi: 10.1212/WNL.0b013e3181feb259. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Margetis K, et al. Intrathecal baclofen associated with improvement of consciousness disorders in spasticity patients. Neuromodulation. 2014;17(7):699–704. doi: 10.1111/ner.12147. discussion 704. [DOI] [PubMed] [Google Scholar]
- 10.Team RC. R: A language and environment for statistical computing. R Foundation for Statistical Computing; Vienna, Austria: 2013. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.