Reliability and Validity of the Sensory Component of the International Standards for Neurological Classification of Spinal Cord Injury (ISNCSCI): A Systematic Review

Abstract

Background:

Since 1982, the International Standards for Neurological Classification of Spinal Cord Injury (ISNCSCI) has been used to classify sensation of spinal cord injury (SCI) through pinprick and light touch scores. The absence of proprioception, pain, and temperature within this scale creates questions about its validity and accuracy.

Objectives:

To assess whether the sensory component of the ISNCSCI represents a reliable and valid measure of classification of SCI.

Methods:

A systematic review of studies examining the reliability and validity of the sensory component of the ISNCSCI published between 1982 and February 2013 was conducted. The electronic databases MEDLINE via Ovid, CINAHL, PEDro, and Scopus were searched for relevant articles. A secondary search of reference lists was also completed. Chosen articles were assessed according to the Oxford Centre for Evidence-Based Medicine hierarchy of evidence and critically appraised using the McMasters Critical Review Form. A statistical analysis was conducted to investigate the variability of the results given by reliability studies.

Results:

Twelve studies were identified: 9 reviewed reliability and 3 reviewed validity. All studies demonstrated low levels of evidence and moderate critical appraisal scores. The majority of the articles (~67%; 6/9) assessing the reliability suggested that training was positively associated with better posttest results. The results of the 3 studies that assessed the validity of the ISNCSCI scale were confounding.

Conclusions:

Due to the low to moderate quality of the current literature, the sensory component of the ISNCSCI requires further revision and investigation if it is to be a useful tool in clinical trials.

Since 1982, the International Standards for Neurological Classification of Spinal Cord Injury (ISNCSCI) and the American Spinal Injury Association (ASIA) Impairment Scale (AIS) have been referred to as the gold standard of SCI classification.¹ In 1992, ISNCSCI was adopted by the International Spinal Cord Society (ISCoS).² The scales were originally used to classify the neurological impairment of individuals with SCI; however they are now commonly used as an outcome measure in clinical trials assessing changes in neurological deficits. The ISNCSCI consists of motor and sensory components.³ The motor component is designed to test the strength of 10 key muscles in the upper and lower limbs. The sensory component uses light touch and pinprick ordinal scale scores. The ISNCSCI scoring requires 2 skills from the examiner: (1) the ability to clinically score the patient and (2) the ability to correctly classify the SCI patient from the scores obtained on assessment.⁴ The reliability of the ISNCSCI scoring has been studied extensively with variable outcomes.^2,5–7 It has been suggested by some authors that ISNCSCI raters require expert training.⁸ The 2009 review and revisions of the ISNCSCI concluded that there was adequate face validity based on a discussion by the ASIA standards committee that examined both clinical experience and research results.⁹ Although a commonly utilized tool, questions have been raised regarding the lack of sensitivity, reliability, and validity of the ISNCSCI, particularly in relation to its sensory component.^10–12

The psychometric properties of the motor and sensory components of the ISNCSCI have been reviewed.^13,14 It was concluded from these articles that the ISNCSCI is an appropriate instrument for subacute and chronic spinal cord–injured patients but not for the acute lesions. The functionality of the ISNCSCI score was also questioned in these articles. The aim of this study, therefore, was to systematically review the literature published between 1982 and 2013 relating to the reliability and validity of the sensory component of the ISNCSCI.

Methods

The strategy used to identify published studies assessing the reliability and validity of the sensory component of the ISNCSCI followed the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement and checklist.¹⁵

MEDLINE via Ovid, Scopus, CINAHL, and PEDro databases were searched using the following search terms: “spinal cord injuries” OR “traumatic SCI” OR “SCI” OR “American Spinal Injury Association” OR “ASIA” OR “International Standards for Neurological Classification of Spinal Cord” OR “ISNCSCI” OR “ASIA Impairment Scale” OR “AIS” AND “reliab*” OR “valid*” (1982 – February 2013). Where possible, MeSH (Medical Subject Headings) terms were applied to include all possible relevant research. Additional articles were sourced through a second search that analyzed the reference lists of other systematic reviews, meta-analysis, and nonsystematic reviews.

Study selection criteria

Articles were selected on strict inclusion and exclusion criteria. Inclusion criteria required that the articles assessed the reliability and/or validity of at least the sensory component of the ISNCSCI, all study participants had been diagnosed with a traumatic SCI correlating with the AIS classifications A-E (see exam sheet; http://www.asia-spinalinjury.org/elearning/ISNCSCI_Exam_Sheet_r4.pdf), and all were written in English. The included articles comprised pretest/ posttest, observational prospective cohort studies, cross-sectional design, or retrospective case series.

Articles were excluded from the review if participants were younger than 16 years, had SCIs with associated head injuries or multitrauma, or had nontraumatic SCIs. Systematic reviews, letters, and articles from non-peer-reviewed journals were also excluded.

Article review criteria

Articles were analyzed and the quality determined through the grading system of The Oxford Centre for Evidence-Based Medicine (OCEBM) hierarchy of evidence.¹⁶ Articles were also critically appraised using the McMasters Critical Review Form–Quantitative Studies.¹⁷ The aim of the McMasters Critical Review Form (eFigure) was to identify methodological flaws in the selected literature and provide a numerical score out of 13 regarding the quality of research evidence. To decrease bias, 2 of the authors (M.H. and J.R.) analyzed the articles and any disagreements were resolved through consensus.

Following analysis of the level of evidence of each article and its critical appraisal, the following factors were extracted:

1. Population sample, including complete or incomplete and tetraplegic or paraplegic

2. Type of reliability or validity

3. Version of the ISNCSCI

4. Raters

5. Case studies or participants

6. Training and amount

7. Results, including pre and post training.

Statistical analysis

Statistical analysis was performed to assess the association of the ISNCSCI posttest results with extracted variables by the Kendall rank correlation and nonparametric linear regression tests using the PASW Statistics 18 Release 18.0.0 package. Statistical significance was defined at P value < .05.

Results

A total of 200 articles were found. Following application of the exclusion and inclusion criteria and removal of duplicates, 12 articles were included within the review (Figure 1). Of these 12 articles, 9 examined the reliability of the ISNCSCI scale (Table 1) and 3 examined the validity of the ISNCSCI scale (Table 2). Using the McMasters Critical Review Form– Quantitative Studies,¹⁷ we examined all 12 of the studies included in this review for quality (eTable). The levels of evidence according to the OCEBM¹⁶ and the critical appraisal scores are documented in Table 3. Six of the 9 articles assessing reliability provided their postresults outcome measures and were used for statistical analysis (Tables 4 and 5).

Table 1. Analysis of reliability studies.

Study (year)	Type of reliability	Reliability of classification or examination assessment	No. of patients	Study populationa (C/I)	Study populationb (T/P)	Case studies or participants	Raters	Pretest results	Posttest results	Version of ISNCSCI standards
Priebe & Waring6(1991)	Intrarater	Classification	10	[Insufficient information]	[Insufficient information]	Case studies	15 (house officers and physician faculty members)	Sensory scores percent correct, 71%	Sensory scores percent correct, 93%	1982 and 1989
Donovan et al2 (1997)	Interrater	Classification	2	C, 1 I, 1	T,1 P,1	Case studies	5 (nil information)	N/A	LT, 90% correct; PP, 90% correct; overall classification	1992
Cohen et al4 (1998)	Intrarater	Classification	2	C, 1 I, 1	T,1 P,1	Case studies	106 (physicians, PTs, OTs, nurses, and rehabilitation specialists)	Sensory level pre-test, 84.75% correct; AIS, 76% correct	of sensory 70% Sensory level posttest, 86.25% correct; AIS, 81.5% correct	1992
Jonsson et al19 (2000)	Interrater	Examination and classification	23	C, 3 I, 20	T,12 P,11	Participants	4 (2 physicians; 2 PTs)	PP scoring was acceptable (0.61-1 kappa values) in only 4/46 calculated dermatomes and LT in only 25/42 calculated dermatomes.	PP scoring was acceptable (0.61-1 kappa values) in only 22/50 calculated dermatomes and LT in only 24/50 calculated dermatomes.	1992
Mulcahey et al12(2007)	Intrarater	Examination and classification	5	C, 2 I, 3	T,3 P,2	Participants	6 (research assistants or principal investigators)	Before training, discrimination ICC=0.786 and LT ICC=0.825	After training, discrimination ICC=0.892 and LT ICC=0.767	2000
Savic et al5 (2007)	Interrater	Examination and classification	45	C, 24 I, 21	T,15 P,30	Participants	2 (clinical scientist with medical background and a senior research physiotherapist)	N/A	LT ICC = 0.997, PP ICC = 0.988; overall sensory level, 76.5% correct	2000
Chafetz et al18 (2008)	Interrater and intrarater	Classification	10	C, 2 I, 8	T,6 P,4	Case studies	28 (11 PTs, 6 OTs, 3 physicians, 4 AHPs, 2 research associates, 2	Pretest, 91% correct	Posttest, 97% correct	2003
Marino et al20 (2008)	Interrater and intrarater	Examination and classification	16	C, 10 I, 6	T,10 P,6	Participants	student therapists) 16 (8 physicians; 8 PTs)	N/A	Interrater reliability for LT, ICC = 0.96; PP ICC = 0.89. Intrarater reliability for LT and PP ICC=0.99	2000
Schuld et al8 (2012)	Interrater and intrarater	Classification	5	C, 1 I, 4	T,3 P,2	Case studies	106 (PT, physician, OT, other rehab professionals)	Pretest, 60.6% correct	Posttest, 96.8% correct	2011

Note: AHP = allied health professional; AIS = ASIA Impairment Scale; ASIA = American Spinal Injury Association; ICC = intraclass correlational coefficient; ISNCSCI = International Standards for Neurological Classification of Spinal Cord Injury; LT = light touch; OT = occupational therapist; PP = pinprick; PT = physiotherapist.

^aC = complete; I = incomplete.

^bT = tetraplegia; P = paraplegia.

Table 3. Level of evidence and critical appraisal of validity and reliability studies.

Study (year)	Study design	Psychometric property	OCEBM hierarchy of evidence	McMasters Critical Review Form – Quantitative Studies critical appraisal
Priebe & Waring6 (1991)	Pretest/posttest	Interrater reliability	4	8/13
Bednarczyk & Sanderson23 (1993)	Observational prospective cohort	Convergent construct validity (wheelchair basketball sports test)	3	10/13
Curt & Dietz22 (1997)	Prospective cohort	Convergent construct validity	3	12/13
Donovan et al2 (1997)	Retrospective case series	Interrater reliability	4	5/13
Cohen et al4 (1998)	Pretest/posttest	Interrater reliability	4	11/13
Jonsson et al19 (2000)	Pretest/posttest	Interrater reliability	3	11/13
Mulcahey et al12 (2007)	Test-retest	Interrater reliability	3	8/13
Savic et al5 (2007)	Observational prospective	Interrater reliability	4	11/13
Chafetz et al18 (2008)	Pretest/posttest	Interrater and intrarater reliability	3	12/13
Marino et al20 (2008)	Observational prospective cohort	Interrater and intrarater reliability	4	8/13
Kalsi-Ryan et al21 (2012)	Cross-sectional	Construct validity (GRASSP)	3	10/13
Schuld et al8 (2012)	Prospective longitudinal cohort study	Interrater and intrarater reliability	3	11/13

Note: GRASSP = Graded Redefined Assessment of Strength, Sensibility and Prehension; OCBEM = Oxford Centre for Evidence-Based Medicine; SSEP = somatosensory-evoked potential.

Table 2. Analysis of validity studies.

Study (year)	Psychometric property	Sample size, n	Study population, n	Examiners	Outcome measures	Results	Version of ISNCSCI standards
Bednarczyk & Sanderson23 (1993)	Construct validity (wheelchair basketball sports, Bracken scale)	30	Chronic SCI: Tetraplegia, 9 Paraplegia, 21	Not specified (physiotherapist experienced in medical and functional assessment techniques)	Spearman’s rho correlation coefficients	Spearman’s rho correlation coefficients showed positive associations between the ISNCSCI and basketball sports test (0.81)	1990
Curt & Dietz22 (1997)	Construct validity (SSEP)	104	Acute SCI: Tetraplegia, 31 Paraplegia, 39 Chronic SCI: Tetraplegia, 34	Not specified (specially trained physicians with >1 yr experience)	Spearman’s rank correlation coefficients and ANOVA	In acute SCI, ISNCSCI scores and SSEP are significantly related to the outcome of ambulation (P < .001).	1992
Kalsi-Ryan et al21 (2012)	Construct validity (GRASSP)	72	Chronic tetraplegia, 72	14 (10 occupational therapists; 2 physiotherapists)	Agreement/ discordance analysis	Average 54% discordance in ISNCSCI sensory innervation was recognised.	2000

Note: ANOVA = analysis of variance: ASIA = American Spinal Cord Injury Association; GRASSP = Graded Redefined Assessment of Strength, Sensibility and Prehension; ISNCSCI = International Standards for Neurological Classification of Spinal Cord Injury; SCI = spinal cord injury; SSEP = somatosensory-evoked potentials.

Table 4. Amount of training and experience of raters in 6 studies analyzed.

Study (year)	No. of patients	Mean age of patients	% Females	No. of raters	Experience of raters	Amount of ISNCSCI training	Posttest results
Priebe & Waring6 (1991)	10	N/A	N/A	15	Experienced	10 min (discussion)	93
Donovan et al2 (1997)	2	38	0%	5	Experienced	No information	70
Cohen et al4 (1998)	2	N/A	N/A	106	Experienced	45 min (video & presentation)	86.25
Savic et al5 (2007)	45	40.3	15%	2	Experienced	0 min (examiners met before)	76.5
Chafetz et al18 (2008)	10	N/A	N/A	28	Experienced	90 min (educational session)	97
Schuld et al8 (2012)	5	N/A	N/A	106	Experienced	12 hours (workshops)	96.8

Note: Experience of raters can be binary (experienced/unexperienced). ISNCSCI = International Standards for Neurological Classification of Spinal Cord Injury; N/A = not applicable.

Table 5. Regression and correlation analysis of the ISNCSCI posttest results with underlying assessment characteristics.

Characteristic	τ	95% CI	P*	Coefficient	95% CI	P*	Power
No. of patients	0.215	-0.577-1.000	.697	0.151	-0.760-3.517	.697	0.224
No. of raters	0.414	-0.153-0.981	.339	0.178	-2.167-1.174	.339	0.271
Physicians, %	0.200	-0.784-1.000	.806	0.136	-0.412-0.467	.806	0.219
Training, hours	0.690	0.300-1.000	.056	8.004	-11.638-21.667	.085	0.324

Note: Values provided are the Kendall rank correlation coefficient, the nonparametric linear regression coefficient, and the achieved power for calculations at α < 0.05. ISNCSCI = International Standards for the Neurological Classification of Spinal Cord Injury.

^*Two-sided P value.

Hierarchy of evidence and critical appraisal of studies

Table 3 provides the level of evidence and critical appraisal score for each reliability and validity study. Of the 12 articles selected for the review, 5 were a level IV on the OCEBM hierarchy of evidence. The remaining 7 scored a level III, suggesting a lower level of evidence.

The critical appraisal scores ranged from 5 to 12 out of 13 on the McMasters Critical Review Form–Quantitative Studies (eTable), with a mean score of 10 out of 13 demonstrating a reasonable level of quality.

Reliability

The publication dates of most of the reliability studies (6 out of 9) were from the last decade, indicating a rise in interest in this topic (Table 1).^5,8,12,18,19 Inter- and intrareliability were consistently researched across the reliability studies. The ability to classify and clinically examine the patient was examined in 4 of the 9 studies, whereas classification alone was examined in 5 of the articles (Table 1). Although this may be seen as a bias toward classification, it would appear that this is the more problematic skill component.

In each of the reliability studies, raters included physiotherapists, general physicians, clinical scientists with medical background, occupational therapists, nurses, and specialist rehabilitation physicians. The number of raters varied from 2 to 106 and 6 studies had 16 or fewer (Table 1).^{2,5,6,12,19,20} The population samples included participants with either incomplete or complete paraplegic or tetraplegic SCIs (Table 1). Only 6 of the 9 studies had a proportionate number of participants in their study designs; the remainder demonstrated a high level of inconsistency. The study by Jonsson et al¹⁹ had 3 participants with a complete SCI compared to 20 participants with an incomplete SCI; Chafetz et al¹⁸ had 8 participants with incomplete SCIs compared to 2 participants with complete SCIs; and Savic et al⁵ had 30 participants with paraplegia compared to 15 participants with tetraplegia. Priebe and Waring⁶ did not supply sufficient data to analyze their sample. We used 3 outcome measures when examining the results for the reliability studies; percentage correct (5 studies), intraclass correlation coefficients (ICC) (3 studies), and kappa values (1 study) (Table 1).

Validity

Only 3 studies analyzed the validity of the ISNCSCI (Table 2). Sample sizes varied from 30 to 104; participants in these studies had incomplete or complete paraplegic or tetraplegic SCIs (Table 2).

Examiners in each of the validity studies included physiotherapists, specially trained physicians, and occupational therapists. Kalsi-Ryan et al²¹ was the only study to specify the number of examiners used.

The 3 studies that analyzed validity of the ISNCSCI used different versions of scale and various outcome measures (Table 2). Kalsi-Ryan et al²¹ compared construct validity of the Graded Redefined Assessment of Strength, Sensibility and Prehension (GRASSP) to the ISNCSCI, Spinal Cord Independence Measure (SCIM), and Capabilities of Upper Extremities (CUE). Due to the ISNCSCI having no palmer test location, the GRASSP was concluded to have on average 54% discordance to the ISNCSCI, making it a more sensitive measure. Curt and Dietz²² used Spearman’s rank correlation coefficients and analysis of variance (ANOVA) to analyze the ISNCSCI in relation to the somatosensory evoked potentials (SSEP). In an acute SCI, the ISNCSCI scores and SSEP scores are significantly related (P < .05) to the outcome of ambulation. It was also found that the ISNCSCI sensory component was less sensitive to the SSEP scores when compared to the ISNCSCI motor scores. Bednarczyk and Sanderson²³ analyzed the ISNCSCI against the Bracken scale and the Wheel Chair Basketball Sports test; negative correlations were found with both scales and the ISNCSCI was reported to demonstrate the best discrimination and sensitivity when classifying an SCI.

Statistical analysis

Six articles providing the ISNCSCI posttest results and underlying study characteristics, such as the number of patients and raters, percentage of physicians among raters, and the length of ISNCSCI training, were selected for a statistical analysis (Table 3). Our analysis indicates that the amount of training positively correlates with posttest results (Kendall tau = 0.690, P = .056) (Table 4). Further investigations were made to analyze the relationship between the ISNCSCI posttest results and the amount of training (hours) using a nonparametric linear regression analysis with resultant regression coefficient of ~8 (P = .085) (Table 5). Statistical power for all computations was <80% (Table 5).

Discussion

The ISNCSCI scale has become a widely accepted tool for classification of SCI. It is endorsed as the gold standard of SCI classification by ASIA and ISCoS. Over the last 2 decades, research has questioned whether the ISNCSCI is a reliable and valid measure, sensitive enough for use within clinical trials.

Classification of a patient’s sensation in the ISNCSCI is measured through pinprick and light touch ordinal scores. There are, however, 4 major modalities of sensation that include mechanoreceptors, proprioceptors, thermal receptors, and nociceptors.²⁴ Although, the 2011 ISNCSCI revision suggested that the scale is not a comprehensive neurological examination, it would appear that the omission of proprioception, heat, and pain sensation may affect the reliability and validity of the scale.^25–30 The use of simple clinical measures such as thermal and proprioceptive sensory testing should be considered.³ As the ISNCSCI scale is used in clinical trials as both a primary and secondary objective measure, the testing of sensory modalities should be investigated in an attempt to improve the detection of neurological changes in patients with SCI. In this review, we found evidence to suggest that the classification of patients with an incomplete SCI is less reliable than classification of patients with a complete SCI.^4,6,8,18,20

The 3 articles on validity discussed in this review^21–23 had confounding results due to the differences in the types of validity examined. In addition, these articles compared the ISNCSCI to a variety of other measures, some of which were physiological measures (SSEP) while others were measures of impairment or function (GRASSP, CUE), leading to an obvious limitation in their comparisons. All, however, suggest that the ISNCSCI sensory component needs to be improved to increase its sensitivity and validity when examining neurological change.

Examiners and/or raters in each of the studies discussed included research assistants, physicians, physiotherapists, occupational therapists, nurses, allied health professionals, and other research and rehabilitation professionals. This diverse group of professionals has varied levels of experience. Schuld et al⁸ verified that “experience in ISCNSCI represents the only factor that significantly influenced the pre-test performance”^(p6) and that training was supported by the majority of the reliability studies; results showed an increase in the reliability of the ISNCSCI after training.^4,6,8,12 The current systematic review also suggests a trend towards a positive correlation between the amount of time of training and the ISNCSCI posttest results. Recent studies have also shown training to be extremely successful.³¹ Even though ASIA offers training modules, these modules are not considered a necessity by any health professionals before utilizing the scale. It is therefore suggested that an accredited ISNCSCI training program be completed before any professional may officially utilize the scale.

For more than 10 years, the international community of clinicians and scientists involved in the area of SCI has been aware of the lack of assessment of the autonomic nervous system within this client group and that this omission could lead to reduced efficacy in treatment.^32,33 This has led to the development of a new international standards document: International Standards of Autonomic Function after Spinal Cord Injury (ISAFSCI).³³ Autonomic dysfunction is common complication associated with SCIs and can have a severe impact on an individual’s cardiovascular system, resulting in life-threatening conditions such as autonomic dysreflexia.^13,32 Although the autonomic system cannot be classified as a sensory modality, the lack of its assessment within a scale such as the ISNCSCI may well affect the scale’s reliability and validity.

The small number of studies that were suitable for inclusion in this systematic review and the subsequent lack of heterogeneity between their outcome measures limit the interpretation of our findings.

In conclusion, the findings of this study offer directions for improving the posttest reliability scores through the training of health professionals in the use of ISNCSCI. Continued research into the reliability and validity of the scale needs to be completed and further research into ISNCSCI training regimen is recommended.