Computerized Classification of Neurologic Injury Based on the International Standards for Classification of Spinal Cord Injury

Abstract

Background/Objective:

To develop and test a computer program to accurately score International Standards for Classification of Spinal Cord Injury (ISCSCI) examinations.

Methods:

A computer program was written following the algorithm published in the ISCSCI 2003 manual. Two methods were used to test the computer program's accuracy. First, 94 hand-scored examinations representing 25 different patients were entered into the program. Differences between hand-scored examinations and computer-scored examinations were evaluated for accuracy. Second, all case examples from the ISCSCI manual were entered into the computer program to verify its accuracy.

Results:

Of the 94 hand-scored examinations, the computer detected all incorrectly scored examinations. Furthermore, the computer scoring agreed with every examination's scores that were correctly calculated by hand. Of the hand-scored examinations, 10% of sensory total scores and 4% of motor scores were incorrect. For sensory level and motor level, there were errors in 9% and 26% of hand-scored examinations, respectively; 13% of hand-scored examinations had incorrectly assigned ASIA Impairment Scale classification.

Conclusion:

This study showed that the computer program we developed was effective in correctly scoring ISCSCI examinations and was able to detect errors in hand-scored examinations.

INTRODUCTION

The International Standards for Classification of Spinal Cord Injury (ISCSCI) is recognized as the primary measurement (1) of the neurologic consequence of a spinal cord injury (SCI). Over the last decade, it has been used as a primary outcome measure for research studies and standard assessment in the care of patients with SCI (2–8). If a tool is to be a meaningful indicator of the construct it purports to measure, it must be valid, reliable, and have demonstrated utility for the population of interest. Currently, our research center has been focused on the establishing the psychometric properties of the ISCSCI examination and classification techniques when applied to children (9–14).

One aspect that contributes to reliability of the ISCSCI is the scoring. In the case of the ISCSCI, it is essential that scores are calculated and summed correctly because of the implication for classification of motor levels (ML), sensory levels (SL), neurologic level (NL), and American Spinal Injury Association (ASIA) Impairment Scale (AIS) (9,15–17). Past literature, including a recent training session at our center, showed that even experienced clinicians have difficulty scoring examinations (9,15–17). Training sessions have shown improvement in scoring of examinations, but the improvement has been less than acceptable for research purposes (9,15–17).

There are several possible reasons for the problems with accurate scoring and classification of ISCSCI examinations. Since its genesis more than 20 years ago, several significant changes have occurred in examination classification. For example, initially in 1983, an incomplete SCI was defined as having movement or sensation 3 levels below the neurologic injury; in 1992, an incomplete SCI was redefined as having sacral sparing (1,18). A second significant change involved the operational definition of ML. In this revision, segmental levels where no key motors muscles are defined, corresponding dermatomes are used to represent motor function, such that, if a dermatome was impaired, the ML was also considered impaired (1). An incorrectly scored ML can lead to incorrect scoring of the AIS, which is dependent on correct ML designation. Both incorrect ML and AIS designations are the most frequent type of errors when scoring ISCSCI examinations (9,15–17).

Recently, our center participated in an inter/intrarater reliability study that involved the scoring of a large number of examinations (9–14). As a mechanism for quality management of the data set, a computer algorithm was built for scoring the ISCSCI examination. The main purpose of this study was to describe the computer program development and testing of the program's accuracy.

METHODS

As part of the larger study and using methodology similar to previous reports (9–13), subjects younger than 21 years of age participated in up to 4 ISCSCI motor, sensory, and anorectal examinations performed by trained examiners. All participants and their parents provided written informed assent and consent, respectively, and Health Insurance Portability and Accountability Act forms approved by the Institutional Review Board. Details of the procedures used to conduct the ISCSCI examination have been published elsewhere (10–12,14) and followed the protocol established by the standards committee of the American Spinal Injury Association (1). Before data collection, study examiners participated in training sessions on performance and classification of the ISCSCI examination (9,11).

Database and Scoring Algorithm

For purposes of quality control, a computer algorithm was design to cross-check the information that was being entered into the computer. The intention of the program was not to provide the final scoring of the clinical examination but rather to serve as a double check of the data being entered into the database for quality control purposes. The database for this study was designed using Microsoft Access. The scoring algorithm was built into the database using visual basic programming. To build the algorithm, the 2003 ISCSCI manual was used for the scoring scheme (Figure 1). To complete the scoring program, arrays were built separating SL and ML into appropriate grouping: C2–C4, C5–T1, T2–L1, L2–S2, and S3–S4/5. With the use of the arrays and “If-then” statements, a program was built that sequentially checked whether scores of 5 or 2 were given at each myotome and dermatome, respectively. The SL was considered the lowest dermatome scored as normal where all levels above were also scored as normal. ML was considered the lowest tested key muscle that was equal to or greater than 3/5 with all rostral levels scored as normal. As stated by the 2003 reference ISCSCI manual revision (1), “for those myotomes that are not clinically testable by a manual muscle examination, the motor level is presumed to be the same as sensory levels. If the sensation for a segment is normal, motor function for that segment is considered normal; if sensation is impaired, motor function is considered impaired” (ASIA, p 52).

Figure 1.

Scoring algorithm for ISCSCI examinations using 2003 standards.

Evaluation of Accuracy

Initially, all examinations were scored by hand and entered into the database by an independent research assistant. The 4 raters who participated in scoring the examinations were 3 physical therapists with 5, 7, and 2 years of experience and 1 OT with 19 years of experience in administering and scoring ASIA examinations. After entering the data, the computer automatically compared the results of the computer-classified examination to the hand-scored examinations. If a discrepancy was found, the field was given a red background, and the corrected score was displayed in the adjacent box (Figure 2). Flagged fields were reviewed by the first and senior author for accuracy. There was diligence in this review process to ensure the computer logic was accurately written to reflect the standards scoring and classification techniques. Throughout the development of the algorithm, several adjustments were made to the program until the final computer scoring reflected the ISCSCI standards. Any hand-scored errors were reviewed and, if needed, corrected, initialed, and dated on the original ISCSCI form.

Figure 2.

Database data entry page. Example of computer screen shot of an incorrectly hand scored exam. Red fields identify errors in scoring of the motor levels and AIS.

To test the accuracy of the computer program, 2 methods were used. First, we compared the first 25 hand-scored examinations (scored before development of the computer program) to the computer program scoring. Second, we entered all the case examples in the 2003 revision manual into the computer program to ensure the computer program scored the examinations the same as the manual.

Sample

This study included 94 examinations of 25 subjects. Each subject participated in up to 4 examinations, conducted by 2 raters within a 1-week period. In the sample, there were 11 male and 14 female subjects, with an average age of 12.92 years. Using the ISCSCI standards, the distribution of AIS classification was 14 subjects with AIS A, 5 subjects with AIS B, and 4 and 2 subjects with AIS C and AIS D, respectively.

RESULTS

Of the initial 94 hand-scored examinations, the computer was able to detect all incorrectly scored examinations. Furthermore, the computer scoring agreed with every examination's scores that were correctly calculated by hand. In other words, the computer program did not identify any correctly scored examinations as incorrect.

Errors identified by the computer were found in calculating total sensory/motor total scores, determining SLs, MLs, and AIS. Of the hand-scored examinations, 10% of sensory total scores and 4% of motor scores were incorrect. This represented errors in 36% of sensory scores and 16% of motor scores. For SL and ML, there were errors in 9% and 26%, respectively, of hand-scored examinations, representing 28% and 52% of patients. Only 1% of hand-scored examinations had severity of injury (complete\incomplete) incorrectly scored (representing 4% of patients) but had 13% incorrectly scored AIS designations representing 28% of patients.

Figure 2 is an example of the computer program identifying an incorrectly hand-scored examination. In this example, severity of injury was incorrectly scored as complete. Because the patient has sensation at bilateral light touch S45 dermatome, this patient's injury would be considered incomplete, ie, having sacral sparing. This patient would be scored as an AIS B because of sacral sparing, but not motor incomplete because there was no volitional motor movement 3 levels below the motor level. Red computer fields indicate incorrect scoring that the computer has identified.

Lastly, the computer program was able to correctly score all difficult case examples presented in the 2003 manual. Figure 3 is a case example taken directly from the 2003 manual. For demonstration of the program, the examination was incorrectly scored as right ML L5 and ML L2. Correct scores are indicated by the red flagged fields below the data entry fields (1).

Figure 3.

Database data entry page. Case example from the 2003 manual; red fields identify errors in scoring of the motor levels.

DISCUSSION

A computer program can be a data quality control mechanism for clinical registries or clinical trials using the ISCSCI motor, sensory, and anorectal examinations, particularly when data sets are voluminous and have been obtained by multicenter efforts. Past studies have shown that health professionals familiar with SCI are still unable to correctly classify patients using the ISCSCI (9,15–17). Even after training, the accuracy of scoring and classification is not 100% (9,15–17). There are several reasons why errors may persist. Although efforts are underway to deploy newly developed electronic learning modules on the methodology for examination and classification of the ISCSCI, there is no training program that evaluates one's competency. Also, the scoring and classification of the ISCSCI have evolved over the years, and changes associated with the evolution of the examination may have contributed to the poor scoring of examinations, even by experienced clinicians (1). In addition, lack of experience by clinicians has also been shown to introduce error, which may be compounded by insufficient detail in the manual (9,15–17). Perhaps most importantly, although the standards are meant to provide a systematic method for both the evaluation and classification of the neurologic consequence of SCI, there remains an important role for clinical interpretations of the patient's presentation.

Although a computer program is ideal for identifying errors, it is vulnerable to data entry errors and cannot replace the clinical reasoning by expert clinicians about findings that either do not or cannot fit into the logic of a classification system. It is for this reason the computer program was used as a data control mechanism to validate classification of examinations completed by hand; it was not used as the sole or final method for scoring and classification. This is particularly important in examinations where the ISCSCI examination findings may not reflect the SCI. Scoring of examinations requires clinical knowledge of the patient's presentation (ie, x-rays and magnetic resonance imaging) and a patient's medical history. For example, a brachial plexus injury from a shoulder restraint would not be considered in the scoring of the examination. For this reason, in our database, the computer program provides the technical ISCSCI scoring, but the final score entered for storage is determined by the clinician.

CONCLUSION

This study showed that the computer program we developed was effective in correctly scoring ISCSCI examinations and was able to detect errors in hand-scored examinations. Although this study included a significantly large sample of ISCSCI examinations, it is possible that not all unusual cases were represented in this study. Further testing of the computer program should be completed by other experts in the field of SCI with inclusion of exceptionally complex ISCSCI examinations. Also, as changes are made to the standards, any computer program must also undergo upgrades to ensure inclusion of associated with changes.