Validation of Algorithm to Identify Persons with Non-traumatic Spinal Cord Dysfunction in Canada Using Administrative Health Data

Abstract

Background: Administrative health data, such as the hospital Discharge Abstract Database (DAD), can potentially be used to identify patients with non-traumatic spinal cord dysfunction (NTSCD). Algorithms utilizing administrative health data for this purpose should be validated before clinical use. Objective: To validate an algorithm designed to identify patients with NTSCD through DAD. Method: DAD between 2006 and 2016 for Southern Alberta in Canada were obtained through Alberta Health Services. Cases of NTSCD were identified using the algorithm designed by the research team. These were then validated by chart review using electronic medical records where possible and paper records where electronic records were unavailable. Measures of diagnostic accuracy including sensitivity, specificity, and positive and negative predictive values and 95% confidence intervals (CI) were computed. Results: Two hundred and eighty cases were identified to have both the administrative codes for neurological impairments and NTSCD etiology. Twenty-eight cases were excluded from analysis as 5 had inadequate medical record information, 17 had traumatic spinal cord injury, and 6 were considered “other” non-spinal cord conditions. Measures of diagnostic accuracy that were computed were sensitivity 97% (95% CI, 94%–98%), specificity 60% (95% CI, 47%–73%), positive predictive value (PPV) 92% (95% CI, 88%–95%), and negative predictive value (NPV) 80% (95% CI, 65%–90%). The most prevalent etiologies were degenerative (36.9%), infection (19.0%), oncology malignant (15.1%), and vascular (10.3%). Conclusion: Our algorithm has high sensitivity and PPV and satisfactory specificity and NPV for the identification of persons with NTSCD using DAD, though the limitations for using this method should be recognized.

Non-traumatic spinal cord dysfunction (NTSCD) is a collection of heterogeneous conditions that affect the spinal cord, causing myelopathy. Due to its heterogeneity and relatively low incidence,¹ identification of persons with NTSCD has been difficult. The use of administrative data can mitigate these challenges and allow for the identification of persons with NTSCD at a population level. To date, there has been no validated algorithm designed to identify persons with NTSCD through the use of administrative data.

As discussed by Jaglal² and Guilcher,³ our research team has developed an algorithm using the International Statistical Classification of Diseases and Related Health Problems, 10th revision, Canada⁴ (ICD-10-CA) codes to identify persons with NTSCD, and the characteristics of the identified cases have been described. Briefly, the Canadian Institute for Health Information (CIHI) captures all publicly funded health care encounters, including the hospital Discharge Abstract Database (DAD), which contains all hospital discharge records in Canada (except Quebec). These include ICD-10 codes for patients 18 years or older with neurologic impairment of paraplegia, tetraplegia, and cauda equina syndrome, as well as NTSCD etiology diagnoses. The algorithm involves linking of the neurologic impairment and NTSCD etiology codes to identify persons with NTSCD. To test the accuracy of our algorithm, a validation study was planned to compare the identified cases with the reference standard by chart review. The study was based in Calgary, which is the medical hub for Southern Alberta, with a catchment population of approximately 2 million people. Calgary has the largest level I trauma center in Canada and has a specialized spinal cord injury (SCI) acute and rehabilitation program that serves Southern Alberta. Alberta has a single health authority, which enables our team to access provincial population-based data. Our primary objective was to validate the administrative data algorithm for accurate identification of patients with NTSCD using clinical diagnoses documented in the medical records as the reference standard.

Methods

Our research team performed a retrospective validation study, using clinical chart review as the reference standard. Administrative data from Alberta Health Services, consisting of ICD-10-CA codes from the DAD for the entire province, were obtained for discharges between January 1, 2006 and June 30, 2016. These data were used to generate a list of patients with NTSCD from Calgary adult acute care sites (including the SCI rehabilitation program, which is based at one of the acute care sites) for validation. Ethics approval was obtained from the University of Calgary's Conjoint Research Ethics Board.

The algorithm that was previously described¹ to identify NTSCD was applied to the DAD. The requested DAD contained all hospital discharge records for the Calgary-based sites. It includes diagnosis information using the ICD-10-CA codes. Up to 25 primary clinical diagnoses, comorbid conditions, and medical complications are recorded on a single abstract, which represents one patient discharge record. Each diagnosis code has a corresponding diagnosis type, which can indicate the reasons for admission, comorbidities present at admission, and complications that arise during the hospital stay.⁴ Patients aged 17 or over on admission were included. All Calgary adult acute care sites were included. The ICD-10-CA codes used in this algorithm will be described below.

Identification of patients with NTSCD from Calgary adult acute care sites

The algorithm identified a sample cohort of NTSCD cases by initially determining those patients with both neurological impairment and etiology codes. Next, patients from this group who also had a traumatic SCI diagnosis were excluded. The resulting cases were identified for inclusion in the final NTSCD sample and were forwarded for chart review.

The algorithm involved the following steps (Figure 1):

• Step 1: Diagnosis field positions 1 through 25 were reviewed for ICD-10-CA codes beginning with G82 (neurological impairment diagnosis of paraplegia or tetraplegia) or an ICD-10-CA code of G83.4 (cauda equina syndrome) (Appendix, Table A1a). All records with the above codes were identified and retained, resulting in 4,127 patients for the province.

• Step 2: Diagnosis field positions 1 through 25 were reviewed for a NTSCD etiology ICD-10-CA code (Appendix, Table A1b). All records that had a NTSCD etiology diagnosis as a significant diagnosis type, per CIHI definition,⁴ were identified. This resulted in 17,792 patients for the province.

• Step 3: The two populations in Steps 1 and 2 were combined, resulting in a cohort of patients who had a neurological impairment (any diagnosis type), a NTSCD etiology code (significant diagnosis types), or both conditions recorded on a provincial inpatient abstract. This resulted in 21,091 patients. Eight hundred and twenty-eight patients in this cohort were identified as having both conditions; 20,263 had either G-codes or NTSCD etiology codes only and were removed as the study criteria was not met.

• Step 4: All provincial inpatient abstracts that had a traumatic spinal cord injury (TSCI) diagnosis (any diagnosis type) were identified. Diagnosis field positions 1 through 25 were reviewed for a TSCI ICD-10-CA code (Appendix, Table A1c). This resulted in 1,670 patients for the province who had a TSCI recorded on an inpatient abstract. Ninety-seven of these were also found to be part of the cohort of the 828 identified patients in Step 3. Due to the difficulty in determining whether the neurological impairment was attributable to the NTSCD diagnosis or the TSCI diagnosis, these 97 TSCI patients were removed from the provincial cohort of 828, resulting in a final provincial list of 731 patients. Forty-four of the 97 excluded TSCI patients were from the Calgary adult acute care sites. A chart review was performed on these 44 cases to confirm (or deny) the diagnosis of TSCI (based on evidence of their mechanism of injury).

• Step 5 (final cohort): All patients with both a diagnosis of neurological impairment (any diagnosis type) and a diagnosis of NTSCD (significant diagnosis types) on any provincial inpatient discharge abstract were identified (Step 4). The final cohort was derived from Step 4; this was limited to those with an inpatient discharge from one of the Calgary adult acute care sites. This final cohort was comprised of 280 patients for the Calgary adult acute care sites: Foothills Medical Centre (where the spinal cord injury acute and rehabilitation programs are located), Peter Lougheed Centre, Rockyview General Hospital, and South Health Campus.

Figure 1.

Non-traumatic spinal cord dysfunction (NTSCD) algorithm using International Statistical Classification of Diseases and Related Health Problems, 10th revision, Canada (ICD-10-CA) codes.

Reference standard

The final cohort described above was provided to the research team for chart review (reference standard) to validate that each patient had a diagnosis of NTSCD. Each of the 280 NTSCD cases resulting from the algorithm included the following information: medical record number, G-code/NTSCD code, date of admission, date of discharge (equivalent to date of death if patient expired in hospital), institution name, and discharge unit number (corresponding with each patient's episode of care). Records were requested from the local hospital Health Information Management (HIM) department using the medical record number. Of note, the Calgary region began its transition to an electronic medical record (EMR) in 2008. As a result, most information was abstracted from the local EMR (Sunrise Clinical Manager [SCM]). Records older than 2008 and certain records missing information in the EMR required a request from the local hospital HIM department. Regardless of the type of record used, the methods used to seek out the necessary information remained consistent. The chart review was completed by 2 clinicians, each with more than 10 years of experience in spinal cord injury and dysfunction. Consultation, assessment, treatment, or summary reports written by surgeons, neurologists, physiatrists, nurses, and rehabilitation therapists, as well as diagnostic imaging reports, were examined for evidence verifying the presence of a TSCI or NTSCD.

Validation of the algorithm was determined by whether there was sufficient clinical evidence in the chart to indicate motor and/or sensory impairment or bowel and/or bladder involvement, which are typically associated with NTSCD. Table 1 summarizes the evidence used to validate the diagnosis.

Table 1.

Chart review criteria to confirm non-traumatic spinal cord dysfunction (NTSCD)

Based on the information available in the chart, we assigned patients into one of the following etiology categories, which are consistent with the NTSCD categories recommended by the National Institute of Neurological Disorders and Stroke Common Data Elements⁵ for SCI and the International SCI non-traumatic SCI datasets⁶: congenital/genetic, degenerative, infection, inflammatory, oncology benign, oncology malignant, vascular, and unknown/unspecified/others.

Statistical methods

Using methods for computing measures of diagnostic accuracy,⁷ sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV),⁸ as well as 95% confidence intervals (95% CI) for each of these measures were computed. Sensitivity was defined as the proportion of persons with NTSCD who were identified by the algorithm with an NTSCD. Specificity was the proportion of persons without NTSCD who were excluded from the NTSCD algorithm. PPV was the probability that a patient identified in the algorithm with an NTSCD actually had an NTSCD, whereas the NPV was the probability that a patient who does not have NTSCD based on the algorithm actually did not have an NTSCD.⁹

Results

The algorithm initially identified 324 patients whose charts were reviewed. Forty-four patients had a TSCI code and were excluded, leaving 280 NTSCD cases. Of these 280 cases, 5 of the patients' charts did not contain adequate clinical information, leaving 275 patients identified as having an NTSCD by the algorithm. Within the 44 TSCI charts, 9 were found to actually be NTSCD, or “false negatives”; the remaining 35 TSCI cases were confirmed to be cases of TSCI, or “true negatives.” Within the NTSCD patient charts, 252 were confirmed to be NTSCD or “true positives,” as per the criteria agreed upon by team consensus. Of the 23 remaining “false positives,” 17 were actually TSCI, and 6 were classified as “other.” Sensitivity, specificity, PPV, and NPV were calculated to be 97% (95% CI, 94%–98%), 60% (95% CI, 47%–73%), 92% (95% CI, 88%–95%), and 80% (95% CI, 65%–90%), respectively (Figure 2).

Figure 2.

Calculation of measures of diagnostic accuracy. NPV = negative predictive value; PPV = positive predictive value.

Etiologies of NTSCD were identified from chart reviews (Table 2), the most prevalent of which were degenerative (n = 93; 36.9%), infection (n = 48; 19.0%), oncology malignant (n = 38; 15.1%), and vascular (n = 26; 10.3%).

Table 2.

Clinical diagnoses of non-traumatic spinal cord dysfunction (NTSCD) by chart review

Discussions

The results showed that our algorithm has high sensitivity and PPV and satisfactory specificity and reasonable NPV. When compared to similar algorithm validation studies using administrative databases for other medical conditions such as rheumatoid arthritis in Canada¹⁰ (sensitivity 97% [95% CI, 94%–100%]; specificity 85% [95% CI, 81%–89%]; PPV 76% [95% CI, 70%–82%]; NPV: 98% [95% CI, 96%–100%]) and in Italy¹¹ (sensitivity 96% [95% CI, 96%–96%]; specificity 90% [95% CI, 90%–90%]; PPV 85% [95% CI, 81%–89%]; NPV 98% [95% CI, 96%–99%]), our results have comparably high sensitivity and PPV, though our specificity is lower.

In a 2012 systematic review of the use of administrative data to identify patients with neurologic conditions (Alzheimer disease and dementia, brain tumors, epilepsy, motor neuron disease, multiple sclerosis, Parkinson disease/parkinsonism, SCI, traumatic brain injury),¹² 3 studies on TSCI met inclusion criteria and were reviewed. Nineteen case definitions were tested in these 3 studies on TSCI, with high variability of reported ranges for sensitivity, specificity, and PPV, depending on the case definitions used.

A later study¹³ from 2014 assessed the validity of different administrative data sources available for the identification of TSCI patients using ICD-10-CA codes from CIHI-DAD, Rehabilitation Coding Groups (RCG) from the National Rehabilitation System (NRS), and physician fee codes from the Ontario Healthcare Insurance Plan (OHIP). The study showed that RCG codes in the NRS had high sensitivity (92%) and specificity (97%) for the identification of TSCI patients; CIHI-DAD ICD-10-CA codes had high specificity (99%) and moderate sensitivity (76%); but OHIP fee codes had poor sensitivity (64%). Therefore, it appears that administrative databases, especially the DAD, can be used successfully to identify persons with TSCI with high sensitivity and specificity. However, we do acknowledge that the relatively low prevalence of our population can impact PPV and NPV calculations.¹⁴

To our knowledge, there has not been any published study on the validation of administrative databases to identify NTSCD in which measures of diagnostic accuracy were calculated (sensitivity, specificity, PPV, and NPV). One Australian study calculated the incidence of NTSCD through the use of ICD-10 codes in an administrative database.¹⁵ Another study¹⁶ systematically reviewed the use of billing, procedural, or diagnostic code algorithms to identify transverse myelitis in administrative databases. Only one algorithm was based on an administrative database with a reported PPV (62%) using 5 ICD-9 codes for transverse myelitis, which may be considered NTSCD. Therefore, ours is the first study to examine the use of an administrative database algorithm to identify NTSCD, with sensitivity, specificity, PPV, and NPV calculated.

Given that there are relatively few cases identified to have NTSCD in our catchment area by this algorithm, it is reasonable to accept the high sensitivity and lower specificity in order to capture as many cases as possible, while acknowledging that some of the cases identified as NTSCD may indeed not be NTSCD. Furthermore, the high sensitivity is especially important for clinical purposes. Currently, there is no standard method to identify persons with NTSCD, and yet, unlike persons with TSCI, this population is not typically connected to spinal cord–related specialty services, such as SCI acute and rehabilitation programs, but they may indeed benefit from them. Therefore, this algorithm with high sensitivity will allow many persons with a potential NTSCD to be identified, so that they may then be offered appropriate clinical services. Because the ICD codes are assigned on discharge, it would help to use administrative data and the algorithm to learn more about patient flow for NTSCD; this would ultimately inform care delivery. Furthermore, this algorithm may allow us to calculate the prevalence of NTSCD, which will be important for the appropriate resource allocation for the care of this population. This algorithm is relatively straightforward and may be applicable to other health care systems that utilize ICD-10 codes; therefore, there could be good external validity.

The findings for etiologies of NTSCD are especially interesting, as these were not completely consistent with previous findings from international rehabilitation units.¹⁷ Both studies identified degenerative causes as the most common etiology for NTSCD, but we found the second and third most common etiologies to be infection and oncology malignant, while the international study noted oncology malignant as the second most common etiology, with ischemia being the third most common etiology. These differences are likely due to our study's use of DAD data, which contain records for all patients who have been admitted to an acute care facility, whereas the international study only included those who were admitted to inpatient rehabilitation for their NTSCD. Therefore, it is possible that many persons with an infection etiology were so medically unstable and had a higher mortality rate in the acute care hospital that they were not admitted to a rehabilitation program, hence the difference in our results. This observation highlights the importance of understanding the type of administrative databases being used.

Limitations of our study

There are several limitations to our study. First, although the list of ICD-10 codes used is comprehensive and generates high sensitivity and PPV, it requires further international consensus. Moreover, this algorithm should be tested on a larger population and in other countries that use the ICD-10 codes for further validation.

Although sampling patients using this algorithm on health administrative databases has the benefit of allowing the calculation of all 4 measures of diagnostic accuracy, the absence of a random sampling strategy may introduce bias because of the disproportionate prevalence of NTSCD in the validation cohort compared to the general population; however, because of the low prevalence of all types of NTSCD, it was not feasible to utilize a random sampling strategy.

Due to the use of the DAD as our administrative database, this algorithm may not have identified all inpatients with both a neurological impairment and NTSCD. Unless determined to be a significant diagnosis type, or specified as mandatory in the CIHI guidelines, the neurological impairment and NTSCD ICD-10-CA codes are optional to capture. Per the CIHI inpatient diagnosis typing rules, inpatient conditions are considered to be “significant” if they meet the following criteria⁴:

1. Requires treatment beyond maintenance of the preexisting condition,

2. Increases the length of stay (LOS) by at least 24 hours, and/or

3. Significantly affects the treatment received. (Further guidelines on criteria for significance are available in the reference material.)

Patients may have been inadvertently excluded from this cohort if one or both of their inpatient diagnoses occurred outside the time frame indicated in this algorithm. Moreover, the inclusion of diagnostic codes in DAD records is dependent on the accuracy and completeness of documentation by physicians. As a result, it is known that identification of patients using administrative database such as DAD may lead to an undercalculation of prevalence.¹⁸

Current or past Alberta residents may have been excluded from this cohort if the neurological impairment and/or NTSCD conditions were diagnosed in other health care jurisdictions. Except for specific projects or research initiatives, health records diagnosis data are not usually shared between provinces or internationally. Most NTSCD patients are seen in the Calgary acute sites. However, we acknowledge that there may have been a small number of patients missed, who were only seen at a rural site. Fortunately, this study was intended to validate the algorithm rather than calculate prevalence.

One or both of the neurological impairment/NTSCD diagnoses may have been recorded in another jurisdiction's health care system, and therefore some of the people excluded in step 3 of our methodology may in fact have an NTSCD. Similarly, cases may have been inadvertently included in the cohort if they had a TSCI outside the time frame of this algorithm or if they were diagnosed with a TSCI in another jurisdiction. It is also possible that cases of TSCI may have been excluded if the original injury was traumatic in nature but occurred prior to the inclusion time frame of this study. Using a longer look-back period may help to confirm whether the original injury was traumatic or non-traumatic in nature and ultimately capture the population of interest more accurately. Finally, because some clinicians also use G codes in association with traumatic injuries and these injuries may in fact have been non-traumatic in nature (for instance, falling from standing height), a clear definition for what does and does not constitute a traumatic code is needed. Refinement of the algorithm with these issues taken into account would most likely lead to improvement of the specificity.

When considering the use of the DAD data for the identification of NTSCD, it must also be recognized that for certain etiologies of NTSCD, the patients may not be admitted to the hospital for diagnosis and management and yet the DAD only captures hospitalized patients. For instance, for patients with degenerative causes of NTSCD, the condition may be slowly progressive while the patients remain in the community and are monitored as an outpatient. Therefore, this population would not be captured by our algorithm. However, for other patients with more acute and severe causes of NTSCD, such as vascular or oncology causes, they likely would be hospitalized and thus captured by our algorithm. While this clinical assumption may be true, it is not clear what percentage of NTSCD cases may be missed due to this reason. Nevertheless, this population bias inherent in the use of the DAD for the identification of NTSCD must be recognized.

Future steps

The use of the DAD to identify patients with NTSCD is a retrospective method. Although this may be useful from an epidemiological standpoint, alternative methods that allow earlier identification of NTSCD may be preferred from a clinical standpoint so that timely and appropriate clinical care may be offered to this population. Future methods to identify persons with NTSCD should take this into consideration. In the meantime, further validation of this algorithm in other populations both in Canada and internationally where ICD-10 codes are used would be useful.

Conclusion

The algorithm that our group developed to identify persons with NTSCD has been shown to have high sensitivity and PPV. Despite its limitations, it has promise for future use for research and clinical purposes.

Appendix

Table A1.

List of ICD-10-CA codes for neurological impairment, non-traumatic spinal cord dysfunction (NTSCD) etiology, and traumatic spinal cord injury (TSCI)

Table A1.