Validation of a novel claims-based stroke severity index in patients with intracerebral hemorrhage

Abstract

Background

Stroke severity is an important outcome predictor for intracerebral hemorrhage (ICH) but is typically unavailable in administrative claims data. We validated a claims-based stroke severity index (SSI) in patients with ICH in Taiwan.

Methods

Consecutive ICH patients from hospital-based stroke registries were linked with a nationwide claims database. Stroke severity, assessed using the National Institutes of Health Stroke Scale (NIHSS), and functional outcomes, assessed using the modified Rankin Scale (mRS), were obtained from the registries. The SSI was calculated based on billing codes in each patient's claims. We assessed two types of criterion-related validity (concurrent validity and predictive validity) by correlating the SSI with the NIHSS and the mRS. Logistic regression models with or without stroke severity as a continuous covariate were fitted to predict mortality at 3, 6, and 12 months.

Results

The concurrent validity of the SSI was established by its significant correlation with the admission NIHSS (r = 0.731; 95% confidence interval [CI], 0.705–0.755), and the predictive validity was verified by its significant correlations with the 3-month (r = 0.696; 95% CI, 0.665–0.724), 6-month (r = 0.685; 95% CI, 0.653–0.715) and 1-year (r = 0.664; 95% CI, 0.622–0.702) mRS. Mortality models with NIHSS had the highest area under the receiver operating characteristic curve, followed by models with SSI and models without any marker of stroke severity.

Conclusions

The SSI appears to be a valid proxy for the NIHSS and an effective adjustment for stroke severity in studies of ICH outcome with administrative claims data.

1. Introduction

Intracerebral hemorrhage (ICH) remains a major challenge for public health. According to a large meta-analysis from 36 studies conducted worldwide between 1980 and 2008, the overall incidence of ICH is 24.6 per 100,000 person-years, with a median 30-day mortality of 40.4%, and only 12%–39% of patients are able to remain independent after ICH.¹ Despite the decrease in annual incidence of ICH from 2000 to 2010, the short- and long-term mortality following ICH did not change.² Stroke severity, assessed using a standardized stroke scale like the National Institutes of Health Stroke Scale (NIHSS), is an important outcome predictor for ICH.^3,4 Every one-point increase in the NIHSS leads to a 10% higher risk of poor 1-year functional outcome.⁴

Administrative claims data, which are derived from claims for routine clinical practices, often include demographic information, diagnoses, and claims for procedures, laboratory tests, and medications. They contain much clinical information and may be an efficient and affordable resource for ICH research. However, before conducting clinical studies using these data, it is essential to determine whether the available claims data could sufficiently reflect the clinical manifestations of a health condition. For example, several studies have focused on developing algorithms to identify subtypes of stroke using administrative claims data.⁵ For the identification of presence of ICH, the algorithms in previous studies generally showed high positive predictive values, ranging from 79% to 97%.⁵ Researchers are increasingly using administrative claims data to investigate risk factors,^6,7 epidemiologic trends,^8,9 and outcomes of ICH.^10–12 Nevertheless, administrative claims data usually do not contain information on stroke severity.^13,14 Therefore, stroke outcome studies based on administrative claims data alone may have inadequate adjustment for stroke severity.¹⁵

Recently, we developed a claims-based stroke severity index (SSI), which has been validated as an effective proxy for stroke severity in patients with acute ischemic stroke (AIS).¹⁶ The SSI is significantly correlated with patient admission NIHSS scores. However, the validity of the SSI for ICH patients has not been determined. In this study, we validated the SSI in patients with ICH from three hospitals in Taiwan.

2. Methods

Overall, we linked stroke registry data with the National Health Insurance Research Database (NHIRD) using non-unique characteristics of a patient with ICH, as described previously.^16–18 We then computed SSI based on claims data of the NHIRD and evaluated the criterion-related validity of the SSI. Finally, we examined the change in model performance resulting from adding the SSI to models for predicting mortality after ICH.

2.1. Stroke registry data

We identified adult patients with an ICH diagnosis in stroke registries from the Chi Mei Medical Center, the Landseed Hospital, and the Ditmanson Medical Foundation Chiayi Christian Hospital. Patients with in-hospital stroke were excluded. The study hospitals registered all stroke patients admitted within 10 days of symptom onset following a protocol that has been reported elsewhere.¹⁹ ICH was defined as “a non-traumatic abrupt onset of symptoms with relevant focal neurological deficit, with or without headache or altered level of consciousness, with a focal collection of blood within the brain parenchyma on computed tomography or magnetic resonance imaging that was not a hemorrhagic conversion of a cerebral infarction”.¹⁹ Stroke severity was determined using the NIHSS. The functional status of patients (who had provided prior written informed consent for follow-up) was evaluated using the mRS at 3 months, 6 months, and 1 year after ICH by in-person assessment or by telephone interview. To ensure patient anonymity, we retrieved only the gender, birth date, admission date, discharge date, NIHSS score at admission, and mRS scores at follow-up from the registry databases. The study protocol was approved independently by the Institutional Review Boards of the three participating hospitals.

2.2. National Health Insurance Research Database linkage

The National Health Insurance (NHI) program is a compulsory, single-payer healthcare program that covers virtually the entire population of Taiwan. The program provides universal coverage for prescription medications, inpatient care, and ambulatory care. The National Health Insurance Research Database (NHIRD), which consists of all administrative claims data on NHI enrollees, is maintained and released for research by the National Health Research Institutes of Taiwan. Traceable personal identifiers in the NHIRD were scrambled to protect patient privacy and confidentiality.²⁰

To identify the inpatient population of ICH patients in the NHIRD, we extracted the data of all patients hospitalized between 2006 and 2010 for ICH (International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] diagnosis code 431) as their principal or secondary discharge diagnosis from a stroke-specific NHIRD data set. Four non-unique patient characteristics (gender, birth date, admission date, and discharge date) were used to link registry data with the NHIRD.^16–18 If more than one patient in either the stroke registries or the NHIRD shared the same values in all four matching characteristics, they were excluded. Errors in coding or entering data in administrative claims data might also be present, which could cause the failure of linkage.²¹ Successfully linked cases comprised the study cohort, and the linked hospitalization record in the NHIRD was defined as the index hospitalization.

2.3. Measurements

For each case in the study cohort, we extracted all the diagnosis codes from the index hospitalization, as well as inpatient and outpatient claims within the 1-year look-back period before the index hospitalization.^10,22 Patients were identified as having a comorbidity if its corresponding ICD-9-CM codes (eTable 1) appeared in at least one inpatient claim or three outpatient claims during the 1-year look-back period.^10,22 Severity of comorbidities was then summarized using a modified version of the Charlson comorbidity index (CCI),²³ which excluded cerebrovascular disease and hemiplegia. The modified CCI has been validated as a measure of comorbidity for predicting mortality and functional outcome in ICH patients²⁴ and was dichotomized into low comorbidity (<2) or high comorbidity (≥2) for analyses.²⁴

Previously, the SSI was developed to predict patient NIHSS score at admission and was validated for AIS patients by linking registry data with the NHIRD.¹⁶ The SSI contains seven predictors: airway suctioning, bacterial sensitivity test, general ward stay, ICU stay, nasogastric intubation, osmotherapy, and urinary catheterization (Table 1). Among all the AIS patients, the SSI was highly correlated with the admission NIHSS score.¹⁶ The presence or absence of each predictor was determined using detailed billing codes (eTable 2) in claims data and was subsequently entered in a multiple linear regression equation (Table 1) to calculate the SSI for each patient.

Table 1. Multiple linear regression model for the stroke severity index.

Predictor	Coefficient
Airway suctioning	3.5083
Bacterial sensitivity test	1.3642
General ward stay	−5.5761
ICU stay	4.1770
Nasogastric intubation	4.5809
Osmotherapy (mannitol or glycerol)	2.1448
Urinary catheterization	1.6569
Constant	9.6804

ICU, intensive care unit.

The outcomes of interest, including patient NIHSS score at admission; functional outcomes assessed using the mRS at 3 months, 6 months, and 1 year after the index ICH; and mortality status were obtained from the registry databases.

2.4. Statistical analysis

Statistical analyses were performed using Stata 13.1 (StataCorp, College Station, TX, USA). We summarized continuous variables using means and standard deviations and categorical variables using counts and percentages. A two-tailed P value of <0.05 was considered statistically significant. Criterion-related validity is the extent to which the score from a new measure correlates with that of other measures evaluating the same or a very similar construct.²⁵ In order to establish the SSI as a valid proxy measure for stroke severity, we evaluated two types of criterion-related validity: concurrent validity (the extent to which the SSI correlates with the NIHSS) and predictive validity (the extent to which the SSI predicts future functional outcomes). Concurrent validity was tested using the Pearson correlation coefficient between the SSI and the admission NIHSS.²⁶ Predictive validity was assessed using the correlations between the SSI and the mRS at 3 months, 6 months, and 1 year.²⁶

To investigate whether including the measure of stroke severity into mortality models enhances the predictive ability of the models, we first fitted base logistic regression models to predict 3-month, 6-month, and 1-year mortality using age, sex, modified CCI, and vascular risk factors (i.e., hypertension, hyperlipidemia, prior stroke, and atrial fibrillation) as covariates.²⁴ Comorbidities already included in the modified CCI, such as diabetes mellitus, coronary heart disease, and chronic kidney disease, were excluded to avoid over-adjustment. Other major covariates, including body mass index, smoking status, and drinking habits, were not considered because they are not recorded in the NHIRD. We then added the NIHSS or the SSI as a continuous variable to the base models. Model fit was examined using the Hosmer-Lemeshow statistic, and model discrimination was assessed and compared using the area under the receiver operating characteristic curve (AUC). The explained variance (R²) of the model was estimated using the Nagelkerke R² statistic.²⁷ We further quantified the improvement in predictive ability after adding the stroke severity measure to the base model using integrated discrimination improvement (IDI) and category-free net reclassification improvement (NRI).^28,29 The IDI measures performance of predictive models based on integrated sensitivity and specificity and offers incremental information over the conventional AUC.²⁸ The category-free NRI, which does not depend on pre-defined risk categories, measures reclassification of event and non-event subjects to a higher or lower predicted probability and is considered an objective measure of improvement in risk prediction.²⁹ Higher IDI and NRI indicate better discrimination and improved ability to classify a health condition.

3. Results

Overall, 1504 adult patients hospitalized for ICH (1059 from Chi Mei Medical Center, 248 from Landseed Hospital, and 197 from Ditmanson Medical Foundation Chiayi Christian Hospital) were identified from the stroke registries. After linkage with the NHIRD, 1360 patients (955, 239, and 166 patients from the three hospitals, respectively) were available for analysis. No patient shared the same values in all four matching characteristics. Overall, the rate of successful linkage was 90.4%. Table 2 gives the overall patient characteristics. Patient stroke severity and the distribution of SSI components differed across the three hospitals (eTable 3). Of the 1360 patients ultimately included in the cohort, follow-up data were available from 1171 patients (86.1%) at 3 months, 1140 patients (83.8%) at 6 months, and 758 patients (55.7%) at 1 year after the index hospitalization. The mortality rates were 16.9% at 3 months, 19.2% at 6 months, and 32.1% at 1 year. The SSI correlated highly with the admission NIHSS (Pearson correlation coefficient, r = 0.731; 95% confidence interval [CI], 0.705–0.755). In addition, the SSI correlated closely with the 3-month mRS (r = 0.696; 95% CI, 0.665–0.724), 6-month mRS (r = 0.685; 95% CI, 0.653–0.715), and 1-year mRS (r = 0.664; 95% CI 0.622–0.702), even though the correlation decreased as the time from stroke onset increased. eFig. 1 further demonstrates these correlations for each study hospital. Fig. 1 displays the box-plots showing distribution of the SSI and the NIHSS scores across all mRS grades at 3 months, 6 months, and 1 year after ICH. The relationship between SSI and mRS was not exactly linear for mRS ≤1 or ≥5.

Table 2. Study patient characteristics.

Characteristic	n = 1360
Mean (SD) age, years	61.0 (14.6)
Female sex	487 (35.8)
Modified CCI
0 or 1	1053 (77.4)
≥2	307 (22.6)
Hypertension	1014 (74.6)
Diabetes mellitus	337 (24.8)
Hyperlipidemia	208 (15.3)
Prior stroke	231 (17.0)
Atrial fibrillation	19 (1.4)
Coronary heart disease	42 (3.1)
Chronic kidney disease	87 (6.4)
Mean (SD) NIHSS score	17.0 (12.9)
Mean (SD) SSI score	14.9 (6.8)

CCI, Charlson comorbidity index; NIHSS, National Institutes of Health Stroke Scale.

SD, standard deviation; SSI, stroke severity index.

Data are presented as numbers (percentage) unless otherwise specified.

Fig. 1. Performance of mortality models for intracerebral hemorrhage. NIHSS, National Institutes of Health Stroke Scale; SSI, stroke severity index.

eTables 4, 5 and 6 report the logistic regression models for 3-month, 6-month, and 1-year mortality, respectively. In the base model for 3-month mortality, hypertension and hyperlipidemia predicted a lower risk, while a high CCI score (≥2) predicted a higher risk of 3-month mortality. When the NIHSS or the SSI was added to the base model, hypertension remained a significant and independent predictor of 3-month mortality, whereas hyperlipidemia was no longer associated with mortality. The effect estimate of the CCI was equal or close to the null. In the base model for 6-month mortality, age, CCI score, hypertension, and hyperlipidemia were significantly associated with mortality. The significance of the CCI score and hyperlipidemia disappeared after the NIHSS or the SSI was included in the model. In the base model for 1-year mortality, age, CCI score, hypertension, and hyperlipidemia were significant factors affecting mortality. After including the NIHSS or the SSI to the base model, the association of hyperlipidemia with mortality became null, and the effect estimate of the CCI was equal or close to the null. Stroke severity, measured by either the NIHSS or the SSI, consistently predicted mortality at all three time points.

Fig. 2 and Table 3 show the performance of these mortality models. For the base models, which did not contain any measure of stroke severity, the AUCs for predicting 3-month, 6-month, and 1-year mortality were 0.670, 0.665, and 0.685, respectively. Adding the NIHSS to the base models not only significantly increased model discrimination (assessed using the AUC, category-free NRI, and the IDI) but also improved explained variance of the model. The AUCs for 3-month, 6-month, and 1-year mortality models were 0.899, 0.888, and 0.880, respectively (all P < 0.001 compared with the base models). Similarly, model discrimination and explained variance were enhanced by including the SSI in the base models. The AUCs for 3-month, 6-month, and 1-year mortality models were 0.831, 0.821, and 0.829, respectively (all P < 0.001 compared with the base models). Although both the NIHSS and the SSI improved model performance, the AUCs of the models with the NIHSS were significantly higher than those of the models with the SSI (all P < 0.001 for mortality outcome at all three time points). Similar results were also obtained for the comparisons of the IDI, NRI, and explained variance between the models that included the NIHSS and the ones that included the SSI.

Fig. 2. Box-plots showing the distribution of the SSI and the NIHSS across all mRS grades at 3 months (A), 6 months (B), and 1 year (C) after intracerebral hemorrhage. mRS, modified Rankin Scale; NIHSS, National Institutes of Health Stroke Scale; SSI, stroke severity index.

Table 3. Performance of mortality models for intracerebral hemorrhage.

	AUC (95% CI)	Hosmer-Lemeshow statistic	P	Nagelkerke R2	IDI (95% CI)	Category-free NRI (95% CI)
Mortality at 3 months
Base model	0.670 (0.628–0.712)	14.13	0.078	0.093	Reference	Reference
Model with NIHSS	0.899 (0.878–0.919)	8.42	0.394	0.485	0.30 (0.27–0.33)	1.27 (1.12–1.42)
Model with SSI	0.831 (0.804–0.858)	8.45	0.391	0.325	0.15 (0.13–0.17)	0.92 (0.77–1.08)
Mortality at 6 months
Base model	0.665 (0.624–0.706)	4.05	0.853	0.092	Reference	Reference
Model with NIHSS	0.888 (0.866–0.910)	11.80	0.160	0.473	0.30 (0.27–0.33)	1.21 (1.06–1.35)
Model with SSI	0.821 (0.793–0.849)	9.07	0.337	0.317	0.15 (0.13–0.17)	0.92 (0.77–1.07)
Mortality at 1 year
Base model	0.685 (0.645–0.726)	5.75	0.675	0.137	Reference	Reference
Model with NIHSS	0.880 (0.855–0.905)	4.56	0.804	0.512	0.31 (0.28–0.35)	1.18 (1.02–1.33)
Model with SSI	0.829 (0.799–0.859)	10.49	0.232	0.388	0.19 (0.16–0.22)	0.96 (0.81–1.11)

AUC, area under the receiver operating characteristic curve; CI, confidence interval.

IDI, integrated discrimination improvement; NIHSS, National Institutes of Health Stroke Scale; NRI, net reclassification improvement; SSI, stroke severity index.

4. Discussion

We confirmed the concurrent validity and predictive validity of the SSI through its significant and positive correlation with the NIHSS at admission and with the mRS at 3-month, 6-month, and 1-year follow-up. Adding the SSI to mortality models based on administrative data for ICH patients significantly improved model discrimination and explained variance of the model, although to a smaller degree than when the NIHSS was added instead. Furthermore, models that included the NIHSS or the SSI to adjust for stroke severity resulted in similar effect estimates for most of the covariates, whereas models without adjustment for stroke severity produced different effect estimates for hyperlipidemia (eTables 4, 5, and 6).

Prior studies found that lower levels of low-density lipoprotein cholesterol predicted hematoma growth and death after ICH.^30,31 Consistent with these findings, our data showed that hyperlipidemia predicted lower mortality after ICH. However, the effect of hyperlipidemia on 3-month, 6-month, and 1-year mortality disappeared when either the NIHSS or the SSI was added in the mortality models. ICH patients with hyperlipidemia in our study likely had lower stroke severity than those without (the mean [SD] NIHSS score was 12.7 [0.8] vs. 17.8 [0.4] for patients with and without hyperlipidemia; t-test, P < 0.001). Therefore, the effect of hyperlipidemia on mortality risk diminished after adjustment for stroke severity.

Although hypertension increases the risk of ICH, a previous study reported that hypertension did not affect mortality rate after ICH.³² Notably, hypertension was demonstrated to be associated with greater survival among patients with ICH in our study, in contrast to the well-known association of hypertension with poor outcomes in the general population. Similar findings have been observed in patients with ischemic stroke and heart failure.^33,34 Whether or not these findings are due to the reverse epidemiology effect deserves further investigation.

Clinical stroke scales, such as the NIHSS, are important tools for monitoring stroke severity and performing clinical stroke research. They have been used for predicting stroke outcomes, including mortality, functional outcome, and hospital readmission in patients with ICH.^3,4,35 The SSI is not meant to replace the existing clinical stroke scales, but rather to provide a valid proxy to measure stroke severity and/or to adjust for risk of outcomes in future claims-based ICH studies, where the details of stroke severity and clinical scales are often unavailable.¹³ Although International Classification of Diseases Tenth Revision Clinical Modification codes for the NIHSS were approved in 2014 and will become active in the fall of 2016, it will still take several years for administrative data sets to include NIHSS data and to be used to adjust for stroke severity in reporting hospital mortality and readmission rates.³⁶ Therefore, further research has been advocated to ascertain stroke severity using administrative billing codes or electronic health records.¹⁴ The SSI was developed in response to this advocacy and might be useful in achieving this aim.

In addition to being a valid adjustment for stroke severity when performing stroke outcome studies or evaluating hospital performance using administrative claims data, the SSI might be applicable in other aspects of research that awaits further investigation. For example, in epidemiologic studies exploring ICH risk factors, the SSI can be used to compare stroke severity between patients with or without a particular risk factor. A previous study using the NHIRD has revealed that patients with liver cirrhosis were prone to developing ICH compared with those without liver cirrhosis.⁶ Through the use of the SSI, investigators now have the opportunity to explore the impact of liver cirrhosis on the severity of ICH.

Our study has certain limitations. First, because the SSI is based on the management and treatment provided for stroke patients, variations in practice patterns between hospitals and areas may influence the validity of the SSI. Although patient characteristics varied across the three study hospitals (eTable 3), the SSI exhibited high correlations with the NIHSS and the mRS in patients from each hospital (eFig. 1). The consistent performance of the SSI across hospitals of different types may imply a good generalizability to other hospitals in Taiwan. Second, unlike the NIHSS, which was generally assessed at a specific time point, the SSI was based on the whole care process during a stroke admission. Therefore, the SSI should be viewed as a global measure of neurologic deficit severity for the entire period of an admission. Third, the performance of the SSI is largely dependent on the accuracy of administrative claims data. Although health care providers are unlikely to under-report treatments during a stroke admission because of the fee-for-service reimbursement system of the NHI, we could not exclude the opposite; that is, the potential of over-reporting of treatments for reimbursement purposes.

5. Conclusions

We established the criterion-related validity of the claims-based SSI in patients with ICH. The SSI may be useful as a proxy for stroke severity and for controlling confounding in outcomes research based on the NHIRD or similar claims databases. More validation studies are needed to test the generalizability of the index to other administrative databases and other healthcare systems.

Conflicts of interest

None declared.

Acknowledgments

This study is based in part on data from the National Health Insurance Research Database provided by the Bureau of National Health Insurance, Department of Health and managed by National Health Research Institutes. The interpretation and conclusions contained herein do not represent those of the National Health Insurance Administration, Department of Health or National Health Research Institutes.

This research was supported in part by the Tainan Sin Lau Hospital (grant number SLH-104004) and the Ministry of Science and Technology (grant number MOST 104-2314-B-705-001).

Appendix A. Supplementary data

Supplementary data related to this article can be found at http://dx.doi.org/10.1016/j.je.2016.08.003.