Stroke Administrative Severity Index: using administrative data for 30-day poststroke outcomes prediction

Abstract

Aim:

Current stroke severity scales cannot be used for archival data. We develop and validate a measure of stroke severity at hospital discharge (Stroke Administrative Severity Index [SASI]) for use in billing data.

Methods:

We used the NIH Stroke Scale (NIHSS) as the theoretical framework and identified 285 relevant International Classification of Diseases, 9th Revision diagnosis and procedure codes, grouping them into 23 indicator variables using cluster analysis. A 60% sample of stroke patients in Medicare data were used for modeling risk of 30-day postdischarge mortality or discharge to hospice, with validation performed on the remaining 40% and on data with NIHSS scores.

Results:

Model fit was good (p > 0.05) and concordance was strong (C-statistic = 0.76–0.83). The SASI predicted NIHSS at discharge (C = 0.83).

Conclusion:

The SASI model and score provide important tools to control for stroke severity at time of hospital discharge. It can be used as a risk-adjustment variable in administrative data analyses to measure postdischarge outcomes.

To improve population health outcomes and healthcare costs for patients with stroke, we need to provide better prevention and the right treatment to the right people at the right time. To facilitate continuous quality improvement, accurate measurements for each stage of the care process are necessary. Currently, there is no established way to perform risk adjustment to billing data to help compare outcomes and costs for groups of patients with stroke in routine practice settings over time and across settings in order to identify optimal care. Patient groups selected from routine practice settings may have different mixes of risk factors that affect outcomes and cost, leading to selection bias in the comparison. Thus, when comparing heterogeneous patient groups, adjustment for baseline characteristics, including comorbidities, sociodemographic factors and disease severity, is essential. Clinical indicators of acute illness severity are often included in prediction risk adjustment models [1–4]. In stroke outcomes research, the need to adjust for acute stroke severity has focused on measuring stroke severity at the time of hospital admission. However, changes in public reporting requirements and reimbursement policy have emphasized the need for an easily applicable method for risk adjustment using widely available data and measures that are appropriate for each stage of the care process [5–7].

Initial stroke severity has been shown to be one of the most powerful predictors impacting the prognosis of stroke patients [1,8,9]. To measure stroke severity, various scales are available in clinical settings, such as the NIH Stroke Scale (NIHSS). Originally developed to measure stroke severity in ischemic stroke patients [10], the NIHSS has been widely used and recommended in hemorrhagic stroke patients as well [11–13]. The NIHSS has been shown to change over time during a hospital stay, its value at hospital discharge strongly predicts long-term survival and functional recovery for acute stroke patients [1,7,8,12,14,15]. Unfortunately, it is rarely recorded repeatedly outside of clinical trial settings and is not available in many large data bases that are used to examine poststroke outcomes.

Administrative billing and hospital or clinical registry databases are commonly used to evaluate health outcomes and cost incurred after hospital discharge. However, comparisons of outcomes for patient groups identified in archival data are prone to selection bias unless adequate risk adjustment is used [7]. Unfortunately, well validate stroke severity scales, such as the NIHSS, are not commonly available in archival data. Several risk adjustment models for assessing hospital- and patient-level outcomes in stroke patients (primarily ischemic stroke) have been developed and have been compared and contrasted in recent literature [7,16]. However, each of these models has focused on measurement of stroke severity at admission and use clinical information that is not universally available. To the best of our knowledge, the only study that attempted to develop a measure of stroke severity for administrative data was recently published by Sung et al., who developed a stroke severity index based on billing codes from Taiwanese administrative data [17]. However, this (not currently validated) index included electronic medical records extracts that contained clinical and pharmaceutical data, as well as billing data that are not widely or readily available or applicable in the USA. As Katzan et al. discussed, in recent work by Get With the Guidelines – Stroke [6], even though clinical stroke severity measures are the strongest predictors of 30-day outcomes, they are often missing from administrative or electronic health records data [7]. Furthermore, unless they are measured at time of hospital discharge, they may not adequately reflect the risk of poor outcomes or cost during the poststroke period.

Severity measures using US billing data are available for other disease conditions, such as sepsis [18] or comorbidities [3,19]. As an example, the Charlson Comorbidity Index predicts patients’ 1 year mortality risk based on their comorbid conditions (e.g., liver disease, diabetes, malignancy and HIV) [2]. The Charlson Comorbidity Index was originally developed to be used in clinical settings, but it was then translated for use in administrative data including the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) and ICD-10-CM billing codes [3,20,21].

Even though the USA has recently moved to using ICD-10 for billing, there are a few reasons why using ICD-9 codes for this study remain relevant. First, it has been shown to be relatively simple to cross-walk ICD-9 codes into ICD-10 formats. It is also true that most currently available data for research does not yet include ICD-10 data. When ICD-10 data do become available, it will be beneficial to have this index available for both ICD-9 and ICD-10 in order to facilitate time trend analysis. Last, while ICD-10 now includes codes for each element in the NIHSS, it is not yet known if any of these codes are being used by billing coders, particularly due to the high probability that most facilities do not systematically measure and record the NIHSS in practice and therefore in the electronic health record.

The lack of a stroke severity measure for administrative data has potentially far-reaching negative consequences. When a hospital's performance is based on outcome measures that are not adjusted for severity, results may be misleading and may not properly characterize the quality of in-hospital stroke care [7] or stroke severity at hospital discharge. As a consequence, the American Heart Association/American Stroke Association (AHA/ASA) urged the development of an appropriate risk adjustment measure for case mix in hospital-level stroke outcomes [5,7]. The purpose of our study was to develop and validate a measure of stroke severity at hospital discharge, the Stroke Administrative Severity Index (SASI), for use with administrative hospital billing data using ICD-9-CM diagnosis and procedure codes that can be used as an indicator of stroke severity at time of hospital discharge. We sought to develop SASI as a risk adjustment for stroke severity postdischarge by using patients’ discharge information from their acute hospital stay. A stroke severity measure at the time of discharge from the acute hospital stay offers the unique opportunity to improve the assessment and measurement of postacute stroke care, including rehabilitation, and therefore is of interest to clinicians, patients and policy makers. In addition, due to the billing cycle in the USA (generally a single hospital facility bill per patient is produced), an index that would allow for postacute risk-adjustment, utilizing all measures available in the acute care billing cycle provides the most useful tool for comparing effect and cost of postacute events, such as readmissions or rehabilitation approaches.

As suggested by the AHA/ASA [5], we used the NIHSS as the theoretical framework and 30-day postdischarge mortality risk as the predicted outcome in building the SASI. It was previously demonstrated that the NIHSS has strong prognostic information for 30-day mortality risk of acute ischemic stroke patients [14,15], and, therefore, we chose the NIHSS as the gold standard against which to measure the SASI developed in our study.

Our study had two objectives: construct and validate a stroke severity scale from acute hospital discharge information using a large sample of US hospital bills to predict 30-day postdischarge mortality in acute stroke patients or discharge to hospice and test its external validity by comparing its predictive power with the NIHSS using publically available clinical trial data that contains both billing information and clinical severity measures.

Methods

Study design

We conducted a retrospective cohort study of adults discharged from hospital with primary diagnosis of ischemic or hemorrhagic stroke. We obtained data from the 5% US national sample of the 2012 and 2013 Medicare Limited dataset (LDS). Medicare 2012 data were used for exploratory cluster analysis and combined 2012 and 2013 Medicare data were used for the score development and validation. This study was designated nonhuman research by the university institutional review board.

Cohort identification & predictor development

As a first step, investigators identified 285 initial ICD-9 diagnosis and procedure codes based on the contents of the 11 domains of the NIH Stroke Scale (Figure 1). The current AHA/ASA guidelines recommend diagnosis and treatments for acute ischemic and hemorrhagic stroke. To cover a wide range of stroke-related events/treatments, the guidelines were matched to appropriate ICD-9 diagnosis and procedure codes [13,22]. Codes were checked using ICD-9-CM coding books and were reviewed by an experienced independent hospital coder.

Figure 1. . Steps in selecting stroke severity clinical predictors.

The codes were then examined analytically to explore how they would group or cluster using the 5% national sample of 2012 Medicare LDS. The cohort sample included patients with either an ischemic (ICD-9 code 434.xx) or hemorrhagic (ICD-9 code 431.xx) stroke, which occurred between 1 January 2012 and 31 December 2012. These stroke diagnostic codes had previously been shown to be highly accurate and were used to maximize specificity of the stroke diagnosis [23,24]. Due to the fact that ICD-9 code 436.xx was found to be less specific in past studies of accuracy, and because it was re-indexed to 434.xx in 2004, we did not include this code in cohort identification. Any ICD-9 diagnosis and procedure code with zero observations was excluded. The remaining ICD-9 diagnosis and procedure codes were grouped into distinctive clusters using a cluster analysis. Using K-means clustering, the distance among clusters was calculated with a Euclidean approach [25,26]. Codes were combined into a smaller number of summarized indicators when they were closely located near the centroids of each cluster. Starting with a large number of clusters, we reiterated the proposed approaches until an optimal and parsimonious number of clusters were identified where the clusters were clinically meaningful. At this point in the study process, a clinical stroke specialist (EC Jauch) with significant clinical and research experience independently reviewed the clinical meaningfulness of individual codes and their relevance within the clustered groups. Modifications were then made to add, exclude or regroup specific codes. As an example, the ICD-9 code 02.34, ‘ventricular shunt to abdominal cavity and organs’ was a very rare code, and when examined in detail, it did not appear to reflect a procedure that was relevant to the acute stroke admission, and it was therefore excluded from final indicator definitions.

After cluster analysis, 23 indicator variables contained relevant groups of ICD-9 codes. The groupings are shown in the Supplementary Table 1, where the variables equal ‘1’ if any of the codes in that cluster existed in the patient bill. Next, the association between the indicators and stroke severity was examined using logistic regression. We used 30-day mortality posthospital discharge or discharge to hospice as a proxy measure for stroke severity at discharge and excluded all patients who died in the hospital from the prediction modeling. We elected to include discharge to hospice with the 30-day postdischarge mortality indicator, because patients receiving hospice care have an end-of-life prognosis which falls on the most extreme end of severity (e.g., Medicare requires patients have a prognosis of ≤6 months to be eligible for hospice care) and an examination of hospital practice patterns for discharge to hospice showed substantial variation by hospital size and geographic location.

Each of the 23 grouped indicator variables were initially modeled as predictors of 30-day mortality/hospice using univariable logistic regression. The inclusion criterion for the prediction variables in univariable logistic regression model was a p-value ≤0.20 [27]. All 23 grouped indicators remained feasible for inclusion in the full multivariable prediction model development step. For the development of the prediction model, a random sample of 60% of ischemic or hemorrhagic stroke patients from the 2012 to 2013 5% Medicare LDS file linked with patient death dates was used. The remaining sample of 40% of patients was used to validate the parsimonious prediction model.

Outcome measure

Our outcome was mortality within 30 days after discharge or discharge to hospice. Patients who died during acute hospitalization were excluded.

Model development & internal validation

Using the 60% development sample (n = 12,738), a multivariable logistic regression model was built predicting mortality at 30 days from hospital discharge or discharge to hospice as a dichotomous variable. The independent variables in the regression model were the stroke severity predictors identified in the previous step (Supplementary Table 1). Since the sample was large, significant predictors were chosen by including all predictors in the model and removing them, manually, one at a time based on a set of predictive model building criteria referred to as ‘purposeful selection’ [28]. Predictors with significance levels less than 0.25 were examined closely prior to their potential removal using acceptance guidelines of removal based on the following statistical criterion: no more than 20% change in other parameter estimates after removal; smaller Akaike information criterion/Bayesian information criterion indicating a better model fit without the covariate; and likelihood ratio test. When a predictor's regression coefficient was a negative value, the impact on the intercept was examined by including and removing the predictor from the model while noting any change in the parameter estimate of the intercept. This was done because the goal of the model was to find predictors of increasing severity, not proxies for better care quality such as tissue plasminogen activator administration. If these negative predictors were removed and the parameter estimate of the intercept in the model was not affected (<10% change) by the inclusion of the covariate, then the negative predictor was permanently removed from the model. This iterative process was continued until an optimal parsimonious predictive model was reached. Through this process of model fitting, we reduced the number of indicators from 23 to 7, which contained a total of 23 ICD-9-CM codes (Supplementary Table 2). Final model fit was assessed using the Hosmer–Lemeshow goodness-of-fit (GoF) test and discrimination was assessed using the area under the receiver operating characteristic curve, ‘C-statistic’.

SASI score development

To generate a stroke severity index for use with administrative data, we adapted the technique of Osborn et al. [27] and assigned integer point values to each predictor variable from the final parsimonious multivariable logistic regression model. The score value for each variable in the model was calculated as the value of the parameter estimate (β-coefficient) multiplied by 10 and rounded to the nearest integer value. A summary score was created for each observation in the 100% sample dataset and used as the predictor of 30-day mortality postdischarge or discharge to hospice in a logistic regression. To assess the predictive ability of the summary score in comparison to the original model, containing each individual stroke severity predictor, a logistic regression was used including the summary score and standard risk covariates age, race and Charlson Comorbidity Score. If possible, the effects of the covariates were examined for both continuous and categorical measures. Model performance was evaluated for GoF, C-statistic and graphically by deciles of predicted 30-day mortality/hospice discharge using the same approach as the original model development (Table 1 & Figure 2).

Table 1. . Characteristics of 100% sample of stroke patient cohort.

Variable	Mean (SD) Range	n (%)
Patient characteristics
Age	76.54 (11.06) 65–98
Female sex		11,756 (55.37)
Race
– White		17,071 (80.41)
– Black		2911 (13.71)
– Hispanic		412 (1.94)
– Other		836 (3.94)
Charlson comorbidity score (max possible 24)	3.86 (2.44) 1–19
Acute hospital length of stay	4.90 (4.70) 0–67
SASI Score (max possible 56)	3.54 (4.94) 0–47
Diagnoses
Aphasia		4343 (20.46)
Coma		170 (0.80)
Dysarthria and/or dysphagia		5147 (24.24)
Hemiplegia or monoplegia		5507 (25.94)
Neglect		285 (1.34)
Procedures
Nutritional infusion		79 (0.37)
Tracheostomy and/or ventilation		867 (4.08)
Outcome
Died in 30 days or discharged to hospice		2533 (11.93)

n = 21,230.

SASI: Stroke Administrative Severity Index; SD: Standard deviation.

Figure 2. . Observed versus predicted 30-day mortality or discharge to hospice ratios by risk deciles.

SASI Stroke Administrative Severity Index.

All analyses were performed using SAS statistical software version 9.4 (NC, USA) [29]. The SAS programming code for our specifications of variables and model is provided in Supplementary Material 1. This programming code works only for ICD-9-CM diagnosis and procedure codes and is not correct for any other type of codes, such as ICD-10-CM, CPT or HCPCS. However, future versions of this score may be cross-walked and validated for use with ICD-10-CM codes.

Results

A total of 1,191,064 acute care hospital discharge records from 2012 and 2013 Medicare 5% LDS sample data were examined. We extracted data for 23,459 admissions where subjects were identified using primary diagnosis codes for stroke (ICD-9-CM 431.xx and 434.xx) between 1 January 2012 and 31 December 2013. After excluding stroke patients who died in hospital, our final cohort included 21,230 individuals who were discharged after acute stroke. Descriptive statistics for patient characteristics are provided in Table 1.

Table 2 provides the composite list of seven variables included in our final prediction model. The proxy measures we generated to reflect stroke severity at hospital discharge were all significantly associated with death within 30 days of discharge or discharge to hospice, with highest predictive value of 10.021, indicating higher odds for patients with coma, to the lowest predictive odds of 1.263 in patients with dysarthria and/or dysphagia. Also provided in Table 2 are the point values for each predictor variable in the SASI score, which can be used to generate a single stroke severity risk adjustment variable when sample size limits covariate adjustment or in settings where the use of SAS or other statistical modeling software is not feasible.

Table 2. . Mortality/discharge to hospice prediction model using parsimonious multiple logistic regression of 100% sample.

Variable	Coefficient	SE	OR	95% CI	p-values	Score point values†
Aphasia	0.4345	0.0494	1.544	1.402–1.701	<0.0001	4
Coma	2.3047	0.1719	10.021	7.154–14.037	<0.0001	23
Dysarthria and/or dysphagia	0.2331	0.0488	1.263	1.147–1.389	<0.0001	2
Hemiplegia or monoplegia	0.5667	0.0464	1.762	1.609–1.930	<0.0001	6
Neglect	0.4762	0.1511	1.610	1.197–2.165	0.0016	5
Nutritional infusion	0.6259	0.2627	1.870	1.117–3.129	0.0172	6
Tracheostomy and/or ventilation	1.0229	0.0822	2.781	2.368–3.268	<0.0001	10
Intercept	-2.4541	0.0318	–	–	<0.0001	–

n = 21,230.

^†Score point values = coefficient values multiplied by 10 and rounded to nearest integer.

OR: Odds ratio; SE: Standard error.

Measures of model performance for the 60% development (n = 12,738), 40% validation (n = 8492) and 100% total samples (n = 21,230) using the individual SASI predictors as covariates in addition to adjustment for age, race and Charlson Comorbidity Score, are provided in Table 3. Last, we measured the performance of the integer-based SASI predictor plus age, race and Charlson Comorbidity Score for the full cohort. The p-values for all GoF statistics were larger than 0.05, suggesting that our models are well calibrated. Additionally, the addition of the SASI score to a model with age, sex, race and Charlson score improved the area under the curve from C = 72.4 to C = 76.8. The C-statistic for each cohort ranged from a low of C = 0.755 to a high of C = 0.784 demonstrating a good ability to discriminate 30-day mortality or discharge to hospice.

Table 3. . Measure of full covariate model performance: development, validation, total cohort and sensitivity analysis.

Model	n	Percentage of population	Hosmer–Lemeshow goodness-of-fit statistics†	Area under the receiver operator curve ‘C-statistic’
			X2, df, p-value
Development	12,738	60%	10.98, 8, p = 0.20	0.784
Validation	8492	40%	10.44, 8, p = 0.24	0.755
Total cohort	21,230	100%	7.78, 8, p = 0.46	0.772
SASI score total cohort	21,230	100%	7.65, 8, p = 0.47	0.769
Sensitivity NIHSS model‡	327	100% IMS III	7.75, 8, p = 0.46	0.854
Sensitivity SASI score model‡	327	100% IMS III	8.30, 8, p = 0.40	0.834

^†performed on a random sample of 1000 observations.

^‡Model controlled for age, treatment group allocation and stroke severity (using either NIHSS or SASI as indicated).

IMS III: International Management of Stroke III; NIHSS: NIH Stroke Scale; SASI: Stroke Administrative Severity Index.

Model performance was evaluated graphically by examining the observed versus predicted 30-day mortality/hospice discharge by decile for each cohort and model prediction method. This is provided in Figure 2 and demonstrates that our model is highly accurate in predicting 30-day mortality/hospice discharge for each decile.

A clinical sensitivity analysis was performed using publically available data from the International Management of Stroke III clinical trial (n = 327) where there were data that included acute hospital ICD-9 diagnosis codes as well as clinical predictors of stroke severity (NIHSS) [30]. The correlation between the SASI and the NIHSS at day 5 or discharge in this group of patients with moderate or severe stroke was 0.42. The association between the SASI and 30-day postdischarge death or discharge to hospice was examined using the same model performance methods used in model development. Model fit remained good (GoF test p-value <0.05) and area under the receiver operating characteristic curve was strong (C-statistic >0.83) in the International Management of Stroke III data when using the SASI to predict death at 30 days or discharge to hospice in this patient group. The model performance in this data supports the theoretical premise that the outcome 30-day mortality/hospice discharge is a good proxy for the NIHSS at discharge.

The generalizability [31] of the SASI score was examined using 7258 stroke patients extracted from the Medicare 5% LDS for 2014. The SASI ranged from 0 to 31. The distribution of the SASI score was skewed (mean = 3.58, standard deviation = 4.47 and median = 2.0). SASI values in the lowest quartile were 0; ‘mild’, values ranged from 1 to 6 in the middle two quartiles; ‘moderate’, and the highest quartile of values was 7–31; ‘severe’. When the 2014 Medicare stroke patients were classified by the SASI-defined severity categories, the SASI groups were clinically different for both in-hospital measures known to differ by stroke severity (Charlson Score, length of stay [LOS] and Medicare payment) and on postdischarge measures associated with stroke severity (discharge to inpatient rehabilitation, payment for care for 12 months poststroke and days at home during the follow-up period) available in our dataset (Table 4) [32]. When the three SASI cohorts were compared, we observe a monotonic increase in important stroke severity measures for the mild, moderate and severe stroke categories. The mean initial hospital LOS, hospital payment and follow-up cost all increase as the SASI-defined stroke severity increases, and days at home decreases, as expected by stroke severity. In addition, the discharge to inpatient rehabilitation is higher for the most severe stroke group as expected. These findings indicate that the SASI score, when applied to a different stroke cohort, in a different year is able to classify patients into groups with meaningful differences in resource use and poststroke outcomes.

Table 4. . Characteristics of a cohort of Medicare 2014 stroke patients discharged alive, their initial hospital payments and 12 month follow-up measures.

Variable Name	Mild stroke SASI = 0 n = 3269 (45.0%)	Moderate stroke SASI = 1–6 n = 2386 (32.9%)	Severe stroke SASI = 7–31 n = 1603 (22.1%)
Age mean (SD)	79.1 (8.4)	79.5 (8.5)	79.7 (8.4)
Charlson score Mean (SD)	1.1 (1.6)	1.8 (1.8)	2.4 (1.8)
Length of stay Mean (SD)	4.5 (3.1)	5.8 (5.4)	6.6 (4.7)
Medicare payment for stroke admission Mean (SD)	$6850 (5999)	$8456 (7932)	$9688 (9768)
Medicare payment over 12 months postdischarge† Mean (SD)	$29,308 (30,901)	$35,690 (34,084)	$39,007 (35,538)
Days at home‡ postdischarge Mean (SD)	296.2 (115)	270.5 (129)	248.7 (137)
Patients discharged to inpatient rehabilitation n (%)	498 (15.2)	638 (26.7)	467 (29.1)
Females n (%)	1761 (53.9)	1310 (54.9)	1047 (65.3)

n = 7258.

^†All Medicare payments except for prescription drugs for patients for 12 months postdischarge or until death, if death is sooner than 12 months.

^‡All days of survival poststroke discharge where patient is not in an inpatient rehabilitation, inpatient acute care hospital or a skilled nursing facility (a patient-centered measure discussed by Myles et al. [32].

SASI: Stroke Administrative Severity Index; SD: Standard deviation.

Discussion

An initial step to improving population-level healthcare outcomes assessment is the development of a valid measure. The SASI is the first attempt to systematically measure risk-adjusted stroke severity utilizing commonly accessible data. The theoretical underpinnings and development process for the SASI provide a strong foundation for this novel, administrative-level risk-adjustment tool. The SASI uses predictor variables that are based on the best identifiable matches to the clinical stroke severity variables of the NIHSS. The process used in this study mirrors the developmental process used in the well-accepted Charlson score. Charlson score has been used as a standard index for controlling for selection bias due to the presence of comorbid conditions when using billing data to assess clinical outcomes or cost. This process was also successfully applied in the development of a severity indicator to control for bias in analysis of data from patients with severe sepsis or septic shock [18]. Last, we tested and validated the predictive ability of the SASI by comparing it to the NIHSS score and found that it remained a strong predictor of stroke severity for a population of patients. While further research is needed related to the applications of the SASI, we foresee several practical uses for clinicians, researchers and policy makers who wish to examine stroke severity in populations. However, it is important to remember that the SASI is designed to predict population risk, and is not validated at the level of the individual. Further, the SASI predicts the risk of death or hospice admission 30 days posthospital discharge, it is not valid for measuring variations in stroke risk at the time of hospital admission.

Stroke is the second leading cause of death and disability worldwide [33] and is a high-cost condition. Treatment outcomes and cost of stroke depend on the severity of the stroke, the timing of the intervention and aspects related to the process of care [34–36]. Acute care costs for stroke can be decreased by using guideline-informed treatment processes [34–36]. However, stroke severity differences at hospital discharge can increase the cost of care for patients by sixfold in the 12 months following the stroke [36]. New stroke interventions may increase survival and initial hospital cost, and may change the characteristics of the population of stroke patients who are discharged to postacute stroke care. Some changes in the acute stroke care process may result in improved long-term patient outcomes and posthospital savings [36]. However, other interventions may have little health benefit and could simply shift costs from hospitals to postdischarge settings. Thus, the impact of implementing interventions for stroke care should be examined in light of their effect on acute care cost, patient outcomes and effect on the medical and rehabilitation care required after discharge from the initial hospitalization. Undesirable postdischarge events and process costs are often examined using archival billing data. However, without a validated approach to control for stroke severity at discharge using administrative billing data, comparisons of poststroke events and cost are subjected to confounding and/or selection bias.

Holding providers and health systems accountable for healthcare outcomes is another common approach to improve quality and transparency [5]. Payers and policy makers incentivize improvements in care through value-based payment and public reporting of quality measures (e.g., mortality and readmissions) [5]. However, concerns persist related to the accuracy of outcomes and the comparability across populations [35,36]. Development of valid risk-adjustment procedures is essential to detecting actual differences in quality. Without proper risk adjustment there is potential to penalize the providers with more complex patients, and comparisons across patient populations (or within the same institution over time) will fluctuate based on patient characteristics rather than detect success of the treatment or intervention [5,35,37]. This study begins to close the gap in our analytic ability to appropriately risk-adjust when using archival billing data to compare outcomes and cost for groups of stroke patients postdischarge.

It is important to note that the individual SASI indicators should be used when sample size and analytical availability allows as one can expect that they will have more sensitivity and, thus, increased individual adjustment power than the single SASI score. In addition, we purposely excluded other commonly agreed upon adjusters from our score so that they may be controlled for separately from our index because requirements based on study question and/or sample data may vary. The use of these other covariates should be selected carefully and on a case-by-case basis [37]. Some examples of these covariates include patient demographics, such as age, sex, race and other social determinants of health. Our intention was to develop a stroke severity adjustment index that can be used across different stroke types, especially if such information is not available. Importantly, if information is available on the stroke type, we recommend that the investigator consider controlling for ischemic compared with hemorrhagic stroke. Also, patient comorbidity should be separately controlled for in fully risk adjusted models as severity of disease and level of disease comorbidity are two separate, but important, risk adjusters [2,3,19,20,38]. Last, many other studies of stroke risk adjustment suggest a variety of potential other risk adjusters, such as history of cardiovascular illness or previous stroke [6,39], co-existing illnesses [6,39] and medication or clinical information, when available [17].

While our stroke severity score has performed well on the tests of validity that we have applied, and we believe that the model and the associated score will be valuable for adjusting for stroke severity differences when using archival data to assess quality improvement, and health policy interventions, our study has several limitations that must be considered. First, The SASI score was based on a cohort of stroke patients who were identified using ICD-9 code criteria applied to administrative data sources and validated on an administrative data source as well as a clinical cohort that was limited to individuals with moderate to severe stroke. Thus, it is unknown how it would function if applied to a clinically identified cohort with mild stroke based on clinically derived data. Furthermore, even though the selection of initial ICD-9 codes to be included for consideration was based on the conditions included in the NIHSS, the large sample size that we used to combine and select predictor variables means that statistical significance may have been achieved for some variables which are more common, while correlated but more clinically important variables which are rarer may have been excluded because they did not retain significance in the final model. Second, the population used for modeling were Medicare patients over age 64 which does not allow us to test the SASI in young stroke patients. It is possible that a different set of prediction variables would prove superior in a younger cohort. Further work should be done to test the validity of the SASI in younger patients. Third, we developed and tested the SASI based on ICD-9 diagnosis and procedure codes. Future studies should address a cross-walk to ICD-10 diagnosis and procedure codes. Finally, there are inherent weaknesses in using hospital discharge data which include the retrospective nature of data collection and dependence on the presence or absence of specific ICD-9 diagnosis and procedure codes [39]. We appreciate that patient status before stroke, at hospital admission and during the hospital course of treatment are well-accepted contributors to stroke outcome; focusing on discharge status alone is a necessary limitation of billing data in contrast to clinical data that would ideally consider those factors as well. However, hospital discharge data are routinely used for quality improvement and payment policy monitoring, thus this limitation is inherent in the types of studies to which the SASI is designed to be applied.

Conclusion

We developed a stroke severity prediction model using 30-day postdischarge mortality risk model of ischemic or hemorrhagic stroke patients and associated severity score for use with administrative data. We validated the model internally and externally using publically available clinical trial data that contains both billing information and clinical severity measures. We believe that the model and score are important tools that should be used to adjust for stroke severity of illness when administrative data are used to compare outcomes and/or cost for groups of patients with stroke when clinical stroke severity scores are not available to adjust for potential confounding. Future studies should perform prospective validation of the SASI, examine how well the SASI performs using other outcomes, such as the 30-day modified Rankin Scale Score, and examine how the measure performs in stroke patients with a different age distribution.