Evaluating the concordance between International Classification of Diseases, Tenth Revision Code and stroke severity as measured by the National Institutes of Health Stroke Scale

Abstract

Background

The National Institutes of Health Stroke Scale (NIHSS) scores have been used to evaluate acute ischaemic stroke (AIS) severity in clinical settings. Through the International Classification of Diseases, Tenth Revision Code (ICD-10), documentation of NIHSS scores has been made possible for administrative purposes and has since been increasingly adopted in insurance claims. Per Centres for Medicare & Medicaid Services guidelines, the stroke ICD-10 diagnosis code must be documented by the treating physician. Accuracy of the administratively collected NIHSS compared with expert clinical evaluation as documented in the Paul Coverdell registry is however still uncertain.

Methods

Leveraging a linked dataset comprised of the Paul Coverdell National Acute Stroke Program (PCNASP) clinical registry and matched individuals on Medicare Claims data, we sampled patients aged 65 and above admitted for AIS across nine states, from January 2017 to December 2020. We excluded those lacking documentation for either clinical or ICD-10-based NIHSS scores. We then examined score concordance from both databases and measured discordance as the absolute difference between the PCNASP and ICD-10-based NIHSS scores.

Results

Among 87 996 matched patients, mean NIHSS scores for PCNASP and Medicare ICD-10 were 7.19 (95% CI 7.14 to 7.24) and 7.32 (95% CI 7.27 to 7.37), respectively. Concordance between the two scores was high as indicated by an intraclass correlation coefficient of 0.93.

Conclusion

The high concordance between clinical and ICD-10 NIHSS scores highlights the latter’s potential as measure of stroke severity derived from structured claims data.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• The National Institutes of Health Stroke Scale (NIHSS) is a vital tool for assessing acute ischaemic stroke severity. International Classification of Diseases, Tenth Revision Code (ICD-10) codes have enabled administrative documentation of NIHSS scores, but their accuracy compared with clinical NIHSS scores was uncertain, particularly across multiple states and healthcare settings.

WHAT THIS STUDY ADDS

• This study, involving 87 996 patients across nine states, demonstrates high concordance (ICC3 of 0.93) between clinical NIHSS scores from the Paul Coverdell Registry and ICD-10-based NIHSS scores from Medicare claims. It reveals minimal discordance between the two scoring methods, with some variations across states and stroke severity levels.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• These findings validate the use of ICD-10-based NIHSS scores as a reliable proxy for stroke severity in large-scale research and quality improvement initiatives. This could enhance stroke epidemiology studies and inform policies on stroke management and reimbursement practices, potentially streamlining administrative processes in stroke care.

Introduction

The National Institutes of Health Stroke Scale (NIHSS) has long served as a vital tool for assessing the severity of acute ischaemic strokes (AIS).¹ This well-established instrument is the most used neurology deficit ranging scale.² It allows for impairment grading in a numerical scale ranging from 0 to 42 with greater scores reflecting a worse clinical scenario. From its validation, raters were required to have standardised training ensuring a reliable and reproducible nature to this instrument, which has resulted in its employment in numerous clinical trials.³,⁵

Beyond its direct clinical and research purposes, since the introduction of the tenth revision of the International Classification of Diseases (ICD) codes, hospitals have also been recording the NIHSS scores for administrative risk stratification through the R29.7xx code group.⁶ However, during the initial national optional reporting period in the USA, NIHSS scores were documented through administrative claims in only 15% of hospital admissions for AIS.⁶

A notable gap in the existing literature pertains to an evaluation of the accuracy of ICD-based NIHSS scores when compared with clinical NIHSS scores. The incorporation of International Classification of Diseases, Tenth Revision Code (ICD-10)-based NIHSS is advocated for its effectiveness in hospital risk adjustment⁷ and in enhancing the measurement of quality of care and patient outcomes.⁸ A recent study took a step in this direction by assessing the validity of ICD-10-based NIHSS scores using data from a single centre stroke registry.⁹ While this study revealed a high level of agreement between ICD-10-based NIHSS scores and clinical NIHSS scores (R2=0.86), it is worth noting that ICD-10 NIHSS scores were available for only a limited subset of patients (n=395 out of 1357, 29.1%).

Furthermore, it is essential to recognise that single institution studies do not fully capture the potential diversity of ICD coding practices across different states and patient populations.⁹ In light of these considerations, we examined the concordance between the NIHSS score reported for clinical assessment in the Paul Coverdell National Acute Stroke Program (PCNASP) and the score reported in Medicare claims database, with data gathered over multiple years across nine states.

Objective

To evaluate the concordance between AIS severity scores from the clinically derived NIHSS scores obtained from PCNASP registry and administratively derived ICD-10 NIHSS scores obtained from Medicare claims.

Methodology

Data source

We established a linkage between the Paul Coverdell National Acute Stroke Program (PCNASP) registry and Medicare Claims data. PCNASP aims to collect and track data on stroke-related cases to enhance patient care quality.¹⁰ The PCNASP registry includes data from 2008 to 2020 across 9 US states (California, Georgia, Massachusetts, Michigan, Minnesota, New York, Ohio, Washington and Wisconsin) and captures the NIHSS scores as reported by clinicians or hospital staff.

Medicare, a national health insurance programme administered by the Centers for Medicare & Medicaid Services (CMS), primarily serves individuals aged 65 or older. From October 2015, Medicare implemented the use of ICD10’s R29.7xx codes to group NIHSS scores creating distinct severity categories that aid stroke outcome research.¹¹ The Medicare Provider Analysis and Review (MEDPAR) database contains extensive information about beneficiaries and as well a broad range of information, including patient demographics, admission and discharge dates, diagnosis, procedure codes, provider identifiers, comorbidities, and, since 2015, measurement of stroke severity through ICD-10-based NIHSS scores.¹² This dataset is used for administrative purposes well as for healthcare research.

Linking databases

Due to the lack of common unique patient identifiers across the two databases, we applied a matching strategy to link individuals in the PCNASP and Medicare datasets.¹³ This approach used variables such as age, gender, admission and discharge dates, diagnosis code, hospitals and state. After the linking process, we retained only those patients with unique matches, excluding cases where PCNASP IDs corresponded to multiple Medicare Beneficiary IDs, and vice versa.

Study population

With the linked dataset, we selected patients aged 65 or older who were hospitalised for AIS and discharged under ICD-10 codes within the I63 groups.¹⁴ Patients were required to have both PCNASP (clinical) and Medicare (ICD-10-based) NIHSS scores recorded. Since the ICD-10-based NIHSS score codes were introduced with the ICD-10th revision, we limited our analysis to cases from 2017 to 2020 that used ICD-10 diagnosis and procedure codes. There was no access to data beyond 2020.

Measurements

NIHSS scores are the primary outcome measured. The scores range from 0 to 42 with increasing values denoting more severe deficits.¹⁵ In ICD-10, the R29.7xx code group allows for direct input of integer values in that range and reflects the identical scoring criteria.

Per CMS guidelines stroke ICD-10 diagnosis code must be documented by the treating physician involved in the patient’s care.¹²

Analysis plan

We assessed the concordance between the NIHSS scores documented in PCNASP and in Medicare claims data in the MEDPAR files. For our analysis of Medicare data, our main emphasis was on the ICD-10 NIHSS score documented at the moment of admission. Any instances where multiple ICD-10 NIHSS scores were present at admission or where there was no ICD-10 NIHSS score at admission were excluded from our analysis.

Statistical analysis

Our analysis focused on evaluating the concordance between the ordinal values of the PCNASP and Medicare NIHSS scores. We assessed the concordance of these two scores using the intraclass correlation coefficient (ICC3) with a two-way mixed-effects model.¹⁶ The ICC3 with a two-way mixed-effects model is a statistical measure used to evaluate the reliability of measurements from a fixed set of raters, by comparing the variance between subjects to the total variance, which includes measurement error and rater inconsistency. We calculated the mean difference and the absolute mean difference (discordance) between the PCNASP and Medicare NIHSS scores. Discordance is a measure of the magnitude of disagreement between the two scores. We followed the Standards for Reporting Diagnostic accuracy studies guidelines reporting of diagnostic accuracy studies¹⁷ (online supplemental materials). We set the level of statistical significance at p<0.05 for all analyses. Statistical analyses were performed using SAS V.9.4 (SAS Institute, Cary, North Carolina) and Python V.3.8.

Regarding data handling and human protection, all patient data were deidentified prior to analysis to ensure confidentiality.

Results

From an initial dataset of 263 748 patients, 87 996 patients with 88 508 AIS admissions met our inclusion criteria. These patient data came from 514 hospitals across nine states, with state-specific demographics and results are provided in online supplemental table S1.

Table 1 details the breakdown of PCNASP (clinical) and Medicare (ICD-10-based) NIHSS scores and their demographic characteristics. The mean age was 78.99±8.66 years. Our population was predominantly white (80.78%). The mean PCNASP clinical NIHSS score was 7.19 (95% CI 7.14 to 7.24), while the mean Medicare ICD-10-based NIHSS score was 7.32 (95% CI 7.27 to 7.37). Both scores had the same median of 4.0 with IQR of (2.0–11.0). Online supplemental figure 1S illustrates the distribution of clinical and ICD-10 NIHSS scores within this cohort.

Table 1. Study cohort demographics and average clinical and ICD-10 NIHSS scores.

Characteristic*	Study population	PCNASP clinical NIHSS mean (95% CI)	Medicare ICD-10 NIHSS mean (95% CI)	SMD‡
Age in years,	78.99±8.66
Mean±SD
Gender
Female	47 872 (54.40%)	7.82 (7.75 to 7.90)	7.94 (7.87 to 8.01)	−0.014
Male	40 124 (45.59%)	6.44 (6.37 to 6.51)	6.60 (6.53 to 6.67)	−0.022
Race†
White	71 085 (80.78%)	6.99 (6.93 to 7.04)	7.13 (7.08 to 7.19)	−0.018
Black or AA§	9597 (10.90%)	7.68 (7.52 to 7.83)	7.81 (7.66 to 7.96)	−0.017
Asian	2797 (3.17%)	8.33 (8.01 to 8.66)	8.32 (7.99 to 8.64)	0.0021
American Indian	215 (0.24%)	6.88 (5.90 to 7.86)	6.92 (5.96 to 7.88)	−0.008
Pacific Islander	119 (0.13%)	8.82 (7.24 to 10.40)	9.20 (7.58 to 10.83)	−0.029
Not reported	4178 (4.74%)	8.78 (8.53 to 9.04)	8.89 (8.64 to 9.14)	−0.013

^*Indicates n (%) unless otherwise specified.

^†Indicates numbers do not sum to group total because of missing data.

^‡SMD: Standardized Mean Difference

^§AA: African American

ICD-10, International Classification of Diseases, Tenth Revision; NIHSS, National Institutes of Health Stroke Scale; PCNASP, Paul Coverdell National Acute Stroke Program registry.

The two scores demonstrated high concordance, as indicated by ICC3 of 0.93, as illustrated in the Bland-Altman plot (figure 1) consistent with high level of agreement rates.

Figure 1. This Bland-Altman plot illustrates the concordance between the clinical and ICD-10 NIHSS scores. The y-axis represents the difference between the two scores, while the x-axis shows the average score obtained by combining them and dividing by 2. Data points, each representing an aggregate of 20 individual cases to enhance visual interpretability, form a heat map indicating the density of sample differences. The colour gradient from yellow to purple visually encodes the sample density, providing an immediate sense of the distribution of agreement across the score range. There is a total of 3252 data points shown. The central band of the plot, highlighted in yellow-green, denotes the area where the majority of the sample scores are tightly clustered within ±1.96 SD. The mean difference was −0.14 (95% CI: −0.16 to –0.12), while the absolute mean difference (Discordance) was 1.09 points (95% CI: 1.08 to 1.11). This indicates strong agreement between the two scales. Areas of data points in purple, dispersed further from the mean, signify fewer occurrences, representing a greater discrepancy in scores. ICD-10, International Classification of Diseases, Tenth Revision; NIHSS, National Institutes of Health Stroke Scale.

In state-specific data, New York led in AIS cases (n=18 288), followed by Ohio (n=14 480), and Washington with the lowest count (n=5481) (online supplemental table S1). Notably, California recorded the

highest mean clinical and ICD-10 NIHSS scores at 8.24 and 8.22, respectively, while Minnesota reported the lowest clinical NIHSS of 6.44, and Massachusetts the lowest ICD-10 NIHSS of 6.52.

The mean difference between the clinical and ICD-10 NIHSS scores for the whole population was −0.14 (95% CI −0.16 to –0.12), while the absolute mean difference (discordance) was 1.09 points (95% CI 1.08 to 1.11) (figure 2). This discordance varied by state, with the highest average absolute score difference observed in California with 1.54 points (95% CI 1.47 to 1.60) and the lowest in Massachusetts with 0.58 points (95% CI 0.53 to 0.62) (online supplemental table S1). More details can be found in online supplemental tables S2–S4 and online supplemental figure S1.

Figure 2. Discrepancy between clinical and ICD-10 NIHSS scores. This figure shows the percentage of cases with differences between clinical NIHSS scores and ICD-10 codes. The central grey bar shows that in 82% of cases the scores match. When disagreement occurs between the two scores, ICD-10 tends to be larger than clinical NIHSS. ICD-10, International Classification of Diseases, Tenth Revision; NIHSS, National Institutes of Health Stroke Scale.

Discussion

In our investigation involving 87 996 patients with AIS from 514 hospitals across nine states, we probed the concordance between clinical NIHSS score as registered in the Paul Coverdell Registry and its ICD-10-based administrative records counterpart in Medicare claims. Despite state-specific variations, our cohort consistently demonstrated an overall high concordance score of 0.93. Notably, our findings aligned with a smaller study that highlighted minimal discordance between the two scores in the state of New York.¹⁰

Bland-Altman plot revealed that the association between stroke severity and documentation concordance was non-linear. Particularly, we observed greater consistency in documentation at the extreme ends of the stroke severity spectrum. This could be due to the clinical characteristics and consequences of the stroke being more distinct and easier to document accurately at the extreme ends of the severity spectrum.

Our study identified regional and state-specific discordance discrepancies, partly explained by variations in patient populations and demographics. However, it is worth stressing that participation in the PCNASP database is voluntary and different numbers of hospitals participated in each state over time. Additionally, potential differences in stroke assessment, and in the ways through which claims are filed to Medicare by providers could also vary by location and practice.¹⁸,²⁰

While our approach of treating stroke severity score as an ordinal variable facilitated a rigorous statistical analysis, it is essential to acknowledge that the slight variations we observed may not carry substantial clinical significance. Nevertheless, these documentation differences could become significant in borderline values where minimal score variations would result in change in stroke severity with subsequent reimbursement repercussions.

In addition, it is likely that with higher clinical NIHSS scores, larger absolute numerical variances can be noted as there is more granularity (ie, more points) available in the score for strokes of greater. Therefore, larger variations for patients with higher scores may carry the same clinical impact as smaller variations in the less severe cases.

Limitations

Our study predominantly analysed data from a substantial population sample spanning 2017–2020, a period preceding the COVID-19 pandemic. This timeframe allows for an insightful examination of stroke-related documentation and claims in a prepandemic context. It is important to note, however, that stroke presentations, outcomes and associated documentation practices have evolved during and post-COVID.²¹ While these changes may influence the applicability of our findings, they remain relevant for understanding prepandemic trends and could offer valuable insights for future healthcare practices.

Concerning generalisability, our study primarily examined traditional Fee-For-Service Medicare Part D, which may limit its applicability to other populations, such as Medicare Advantage enrollees and individuals who opt out of Part D. Recent research indicates a growing preference for Medicare Advantage, particularly among more vulnerable populations, including dual Medicare and Medicaid enrollees, Black and Hispanic individuals and residents of areas with high Social Deprivation Indexes.^{22 23} Consequently, despite the steady growth in Part D enrolment since its inception, the representation of these groups in our study may have diminished.²⁴

Furthermore, the MEDPAR dataset inherently focuses on the inpatient setting, thereby confining our analysis to NIHSS scores recorded from admission to discharge. This restriction precludes consideration of NIHSS scores obtained in the outpatient setting and does not address the coding accuracy for stroke patients who do not survive to hospital admission.²⁵

Moreover, our outcome measurements within the prepandemic period are not without limitations. The reliance on administrative documentation, often constrained by time and system complexities, introduces potential inaccuracies.^{1118 26},²⁸ These limitations underscore the need for cautious interpretation of our findings. Furthermore, the generalisability of our results is restricted to the healthcare settings and regions included in our study. Variations in patient demographics, stroke care protocols and documentation practices across different healthcare institutions and regions may limit the applicability of our findings to other contexts.

Conclusion

Our study revealed a remarkable alignment between clinically assessed NIHSS scores and ICD-10 NIHSS scores derived through administrative means, thus bolstering the validity of the latter as a viable proxy for gauging stroke severity. Although certain clinical and demographic factors did exhibit a slight discordance, their practical clinical significance is arguably minimal.

Conducting extensive, multicentre analyses on a larger scale with the ICD-10 NIHSS score serving as a gauge of stroke severity can provide valuable insights into stroke epidemiology and further advancements in stroke management.

Data availability statement

No data are available.