NIH Stroke Scale as an Outcome in Stroke Research: Value of ANCOVA Over Analysing Change from Baseline

Abstract

NIH Stroke Scale (NIHSS), measured a few hours to days after stroke onset, is an attractive outcome measure for stroke research. NIHSS at the time of presentation (baseline NIHSS) strongly predicts the follow-up NIHSS. Because of the need to account for the baseline NIHSS in the analysis of follow-up NIHSS as an outcome measure, a common and intuitive approach is to define study outcome as the change in NIHSS from baseline to follow-up (ΔNIHSS). However, this approach has important limitations. Analysing ΔNIHSS implies a very strong assumption about the relationship between baseline and follow-up NIHSS that is unlikely to be satisfied, drawing into question the validity of the resulting statistical analysis. This reduces the precision of the estimates of treatment effects and the power of clinical trials that use this approach to analysis. Analysis of covariance (ANCOVA) allows for the analysis of follow-up NIHSS as the dependent variable while adjusting for baseline NIHSS as a covariate in the model and addresses several challenges of using ΔNIHSS outcome using simple bivariate comparisons (e.g. a t-test, Wilcoxon rank-sum, linear regression without adjustment for baseline) for stroke research. In this article, we use clinical trial simulations to illustrate that variability in NIHSS outcome is less when follow-up NIHSS is adjusted for baseline compared to ΔNIHSS and how a reduction in this variability improves the power. We outline additional, important clinical and statistical arguments to support the superiority of ANCOVA using the final measurement of the NIHSS adjusted for baseline over, and caution against using, the simple bivariate comparison of absolute NIHSS change (i.e. delta).

Introduction

In patients suffering from stroke, a key goal in treatment is minimization of the functional disability.¹ A common predictor of long-term disability, usually measured few months after the incident stroke, is short term functional assessment done using the NIH stroke scale (NIHSS).^{2, 3} Despite known limitations such as disproportionate representation of anterior cerebral circulation and left hemispheric deficits as well as inability to account for the baseline vs new neurological deficits,^4–6 NIHSS is the most widely utilized measure of stroke severity.⁷ Baseline and follow-up NIHSS are clinically important parameters commonly obtained via clinical examination as a routine practice in patients with ischemic stroke.¹ Both baseline and follow-up NIHSS are strong predictors of functional outcomes in ischemic stroke patients.^{3, 8} Clinical trials evaluating the efficacy of treatment strategies for acute ischemic stroke (AIS) often evaluate the follow-up NIHSS, measured as early as few hours after stroke onset, as the primary or a key secondary outcome.^9–13 The follow-up NIHSS is strongly influenced by the baseline NIHSS, e.g., measured at the time of patient presentation.¹⁴ Thus, to increase the sensitivity of statistical analyses designed to detect beneficial effects of experimental treatment strategies, the statistical analysis of the follow-up NIHSS needs to account or adjust for the baseline measurement.

An early target for AIS therapies is to minimize the neurological worsening, as measured by an increase in the NIHSS from the baseline, and is known to strongly associate with reduction in long-term disability.^15–18 Because of the need to account for the baseline NIHSS during analysis, a common and intuitively appealing approach is to define the primary study outcome as the change in NIHSS (ΔNIHSS) from baseline to follow up.^9–13 For example, ΔNIHSS from baseline was used as a key secondary endpoint in the SWIFT-PRIME (27+/−3 hours),¹¹ ASTER (24 hours),¹³ phase 1/2a study of SB623 (24 months)¹², and Amphetamine-Enhanced Stroke Recovery (3 months) trials,⁹ and as the primary endpoint in the NINDS-rtPA (24 hours) trial.¹⁰ It has been extensively used as primary outcome in observational studies.^18–20 While defining the treatment effect as a ΔNIHSS from baseline to follow up for the purposes of analysis is appealing, this approach has important limitations. Analysing AIS outcomes as the ΔNIHSS implies a very strong assumption about the relationship between baseline and follow-up NIHSS that is unlikely to be satisfied, drawing into question the validity of the resulting statistical analysis. This reduces the precision of the estimates of treatment effects and the power of clinical trials that use this approach to analysis.

Our goals here are to clarify the limitations associated with the analysis of NIHSS as a change from baseline, propose a more appropriate and powerful approach, and illustrate the advantages of the proposed approach through a simulation study of trial outcomes.

Clinical Concerns Regarding Use of Change in NIHSS as a Clinical Outcome

In the setting of AIS, a variety of patient characteristics are likely responsible for differences in baseline NIHSS and in the rate and magnitude of worsening in the progression to follow-up NIHSS, e.g., etiology of stroke, age, extent of collateral vessels, and recanalization status.^{21, 22} By analysing ΔNIHSS, one makes an inherent assumption that it is independent of baseline NIHSS and that the average benefit of treatment, as measured by ΔNIHSS, is independent of the baseline NIHSS. However, ΔNIHSS depends on baseline NIHSS.²³ Further, it is highly unlikely that each unit of change in NIHSS would have the same implication on functional outcomes for patients. For example, a 4-unit change in NIHSS would have widely different effects on functional outcome in a patient with a baseline NIHSS of 6 compared to one with a baseline NIHSS of 22. An additional consideration, unique to stroke patients, is the laterality of the lesion. Because left cerebral hemisphere is disproportionately represented in the NIHSS, a given baseline and ΔNIHSS may lead to differential long-term clinical outcomes in patients with left vs right hemispheric infarcts.^{4, 5} Finally, if patient centrality is to be considered of paramount importance when selecting outcome measures,^{24, 25} the follow-up NIHSS is likely to carry more clinical relevance and have higher implications on functional outcomes than the ΔNIHSS. In other words, final neurological status is likely to be more relevant to the patient than how such status was achieved.

Statistical Concerns Regarding Use of Change in NIHSS for Analysis

The issues associated with change from baseline measures have been described by Harrell and Slaughter,²⁶ among others. Briefly, analysis of ΔNIHSS assumes that 1) baseline NIHSS is linearly related to follow-up NIHSS, and 2) that this relationship has a slope of 1 (ie, for every 1 unit increase in baseline NIHSS, the expected follow-up NIHSS also increases by 1 unit). Although follow-up NIHSS is associated with baseline NIHSS, the slope of this relationship is far from 1.^{4, 23} If the baseline is not strongly associated with the follow-up (i.e., if the correlation between baseline NIHSS and follow-up NIHSS is any less than 0.5), then ΔNIHSS will have higher variance than follow-up NIHSS. However, it can be speculated that patients with lower baseline NIHSS may have smaller ΔNIHSS compared to those who start at a higher baseline NIHSS, because of the more limited room for improvement or the milder nature of initial lesion. Further, because of regression to the mean, it is impossible to make the ΔNIHSS truly independent from baseline NIHSS.²⁷ Calculated ΔNIHSS is subject to measurement error from two sources, both the baseline and final measurement whereas the follow-up NIHSS is only subject to its own measurement error. Practical measurement of NIHSS at baseline is subject to additional variability such as measurement by non-certified raters and within a pressing timeline. This error will be further enhanced if baseline NIHSS is used as an inclusion/exclusion criterion for a study,²⁶ as is the case for most acute stroke trials. In this case, it becomes necessary to obtain a second baseline to get a meaningful ΔNIHSS.²⁶

To reduce bias in observational studies and to maximize power and precision in randomized trials, it is imperative to adjust any NIHSS outcome measure for baseline NIHSS using regression modelling.²⁶ In this case, ΔNIHSS is confusing as it requires baseline NIHSS to be on both the left-hand and right-hand side of the model equation. Because ΔNIHSS (computed as follow-up minus baseline NIHSS) is a function of baseline NIHSS, the researcher analysing the ΔNIHSS as an outcome measure indirectly considers baseline NIHSS as a response to the treatment, which is not logical.²⁸ It is also important to note that missing data on baseline NIHSS would be difficult to account for when ΔNIHSS is used as an outcome measure.

Alternate Approach: Analysis of Covariance

Analysis of covariance (ANCOVA) addresses the challenges noted above. It is equivalent to a regression model that includes the baseline value as one of the predictors. The ANCOVA model also includes a categorical variable (such as treatment A vs treatment B) as a covariate. In this case, ANCOVA allows for the analysis of follow-up NIHSS as the dependent variable while adjusting for baseline NIHSS as a covariate in the model.. This is a superior approach compared to analysis of ΔNIHSS using simple bivariate comparisons (e.g a t-test, Wilcoxon rank-sum, linear regression without adjustment for baseline) for stroke trials as well as observational research.^28–30

As mentioned above, it is imperative to adjust for baseline NIHSS using regression modelling whenever NIHSS is used as an outcome measure. This approach correctly partitions variability associated with the baseline NIHSS and thus is more powerful. In fact, ANCOVA is more efficient than analysis of ΔNIHSS for all levels of correlation between baseline and follow-up NIHSS other than a correlation coefficient of 1.0.²⁶ Consequently, for a given treatment effect and sample size, ANCOVA using follow-up NIHSS adjusted for baseline NIHSS renders more power compared to the simple bivariate analysis, for example using the t-test, of ΔNIHSS. In addition, missing value of bNIHSS can be easily handled in ANCOVA using multiple imputation.

We performed trial simulations to illustrate these arguments. First, we generated baseline and follow-up NIHSS data (with measurement errors) for a total of 2000 patients, randomly assigned to control and treatment groups in a 1:1 ratio (Figure-1). The distribution of the baseline and follow-up NIHSS are shown in panels A and D. In panels B and E, we show the distribution of ΔNIHSS (follow-up minus baseline NIHSS). To illustrate the variance of follow-up NIHSS adjusted for the baseline, we plot the residuals (the difference between follow-up NIHSS predicted by a given baseline in a linear model and the actual follow-up NIHSS for the given baseline) in panels C and F. The variance in residuals is smaller than the variance of the ΔNIHSS. To illustrate how this reduction in variability improves power, we simulate 10,000 trials for each combination of treatment effect (difference in NIHSS between treatment and control arms) ranging from −10 to −2 (increments of 0.25), with sample size ranging from 20 to 200 (increments of 1; Figure-2). The 80% and 95% probability of a trial deemed “positive” with a p-value ≤0.05 is plotted for ANCOVA (solid line; difference in follow-up NIHSS between treatment and control group, adjusted for the baseline NIHSS) and a t-test (dotted line; difference in the ΔNIHSS between treatment and control group). Figure 2 demonstrates that ANCOVA yields 80% and 95% power to detect a difference in the follow-up NIHSS between the treatment and control arm (treatment effect) with fewer number of patients compared to simple bivariate comparison of ΔNIHSS using t-test. To illustrate further how the correlation coefficient of baseline and follow-up NIHSS impacts this power, we perform similar trial simulations for correlation coefficients ranging from 0.25 to 0.75. It is notable that lower correlation coefficients have a drastic impact on the differences in the power of ANCOVA vs t-test. Additional statistical details and the R-code for trial simulations are provided in the Supplemental Material. The authors declare that all supporting data are available within the article [and its online supplementary files].

Figure 1. Variability associated with change in NIH Stroke Scale (NIHSS), vs final NIHSS adjusted for baseline, in a single simulated randomized clinical trial for 2000 patients.

Panel A and D illustrate the baseline and follow-up NIHSS, with measurement errors, for the control and treated groups. The mean baseline NIHSS in the control group was 10+/− 4.9 (median with IQR 10 [2–18]). Mean follow-up NIHSS in the control group was 14.7 +/− 5.04 (median with IQR 15 [8–22]). Mean baseline NIHSS was 10.39 +/−4.9 (median with IQR 10 [2–18]) and mean follow-up NIHSS was 10.2 +/−4.9 (median with IQR 11 [4–18]) in the treated group. ΔNIHSS (follow-up minus baseline NIHSS) for the control and treated groups are shown in panels B and E. The mean ΔNIHSS for the was 4.4 +/− 5.2 (median with IQR 5 [−2–12]) for the control group and −0.14 +/− 4.9 (median with IQR 0 [−6–6]). Panels C and F outline the residuals (the difference between follow-up NIHSS predicted by a given baseline NIHSS and the actual follow-up NIHSS for the given baseline) of a linear model of follow-up NIHSS adjusted for the baseline adjusted NIHSS was calculated using linear regression of follow-up NIHSS over the baseline. The mean of these residuals for the control group was 0 +/− 4.5 (median with IQR −.04[−3.04–3.12]) and for the treated group was 0 +/− 4.2 (median with IQR −0.09[−3.05–2.95]). The variance for the ΔNIHSS (panels B and E) is larger than that of the final NIHSS adjusted for the baseline in a model (panels C and F), as shown by the shorter length of the box plot and smaller SD for each plot (5.2 vs 4.5 and 4.9 vs 4.2) for panels C and F compared to B and E. Additional methodological details and R code for trial simulations is provided in the Online Supplement.

Figure 2. Comparison of ANCOVA using follow-up NIHSS adjusted for baseline as the outcome measure and T-test using ΔNIHSS as outcome measure for various, treatment effects, sample sizes, and correlation coefficients (of follow-up and baseline NIHSS).

Additional methodological details and R code for trial simulations is provided in the Online Supplement.

The ANCOVA model can be formulated to allow for a non-linear relationship between baseline NIHSS and follow-up NIHSS. Additionally, it does not assume that ΔNIHSS is independent of baseline NIHSS. Analysis of follow-up NIHSS adjusted for baseline using ANCOVA better accounts for individual patient factors responsible for this clinical progression than the analysis of ΔNIHSS. ANCOVA allows the researcher to select patients based on baseline NIHSS and still be able to account for it in the analysis without acquiring a second baseline to cancel some of the regression to the mean.

It is important to note that the arguments herein favouring ANCOVA over simple bivariate comparisons not accounting for the baseline measurement can be applied to other outcomes assessed at a pre-treatment baseline and single follow-up timepoint, such as the radiographic evaluation of infarct volume, modified Rankin scale score for functional assessment, or the Fugl-Meier Score of motor strength. Although details of measurements of each scale and their clinical interpretation varies, the principles regarding estimation of treatment effect using final value adjusted for the baseline compared to change from baseline apply generally. For categorical scales such modified Rankin scale score analysing change from baseline is particularly flawed because each unit of change from baseline implies vastly varying clinical impacts that is dependent on the baseline score, i.e change from mRS 0 (no symptoms) to 2 (mild disability, independent in activities of daily living and ambulation) is vastly dissimilar to that from mRS 2 to 4 (Severe disability, dependent in activities of daily living and cannot ambulate independently), as is noted in the utility values of mRS score categories.³¹

Conclusion

NIHSS-based outcome measures have been widely used and will be increasingly considered in the design of future stroke trials and observational studies. When used, as with any repeated measure with a pre-randomization baseline and a post-treatment follow-up measurement, we recommend using ANCOVA to analyse final measurement adjusting for the baseline value as a covariate in the regression model. We provide clinical and statistical arguments, to support the superiority of ANCOVA using final measurement of the outcome variable adjusted for baseline value over the simple bivariate comparison of absolute change (i.e. delta). The concerns regarding the latter approach are upheld by examples of simulated clinical trials; and thus, we caution against using absolute change in a variable as an outcome measure for stroke research.

Abbreviations

ANCOVA

Analysis of Covariance

AIS

Acute Ischemic Stroke

ASTER

Contact Aspiration vs Stent Retriever for Successful Revascularization

NIHSS

NIH Stroke Scale

NINDS

National Institute of Neurological Disorders and Stroke

rtPA

Recombinant Plasminogen Activator

SWIFT-PRIME

Solitaire With the Intention for Thrombectomy as PRIMary Endovascular Treatment