External validation of N2H3 nomogram to predict outcomes in patients with acute ischemic stroke treated by intravenous thrombolysis

Abstract

BACKGROUND:

The N2H3 model was evaluated for forecasting the 3-month outcomes for patients experiencing acute ischemic stroke who received intravenous thrombolysis (IVT), in our previous study. The present study aimed to validate the predictive ability of the N2H3 model and to compare its accuracy to the THRIVE-c and START models (both of which are widely employed for prognostic predictions following IVT).

METHODS:

Our study prospectively enrolled consecutive stroke patients who received IVT from 16 hospitals. Cases from one hospital were included in External Validation Dataset 1, whereas External Validation Dataset 2 included patients from the other 15 hospitals. The effectiveness of each model in distinguishing outcomes was assessed by calculating the area under the receiver operating characteristic curve (AUC-ROC). In addition, the overall performance of the N2H3 model was assessed through the scaled Brier score.

RESULTS:

Finally, 794 patients were included, of which 582 were included in External Validation Dataset 1 and 212 in External Validation Dataset 2. The N2H3 model’s AUC-ROC for forecasting unfavorable outcomes at 3-months was 0.810 (95% confidence interval [CI]: 0.771–0.848) in the first dataset and 0.782 (95% CI: 0.699–0.863) in the second dataset. For the START model, the AUC-ROCs in the two validation datasets were 0.729 (95% CI: 0.685–0.772) and 0.731 (95% CI: 0.649–0.772), respectively. The THRIVE-c model showed AUC-ROCs of 0.726 (95% CI: 0.682–0.770) and 0.666 (95% CI: 0.573–0.759), respectively. The Brier scores of the N2H3 model were 0.153 and 0.147 in cohorts 1 and 2, respectively.

CONCLUSIONS:

The N2H3 model exhibited good predictive ability in both external validation cohorts. Moreover, it demonstrated advantages over the THRIVE-c and is not inferior to the START nomogram in this regard.

TRIAL REGISTRATION:

Clinical Research of Intravenous Thrombolysis for Ischemic Stroke in Northeast of China (CRISTINA) (identifier: NCT05028868).

Introduction

Ischemic stroke represents a critical and potentially lethal condition, marked by high rates of morbidity, recurrence, and mortality, thereby imposing a significant burden on families and society at large.[1] Intravenous thrombolysis (IVT) stands out as the most significant therapeutic advancement in acute ischemic stroke (AIS) management in the last two decades.[2,3,4,5] Clinical trials have demonstrated that IVT can enhance functional outcomes in patients following AIS events;[6] however, its efficacy varies widely between patients.[7,8,9] Individual risk factors have inherent limitations in terms of predicting stroke prognoses, so predictive models that integrate multiple risk factors can more effectively identify patient populations with poor prognoses. Early and accurate prognosis prediction for patients with AIS who are receiving IVT is vital for guiding clinical decision-making and managing the expectations of patients and their families regarding stroke outcomes.

Recently, some studies have concentrated on predictive models for the prognoses of patients with AIS undergoing IVT.[10,11,12,13,14,15,16] The most common models have included THRIVE-c and START. The THRIVE-c model achieved an area under the receiver operating characteristic curve (AUC-ROC) of 0.786 in its initial cohort,[15] whereas a subsequent validation study in the TMS-China cohort reported an AUC-ROC of 0.750.[17] The START model achieved an AUC-ROC of 0.800 during its training phase,[16] whereas a separate validation study conducted at a single-center cohort in China reported an AUC-ROC of 0.766.[18] In our earlier study, we created a dependable N2H3 nomogram model incorporating variables like initial National Institutes of Health Stroke Scale (NIHSS) score, changes in NIHSS from baseline to 24 h postprocedure (i.e., delta NIHSS), hypertension, hyperhomocysteinemia (Hhcy), and ratio of high-density lipoprotein cholesterol to low-density lipoprotein cholesterol (HDL-C/LDL-C) levels.[19] It achieved an AUC-ROC of 0.872 in its training cohort, suggesting that its predictive validity may surpass those of the THRIVE-c and START nomograms. However, the absence of external validation represents a key barrier to its widespread adoption in clinical settings. Moreover, the difference in predictive performance between N2H3 and other commonly used models remains unclear.

Thus, we aimed to achieve two main objectives: (i) to externally validate our N2H3 predictive model with two different patient groups and (ii) to evaluate its predictive accuracy against the THRIVE-c and START models in forecasting 3-month poor outcomes in AIS patients treated with IVT.

Methods

Study design and participants

We carried out our research following the guidelines of the Declaration of Helsinki and received authorization from our hospital’s Ethics Committee (approval number 2015-156). All participants provided their informed consent. The study utilized a forward-looking patient database, including consecutive ischemic stroke patients receiving IVT at the China National Comprehensive Stroke Center from July 2019 to October 2023, along with 15 additional partner hospitals from September 2021 to February 2022. Individuals receiving care at the China National Comprehensive Stroke Center were part of External Validation Dataset 1, whereas patients from the other 15 hospitals were included in External Validation Dataset 2.

IVT was administered to each patient within 4.5 h of symptom onset following the recommendations of guidelines.[20,21,22] All patients received medical treatment in accordance with guidelines following thrombolysis.[20,21,22] Patients were excluded if they had prestroke modified Rankin scale (mRS) scores above 2, received endovascular treatment post-IVT, passed away within 24 h, had stroke-like symptoms or blood disorders, or had incomplete records or were lost to follow-up.

Data collection

We gathered comprehensive information on each patient, covering initial demographics, vascular risk factors (such as smoking, high blood pressure, diabetes, prior stroke, high cholesterol, Hhcy, and atrial fibrillation[23,24]), clinical details (NIHSS, change in NIHSS, blood pressure, time from onset to rt-PA bolus [OTT]), lab test outcomes, and follow-up records. The laboratory evaluations encompassed measurements of blood glucose, prothrombin time, white blood cell count, platelet count, total cholesterol, homocysteine, triglycerides, serum uric acid, and HDL-C/LDL-C.

Outcomes

The clinical outcome was evaluated at 3 months postprocedure, using the mRS. Unfavorable outcomes were categorized as mRS scores 3–6, whereas favorable outcomes were classified as mRS scores 0–2. We also recorded death (mRS = 6) within 3 months and symptomatic intracerebral hemorrhage (sICH) within 24 h after IVT.

Statistical analysis

The data analysis was performed using SPSS Statistics version 26.0 (IBM, Armonk, NY, USA). ROC curves were generated with MedCalc Statistical Software version 19.6.4 (MedCalc Software, Ostend, Belgium). The Kolmogorov–Smirnov goodness-of-fit test was used to evaluate the normality of the distribution for continuous variables. Continuous data are shown as averages (standard deviations) or medians (interquartile ranges), whereas categorical data are represented as counts (percentages), depending on the context.

The ability of the N2H3, THRIVE-c, and START models to predict adverse outcomes over a 3-month period was assessed using their individual logistic regression equations.[15,16,19] The N2H3 model’s predictor factors comprised initial NIHSS score, change in NIHSS score, high blood pressure, homocysteine levels, and HDL-C/LDL-C ratio. The formula used for the N2H3 model (where P represents the probability of a poor prognosis) was as follows:

The predictors used for the THIRVE-c model included baseline NIHSS score, age, hypertension, diabetes, and atrial fibrillation. The formula used for the model (where P represents the probability of a poor prognosis) was as follows:

The START model’s predictors comprised initial NIHSS score, age, prestroke mRS rating, and OTT. The formula used for the model (where P represents the probability of a poor prognosis) was:

The performance of a model was evaluated through discrimination and calibration.[25] The model’s capacity to differentiate between the presence and absence of an outcome was assessed using the AUC-ROC, which quantities discrimination. AUC-ROCs of 0.5 indicated random guessing (i.e., no discriminatory capacity), 0.7–0.9 indicated good discrimination, and 1.0 indicated perfect discrimination.[26,27] Previous research has also utilized the Hosmer–Lemeshow test to assess calibration. P < 0.05 indicates a notable discrepancy between the expected and observed probabilities, suggesting that the model’s accuracy is lacking. However, the Hosmer–Lemeshow test has certain drawbacks; it cannot provide quantitative information for the evaluation of accuracy. In addition, when the sample size is large, the Hosmer–Lemeshow test may mislead the results.[25] The Brier score was used to evaluate the overall performance and precision of the model more accurately. The Brier score is a measure of both discrimination and calibration and is calculated from the mean square deviation of the actual observed outcome variables (0 or 1) and the predicted probabilities, varying between 0 and 1. A lower Brier score indicates a higher calibration of the model.[28,29]

Results

Participant characteristics

Finally, 794 patients were included. External Validation Dataset 1 included 582 patients, and the percentage of patients with unfavorable 3-month prognoses was 32.81% (191/582; median age, 62.92 years; 71.50% male). To perform additional external validation of the model, another 212 patients were used, comprising 21.70% of the total cohort (46/212; median age, 63.55 years; 68.90% male). Tables 1 and 2 illustrate the differences in initial characteristics and laboratory tests between the two validation groups.

Table 1.

Comparison of patient’s baseline characteristics in external validation cohorts

Variables	Validation dataset 1 (n=582), n (%)	Validation dataset 2 (n=212), n (%)	χ2/t/Z	P
Age (years)	62.94±10.93	63.55±10.68	0.699	0.485
Male	416 (71.5)	146 (68.9)	−0.715	0.475
Vascular risk factors
Smoking	262 (45.0)	81 (38.2)	−1.715	0.087
Hypertension	309 (53.1)	121 (57.1)	0.996	0.320
Diabetes mellitus	116 (19.9)	55 (25.9)	1.995	0.046
Previous stroke	148 (25.4)	52 (24.5)	−0.258	0.796
Hyperlipidemia	447 (76.8)	156 (73.6)	−0.938	0.348
Hhcy	459 (78.9)	97 (45.8)	−9.496	<0.001
Atrial fibrillation	110 (18.9)	23 (10.9)	32.040	<0.001
Clinical data
Initial NIHSS score, points	8.00 (6.00–11.00)	5.00 (4.00–8.00)	−6.350	<0.001
Delta NIHSS, points	2.00 (0.00–5.00)	2.00 (0.00–6.00)	−2.359	0.002
Admission SBP (mmHg)	156.00 (142.00–174.00)	150.00 (136.00–165.00)	−3.392	0.001
Admission DBP (mmHg)	90.00 (81.00–100.00)	90.00 (80.00–98.00)	−1.934	0.053
OTT (min)	193.00 (141.00–237.00)	162.00 (110.00–200.00)	−5.088	<0.001
Unfavorable outcome	191 (32.82)	46 (21.70)	3.043	0.002
sICH	7 (1.20)	3 (1.41)	−0.813	0.421
Death	22 (3.78)	5 (2.36)	0.977	0.329

Hhcy: Hyperhomocysteinemia (defined as total homocysteine level≥10 µmol/L), NIHSS: National Institutes of Health Stroke Scale, Delta NIHSS: Changes in the NIHSS score from baseline to 24 h, SBP: Systolic blood pressure, DBP: Diastolic blood pressure, OTT: Onset to rt-PA bolus time, sICH: Spontaneous intracerebral hemorrhage

Table 2.

Comparison of patient’s laboratory tests in external validation cohorts

Variables	Validation dataset 1 (n=582)	Validation dataset 2 (n=212)	χ2/t/Z	P
Serum glucose (mmol/L)	7.30 (6.30–9.00)	7.55 (6.25–8.63)	0.790	0.575
WBC counts ×109/L	7.83 (6.48–9.94)	7.77 (6.14–9.53)	0.792	0.240
Platelet counts ×109/L	210.00 (175.00–245.00)	219.00 (181.00–246.00)	1.982	0.048
Prothrombin time (s)	11.00 (10.60–11.50)	11.25 (10.60–12.10)	1.293	0.197
Homocysteine (mmol/L)	13.30 (10.48–18.09)	15.90 (10.73–20.00)	0.390	0.697
Total cholesterol (mmol/L)	5.02 (4.36–5.76)	5.02 (4.29–5.79)	-0.937	0.349
Triglyceride (mmol/L)	1.41 (0.97–2.02)	1.47 (0.99–2.03)	0.649	0.517
LDL-C (mmol/L)	3.14 (2.61–3.67)	2.87 (2.30–3.57)	−1.176	0.240
HDL-C (mmol/L)	1.15 (0.99–1.31)	1.14 (0.96–1.36)	1.804	0.072
HDL-C/LDL-C (ratio)	0.35 (0.31–0.43)	0.39 (0.32–0.51)	4.582	<0.001
Serum uric acid (mmol/L)	331.00 (275.00–403.00)	330.00 (277.00–383.00)	−1.295	0.196

WBC: White blood cell, LDL-C: Low-density lipoprotein cholesterol, HDL-C: High-density lipoprotein cholesterol, HDL-C/LDL-C: The ratio of high-density lipoprotein cholesterol to low-density lipoprotein cholesterol

Three-month unfavorable outcomes

The N2H3 model achieved an AUC-ROC of 0.810 (95% confidence interval [CI]: 0.771–0.848) in External Validation Dataset 1 and 0.782 (95% CI 0.699–0.863) in External Validation Dataset 2 [Figures 1 and 2]. Furthermore, the N2H3 model demonstrated strong overall performance in both external validation cohorts (Brier scores of 0.153 and 0.147, respectively).

Figure 1.

N2H3 model in the External Validation Dataset 1: receiver operating characteristic curve area = 0.810. ROC: Receiver operating characteristic curve

Figure 2.

N2H3 model in the External Validation Dataset 2: receiver operating characteristic curve area = 0.782. ROC: Receiver operating characteristic curve

For External Validation Dataset 1, the START model achieved an AUC-ROC of 0.729 (95% CI: 0.685–0.772), whereas the THRIVE-c model had an AUC-ROC of 0.726 (95% CI: 0.682–0.770) [Figure 3]. In External Validation Dataset 2, the AUC-ROC was 0.731 (95% CI 0.649-0.813) for the START model and 0.666 (95% CI: 0.573–0.759) for the THRIVE-c model [Figure 4]. Within External Validation Dataset 1, the N2H3 model (AUC-ROC 0.810) showed better discrimination when compared to both the START (0.729; Z-score [Z] = 3.48, P = 0.001) and THRIVE-c (0.726; Z = 3.52, P < 0.001) models. In External Validation Dataset 2, the N2H3 model (0.782) demonstrated significantly better predictive accuracy compared with the THRIVE-c model (0.666; Z = 2.36, P < 0.001), but it was not significantly different from the START model (0.731; Z = 1.12, P = 0.264); nonetheless, a positive trend was observed.

Figure 3.

N2H3 model receiver operating characteristic curve (ROC) area = 0.810; START model ROC area = 0.729; THRIVE-c model ROC area = 0.726. ROC: Receiver operating characteristic curve

Figure 4.

N2H3 model receiver operating characteristic curve (ROC) area = 0.782; START model ROC area = 0.731; THRIVE-c model ROC area = 0.666. ROC: Receiver operating characteristic curve

Mortality and symptomatic intracerebral hemorrhage

In addition, we further explored whether N2H3 can be used to predict mortality and sICH. Among 794 patients, 27 (3.40%) died 3 months after IVT. The N2H3 model achieved an AUC-ROC of 0.899 (95% CI 0.851–0.948) in predicting mortality. The percentage of patients with sICH within 24 h was 1.26% (10/794), and the AUC-ROC of N2H3 to predict sICH was 0.591 (95% CI: 0.532–0.649). The results indicated that N2H3 may have the potential to predict all-cause death, warranting further studies. However, the ability of N2H3 to judge sICH may be limited.

Discussion

Our results revealed that the N2H3 model demonstrated good predictive capability for identifying poor 3-month prognoses in patients with AIS undergoing IVT. Furthermore, it significantly outperformed the THRIVE-c nomogram and was not inferior to the START nomogram in this regard.

The N2H3 model performed well in two validation cohorts. The possible explanations for this are presented below. First, the predictive factors included in the N2H3 model were more rigorously tested to evaluate their potential roles in improving the predictive performance of the model. To construct the N2H3 prognostic model, we collected clinical and follow-up information from patients, including baseline demographics, vascular risk factors, and laboratory tests, and statistical analysis was rigorously performed among all these factors. By contrast, the four predictive factors included in the START nomogram were determined solely by three neurologists based on their clinical expertise.[16] This is obviously not scientific enough. Second, laboratory information for Hhcy and HDL-C/LDL-C included in the N2H3 model represents an objective indicator. In contrast, traditional vascular risk factors such as diabetes, age, smoking habits, and high blood pressure which are too dependent on the memories of family members do not completely account for outcomes after a stroke. Third, because IVT is the treatment with the highest level of evidence for patients with AIS,[20,21,22] our model included a variable which reflected the efficacy of IVT.[19] A previous study found that delta NIHSS serves as the stronger predictor of outcomes when evaluating an intervention in the acute period.[30,31,32] Therefore, we considered that the innovative inclusion of delta NIHSS further promoted the N2H3 model’s predictive ability.

Compared with several other predictive models for patients who underwent IVT, such as the iScore, the DRAGON score, the ASTRAL score, the Stroke-TPI model, and the SPAN-100 index, the N2H3 model demonstrates advantages across various dimensions. First, the N2H3 model (AUC-ROC: 0.872) has a good ability to predict the 3-month outcomes of patients who received IVT,[19] while the AUC-ROCs of the ASTRAL, DRAGON, and Stroke-TPI scores are 0.850, 0.840, and 0.788, respectively.[10,13,33] Second, the five predictor variables included in the N2H3 (initial NIHSS, delta NIHSS, hypertension, Hhcy, and HDL-C/LDL-C) model are simple and easy to obtain compared with the iScore and DRAGON score. The iScore includes the predictor variable of stroke subtypes which requires an exhaustive examination to complete.[34] Similarly, the results of arterial computed tomography/magnetic resonance imaging in DRAGON are also not simple or necessarily feasible.[35] Third, the N2H3 model has wider clinical applicability. Saposnik et al. developed the SPAN-100 index,[14,36] which incorporates only two predictor variables: age and baseline NIHSS score. Therefore, for younger patients, the total score might be still low even if NIHSS scores were high, suggesting that the predictive accuracy of the SPAN-100 index may not be satisfactory enough. In addition, nomograms have been found to be more accurate for predicting stroke outcomes than physicians’ opinions, even for those with substantial levels of experience, particularly given the extensive range of patient parameters considered in the scenarios.[37]

We consider that our study is necessary and promotes the application of the N2H3 model in clinical practice, since we confirmed the good predictive capability of the N2H3 model for identifying poor 3-month prognosis in patients with AIS undergoing IVT by external validation. External validation is essential for the widespread application of predictive models. For example, the ABCD2 score (predicting stroke risk after the transient ischemic attack) and the CHA2DS2-VASc score (assessing the risk of ischemic stroke in patients with nonvalvular atrial fibrillation) have been validated in external cohorts before commonly used in clinical practice.[38,39,40,41] In addition, we further compared the predictive performance of the N2H3 model and that of other commonly used models in this study. We found the N2H3 model not only significantly outperformed the THRIVE-c nomogram in predicting 3-month prognosis but also had a good ability to predict the mortality for patients after IVT.

Our study is accompanied by several notable limitations. First, we did not achieve the expected result in terms of the prediction of sICH. This can be attributed to the fact that the N2H3 model was initially designed to focus on patients’ 3-month prognosis outcomes, which was not suitable for the prediction of sICH. Future research should aim to refine this model using larger samples from diverse geographic locations. Second, additional neuroprotective drugs[42,43,44] and intensive rehabilitation may have also affected the prognoses of the patients. However, patients with calculated predictions indicating positive outcomes should not withdraw from beneficial treatments.

Conclusions

The N2H3 model exhibited good predictive ability in external validation cohorts and was able to accurately identify patients with poor prognoses. Moreover, it demonstrated advantages over the THRIVE-c and START models. Therefore, the N2H3 model may provide significant information to neurologists during prognosis discussions with patients and their families. In addition, it can aid in promptly identifying patients suitable for intensive rehabilitation, proving beneficial in clinical settings.

Author contributions

1. H-MZ, Z-NG and YY contributed to the study conception and design and drafting the manuscript; 2. H-MZ, YQ, PZ, RA, L-JW, YL, Y-MC, A-RL, X-DL, and L-LZ, contributed to the data acquisition, analysis, and interpretation; 3. C-YY, JY, A-YW, Y-FJ, J-CW, C-PD, F-FL, LL, and Y-BQ, drafted and critically revised the manuscript for important intellectual content; 4. H-MZ , Z-DS, C-FW, HL, L-YZ and W-JM agree to be accountable for all aspects of the work by ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Ethical policy and institutional review board statement

The study was conducted according to the principles of the Declaration of Helsinki and was approved by the Ethics Committee of the First Hospital of Jilin University (approval code: 2015-156, dated on Mar 2^nd, 2015). Written informed consent for this study was obtained from all study participants or their direct relatives.

Data availability statement

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Conflicts of interest

There are no conflicts of interest.

Funding Statement

This project was supported by the National Natural Science Foundation of China (U24A20686, 82071291), the Norman Bethune Health Science Center of Jilin University (2022JBGS03), Science and Technology Department of Jilin Province (YDZJ202302CXJD061, 20220303002SF) and Jilin Provincial Key Laboratory (YDZJ202302CXJD017) to YY, the Talent Reserve Program of the First Hospital of Jilin University (JDYYCB-2023002) to ZNG.