Towards personalised intra-arterial treatment of patients with acute ischaemic stroke: a study protocol for development and validation of a clinical decision aid

Abstract

Introduction

Overall, intra-arterial treatment (IAT) proved to be beneficial in patients with acute ischaemic stroke due to a proximal occlusion in the anterior circulation. However, heterogeneity in treatment benefit may be relevant for personalised clinical decision-making. Our aim is to improve selection of patients for IAT by predicting individual treatment benefit or harm.

Methods and analysis

We will use data collected in the Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands (MR CLEAN) trial to analyse the effect of baseline characteristics on outcome and treatment effect. A multivariable proportional odds model with interaction terms will be developed to predict the outcome for each individual patient, both with and without IAT. Model performance will be expressed as discrimination and calibration, after bootstrap resampling and shrinkage of regression coefficients, to correct for optimism. External validation will be conducted on data of patients in the Interventional Management of Stroke III trial (IMS III). Primary outcome will be the modified Rankin Scale (mRS) at 90 days after stroke.

Ethics and dissemination

The proposed study will provide an internationally applicable clinical decision aid for IAT. Findings will be disseminated widely through peer-reviewed publications, conference presentations and in an online web application tool. Formal ethical approval was not required as primary data were already collected.

Trial registration numbers

ISRCTN10888758; Post-results and NCT00359424; Post-resultsc.

Strengths and limitations of this study.

• Multiple characteristics will be evaluated simultaneously to show clinically relevant heterogeneity in treatment benefit between patients.

• Multivariable prediction modelling substantially increases statistical power compared with other approaches and is more robust, especially in small data sets.

• We will use a relatively small cohort for the development of a prediction model.

• Using a proportional odds model requires the assumption that the ORs are the same for each cut-off of the modified Rankin Scale.

Introduction

In 2015, five consecutive randomised controlled trials (RCTs) showed that intra-arterial treatment (IAT) improves functional outcome in patients with a proximal occlusion in the anterior circulation.^1–6 This was a major breakthrough in the field, and IAT is now implemented in updated guidelines on acute ischaemic stroke (AIS) management.⁷

Ideally, IAT will be targeted at patients who are expected to have optimal benefit: personalised treatment. In this study protocol, we present seven steps for development and validation of a clinical decision aid to predict which individual patients with AIS will benefit most from IAT.^{8 9}

Methods and analysis

Step 1: problem definition and data inspection

Problem definition

RCTs provide estimates of treatment effects for average patients. However, it is important to take potential heterogeneity of treatment effects into account. Clinically relevant differences in the absolute effect of a treatment can be caused by (1) differences in the relative treatment effect (predictive effects) and (2) differences in baseline risk on the outcome of interest (prognostic effects).^{10 11} For example, in the Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands (MR CLEAN) trial, there is no predictive effect of age; the relative treatment effect is constant across age subgroups.¹ This is demonstrated by a non-significant test for interaction between age and treatment (figure 1A). However, variation in baseline risk on favourable outcome according to age results in a larger absolute treatment benefit in younger patients (figure 1B).

Figure 1.

Relative risk (A) and absolute risk difference (B) for good functional outcome (mRS 0–2) in MR CLEAN sort by age. MR CLEAN, Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands; mRS, modified Rankin Scale.

Conventional subgroup analyses are focused mainly on predictive effects and assess the effect of only one variable at a time. If predictive and prognostic effects of multiple characteristics are evaluated simultaneously in multivariable prediction modelling, it is likely that larger heterogeneity in treatment benefit between individual patients will be found. Our aim is to improve selection of patients for IAT by predicting treatment benefit or harm for individual patients with stroke.

Development data

We will use data of the MR CLEAN trial (n=500), which was a phase 3 multicentre clinical trial with randomised treatment group assignment, open-label treatment and blinded end point evaluation. IAT plus usual care (which could include intravenous administration of alteplase) was compared with usual care alone. IAT consisted of arterial catheterisation with a microcatheter to the level of occlusion and delivery of a thrombolytic agent, mechanical thrombectomy or both.¹

Severity of stroke was assessed at baseline with the National Institutes of Health Stroke Scale (NIHSS; range 0–42). Baseline CT was evaluated with the Alberta Stroke Program Early CT Score (ASPECTS; range 0–10). Baseline imaging (CT angiography) was used to determine the location of occlusion and to grade the quality of collateral flow to the ischaemic area with a four-point scale. Detailed information about the MR CLEAN trial can be found in the study protocol and the publication of the main results.^{1 12}

End points of interest

The primary outcome will be the modified Rankin Scale (mRS), a seven-point scale ranging from 0 (no symptoms) to 6 (death) at 90 days after stroke.¹³ We will provide estimates of treatment benefit as the absolute increase in probability on functional independence (defined as mRS 0–2) and survival (defined as mRS 0–5).

Step 2: coding of variables

As variables, we will use patient characteristics that are expected to predict outcome, or that are expected to interact with treatment, based on expert opinion and the recent literature (table 1). Non-linearity of continuous variables will be tested by comparing the two log likelihood of models with linear and restricted cubic spline functions.¹⁴

Table 1.

Patient characteristics that are expected to predict outcome (prognostic), or that are expected to interact with treatment (predictive)

	Per cent of data complete in MR CLEAN	Prognostic	Predictive
Clinical
Age6 15	100%	x
Baseline NIHSS16 17	100%	x
History of diabetes mellitus18	100%	x
History of previous stroke19	100%	x
History of atrial fibrillation20	100%	x
Prestroke mRS score19	100%	x
Systolic blood pressure21	100%	x
IV treatment with alteplase22–24	100%	x
Time from onset stroke to groin puncture25 26	100%*	x	x
Radiological
ASPECTS6 27	99.2%	x
Location of intracranial occlusion on non-invasive vessel imaging28 29	99.8%	x
Collateral score on CTA29 30	98.4%	x	x

*Of patients undergoing intra-arterial treatment.

ASPECTS, Alberta Stroke Program Early CT score; CTA, CT angiography; IV, intravenous; NIHSS, National Institutes of Health Stroke Scale; MR CLEAN, Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands; mRS, modified Rankin Scale.

Timing of treatment is an essential predictor of outcome. Since time to randomisation was not a reliable indicator for time to treatment in the MR CLEAN trial and will not be applicable in clinical practice, we will use time from stroke onset to groin puncture. Since time to groin puncture is not observable in the control group, we will explore imputation approaches based on the correlation with time to randomisation. All other baseline variable values are more than 98% complete in the MR CLEAN data, so we choose simple imputation by the mean for continuous variables and simple imputation by the mode for categorical variables.

Steps 3 and 4: model specification and estimation

We will test the effect of variables on functional outcome and treatment effect with proportional odds regression modelling. All variables from table 1 will be tested for effect on outcome and interaction with treatment effect. Prognostic variables (main effects) and predictive variables (interaction effects) with a p value of 0.15 in univariable and multivariable analyses will be included in our final model. A p value of 0.15 was chosen to make the predictor selection less data driven and prevent overfitting.^{14 31} We will perform shrinkage of all regression coefficients with ridge regression to prevent overfitting of the model.¹⁴ Predicted probabilities for each of the mRS categories, with and without treatment, will be derived from the ordinal model. All statistical analyses will be performed within the computing environment R V.3.2.2 (The R Foundation)

Step 5: model performance

Model performance will be expressed in discrimination and calibration. Discrimination will be quantified with the c-statistic. The c-statistic is similar to the area under the curve for binary outcomes and estimates the probability that out of two randomly chosen patients, the patient with the higher predicted probability of a good outcome will indeed have a better outcome. Calibration refers to the agreement between predicted and observed risks and will be assessed graphically with validation plots, and expressed as calibration slope and an intercept. The calibration slope describes the relative overall effect of the variables in the validation sample, and is ideally equal to 1. The intercept indicates whether predictions are systematically too high or too low, and should ideally be 0.³² We will calculate a general c-statistic to express the performance of our ordinal model and additional calibration plots with specific c-statistics for the predictions of favourable functional outcome (mRS 0–2) and survival (mRS 0–5).

Step 6: model validity

The c-statistic will be internally validated with a bootstrap procedure (500 samples with replacement) to estimate the degree of optimism in parameter estimates.⁸ After penalisation of the regression coefficients, we will externally validate the model on data of patients in the Interventional Management of Stroke III trial (IMS III) with an occlusion in the anterior circulation on non-invasive vessel imaging.³³ Coefficients of the final model will be fitted on the combined development and validation data sets.

After validation, we will assess whether the model can be used to discriminate between patients with low and high expected benefit by making individual predictions of outcome for all patients included in the development and validation data.

Step 7: model presentation

The final model will be digitally available for use in clinical practice, both for mobile devices and as a web application. It will provide predictions of all mRS categories for each individual patient, both with and without IAT.

Ethics and dissemination

Findings will be disseminated widely through peer-reviewed publications, conference presentations and in an online web application tool. Formal ethical approval was not required for this study as primary data were already collected.

Discussion

Compared with the current subgroup analyses on the effect of IAT, our modelling approach has multiple advantages. First, it accounts for the fact that patients have multiple characteristics that simultaneously affect the likelihood of treatment benefit.³⁴ Thus, our model will show more clinically relevant heterogeneity in treatment benefit between patients. Second, a multivariable prediction model substantially increases statistical power to identify heterogeneity in treatment effects compared with other approaches.³⁵ These include neural network and decision trees. We use regression modelling since it is considered more robust, especially in relatively small data sets.^{36 37}

There are some differences between patients included in the MR CLEAN trial and the IMS III trial that may influence the external validity of our model. IMS III had different inclusion criteria, used older devices and older treatment paradigms than MR CLEAN. In order to overcome these limitations, we will use only those patients in IMS III with an occlusion in the intracranial anterior circulation on non-invasive vessel imaging. We will compare the baseline characteristics of the derivation and validation cohort and describe relevant differences that might lead to an underestimation or overestimation of the model performance. Interestingly, a substantial treatment effect in the IMS III patients with proven intracranial large vessel occlusion has been reported.³⁸

Furthermore, even though the MR CLEAN trial has included most patients of the recent RCTs, the cohort remains relatively small for the development of a prediction model, especially for the selection of the main effect and interaction effects. We will reduce regression coefficients to prevent overfitting and also perform external validation. In the future, we will further validate and update our model in the pooled individual patient data of the Highly Effective Reperfusion evaluated in Multiple Endovascular Stroke Trials (HERMES) collaboration, harbouring data of all patients from recent randomised trials regarding IAT (over 1700 patients in total). Moreover, we aim to investigate the validity of our model predicting outcome after treatment in clinical practice. Our model will therefore be tested by applying it to recently treated patients in all Dutch neurovascular centres participating in the MR CLEAN Registry (mrclean-trial.org).

We will use a proportional odds model to analyse the full mRS score as outcome. Formally, this model requires the assumption that the ORs are the same for each cut-off of the mRS. However, previous studies have shown that even if the proportionality assumption is violated, proportional odds analysis is still more efficient than dichotomisation.³⁹ In addition, all recent RCTs on the effect of IAT used the full mRS and analysed their results with proportional odds regression.

Conclusion

The proposed study will provide an internationally applicable clinical decision aid for the selection of patients for IAT. We consider this study an important next step towards personalised treatment of patients with AIS.