Predicting the risk of stroke among patients with type 2 diabetes: a systematic review and meta-analysis of C-statistics

Abstract

Objective

Stroke is a major cause of disability and death worldwide. People with diabetes are at a twofold to fivefold increased risk for stroke compared with people without diabetes. This study systematically reviews the literature on available stroke prediction models specifically developed or validated in patients with diabetes and assesses their predictive performance through meta-analysis.

Design

Systematic review and meta-analysis.

Data sources

A detailed search was performed in MEDLINE, PubMed and EMBASE (from inception to 22 April 2019) to identify studies describing stroke prediction models.

Eligibility criteria

All studies that developed stroke prediction models in populations with diabetes were included.

Data extraction and synthesis

Two reviewers independently identified eligible articles and extracted data. Random effects meta-analysis was used to obtain a pooled C-statistic.

Results

Our search retrieved 26 202 relevant papers and finally yielded 38 stroke prediction models, of which 34 were specifically developed for patients with diabetes and 4 were developed in general populations but validated in patients with diabetes. Among the models developed in those with diabetes, 9 reported their outcome as stroke, 23 reported their outcome as composite cardiovascular disease (CVD) where stroke was a component of the outcome and 2 did not report stroke initially as their outcome but later were validated for stroke as the outcome in other studies. C-statistics varied from 0.60 to 0.92 with a median C-statistic of 0.71 (for stroke as the outcome) and 0.70 (for stroke as part of a composite CVD outcome). Seventeen models were externally validated in diabetes populations with a pooled C-statistic of 0.68.

Conclusions

Overall, the performance of these diabetes-specific stroke prediction models was not satisfactory. Research is needed to identify and incorporate new risk factors into the model to improve models’ predictive ability and further external validation of the existing models in diverse population to improve generalisability.

Strengths and limitations of this study.

• The breadth of the comprehensive systematic literature search is a strength of this study.

• To our knowledge, this is the first study where a meta-analysis and study quality assessment was performed on stroke prediction models in patients with diabetes.

• We were only able to use C-statistics to compare the model performance, which might be insensitive to identify differences in models’ ability to accurately risk-stratify patients into clinically meaningful risk groups.

Introduction

Stroke, also known as a cerebrovascular accident, is the third leading cause of disability and accounted for over 6 million deaths worldwide in 2015.^{1 2} Diabetes mellitus, characterised by chronic hyperglycaemia due to an absolute or relative deficiency in insulin, is a major risk factor for stroke. People with diabetes are at a twofold- to fivefold increased risk for stroke compared with people without diabetes.^3–7 Large clinical trials performed in people with diabetes supports the need for targeted cardiovascular risk reduction strategies to prevent the onset, recurrence and progression of acute stroke.⁸

Risk prediction models are statistical tools to estimate the probability that an individual with specific risk factors (eg, diabetes mellitus) will develop a future condition, such as stroke, within a certain time period (eg, 5 years).⁹ Such tools for the estimation of stroke risk are frequently used to assist in decisions about clinical management for both individuals and populations. Accurate risk prediction of stroke is thus necessary to provide patients with accurate information on the expected benefit from a therapy or intervention. The importance of well-performing prediction models is increasingly being recognised and health researchers continue to develop parsimonious risk prediction models under different scenarios to meet this demand. Model performance statistics, such as C-statistic or AUC (area under the receiver operating characteristic curve) are indicators frequently used to identify models with the best predictive ability. These metrics can be compared and assessed through a formal systematic review and meta-analysis. Performing a systematic review and meta-analysis can also provide a comprehensive quantitative summary of the predictive ability of these models and evaluate their predictive performance within the available literature.

Risk factors for stroke include lifestyle-related factors,^{10 11} predisposing medical conditions,^{10 12} specific genetic diseases,^{13 14} as well as sociodemographic factors.^{11 12} Over the past decade, a number of prediction models (or risk scores) have been developed incorporating these risk factors to predict a person’s risk of developing stroke.¹⁵ Prediction of stroke is important for a number of reasons: to detect or screen high-risk subjects to prevent developing stroke through early interventions, to facilitate patient–doctor communication based on more objective information and to help patients to make an informed choice regarding their treatment. While multiple stroke prediction models have been proposed in patients with diabetes, little is known about which is the most accurate one. There has also been a lack of consistency in estimating risk across these different models. With this in mind, we aimed to systematically identify all prediction models for stroke that have been applied to patients with diabetes. We characterised the study populations in which these models were derived and validated. We also assessed the predictive performance and generalisability of these stroke prediction models so that the selection of models for clinical implementation can be informed.

Methods

Data sources and searches

Similar to previously employed methodology,¹⁶ we searched MEDLINE, EMBASE and PubMed (from database inception to 22 April 2019) for studies predicting the risk of stroke among patients with diabetes. We also searched the reference lists of all identified relevant publications. The search strategy focused on three key elements: diabetes, risk prediction with specific names of known risk scores and stroke. Only studies published in English were considered. The detailed search strategy is given in online supplemental table S1.

Study selection

Eligible articles were identified by two reviewers independently using a two-step process. First, an initial screen of titles and abstracts was performed. Abstract were retained if they reported data from an original study and reported on the development and/or validation of a stroke risk prediction model for patients with type 2 diabetes. We defined a stroke risk prediction model as one combining two or more independent variables to obtain estimates of the predicted risk for developing stroke. We considered any clinical-based or laboratory-based definition of stroke. Selected abstracts were further screened based on a full-text review. We used broad inclusion criteria to provide an extensive systematic review of the topic. There were no restrictions on study design (eg, cohort study, case–control study), geographic region or age ranges. Studies that developed prediction models for stroke in populations with type 2 diabetes and in the general population were included; however, models that were developed in the general population but did not validate their model in a type 2 diabetes population or models developed on a type 1 diabetes population were excluded. A study was included if the outcome of the prediction model was any type of stroke or stroke that was part of a composite cardiovascular disease (CVD) outcome, but excluded if the outcome was any other cardiovascular conditions (eg, coronary heart disease (CHD), coronary artery disease (CAD), heart failure). We excluded studies that did not predict stroke. Studies on recurrent stroke or other vascular conditions (eg, patients with hypertension) were also excluded. Studies that focused only on the added predictive value of new risk factors to an existing prediction model without reporting the performance of the existing model were excluded. Studies on score-based tools, such as risk charts were also excluded. Agreement between reviewers at the full-text stage was quantified using the kappa statistic. Any disagreement between reviewers was solved through consensus.

Data extraction

Data were extracted from the finally selected studies using a standardised form by two reviewers. Information collected from each study included, outcome of the prediction model, location where the model was developed, predictors included in the model, age and gender of the study participants, number of events, duration of follow-up, modelling method used, measures of discrimination and calibration of the prediction model and the external validation of the prediction model. For the external validation studies, a different data extraction sheet was used. The collected information included specifics of the validation population, number of events, type of outcome, statistical tests and measures of discrimination, and calibration of the prediction model. Study quality was assessed using the CHARMS (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies) Checklist.¹⁷ The following items were evaluated for each study: Was inclusion/exclusion criteria for study participants specified?; Was there non-biased selection of study participants?; Did the authors discuss or consider missing values/information?; Was there blinded assessment of the outcome?; Was duration of follow-up adequate?; Were modelling assumptions satisfied?; Was the model externally validated? and Was the potential clinical utility of the model discussed in light of study limitations?

Data analysis

The selection process for this systematic review and meta-analysis is summarised using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram.¹⁸ Discrimination is defined as any assessment of the ability of the model to differentiate between subjects who will develop stroke from those who will not. The discrimination of a prediction model is often assessed using the concordance or C-statistic (also known as AUC). Calibration is defined as any report of the agreement between predicted probabilities and observed probabilities. Calibration is assessed using goodness-of-fit tests (eg, Hosmer-Lemeshow test), calibration slopes, tabular or graphical comparisons of predicted versus observed values within groupings of predicted risk or calibration plots. In studies that only provided a C-statistic but no measure of its variance, the SE and 95% CI of the AUC (C-statistic) was calculated using the formula:

$${\displaystyle \mathrm{S}\mathrm{E}\left(\mathrm{A}\mathrm{U}\mathrm{C}\right)=\sqrt{\frac{\mathrm{A}\mathrm{U}\mathrm{C}\left[1-\mathrm{A}\mathrm{U}\mathrm{C}\right]+\left[{\mathrm{N}}_1-1\right]\times \left[\mathrm{A}\mathrm{U}\mathrm{C}/\left(2-\mathrm{A}\mathrm{U}\mathrm{C}\right)-{\mathrm{A}\mathrm{U}\mathrm{C}}^2\right]+\left[{\mathrm{N}}_2-1\right]\times \left[2{\mathrm{A}\mathrm{U}\mathrm{C}}^2/\left(1+\mathrm{A}\mathrm{U}\mathrm{C}\right)-{\mathrm{A}\mathrm{U}\mathrm{C}}^2\right]}{\left({\mathrm{N}}_1\times {\mathrm{N}}_2\right)}}}$$

where $N_1=$ the number of patients with stroke and $N_2=$ the number of patients without stroke and the upper 95% CI ${\displaystyle =\mathrm{A}\mathrm{U}\mathrm{C}+\left[1.96\times \mathrm{S}\mathrm{E}\left(\mathrm{A}\mathrm{U}\mathrm{C}\right)\right]}$, and lower 95% CI ${\displaystyle =\mathrm{A}\mathrm{U}\mathrm{C}-\left[1.96\times \mathrm{S}\mathrm{E}\left(\mathrm{A}\mathrm{U}\mathrm{C}\right)\right]}$.¹⁹ The summary statistic from the individual studies was the C- statistic or AUC. We grouped studies based on the outcome of the risk prediction models developed in diabetes populations, whether stroke was the primary outcome of the model or stroke was a part of composite CVD outcome. Random effects meta-analysis was used to obtain the pooled weighted average C-statistic with 95% CIs for common groups of models using the DerSimonian and Laird method.²⁰ Heterogeneity was assessed using the Cochran Q and the I² statistic and was explored using meta-regression and stratified analyses according to model outcomes. Small study effects were examined using funnel plots and Begg’s test. The analyses were performed using Stata version 13.1 (Stata, College Station, Texas, USA) using the metan, metareg, metabias and metafunnel commands.

Patient and public involvement

There was no direct patient or public involvement in this review.

Results

The search retrieved 21 797 citations (after duplicate removal) with an additional 63 potentially relevant papers retrieved from our grey literature search. After title and abstract screening, 262 studies were selected for full-text screening. After examining the full-text papers, 56 studies remained (reasons for exclusion stated in figure 1), describing 38 models predicting stroke in patients with diabetes. Agreement between reviewers on the final articles eligible for inclusion in the systematic review was good (κ=0.83). Of these 38 models, 34 were specifically developed in patients with diabetes and 4 were developed in the general population but later externally validated in patients with diabetes. Among the models developed in patients with diabetes, nine models reported their outcome as stroke and presented a corresponding performance measure (C-statistic) for the models. Twenty-three models reported their outcome as a composite CVD outcome where stroke was one of the components and presented the model’s performance measure (C-statistic) for the composite CVD outcome. Among the models developed in the general population, one model reported its outcome as stroke and three models reported a composite CVD outcome, which included stroke. Of these 38 prediction models, 17 were validated by 33 studies (some studies validated more than one model in the same study), of which 10 models had multiple validations, 7 models had a single validation and 21 models were not validated. Among the models with multiple validations, eight models were developed in a diabetes population (validated by 31 studies) and two were developed in the general population (validated by four studies). United Kingdom Prospective Diabetes Study (UKPDS) Risk Engine for Stroke by Kothari et al ²¹ was the most validated risk score (validated by 12 studies). Figure 1, describes the systematic selection process of studies presenting a stroke prediction model applicable to patients with diabetes.

Figure 1.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) diagram for systematic review of studies presenting stroke prediction models developed or validated in individuals with diabetes.

Predicting the risk of stroke within models developed in populations with diabetes

Table 1 describes the study characteristics of the nine risk prediction models developed in diabetes populations and presented a corresponding performance measure. The number of participants ranged from 1748 to 26 140 in the model development. The outcome of most models was stroke regardless of type. Duration of follow-up (total/median/mean) ranged from 501 days to 10.5 years with six models having ≥5 years of follow-up (defined as long duration) and three models with <5 years of follow-up. Most of the prediction models were developed using Cox proportional hazards modelling techniques. The number of predictors included in the prediction models ranged from 4 to 29 with an average of 11 predictors per model. Several predictors were common to multiple models including age, sex, duration of diagnosed diabetes, systolic blood pressure and haemoglobin A1c (HbA1c). Only two models were externally validated after their development and four of them had never been validated in an external population. Calibration of the prediction model was reported by six studies (most commonly using the Hosmer-Lemeshow test). Discrimination was assessed using the C-statistic (or AUC) and reported by six models with values ranging from 0.64 to 0.80. The median C-statistic of the models was 0.71 with a large amount of unexplained heterogeneity in the discriminative performance of these models (I²=94.6%; Cochran Q-statistic p<0.001; figure 2). Stratifying pooled results by sample size (small vs large, p=0.19), follow-up time (short vs long, p=0.60), variables included in the model (few vs many, p=0.24) and geographic location (Asia vs others, p=0.60) did not explain the observed heterogeneity in the discriminative performance of these models. The discriminative ability of the model by Kiadaliri et al ²² was highest (C-statistic=0.80). The funnel plot and Begg’s test (p>0.999) suggested the absence of small study effects, with no correlation between studies of smaller cohorts reporting higher C-statistics (online supplemental figure S1).

Table 1.

Characteristics of prediction models when outcome and corresponding performance measure (C-statistic) were reported for stroke

Study	Location	Outcome	No of Predictors	Age	Gender	Events (n)/total participants (n)	Duration of follow-up	Modelling method	Calibration	Discrimination (with CI)	External Validation
Yang et al 38	Hong Kong, China	Stroke (stroke or deaths from stroke), haemorrhagic stroke and ischaemic stroke	4 (age, A1C, spot urine ACR and history of CHD)	Median age 57 years	Both male and female	372/7209	Median follow-up 5.37 years	Cox proportional hazard model	The Life Table Method. Adequate calibration, value NR.	Adjusted: AUROC=0.776 (considering follow-up time and censoring); unadjusted AUROC=0.749 (0.716 to 0.782)	No
Kothari et al 21	UK	Stroke (defined as a neurological deficit with symptoms or signs lasting 1 month or more)	7 (duration of diabetes, age, sex, smoking, systolic blood pressure, total cholesterol to high-density lipoprotein cholesterol ratio and presence of atrial fibrillation)	25 to 65 years	Both male and female	188/4549	Median follow-up 10.5 years	Maximum likelihood estimation using the Newton-Raphson method	NR	NR	Yes
Wells et al 47	USA	CHD, heart failure, stroke, mortality	29 (different variables for different models)	18 years of age or older	Both male and female	Stroke: 1088/26 140	Median follow-up 501 days (Stroke model)	Competing risks regression model	Calibration plot (predicted risk against actual risk): less-well calibration (stroke and mortality)	C-statistic=0.6881 (stroke)	No
Stevens et al. UKPDS 6624	UK	MI case fatality and stroke case fatality	5 (sex, HbA1c, SBP, previous stroke, white cell count for Stroke model)	Between 25 and 65 years	Both male and female	Stroke: 234/5102	Median follow-up of 7 years	Stepwise selection algorithm	HL test: p=0.248 (Stroke model)	NR	No
Tanaka et alJJ Risk Engine48	Japan	CHD, stroke, non-cardiovascular mortality, overt nephropathy and progression of retinopathy	11 (sex, age, HbA1c, years after diagnosis, BMI, non-HDL cholesterol, ACR, atrial fibrillation, current smoker and leisure-time physical activity)	40–84 years	Both male and female	Stroke: 89/1748	Median follow-up of 7.2 years	Cox regression model	HL test: p=0.12 (Stroke model)	C-statistic=0.636 (0.564 to 0. 708) (Stroke model)	No
Palmer et al (IRS)23	Scotland	Fatal or non-fatal stroke	5 genotypes (IL-6 GG/GC, MCP- 1 GG, ICAM-1 EE, sel·E RR and MMP-3 5A5A)	Mean age 64.5±11.7 years	Both male and female	108/2182	Mean follow-up of 6.2±1.1 years	Cox regression model	NR	NR	No
Kiadaliri et al 22	Sweden	First and second events of: AMI, heart failure, non-acute ischaemic heart disease and stroke	12 (age, TC/HDL, diabetes duration, HbAIc, BMI, SBP and diastolic BP, history of events before diagnosis, LDL cholesterol, albuminuria, smoking status, BMI and gender)	Male : mean age, 55.36±9.28 years; Female: mean age, 57.15±9.55 years	Both male and female	993/21 775 (first stroke); 314/21 775 (second stroke)	82 232 person-years for first stroke and 4127 person-years for second stroke	Weibull proportional hazard model	HL χ2 statistic: 11.22 (p=0.19) (first stroke); 8.09 (p=0.43) (second stroke)	C-statistic=0.80 (0.78 to 0.82) (first stroke); C-statistic=0.74 (0.71 to 0.77) (second stroke)	No
Li et al 49	Taiwan	Ischaemic stroke	14 (age, gender, smoking habit, duration of type 2 diabetes, blood pressure, HbA1c level, total cholesterol to HDL ratio, creatinine, fasting plasma glucose variation, arterial embolism and thrombosis, diabetes retinopathy, hypoglycaemia, antidiabetes medication use and cardiovascular medication)	30–84 years	Both male and female	2091 (derivation set), 1076 (validation set)/18 750 (derivation set), 9374 (validation set)	Mean follow-up of 8 years	Cox proportional hazard regression model	NR	AUROC=0.72 (3 years); AUROC=0.71 (5 years); AUROC=0.68 (8 years)	No
Basu et al RECODe36	USA and Canada	Microvascular: nephropathy, retinopathy, neuropathy; Cardiovascular: composite of atherosclerotic cardiovascular disease (first fatal or non-fatal MI or stroke), fatal or non-fatal MI, fatal or non-fatal stroke, congestive heart failure, or death from any cardiovascular cause	14 (Age, sex, ethnicity, smoking, SBP, history of CVD, blood pressure-lowering drugs use, statin use, anticoagulants use, HbA1c, total cholesterol, HDL cholesterol, serum creatinine, urine ACR)	40–79 years	Both male and female	197 (stroke)/9635	Median follow-up of 4.7 years	Cox proportional hazard models	Calibration slope=1.16, χ2=7.4, p value=0.38 (internal validation); calibration slope=0.99, χ2=8.2, p value=0.22 (external validation)	C-statistic for stroke=0.70 (0.66 to 0.74) (internal validation); C-statistic for stroke=0.67 (0.63 to 0.71) (external validation)	Yes

ACR, albumin-to-creatinine ratio; AMI, acute myocardial infarction; AUROC, area under the receiver operating characteristic curve; BMI, body mass index; BP, blood pressure; CHD, coronary heart disease; CVD, cardiovascular disease; HDL, high-density lipoprotein; HL, Hosmer-Lemeshow; HbA1C, haemoglobin A1C; IRS, inflammatory risk score; JJ, The JDCS/J-EDIT; LDL, low-density lipoprotein; MI, myocardial infarction; NR, not reported; RECODe, Risk Equations for Complications of Type 2 Diabetes; SBP, systolic blood pressure; TC, total cholesterol; UKPDS, United Kingdom Prospective Diabetes Study.

Figure 2.

Forest plot of C-statistics, with 95% CIs of risk prediction models when outcome was reported for stroke.

A set of nine criteria was used to assess the quality of the studies and was summarised in table 2. All the studies specified inclusion/exclusion criteria. Non-biased selection of study participants was clear in all studies except the study by Palmer et al.²³ Handling of missing values was reported in four (44%) studies, modelling assumptions was satisfied by two studies and model external validation was performed in two studies. Stevens et al was the only study to mention whether the outcome was assessed without knowledge of the candidate predictors.²⁴ Duration of follow-up was long (≥5 years) in six models (67%). The clinical utility of the models was discussed in six (67%) studies and almost all studies reported their study limitations.

Table 2.

Study quality assessment of prediction models when outcome and corresponding performance measure (C-statistic) were reported for stroke

Study	Inclusion/exclusion criteria specified	Non-biased selection	Missing value/loss to follow-up considered	Modelling assumptions satisfied	Model external validation	Outcome assessed without knowledge of the candidate predictors (ie, blinded)	Duration of follow-up long enough	Potential clinical use of the model discussed	Study limitations discussed
Yang et al 38	Yes	Yes	Not clear	No	No	Not clear	Yes	Yes	Yes
Kothari et al 21	Yes	Yes	No	Yes	Yes	Not clear	Yes	Yes	Yes
Wells et al 47	Yes	Yes	Yes	No	No	Not clear	No	Yes	Yes
Stevens et al (UKPDS 66)24	Yes	Yes	Not clear	No	No	Yes	Yes	Yes	No
Tanaka et al (JJ Risk Engine)48	Yes	Yes	Yes	No	No	Not clear	Yes	No	Yes
Palmer et al (IRS)23	Yes	Not clear	No	No	No	Not clear	Yes	No	Yes
Kiadaliri et al 22	Yes	Yes	Yes	No	No	Not clear	No	No	Yes
Li et al 49	Yes	Yes	Yes	Yes	No	Not clear	Yes	Yes	Yes
Basu et al RECODe36	Yes	Yes	No	Not clear	Yes	Not clear	No	Yes	Yes

IRS, inflammatory risk score; JJ, The JDCS/J-EDIT; RECODe, Risk Equations for Complications of Type 2 Diabetes; UKPDS, United Kingdom Prospective Diabetes Study.

Predicting risk of stroke (as part of a composite CVD outcome) in populations with diabetes

We identified 23 models developed in diabetes populations that reported their outcome as a composite of CVD. A summary of the characteristics of these prediction models is described in table 3. The number of participants considered in the model development ranged from 132 to 1 81 619 with an average age of >50 years. Duration of follow-up (mean, median, maximum) ranged from 11 months to 11.8 years with 14 models with ≥5 years and nine models with <5 years of follow-up. The number of predictors included in prediction models ranged from 4 to 18 with an average of 11 predictors per model. The most common predictors included in the models were age, sex, systolic blood pressure and HbA1c, smoking and high-density lipoprotein-cholesterol. Four models were externally validated after its development and 17 of them had never been validated in an external population. Calibration of the prediction models was reported by 13 studies while discrimination reported by almost all studies with C-statistics ranging from 0.60 to 0.92. The median C-statistic of the models was 0.70 with a large amount of unexplained heterogeneity in the discriminative performance of these models (I²=93.7%; Cochran Q-statistic p<0.001; figure 3). Sample size (small vs large, p=0.46), models’ external validation (externally validated vs not externally validated, p=0.71), variables included in the model (few vs many, p=0.21) and geographic location (Europe vs others, p=0.08) were not identified as significant sources of heterogeneity in the discriminative performance of these models. The discriminative ability of the model by Ofstad et al ²⁵ was highest (C-statistic=0.92) when novel risk markers were added to their standard model. The funnel plot and Begg’s test (p=0.24) suggested the absence of small study effects, with no correlation between studies of smaller cohorts reporting higher C-statistics (online supplemental figure S2). Only four models were developed using logistic regression models while others were developed mostly using Cox proportional hazards models.

Table 3.

Characteristics of prediction models when stroke is reported as a part of composite CV outcome and performance measure (C-statistic) is presented for the composite CV outcome

Study	Location	Outcome	No of predictors	Age	Gender	Events (n)/total participants (N)	Duration of follow-up	Modelling Method	Calibration	Discrimination (with CI)	External Validation
Brownrigg et al 15	England	CVD events (non-fatal MI, coronary revascularisation, congestive cardiac failure, transient ischaemic attack and stroke)	6 (age, sBP, smoking status, LDL-C and HDL-C and peripheral neuropathy)	Mean age of 63.8 years	Both male and female	399/13 043	Total 2.5 years	Probability weighted Cox regression	χ2=121.2, p<0.001	C-statistic=0.661 (0.636 to 0.686) (with PN)	No
Khalili et al 50	Iran	CVD events (definite MI, probable MI, unstable angina, angiographic-proven CHD, stroke, death from CVD)	4 (BMI, waist circumference, WHR, and waist-to-height ratio)	Mean age 55.7 years (male), 52.7 years (female)	Both male and female	188/1010	Median follow-up 8.4 years	Cox proportional hazard model	NR	C-statistic=0.64 (0.58 to 0.70) (for diabetic men with WHR, model 2) and C-statistic=0.70 (0.65 to 0.75) (for diabetic women with WHR, model 2)	No
Cederholm et al 51	Sweden	Fatal or non-fatal CVD (CHD or stroke, whichever came first)	9 (A1C, age at the onset of diabetes, diabetes duration, sex, BMI, smoking, sBP, antihypertensive drugs and lipid-reducing drugs)	18–70 years	Both male and female	1482/11 646	Mean follow-up 5.64 years	Cox regression	HL test: χ2=4.29 (p=0.83) and the ratio of observed to predicted survival rates=0.999. Excellent calibration	C-statistic=0.70	No
Davis et al 33	Australia	CVD (hospitalisation for/with MI or stroke, and death from cardiac or cerebrovascular causes or sudden death)	7 (age, sex, prior CVD, ln (urinary albumin : creatinine ratio), lnHbA1c, ln(HDL-C), Southern European ethnic background and aboriginality)	Mean age 64.1 (38.7–83.7) years	Both male and female	185/1240	Mean follow-up 4.5 years	Cox proportional hazards model	HLˆC test, p=0.74	AUC=0.80, p<0.001	Yes
Kengne et al 34	20 Countries (Asia, Australasia, Europe and Canada)	CVD (fatal or non-fatal MI or stroke or CV death)	10 (age at diagnosis, known duration of diabetes, sex, pulse pressure, treated hypertension, atrial fibrillation, retinopathy, HbA1c, log of urinary albumin/creatinine ratio and non-HDL-C at baseline)	Mean age 65.8 (6.3) years	Both male and female	473/7168	4.5 years	Cox regression model	HL test: p=0.76 (ADVANCE cohort)	AUC=0.702 (0.676 to 0.728) (ADVANCE cohort)	Yes
Ofstad et al 25	Norway	Death or first CV event (MI, stroke or hospitalisation for unstable angina pectoris)	11 (age, gender, known CVD, dB, microalbuminuria, serum levels of HDL-C and creatinine); novel risk markers: (IL-6, log Activin A, E/Em, pathol recovery loop)	Mean age 58.5±10.0 (SD) years	Both male and female	36/132	8.6±2.1 years	Cox proportional hazard model	NR	C-statistic: STD model: 0.794; STD + IL-6 model: 0.913; STD + log Activin A model: 0.859; STD + IL-6 + log Activin A model: 0.923; STD + E/Em + pathol recovery loop model: 0.891	No
Looker et al 52	Five cohorts from Europe	CVD (acute CHD or an ischaemic stroke)	14 (age, sex, smoking, sBP and dBP, LDL-C, HDL-C, triacylglycerol, diabetes duration, HbA1C, BMI, height, (eGFR), cohort and current medication (including antihypertensive agents, aspirin, lipid-lowering agents and insulin therapy)). +6 Biomarkers (NT-proBNP apoCIII hsTnT IL-6 sRAGE IL-15)	Median age 68.4 years (controls) and 68.8 years (cases)	Both male and female	1123/2310	Median follow-up 3.2 years for cases and 6.5 years for controls	Forward selection using logistic regression	NR	AUROC=0.72 (full clinical covariate set plus forward selection biomarkers)	No
Mukamal et al 32	USA	MI, stroke, CV death	7 in basic model (Age, smoking, sBP, total and HDL-C, creatinine and the use of glucose-lowering agents)	Mean age 72.6 years for female and 73.0 years for male	Both male and female	265/782	10 years	Cox proportional hazard model	Basic model: HL p=0.25; basic model+CRP: HL p=0.87; Basic Model+CRP + (ABI, internal carotid wall thickness, ECG left ventricular hypertrophy): HL p=0.65	Basic model: C -statistic=0.64; Basic model+CRP: C- statistic=0.64; Basic model+CRP + (ABI, internal carotid wall thickness, ECG left ventricular hypertrophy): C-statistic=0.68	Yes
Paynter et al 53	USA	MI, ischaemic stroke, coronary revascularisation or CV death	8 in different models (age, sBP, total cholesterol, HDL-C, smoking, CRP, parental history of premature MI and HbA1c)	Median age 55 years for female and 67.8 years for male	Both male and female	125/685 (women); 170/563 (men)	Median follow-up 10.2 years (women); median follow-up 11.8 years (men)	Cox proportional hazards model	NR	C-statistic of model with HbA1c=0.692 (ATP III) and=0.697 (RRS) for women; C-statistic of model with HbA1c=0.602 (ATP III) and=0.605 (RRS) for men.	No
Price et al 54	Scotland	All CV events (fatal and non-fatal MI, angina, fatal IHD, fatal and non-fatal stroke and TIA)	18 (age, sex, baseline CVD status, duration, diabetes treatment, lipid-lowering drugs, BP-lowering drugs, smoking status, BMI, sBP, dBP, HbA1c, HDL-C, total cholesterol, eGFR, microalbuminuria and social status +NT-proBNP) (model D)	60–75 years	Both male and female	112/1066	4 years	Cox proportional hazards model	NR	C-statistic=0.748 (0.691 to 0.805) (model D)	No
Selby et al 55	USA	Macrovascular and microvascular complications (MI, other ischaemic heart disease, congestive heart failure, cerebrovascular accident, etc)	16 (outpatient diagnoses, inpatient events, age, antihypertensives, serum creatinine, diabetes treatment, mean HbA1c, albuminuria, primary care visits, outpatient diagnoses of obesity, outpatient ID diagnoses, mean total cholesterol, self-report of neuropathy, education, type of diabetes, sex)	Mean age of 60.8 years	Both male and female	1997/28 838	1 year	Logistic regression model	NR	AUC=0.64 (full model)	No
Zethelius et al 35	Sweden	Fatal/non-fatal CVD (the composite of CHD or stroke)	12 (onset age of diabetes, diabetes duration, total-cholesterol-to-HDL-C ratio, HbA1c, sBP, BMI, males sex, smoker, microalbuminuria, macroalbuminuria, atrial fibrillation, previous CVD)	30–74 years	Both male and female	2488/24 288	Mean follow-up of 4.8 years	Cox proportional hazard model	Modified HL χ2 statistic=0.13 (p=0.9)	C-statistic=0.71	No
Alrawahi et al 41	Oman	First fatal or non-fatal CHD, stroke, or PAD	7 (age, diabetes duration, HbA1c, total cholesterol, albuminuria, hypertension, BMI)	54.5±11.4 years	Both male and female	192/2039	Mean follow-up of 5.3 years	Cox regression model	NR	NR	No
Zarkogianni et al 56	Greece	Fatal or non-fatal CVD: stroke and CHD	16 (age, diabetes duration, BMI, glycosylated haemoglobin, pulse pressure, fasting glucose, total cholesterol, triglycerides, HDL-C, smoking habit, sex, hypertension, lipid-lowering therapy, aspirin, insulin therapy, parental history of diabetes)	58.56±10.70 years	Both male and female	41/560	5-year follow-up	Machine learning: HWNNs and SOMs	Brier score: 0.08±0.01 (HWNN-based ensemble 4); 0.07±0.01 (SOM-based ensemble 4); 0.007±0.02 (hybrid ensemble)	AUC=0.6764±0.1509 (HWNN-based ensemble 4); AUC=0.7054±0.1372 (SOM-based ensemble 4); AUC=0.7148±0.1573 (hybrid ensemble)	No
Price et al 57	Scotland	Fatal or non-fatal MI or stroke, angina, fatal IHD, TIA, coronary intervention	13 (age, sex, smoking, atrial fibrillation, CKD, arthritis, hypertension, BMI, SBP, total HDL-C, social status, baseline CVD status, lipid-lowering medication) in basic model + (ABI, hs-cTnT, GGT, proBNP, g) in full model	60–75 years	Both male and female	205/1066	8 years	Binary logistic regression	HL p=0.97 (basic model); HL p=0.39 (full model). Well calibrated	C-statistic=0.722 (0.681 to 0.763) (basic model); C-statistic=0.74 (0.699 to 0.781) (full model)	No
Wan et al 58	China	IHD, MI, coronary death and sudden death, heart failure, fatal and non-fatal stroke	13 (age, eGFR, total cholesterol/HDL-C ratio, urine ACR, smoker, duration of diabetes mellitus, sBP, HbA1c, anti-hypertensive drugs used, dBP, BMI, insulin used, anti-glucose oral drugs used)	18–79 years	Both male and female	Events (n) NR/137 935	Median follow-up of 5 years	Cox proportional hazard regression	Calibration plots: good calibration	Harrell’s C-statistic Male: 0.705 (0.693 to 0.716) (model 1), 0.689 (0.678 to 0.701) (model 2); Female: 0.719 (0.707 to 0.731)(model 1), 0.708 (0.696 to 0.719) (model 2)	No
Young et al 59	USA	MACE: non-fatal MI, non-fatal stroke and CVD-related death; MACE-plus: any MACE, hospitalisation for unstable angina, or hospitalisation for congestive heart failure; CVD-related death	12 (age, gender, type of insurance, race, region, diabetes-related hospitalisations, prior CVD diagnoses, chronic pulmonary disease, use of antihypertensive drugs, use of antihyperglycaemic drugs, HbA1c, urine ACR)	50 years or older	Both male and female	13 856 (MACE), 20 100 (MACE-plus)/181 619	Median duration of the at-risk period: 12 months (primary prevention population) and 11 months (secondary prevention population)	Logistic regression	NR	C-statistic=0.70 (MACE); C-statistic=0.72 (MACE-plus); C-statistic=0.77 (CVD-related death)	No
van der Leeuw et al 60	The Netherlands	Major CV events (MI, stroke and vascular death)	12 (age at diabetes diagnosis, duration of diagnosed diabetes, sex, smoking, HbA1c, sBP, total cholesterol/ HDL-C ratio, previous CV event, urinary ACR or eGFR) in base model+NT-proBNP, osteopontin, and MMP-3 in multimarker model	Mean age 59±10 years (SMART), 58±7 (EPIC-NL)	Both male and female	248 (SMART), 134 (EPIC-NL)/1002 (SMART), 218 (EPIC-NL)	Median follow-up 9.2 years in SMART and 11.3 years in EPIC-NL	Cox proportional hazard model	Calibration plots	Base model: C-statistic=0.70 (0.67 to 0.74) (SMART), C-statistic=0.69 (0.64 to 0.74) (EPIC-NL); Multimarker model: C-statistic=0.73 (0.68 to 0.79) (SMART), C-statistic=0.72 (0.64 to 0.77) (EPIC-NL)	No
Alshehry et al 61	20 countries from Asia, Australasia, Europe and North America	Non-fatal MI, non-fatal stroke, and CV death	14 (age, sex, BMI, SBP, glycohaemoglobin, HDL-C, eGFR, diabetes duration, CRP, history of macrovascular disease, history of heart failure, use of antihypertensive medication, use of antiplatelet medication, exercise) in base model + 7 lipid species	Mean age 67 years	Both male and female	698/3779	Median follow-up of 5 years	Weighted Cox regression	NR	Base model: C-statistic=0.68 (0.678 to 0.682); base model + 7 lipid species: C-statistic=0.70 (0.698 to 0.702)	No
Woodward et al The AD-ON Risk Score62	20 countries from Asia, Australasia, Europe and North America	Non-fatal MI, non-fatal stroke or death from any CV cause, renal death or requirement for renal replacement therapy or renal transplantation	13 (age, sex, sBP with and without use of antihypertensives, duration of diabetes, HbA1c, urinary ACR, eGFR and its square, age at completion of formal education, exercise, history of diabetic retinopathy and current or previous atrial fibrillation)	Mean age of 65.8 years	Both male and female	1145/7301	Median follow-up of 9.9 years	Cox regression model	Calibration plots and HL test (p=0.13). Excellent calibration	C-statistic=0.668 (0.651 to 0.685)	No
Parrinello et al 63	USA	Incident CHD, stroke, heart failure, CKD, lower extremity amputation or peripheral vascular bypass	18 (age, sex, race, education, smoking status, alcohol consumption, physical activity, family history of CVD, glucose-lowering medication use, antihypertensive medication use, cholesterol-lowering medication use, recent onset of diabetes, BMI, LDL-C, HDL-C, triglycerides, sBP, HbA1c) + 12 biomarkers	Mean age of 58.1 years	Both male and female	141 (CVD events)/654	Maximum follow-up of 10 years	Fine and Gray model	Calibration plots: well calibrated	C-statistic=0.667 (0.64 to 0.70) (model 1); C-statistic=0.683 (0.65 to 0.71) (model 2); C-statistic=0.694 (0.66 to 0.72) (model 3); C-statistic=0.716 (0.69 to 0.74) (model 4)	No
Colombo et al 64	UK and Ireland	Acute CHD (MI, unstable angina, revascularisation or acute CHD death), fatal or non-fatal stroke	8 (age, sex, SBP, total cholesterol, HDL-C, smoking status, apoCIII and NT-proBNP)	Median age of 62.9 years	Both male and female	144/2105	Maximum follow-up of 5 years	Cox proportional hazard model	NR	AUROC=0.661 (0.615 to 0.706) (Framingham covariates alone); AUROC=0.745 (0.701 to 0.789) (full model with additional biomarkers)	No
Elley et al NZ DCS40	New Zealand	Fatal or non-fatal CVD event (ischaemic heart disease, cerebrovascular accident/transient ischaemic attack, PAD)	9 (age at diagnosis, diabetes duration, sex, sBP, smoking status, total cholesterol: HDL ratio, ethnicity, glycated HbA1C), urine ACR)	Median age of 59 years	Both male and female	6479/36 127	Median follow-up of 3.9 years	Cox proportional hazards regression models	Calibration plot	AUROC=0.68 (0.67 to 0.70) (CVD)	Yes

ABI, ankle–brachial index; AD-ON, Action in Diabetes and Vascular Disease: Preterax and Diamicron Modified Release Controlled Evaluation-Observational;ADVANCE, Action in Diabetes and Vascular Disease: Preterax and Diamicron Modified Release Controlled Evaluation; apoCIII, Apolipoprotein C-III; ATP, Adult Treatment Panel; AUC, area under the curve; AUROC, area under the receiver operating characteristic curve; BMI, body mass index; CHD, coronary heart disease; CKD, chronic kidney disease; CRP, C reactive protein; CV, cardiovascular; CVD, cardiovascular disease; dBP, diastolic blood pressure; eGFR, estimated glomerular filtration rate; EPIC-NL, European Prospective Investigation into Cancer and Nurition Netherlands;GGT, gamma-glutamyl transferase; HbA1C, haemoglobin A1C; HDL, high-density lipoprotein; HDL-C, high-density lipoprotein cholesterol; HL, Hosmer-Lemeshow; HL^∧C, Hosmer-Lemeshow C-test;hs-cTnT, high-sensitivity cardiac troponin T; HWNNs, hybrid wavelet neural networks; ID, infectious disease; IHD, ischaemic heart disease;IL-6, interleukin 6; LDL-C, low-density lipoprotein cholesterol; MACE, major adverse cardiovascular event; MI, myocardial infarction;MMP-3, matrix metalloproteinase-3; NR, not reported;NT-proBNP, N-terminal pro b-type Natriuretic Peptide; NZ DCS, New Zealand Diabetes Cohort Study; PAD, peripheral artery disease; PN, peripheral neuropathy; RRS, Reynolds risk score; sBP, systolic blood pressure; SMART, second manifestations of arterial disease; SOMs, self-organising maps; STD, standard; TIA, transient ischaemic attack; WHR, waist-to-hip ratio.

Figure 3.

Forest plot of C-statistics, with 95% CIs of risk prediction models when stroke was reported as part of a composite cardiovascular disease outcome. AD-ON, Action in Diabetes and Vascular Disease:Preterax and Diamicron Modified Release Controlled Evaluation-Observational; EPIC-NL,European Prospective Investigation into Cancer and Nutrition-Netherlands; HWNNs,hybrid wavelet neural networks; MACE, major adverse cardiovascular event; NZDCS, New Zealand Diabetes Cohort Study; SMART, second manifestations ofarterial disease; SOMs, self-organising maps.

The study quality for this group of models is summarised in table 4. Similar to the models developed in diabetic populations that look at the outcome of stroke specifically, we found that study quality was similar.

Table 4.

Study quality assessment of prediction models when stroke is reported as a part of composite CV outcome and performance measure (C-statistic) is presented for the composite CV outcome

Study	Inclusion/ exclusion criteria specified	Non-biased selection	Missing value/loss to follow-up considered	Modelling assumptions satisfied	Model external validation	Outcome assessed without knowledge of the candidate predictors (ie, blinded)	Duration of follow-up long enough	Potential clinical use of the model discussed	Study limitations discussed
Brownrigg et al.15	Yes	No	No	No	No	Not clear	No	Yes	Yes
Khalili et al 50	Yes	No	No	Yes	No	Not clear	Yes	Yes	Yes
Cederholm et al 51	Yes	Not clear	No	Yes	No	Not clear	Yes	Yes	Yes
Davis et al 33	Not clear	Not clear	No	Yes	Yes	Not clear	No	No	No
Kengne et al 34	Yes	Yes	Not clear	Not clear	Yes	Not clear	No	Yes	Yes
Ofstad et al 25	Yes	Not clear	Yes	Yes	No	Not clear	Yes	Yes	Yes
Looker et al 52	Yes	Not clear	Yes	Not Clear	No	Not clear	No	Yes	Yes
Mukamal et al 32	Yes	Yes	Yes	Yes	Yes	Not Clear	Yes	Yes	Yes
Paynter et al 53	Yes	Yes	No	No	No	Not Clear	Yes	No	No
Price et al 54	Yes	Not clear	Not clear	No	No	Not clear	No	No	Yes
Selby et al 55	Yes	Yes	Yes	No	No	Not clear	No	Yes	Yes
Zethelius et al 35	Yes	Not clear	No	Yes	No	Not clear	No	Yes	Yes
Alrawahi et al 41	Yes	Yes	No	Yes	No	Not clear	Yes	Yes	Yes
Zarkogianni et al 56	No	Not clear	Not clear	Yes	No	Not clear	Yes	Yes	Yes
Price et al 57	Not clear	Yes	No	Not clear	No	Not clear	Yes	Yes	Yes
Wan et al 58	Yes	Yes	Yes	Yes	No	Not clear	Yes	Yes	Yes
Young et al 59	Yes	Yes	Not clear	Not clear	No	Not clear	Not clear	Yes	Yes
van der Leeuw et al 60	Yes	Not clear	Yes	Not clear	No	Not clear	Yes	Yes	Yes
Alshehry et al 61	Yes	Yes	Not clear	Not clear	No	Not clear	Yes	Yes	Yes
Woodward et al. The AD-ON Risk Score62	Yes	Yes	Not clear	Not clear	No	Not clear	Yes	Yes	Yes
Parrinello et al 63	Yes	Yes	No	Not clear	No	Not clear	Yes	Yes	Yes
Colombo et al 64	Yes	Yes	Yes	Not clear	No	Not clear	No	Yes	Yes
Elley et al. NZ DCS40	Yes	Not clear	Yes	Yes	Yes	Not clear	No	Yes	No

AD-ON, Action in Diabetes and Vascular Disease: Preterax and Diamicron Modified Release Controlled Evaluation-Observational; NZ DCS, New Zealand Diabetes Cohort Study.

Validation studies of stroke prediction models developed in populations with and without diabetes

Seventeen risk prediction models for stroke (developed both in patients with diabetes and in general populations) were validated in diabetes populations by 33 studies (table 5). Among the 17 validated models, 14 of them were externally validated in independent cohorts and 3 of them were internally validated in a test sample or separate data set from the same cohort. Three studies validated more than one risk model in the same cohort. Models with multiple validations (two or more) and reported C-statistics are provided in figure 4. Models that had only been validated once were excluded from meta-analysis. In addition, only those studies that provided enough information to estimate the variance of the provided C-statistic for meta-analysis were considered for analysis.

Table 5.

Characteristics of the validation studies of the stroke prediction models

Study name	No of Studies	Validation study	Location	Outcome	Age	Gender	Events (n) /total participants (N)	Calibration	Discrimination (with CI)
Kothari et al, UKPDS Risk Engine for Stroke (UKPDS 60)21	12	Kengne et al 31	20 Countries (Australasia, Asia, Europe, North America)	Major CHD, major CVD and major cerebrovascular event (death from cerebrovascular disease and non-fatal stroke)	Mean age 66 years for both males and females	Both male and female	288/7502	HL χ2=138.7 (p<0.0001) (major event); HL χ2=114.3 (p<0.0001) (any event)	AUC=0.62 (major event); AUC=0.61 (any event)
		Davis et al 65	Australia	Fatal stroke, all stroke	Mean age of 62.2 years	Both male and female	13 (fatal stroke), 23 (all stroke)/791	HL∧C-test: p=0.06 (fatal stroke) and p=0.33 (all stroke), good calibration	AUC=0.88 (0.81 to 0.96) (fatal stroke); AUC=0.86 (0.78 to 0.93) (all stroke)
		Kothari et al 21	UK	Fatal stroke	Mean age 51.5 years for males and 52.6 years for females.	Both male and female	197/1370	NR	NR
		Jiao et al 66	Hong Kong	Stroke	Mean age 64.3 years (RAMP-DM) and 65.3 years (control)	Both male and female	Total CVD events n=10 (RAMP-DM) and n=13 (control group) /RAMP-DM group n=1072, control group n=1072	NR	NR
		Yang et al 38	Hong Kong, China	Stroke	Median age 57 years	Both male and female	182/3541	NR	Unadjusted AUROC=0.588 (0.549 to 0.626)
		Lahoz-Rallo et al 67	Spain	Cerebrovascular risk (stroke)	Mean age 65.5 years	Both male and female	Events (n) NR/ n=1846	NR	NR
		Metcalf et al 68	New Zealand	Stroke	35 to 74 years	Both male and female	Events (n) NR/n=423	NR	NR
		Tanaka et al 48	Japan	Stroke	40 to 84 years	Both male and female	89/1748	HL test: (p=0.54)	C-statistic=0.638 (0.566 to 0.7 11)
		Wells et al 47	USA	Stroke	18 years of age or older	Both male and female	Events (n): stroke (1088)/total participants (N): stroke (26 140)	Risk underestimated when examining calibration in the large	C- statistic=0.752
		Bannister et al 69	UK	CHD, fatal CHD, stroke, fatal stroke	Mean age 60.3 years (male) and 62.6 years (female)	Both male and female	6717/79 966 (stroke), 7037/79 966 (fatal stroke)	NR	C-statistic=0.73 (0.72 to 0.75) (stroke, female), C-statistic=0.71 (0.70 to 0.72) (Stroke, male); C-statistic=0.77(0.74 to 0.80) (fatal stroke, female), C-statistic=0.78 (0.76 to 0.81] (fatal stroke, male)
		Wu et al 70	China	Stroke and CHD	20 years and above	Both male and female	Events (n) NR/n=1584	NR	NR
		Ipadeola et al 71	Nigeria	CHD and stroke	Mean age 60.5±9.89 years	Both male and female	Events (n) NR/340	NR	NR
Clarke et al, UKPDS Outcomes Model27	4	Leal et al 72	UK	MI/stroke/IHD/heart failure/amputation/blindness/renal failure/death from any cause	Mean age 62 years	Both male and female	Events (n) NR/n=4031	Calibration plots: overestimated	C-statistic=0.68 (0.65 to 0.71) (stroke)
		McEwan et al 73	UK	CHF/IHD/MI/stroke/blindness/ESRD/amputation	Mean age 51.49 years (low risk) and 66.08 years (intermediate)	Both male and female	723 (stroke)/54 169 (all in low-risk patient)	NR	ROC=0.62 (stroke)
		Pagano et al 74	Italy	MI, other IHD, stroke, CHF and amputation (2000 survey) and mortality (1991 survey)	Mean age 57.9 years (1991 survey) and 57.4 years (2000 survey)	Both male and female	Events (n) NR/n=2514 (2000 survey) and n=1443 (1991 survey)	NR	NR
		Tao et al 75	UK, Denmark and the Netherlands	MI and stroke	40–69 years (50–69 years in the Netherlands)	Both male and female	Events (n) NR/2899	HL test: p=0.33 (Stroke)	AUROC=0.70 (0.64 to 0.77) (stroke)
Stevens et al, UKPDS Risk Engine (UKPDS 56)26	5	Shivakumar et al 76	India	CHD and stroke	Mean age 63.3 years	Both male and female	NR	NR	NR
		Moazzam et al 77	Pakistan	CHD, fatal CHD, stroke, fatal stroke	30–74 years	Both male and female	Events (n) NR/470	NR	NR
		Ezenwaka et al 78	Trinidad and Tobago	Absolute CHD and stroke	Mean age 63.1 years (male) and 59.5 years (female)	Both male and female	Events (n) NR/325	NR	NR
		Sun et al 79	China	CHD and stroke	21–94 years (58.4±12.9 years)	Both male and female	Events (n) NR/853 (no of patients with CKD)	NR	NR
		Pang et al 80	China	CHD and stroke	21–90 years	Both male and female	Events (n) NR/1178	NR	NR
Anderson et al, (Framingham Risk Score)28	2	Herder et al 81	Germany	MI, stroke, cardiovascular death	NR	Both male and female	84/1072	Observed/expected events reported. Good calibration (p>0.05) in all quintiles except quintile 4	C-statistic=0.636
		Kengne et al 31	20 countries (Australasia, Asia, Europe, North America)	Major CHD, major CVD and major cerebrovascular event (death from cerebrovascular disease and non-fatal stroke)	Mean age 66 years for both male and female	Both male and female	288/7502	HL χ2=42.7 (p<0.0001) (major event); HL χ2=149.0 (p<0.0001) (any event)	AUC=0.568 (major event); AUC=0.555 (any event)
D’Agostino et al, (Framingham Risk Score)29	2	Ataoglu et al 30	Turkey	Cardiovascular death, non-fatal MI, angina, ischaemic stroke	NR	Both male and female	66/102	NR	NR
		Kengne et al 31	20 countries (Australasia, Asia, Europe, North America)	Major CHD, major CVD and major cerebrovascular event (death from cerebrovascular disease and non-fatal stroke)	Mean age 66 years for both male and female	Both male and female	288/7502	HL χ2=19.9 (p=0.0004) (major event); HL χ2=32.7 (p<0.0001) (any event)	AUC=0.587 (major event); AUC=0.567 (any event)
D'Agostino et al, (Framingham Stroke Risk)37	1	Costa et al 82	Spain	Stroke	55–85 years	Both male and female	9/178	NR	NR
Yang et al, (Hong Kong Diabetes Registry for Stroke)38	1	Yang et al 38	Hong Kong	Stroke	Median age 57 years	Both male and female	182/3541	The Life Table method, adequate calibration	Unadjusted AUROC=0.749 (0.716 to 0.782) and adjusted AUROC=0.776
Mukamal et al 32	2	Mukamal et al 32	USA	MI, stroke, cardiovascular death	45–84 years	Both male and female	71/843	NR	Basic model: C - statistic=0.65; basic model+CRP: C- statistic=0.66; basic model+CRP + (ABI, internal carotid wall thickness, ECG left ventricular hypertrophy): C-statistic=0.68
		Read et al 83	Scotland	Hospital admission or death from MI, stroke, unstable angina, transient ischaemic attack, peripheral vascular disease, and coronary, carotid, or major amputation procedures	30–89 years	Both male and female	14 081/181 399	Calibration plots: better calibration	C-statistic=0.674 (0.669 to 0.679)
Kiadaliri et al 22	1	Kiadaliri et al 22	Sweden	First and second events of: AMI, heart failure, non-acute IHD and stroke	Mean age 55.33 years (male) and 56.89 years (female)	Both male and female	NR/7259	HL χ2 statistic: 11.61 (p=0.17) (first stroke); 9.99 (p=0.27) (second stroke)	C-statistic=0.79 (0.76 to 0.82) (first stroke) C-statistic=0.70 (0.64 to 0.75) (second stroke)
Davis et al, (Fremantle)33	2	Davis et al 33	Australia	CVD (hospitalisation for/with MI or stroke, and death from cardiac or cerebrovascular causes or sudden death)	Mean age 65.3 (35.9–89.0) years	Both male and female	24/180	HL∧C -test, p=0.85, good calibration	AUC=0.84 (0.76 to 0.91); p<0.001
		Read et al 83	Scotland	Hospital admission or death from MI, stroke, unstable angina, transient ischaemic attack, peripheral vascular disease, and coronary, carotid or major amputation procedures	30–89 years	Both male and female	14 081/181 399	Calibration plots	C-statistic=0.665 (0.660 to 0.670)
Kengne et al, (ADVANCE)34	2	Kengne et al 34	16 countries	CVD (fatal or non-fatal MI or stroke or cardiovascular death)	Mean age 64.4 (8.1) years	Both male and female	183/1836	HL test: p=0.032; predicted/observed risk=0.82	AUC=0.69 (0.646 to 0.724)
		Read et al 83	Scotland	Hospital admission or death from, stroke, unstable MI angina, transient ischaemic attack, peripheral vascular disease, and coronary, carotid, or major amputation procedures	30–89 years	Both male and female	14 081/181 399	Calibration plots	C-statistic=0.666 (0.661 to 0.671)
Zethelius et al 35	2	Zethelius et al 35	Sweden	Fatal/non-fatal CVD (the composite of CHD or stroke)	30–74 years	Both male and female	522/4906	P/O ratio=0.97, modified HL χ2 statistic=10.7 (p=0.2). Well calibration	C-statistic=0.72
		Read et al 83	Scotland	Hospital admission or death from MI, stroke, unstable angina, transient ischaemic attack, peripheral vascular disease, and coronary, carotid, or major amputation procedures	30–89 years	Both male and female	14 081/181 399	Calibration plots: better calibration	C-statistic=0.663 (0.658 to 0.668)
Stevens et alUKPDS 6624	1	Yao et al 84	China	CHD, stroke	30–79 years	Both male and female	Events (n) NR/1514	NR	NR
Hippisley-Cox et alQRISK239	1	Read et al 83	Scotland	Hospital admission or death from MI, stroke, unstable angina, transient ischaemic attack, peripheral vascular disease, and coronary, carotid, or major amputation procedures	30–89 years	Both male and female	14 081/181 399	Calibration plots	C-statistic=0.674 (0.669 to 0.679)
Elley et alNZ DCS40	1	Read et al 83	Scotland	Hospital admission or death from MI, stroke, unstable angina, transient ischaemic attack, peripheral vascular disease, and coronary, carotid, or major amputation procedures	30–89 years	Both male and female	14 081/181 399	Calibration plots: better calibration	C-statistic=0.670 (0.665 to 0.674)
Basu et alRECODe36	2	Basu et al 85	USA	Nephropathy (microalbuminuria, macroalbuminuria, renal failure, ESRD, reduction in glomerular filtration rate), moderate to severe diabetic retinopathy, fatal or non-fatal MI, fatal or non-fatal stroke, CHF and all-cause mortality	45–84 years (MESA), 35–84 years (JHS)	Both male and female	89 stroke (MESA), 142 stroke (JHS)/1555 (MESA), 1746 (JHS)	Calibration slope=1.00, χ2=17.3, p value <0.001 (MESA); calibration slope=1.05, χ2=22.9, p value <0.001 (JHS)	C-statistic=0.75 for stroke (MESA); C-statistic=0.72 for stroke (JHS)
		Basu et al 36	USA	Microvascular: nephropathy, retinopathy, neuropathy; cardiovascular: composite of atherosclerotic CVD (first fatal or non-fatal MI or stroke), fatal or non-fatal MI, fatal or non-fatal stroke, CHF, or death from any cardiovascular cause	Mean age of 58.9 years	Both male and female	157/4760	Calibration slope for stroke=0.99, χ2=8.2, p value=0.22	C-statistic for stroke=0.67 (0.63 to 0.71)
Alrawahi et al 41	1	Alrawahi et al 86	Oman	Fatal and non-fatal CHD, stroke and PAD	Mean age 55.3±11.0 years (derivation sample) and 52.3±11.4 years (validation sample)	Both male and female	126 (derivation sample), 52 (validation sample /1314 (derivation sample), 405 (validation sample)	HL χ2 p value=0.15 (derivation sample) and HL χ2 p value=0.06 (validation sample). Satisfactory calibration	AUC=0.73 (0.69 to 0.77) (derivation sample); AUC=0.70 (0.59 to 0.75) (validation sample)

ABI, ankle–brachial index;ADVANCE, Action in Diabetes and Vascular Disease: Preterax and Diamicron Modified Release Controlled Evaluation; AMI, acute myocardial infarction; AUC, area under the curve;AUROC, area under the receiver operating characteristic curve; CHD, coronary heart disease; CHF, congestive heart failure; CKD, chronic kidney disease; CRP, C reactive protein; CVD, cardiovascular disease; ESRD, end-stage renal disease; HL, Hosmer-Lemeshow; HLˆC, Hosmer-Lemeshow C-test; IHD, ischaemic heart disease;JHS, Jackson Heart Study; MESA, Multiethnic Study of Atherosclerosis; MI, myocardial infarction; NR, not reported;NZ DCS, New Zealand Diabetes Cohort Study; PAD, peripheral artery disease; P/O, predicted over observed; RAMP-DM, Multidisciplinary Risk Assessment and Management Program for Patients with Diabetes Mellitus;RECODe, Risk Equations for Complications of Type 2 Diabetes; ROC, receiver operating characteristic;UKPDS, United Kingdom Prospective Diabetes Study.

Figure 4.

Forest plot of C-statistics, with 95% CIs, of stroke prediction models that are externally validated in two or more independent cohorts. JHS, Jackson Heart Study; MESA,Multiethnic Study of Atherosclerosis; UKPDS, United Kingdom ProspectiveDiabetes Study

UKPDS Risk Engine for Stroke by Kothari et al ²¹ was the most frequently externally validated model with a total of 12 studies reporting its performance in different diabetes cohorts. In 12 external validation studies, a total of 126 323 patients were included with considerable variations in sample sizes across the different studies. The pooled C-statistic for the model by Kothari et al ²¹ was 0.72 (95% CI, 0.68 to 0.75), with high heterogeneity identified (I²=95%; Cochran Q statistic p<0.001). Stratification by sample size (small vs large, p=0.69), geographic location (Asia vs others, p=0.09) and stroke type (fatal vs non-fatal, p=0.07) did not explain the observed heterogeneity in the discriminative performance of this model. UKPDS Risk Engine by Stevens et al ²⁶ was the second most externally validated model with five validation studies including 2826 patients. One study did not report the number of participants and none of the studies reported C-statistics. As a result, a pooled C-statistic and heterogeneity was not possible to assess for this model. The UKPDS Outcomes Model by Clarke et al ²⁷ was externally validated by four studies including 65 056 patients. The pooled C-statistic was 0.66 (95% CI, 0.61 to 0.71) with high heterogeneity between studies (I²=84.5%; Cochran Q statistic p=0.002). Similar to the UKPDS Risk Engine for Stroke,²¹ stratification across select study characteristics did not explain the observed heterogeneity. The Framingham risk score by Anderson et al ²⁸ was externally validated in two studies including 8574 patients. The pooled C-statistic was 0.58 (95% CI, 0.54 to 0.61) with non-significant heterogeneity between studies (I²=56.1%; Cochran Q statistic p=0.102). The Framingham risk score by D’Agostino et al ²⁹ was externally validated in two studies including 7604 patients. One study (Ataoglu et al ³⁰) did not report the C-statistic for the model and one study (Kengne et al ³¹) reported two C-statistic values, one for major events and one for any event. The pooled C-statistic for these two values was 0.58 (95% CI, 0.55 to 0.60). Models by Mukamal et al.,³² Davis et al.,³³ Kengne et al ³⁴ and Zethelius et al ³⁵ each were externally validated by two studies with pooled C-statistics of 0.67 (95% CI, 0.67 to 0.68), 0.75 (95% CI, 0.58 to 0.92), 0.67 (95% CI, 0.65 to 0.69) and 0.69 (95% CI, 0.63 to 0.75), respectively. Observed heterogeneity was high in models by Davis et al ³³ and Zethelius et al ³⁵ while low in models by Mukamal et al ³² and Kengne et al.³⁴ The model by Basu et al ³⁶ was externally validated by two studies in three different population yielding a pooled C-statistic of 0.71 (95% CI, 0.67 to 0.76) with moderate heterogeneity between studies (I²=56.8%; Cochran Q statistic p=0.099).

Separate models by D'Agostino et al (Framingham Stroke Risk Score),³⁷ Yang et al (Hong Kong Diabetes Registry for Stroke),³⁸ Kiadaliri et al,²² Stevens et al (UKPDS 66),²⁴ Hippisley-Cox et al (QRISK2),³⁹ Elley et al (New Zealand Diabetes Cohort Study)⁴⁰ and Alrawahi et al ⁴¹ were each validated in one external or separate cohort with sample sizes ranging from 178 to 1 81 399 patients. For the studies that reported discrimination, C-statistics ranged from 0.67 to 0.79. In addition, calibration assessed by calibration plots and Hosmer-Lemeshow tests found good calibration in most studies.

The overall pooled C-statistic for all validation studies was 0.68 (95% CI, 0.67 to 0.70) with high heterogeneity between studies (I²=95.3%; Cochran Q statistic p<0.001). Models that were developed in diabetes population showed significantly higher C-statistics than models developed in general populations (meta-regression p=0.001). Models, where stroke was reported as the main outcome as opposed to part of a composite CVD outcome, did show borderline significantly higher C-statistics (meta-regression p=0.052), although the value of the C-statistic is still low. This observed difference in the two models makes sense as models that include stroke as part of a composite outcome are expected to be different from models where stroke is the only outcome. A summary describing the characteristics of the studies where prediction models were developed in general populations but validated in patients with diabetes is presented in table 5.

Discussion

This systematic review and meta-analysis provides an overview of all stroke prediction models that were specifically developed for, or validated in patients with diabetes to calculate future stroke risk. Thirty-four stroke prediction models were identified that were specifically designed for patients with diabetes and only 32% of these prediction models have been externally validated, with varying results. Overall, the pooled C-statistics were poor for most models. Four of the prediction models identified were originally developed in the general population but externally validated in diabetes populations. The most notable prediction model was the UKPDS Risk Engine for Stroke²¹ with 12 validation studies. Ten stroke prediction models had multiple validations, seven models had single validations and twenty-one had no validations at all. It is difficult to assess model performance for those with no validation or single validations. Additional validation studies on the performance of stroke prediction models in different diabetes populations are needed. Since stroke prediction models developed in the general population may not account for specific risk factors related to diabetes, using risk scores developed specifically in the diabetes population will help to estimate stroke risk among people with diabetes more accurately.

None of the models showed good discriminative performance consistently when externally validated. The model by Kothari et al ²¹ where the stroke was the primary outcome showed moderate discriminative performance (pooled C-statistic=0.72). Since this model was externally validated multiple times, the performance of this model can be considered as consistent. The discriminative ability of stroke prediction models where stroke was the primary outcome and models where stroke was a part of composite CVD outcome were modest, with C-statistics often less than 0.70.⁴² Meta-analyses of the C- statistic suggests that there is significant between-study heterogeneity in the models where stroke is reported as the primary outcome and in those where stroke is reported as part of composite CVD outcome. Further, the possible sources of heterogeneity are unexplained. Perhaps the difference in patient characteristics in the different cohorts could be a potential source of heterogeneity; however, geographic location, sample size, follow-up time, external validation and variables included in the models were not significant sources of heterogeneity in meta-regression.

The discrimination of the 17 models that were validated were generally comparable with those observed in the development cohorts. However, the performance of some models externally validated in multiple cohorts was heterogeneous and possible source for this heterogeneity remains unexplained. There was also variability in prediction model quality and the methodology used in developing them. Our study findings suggest that, from a large number of published models in patients with diabetes, very few well-validated models are available for stroke prediction. This is helpful to inform the determination of models for clinical uptake when risk stratification approaches for stroke are implemented.

No evidence of small-study effects was detected, in which smaller studies reported better discrimination of models for predicting stroke. Study quality assessment shows many of the models failed to meet some key criteria: consideration of missing values, modelling assumptions, model validation and blinded outcome assessment, which is a concern. Many studies lacked standard reporting. This, to some extend, may be due to lack of guidelines for standards of reporting for risk prediction studies during that time. Many authors reported different aspects of prediction models, and in varying ways created difficulty in collecting information. The publication of new guidelines such as Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD)⁴³ has been introduced and may help improve reporting standards in subsequent studies in this area.

In prior reviews examining risk prediction models in adults with diabetes (Chamnan et al ⁴⁴ van Dieren et al,⁴⁵ and Chowdhury et al ¹⁶), all components of cardiovascular disease such as CHD, stroke, CAD, myocardial infarction, heart failure were considered as outcomes of the prediction model. Our review adds to knowledge on predicting risk of stroke in persons with diabetes in the following ways: (1) We only considered models where the primary outcome of the model was stroke or when stroke was part of a composite CVD outcome and corresponding C-statistic were provided; (2) We did not consider other components of CVD as outcomes of the model and therefore our estimates of model performance are more specific to stroke; (3) We have identified and included several recently derived models and conducted meta-analyses to explore reasons for variability in the discriminative performance across models and (4) We provide a detailed assessment of quality of studies among models developed in diabetes populations. Only one prior study¹⁶ in this area performed a meta-analysis of model performance statistics across multiple studies or assessed study quality.

One of the major strengths of our study is the breadth of the systematic search, which included three different databases and extensive use of reference lists of the identified studies. Therefore, it is unlikely that any stroke prediction model-related studies have been missed. To best of our knowledge, this is the first study, where a meta-analysis and study quality assessment was performed on stroke prediction models in patients with diabetes. Nonetheless, there are few limitations in our study, which need to be kept in mind. In this paper, we only considered studies that developed or validated stroke prediction models within patients with diabetes. While prediction models for stroke have been developed for patients with other potential risk factors (eg, patients with hypertension), we felt that an exploration of a broad range of risk factors was outside the scope of this review. Though the inclusion of all stroke prediction models (regardless of the underlying risk factor(s)) could potentially improve the generalisability of our findings, it could have also increased the between-study heterogeneity, making the pooled estimates more difficult to interpret. We also did not consider non-English publications. Although, the English language is generally perceived to be the universal language of science, selection of research findings in a particular language can introduce language bias and may lead to erroneous conclusions. With this in mind, readers should to be cautious when interpreting the findings of our results. Finally, we were only able to use C-statistics to compare the model performance, which might be insensitive to identify differences in the ability of models to accurately risk-stratify patients into clinically meaningful risk groups.⁴⁶ In addition, meta-analysis of calibration measures (eg, E/O ratio) along with C-statistics could give a comprehensive summary of the performance of these models.

Our findings suggest that there is no significant difference between the discrimination of models where stroke was the primary outcome and stroke was part of composite CVD outcome. Models, particularly those that have never been validated or validated once need to undergo further external validation in which they will be used with or without recalibration or model updating to better understand the comparative performance of these models.

Conclusions

In conclusion, we have identified many models for predicting stroke in patients with diabetes and attempted to compare these models. Only a small number of models have undergone external validation and might provide generalisable predictions that would support their use in another clinical setting. It is difficult to choose one model over another as none of these models exhibited superior discriminative performance, and unfortunately, no single model appears to perform consistently well. It could be argued that risk prediction in patients with diabetes is not essential. Persons with diabetes are generally perceived to be at elevated risk of stroke and the current practice is to treat to common HbA1C, blood pressure and low-density lipoprotein targets based on diabetes status alone and not on calculated risk. This non-risk based approach may be leading to unnecessary overtreatment and the absence of high-quality validated risk prediction models which limits our ability to assess whether more targeted approaches are possible. Further research is warranted to identify new risk factors with high associated relative risk to improve the currently available prediction models.