Comprehensive comparison of stroke risk score performance: a systematic review and meta-analysis among 6 267 728 patients with atrial fibrillation

Abstract

Aims

Multiple risk scores to predict ischaemic stroke (IS) in patients with atrial fibrillation (AF) have been developed. This study aims to systematically review these scores, their validations and updates, assess their methodological quality, and calculate pooled estimates of the predictive performance.

Methods and results

We searched PubMed and Web of Science for studies developing, validating, or updating risk scores for IS in AF patients. Methodological quality was assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). To assess discrimination, pooled c-statistics were calculated using random-effects meta-analysis. We identified 19 scores, which were validated and updated once or more in 70 and 40 studies, respectively, including 329 validations and 76 updates—nearly all on the CHA₂DS₂-VASc and CHADS₂. Pooled c-statistics were calculated among 6 267 728 patients and 359 373 events of IS. For the CHA₂DS₂-VASc and CHADS₂, pooled c-statistics were 0.644 [95% confidence interval (CI) 0.635–0.653] and 0.658 (0.644–0.672), respectively. Better discriminatory abilities were found in the newer risk scores, with the modified-CHADS₂ demonstrating the best discrimination [c-statistic 0.715 (0.674–0.754)]. Updates were found for the CHA₂DS₂-VASc and CHADS₂ only, showing improved discrimination. Calibration was reasonable but available for only 17 studies. The PROBAST indicated a risk of methodological bias in all studies.

Conclusion

Nineteen risk scores and 76 updates are available to predict IS in patients with AF. The guideline-endorsed CHA₂DS₂-VASc shows inferior discriminative abilities compared with newer scores. Additional external validations and data on calibration are required before considering the newer scores in clinical practice.

Clinical trial registration

ID CRD4202161247 (PROSPERO).

What’s new?

• 19 risk scores, 329 external validations and 76 risk score updates have been conducted to accurately predict ischaemic stroke (IS) in patients with atrial fibrillation (AF).

• Despite the development of new risk scores, the choice on initiating antithrombotic therapy for the prevention of IS in patients with AF remains mostly based on the conventional guideline endorsed CHA₂DS₂-VASc (2010).

• Probably due to the inclusion of specific biomarkers, new risk scores on IS risk in patients with AF tend to have better predictive abilities when compared with conventional scores, yet differences are marginal and should be interpreted with caution.

• Although newer scores tend to have better predictive abilities, external validations and studies on calibration are required to confirm this.

• In future studies developing and/or validating risk scores, adhering to the PROBAST and TRIPOD guidelines is warranted in order to minimize the risk of bias.

Introduction

Patients with atrial fibrillation (AF) are at increased risk of ischaemic stroke (IS) and face poor stroke outcomes including severe morbidity and mortality.¹ Anticoagulation reduces this risk substantially and is therefore prescribed to most patients.² In current guidelines, the threshold for initiating anticoagulation therapy is based on the balance between the predicted IS risk with the expected risk of bleeding and associated quality of life.³ Therefore, accurate prediction of these outcomes is of major importance in AF management.

Since the widespread use of the first risk scores for cardiovascular disease, such as the Framingham risk score (1998), a multitude of risk scores have been developed to predict IS risk in patients with AF.⁴ Examples include the CHADS₂ (2001) and the commonly used CHA₂DS₂-VASc (2010), with the latter being endorsed by most current clinical guidelines.^5–7 Studies on the predictive performance suggest that current risk scores have comparable but limited overall ability to predict IS in patients with AF.^8–10 Consequently, the use of these scores in clinical practice to allocate anticoagulation treatment is not without risk: overprediction of IS risk will result in overtreatment and higher bleeding rates, whereas higher IS rates will occur following underpredicted risks. To improve IS prediction, new risk scores have been developed and earlier scores updated. As much emphasis has been put on commonly available risk scores, limited research seems to be performed on newer risk scores and updates. For example, external validation, a crucial method in the assessment of predictive performances such as discrimination and calibration, is lacking the newer risk scores, despite the overall increase of external validations conducted over the past years.^8–11 As a result, there may be an undervalued but promising risk score in the literature that is currently not integrated in clinical practice but could improve decision-making.

For the clinician interested in the best risk score to inform on the patient’s risk, but also for the researcher aiming to develop or validate a risk score, a comprehensive comparison of all available risk scores, their updates, and validations is essential. Therefore, the objective of this study is to(i) identify and systematically review all available risk scores predicting IS risk in patients with AF,(ii) present an overview on their external validations and updates,(iii) assess the methodological quality of these studies, and (iv) provide a pooled estimate of the predictive performance.

Methods

The current study is performed and reported in line with the PRISMA,¹² TRIPOD,¹³ and CHARMS¹⁴ guidelines, which were followed where applicable. The review was registered in PROSPERO under ID CRD4202161247.

Data sources and eligibility

The present review aims to identify risk scores, and corresponding update and validation studies, on future event of IS in adults with AF. In collaboration with a medical librarian, we conducted two searches in an iterative fashion. First, on 19 May 2021, we conducted a search in PubMed for English-language studies regarding ‘risk scores’ and ‘ischaemic stroke’ using methods developed in earlier work.^15,16 Studies were screened independently by two researchers (V.H.W.v.d.E. and J.M.) and were deemed eligible if they met the following criteria:(i) development of a multivariable prognostic risk score,(ii) predicting outcomes including first event of IS from 1 month onward, assessed in a longitudinal design,(iii) in a population of adults (>18 years) with AF. Secondly, on 28 May 2021, for each of the identified risk scores, we looked for studies externally validating or updating these scores, using citation search methods in Web of Science. Studies renewing old risk scores, for example, by adding or replacing a predictor, were regarded as update studies.¹⁷ In the end, we thus compiled three data sets: (i) development: the set of studies in which the scores were originally developed, (ii) validation: the studies in which these scores were validated, and (iii) update: the set of studies in which (at least) one of the scores was updated. Update studies that validated an original score as comparison to the updated score were included in both the validation and update data set. Finally, included articles and reviews were cross-referenced for possible relevant studies. Detailed search methods are displayed in Supplementary material online, section ‘Detailed search methods’.

Data extraction and study appraisal

Titles, abstracts, and full texts were screened independently by two researchers (V.H.W.v.d.E. and J.M.). Data extraction was conducted by V.H.W.v.d.E. and J.M., with consultation of Y.d.J. when necessary. Information on the study design, population, outcome, prediction horizon (i.e. the time between prediction and the timeframe in which the outcome may occur), candidate predictors, sample size, risk score development, and risk score performance were extracted and summarized. Risk score predictive performance was evaluated using discrimination and calibration measures. For discrimination, which describes the scores’ ability to discriminate between events and non-events, we extracted the c-statistic (area under the receiver operating characteristic curve for logistic model and Harrell’s C-index for Cox model). In general, c-statistics of <0.60, 0.60–0.80, and >0.80 are interpreted as poor, reasonable, and good discrimination.¹⁸ For calibration (i.e. the agreement between the predicted and the observed risk), we extracted the calibration-in-the-large, the calibration slope, the Hosmer–Lemeshow, or the Nam–d'Agostino statistic. Studies presenting a calibration plot (i.e. a plot with the relation between the observed and predicted risks) were qualitatively categorized as poor, medium, or good fit.¹⁸ For the update studies, net reclassification index and the integrated discrimination improvement, measures quantifying the improvement when a score is updated, were extracted.¹⁹ Methodological quality was assessed in all studies using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). This tool consists of 4 domains (participants, predictors, outcome, and analysis) containing 20 signalling questions to determine the risk of bias and three signalling questions to determine applicability to the review question.²⁰

Statistical analysis

A random-effects meta-analysis was conducted to summarize discrimination measures of the included original risk scores. It should be noted that pooling calibration-in-the large, the expected/observed ratio and calibration slopes is possible as well.²¹ However, due to the limited number of studies assessing calibration and the heterogeneity of these calibration measures, no random-effects meta-analysis was conducted on calibration. Risk scores with more than two external validations were included. C-statistics were logit-transformed, and confidence intervals were calculated following the Hartung–Knapp–Sidik–Jonkman approach.^22–24 A logit pooled c-statistic was calculated for each of the included models and then transformed back to the original scale.^21,25,26 Forest plots were drawn to visualize the estimated results of all included studies. To assess small study bias, funnel plots were made and Egger’s regression tests were performed to test for funnel plot asymmetry.^27,28 All analyses were conducted using RStudio version 1.2.5033 and the metafor package.²⁹

Sensitivity analyses

Regarding the validation studies included in the meta-analysis, eight sensitivity analyses were conducted. Per analysis, pooled c-statistics were calculated and forest plots were drawn for each individual score. Next, the yielded c-statistics were analysed on discrepancies. For the first sensitivity analysis, studies were stratified according to the cohorts’ mean or median age: (i) <65 years, (ii) 65–75 yeras, and (iii) >75 years. In the second analysis, we evaluated the effect of outcome measure: IS, thromboembolism, or other outcomes measures. For the third analysis, we calculated c-statistics for three patient groups: (i) corrected for anticoagulation, (ii) not corrected for anticoagulation, and (iii) with no information on anticoagulation. In the fourth analysis, we studied the effect of ethnicity and grouped the studies conducted in predominantly Caucasian or Asian participants. For the fifth analysis, we categorized the studies in high or low risk of bias according to the PROBAST. For the sixth analysis, we compared ‘true low’ risk patients (CHA₂DS₂-VASc/CHADS₂ score ≤ 1, ATRIA score ≤5) with intermediate- or high-risk patients (CHA₂DS₂-VASc/CHADS₂ score ≥2, ATRIA score ≥6). For analysis 7, we evaluated the effect of study design: observational vs. randomized controlled trial studies. In the last analysis, studies were stratified according to the year of publication to evaluate the effect of inclusion of older study cohorts with possible different baseline risks for IS due to different treatment options at that time. Detailed methods are given in Supplementary material online, section ‘Sensitivity analyses’.

Results

Study selection

The study selection is described in Flowcharts 1 and 2 (Figure 1). The primary search for studies developing a risk score identified 1496 titles of which 213 abstracts were screened and 71 articles were screened full text for eligibility, leading to 19 studies that were included in the review. Next, we searched for update and validation studies, 2188 unique titles were identified of which 70 validation studies and 40 update studies were included in the present review. As most update studies (n = 37, 93%) also validated the original model as comparison for the updated model, we included a total of 107 studies in which a risk score was validated.

Figure 1.

Flowchart study selection. Two iterative searches were performed to identify (i) development studies on risk scores for IS in patients with AF (ii) corresponding validation and update studies.

Methodological quality of the included studies

Methodological risk of bias was assessed for all included development (n = 19), validation (n = 70), and update studies (n = 40). For all these studies, risk of bias was low in the participants and predictors domains. In the outcome domain, however, the risk of bias was high or unclear as study outcomes were often ill-defined or, in the case of validation and update studies, determined differently from the original risk scores. In the analysis domain, all studies were of high risk of bias, mostly due to statistical considerations that were not correctly addressed, such as omission of calibration, failure to take competing risk into account, or inappropriate methods to handle incomplete data. The applicability of the participant domain was of low concern for all studies, and for most studies in the predictor domain. For the outcome domain, applicability was a concern for most studies due to the use of composite outcomes, whereas the present study focused on the predictive performance of the non-composite outcome of IS. More detailed information on the risk of bias per signalling question is provided in Figure 2 and in Supplementary material online, section ‘Risk of bias of included validation and update studies’ for the validation and update studies. The funnel plots showed no clear asymmetry. This was confirmed by the Egger’s regression test which showed a P-value of >0.05 in all risk scores but two: the modified-CHADS₂ showed a P-value of <0.05 and for the GARFIELD-AF, no Egger’s test could be performed due to the inclusion of only two studies (see Supplementary material online, section ‘Analysis on publication bias’).

Figure 2.

Risk of bias (ROB) and applicability assessment using the PROBAST. (A and B) development studies (n = 19), (C and D) validation studies (n = 70), and (E and F) update studies (n = 40). Blue: high risk of bias, green: low risk of bias, pink: unclear bias due to lack of information.

Development studies

Study characteristics

The first of the 19 studies developing a prediction model for IS in AF patients was published in 1994³⁰ and the most recent included studies were published in 2021^31–34 (Figure 3). Risk scores were developed in cohorts with a sample sizes ranging between 705 and 52 032 patients; event rates ranged between 1.3 and 11.8%. In 11 studies, the follow-up period was described, ranging from 0.9 to 5.5^33–42 and 1.0–1.9^43,44 years in studies presenting a mean and median follow-up period, respectively. Except for four studies,^31,33,34,45, all studies were conducted in a cohort of predominantly Caucasian patients. Six studies^{31,36,37,40,43,46} developed their score in a cohort of patients not on anticoagulation, whereas 12 studies^{30,32–35,38,39,41,42,44,47,48} included patients on anticoagulation. In one study, no information on anticoagulation was given.⁴⁵

Figure 3.

Timeline on the development, validation, and update of risk scores. (Upper panel) The number of external validations plotted over time; (Middle panel) The number of update studies plotted over time; (lower panel) timeline of developed risk scores. Note that the number of validations slightly differs with Table 3, which gives the number of validations that were included in the random effects meta-analysis.

Risk score characteristics

Of the 19 risk scores, most (n = 13) studies developed a point-based risk score. In total, 27 different predictors were used of which age (n = 15), history of stroke/transient ischaemic attack (TIA) (n = 12), diabetes mellitus (n = 10), and sex (n = 7) were most common (Table 1). Other studies developed a mathematical formula,^32,39,41,42 a decision tree,³⁶ or employed artificial intelligence algorithms.⁴⁸ Regarding the statistical analysis method used to develop a model, 13 studies used Cox proportional hazard regression,^{30–33,35,37,40–44,46,47} 4 logistic regression,^34,38,39,45 and 2 studies used another regression method.^36,48 The prediction horizon ranged between 31 days and 5 years and was not given for three scores.^33,42,45

Table 1.

Predictor use in development studies (N =19)

	Age	Sex	World religion	Race	Smoking	BMI	Type of AF	Previous stroke/TIAa	Previous bleeding	Vascular disease	Heart failure	Hypertension	Diabetes mellitus	Chronic kidney disease	Dementia	Antiplatelet	Oral anticoagulant	Other treatmentb	Time in therapeutic range	IMRSa	cTnI/T-hsa	NT-proBNPa	Proteinuria	Creatinine clearance	PT-INRa	Blood pressure	Left atrial dimension
	Patient characteristics							Comorbidities								Treatment				Laboratory findings						Clinical findings
AFI investigators, 1994	x							x				x	x
Hart, 1999	x	x										x														x
Gage, 2001	x							x			x	x	x
Van Walraven, 2003								x		x			x													x
Wang, 2003	x	x						x					x													x
Rietbrock, 2008	x	x						x					x
Lip, 2010	x2	x						x		x	x	x	x
Lip, 2013	x							x								x			x
Singer, 2013	x	x									x	x	x										x	x
Hijazi, 2016	x							x													x	x
Fox, 2017			x	x				x	x		x			x			x
Claxton, 2019	x	x						x	x							x	x3	x5
Horne, 2019	x	x						x				x	x				x			x7
Shin, 2019	x																					x		x			x
Goto, 2019																									x
Jiang 2020	x							x			x
Fox, 2021	x				x			x	x	x	x	x	x	x	x		x
Okumara, 2021	x2					x	x	x				x
Arnson, 2021	x									x		x	x	x2

The inclusion of a predictor is shown as ‘x’. The subscript under x indicates the number of predictors included from that category (e.g. ‘x2’ implies that 2 predictors were used from the same category).

^acTnI/T-hs, high-sensitive cardiac troponin I/T; IMRS, intermountain risk score (consists of complete blood count parameters and basic metabolic factors); NT-proBNP, N-terminal-pro hormone Brain natriuretic peptide; PT-INR, prothrombin time international normalized ratio; TIA, transient ischemic attack.

^bAntiarrhythmic, calcium channer blocker, beta Blocker, lipid lowering medication, or anti-diabetic medication.

The outcome measure was IS (n = 5)^{30,33–35,42}, undefined stroke (n = 2),^45,47 thromboembolic events (n = 5, including IS, TIA, venous thrombosis, and/or pulmonary embolism)^{31,38,40,41,48} or the composite of thromboembolic events with bleeding events (n = 7, including haemorrhagic stroke and/or major bleeding).^{32,36,37,39,43,44,46} Discrimination was reported in 16 studies with c-statistics ranging from 0.61³⁸ to 0.86.⁴⁵ Calibration was presented as observed vs. expected risks for the ABC-score, GARFIELD-AF, ACTS, and GARFIELD-AF II, demonstrating adequate calibration.^32,41,42,44 The Hosmer–Lemeshow statistic was given in five studies, showing no evidence of poor calibration for ACTS, ATRIA, HELT-E₂S₂, and the ABCD score (χ²-statistics with a P-value of >0.05).^33,40,42,45 Information on characteristics per individual score is presented in Table 2; detailed information in Supplementary material online, section ‘Overview of included studies’.

Table 2.

Baseline characteristics and predictive performances of the risk scores

Year of publication	1994	1999	2001	2003	2003	2008	2010	2013
Risk score name	AFI	SPAF	CHADS2	—	Framingham	Modified-CHADS2	CHA2DS2-VASc	—
Risk score type	Risk score	Risk score	Risk score	Prediction rule	Risk score	Risk score	Risk score	Formula
Study design	RCT	RCT	Cohort (retrospective)	RCT	Cohort (prospective)	Case–control	Cohort (prospective)	RCT
Country	USA	USA	USA	USA, Canada, Denmark, and The Netherlands	USA	UK	>10 countries	>10 countries
Outcome	IS	IS	Composite (IS, TIA, undefined cerebral event)	Composite (IS, haemorrhagic stroke, TIA)	Composite (IS, haemorrhagic stroke)	Composite (IS, haemorrhagic stroke)	Composite (IS, systemic embolim, pulmonary embolism)	Composite (IS, systemic embolus, major bleeding)
Method	Cox	Cox	Cox	Recursive partitioning	Cox	Cox	Logistic	Logistic
Follow-up	—	2 yearsa	1 yearsb	1.9 yearsa	4.3 yearsa	—	1 yeara	0.9 yearsa
Prediction horizon	1 year	1 year	1 year	1 year	5 years	5 years	1 year	1 year
n events/n total (%)	208/5955c (3.5)	130/2012 (6.5)	94/1733 (5.4)	103/1661 (6.2)	83/705 (11.8)	5526/51807 (10.7)	25/1084 (2.3)	50/2292 (2.2)
n cand. pred (EPV)	15 (14)	30 (4)	5 (19)	18 (6)	9 (9)	9 (614)	9 (3)	14 (4)
Internal validation	—	—	Bootstrapping	Split sample	Bootstrap	—	—	—
Discrimination (C-statistic)	—	—	0.82 (0.80–0.84)	—	0.66 (SD 0.03)	0.72 (0.72–0.73)	0.61 (0.51–0.70)d	0.73 (0.57–0.73)
Calibration	—	—	—	—	Hosmer– Lemeshow	—	—	—
Mean age (SD)	69a	69a (SD 10)	81a	70a (SD 10)	75	e	66a (SD 14)	70 (SD 9)
Male sex (%)	—	72	42	67	52	51	59	65
Treatment (%)
Antiplatelet	50%	100%	31%	100%	e	e	e	24%
DOAC	—	—	—	—	—	e	—	—
VKA	50%	14%	—	—	—	e	18%	100%

Year of publication	2013	2016	2017	2019	2019	2019
Risk score name	ATRIA	ABC stroke risk score	GARFIELD-AF	ACTS	IMRS-VASc	ABCD score
Risk score type	Risk score	Risk score	Formula	Formula	Risk score	Risk score
Study design	Cohort (prospective)	RCT	Cohort (prospective)	Cohort (prospective)	Cohort (prospective)	Case–control
Country	USA	>10 countries	>10 countries	USA	USA	South Korea
Outcome	Composite (IS, systemic embolus)	Composite (IS, haemorrhagic stroke, systemic embolus)	Composite (IS, systemic embolus)	IS	Stroke (undefined)	Stroke (undefined)
Method	Cox	Cox	Cox	Cox	Cox	Logistic
Follow-up	3 yearsa	1.9 yearsb	—	1.8 yearsa	—	—
Prediction horizon	1 year	1 year	1 year	—	2 year	—
n events/n total (%)	685/10 927 (6.3)	391/27 929c (N.A.)	511/38 935 (1.3)	2028/252 904c (N.A.)	1506/55 970 (2.7)	583/− (−)
n cand. pred (EPV)	13 (53)	15 (26)	30 (17)	44 (46)	32 (47)	15 (39)
Internal validation	Bootstrapping	Bootstrapping	Cross-validation	Bootstrapping	—	Bootstrapping
Discrimination (C-statistic)	0.73 (0.71–0.75)	0.68 (0.65–0.71)	0.69 (0.67–0.71)g	0.68 (0.66–0.70)	0.70 (0.69, 0.73)	0.86 (0.84–0.88)
Calibration	Hosmer–Lemewhow	Observed vs. expected	Observed vs. expected	Hosmer–Lemeshow, observed/expected	—	Hosmer–Lemewhow
Mean age (SD)	e	70b	71f (63–78)	e	F: 73 (SD 13) M: 69 (SD 14)	61 (SD8)
Male sex (%)	57	64	45	60	53	73
Treatment (%)
Antiplatelet	e	e	36%	b	20%	e
DOAC	—	50%	23%	33%	b	e
VKA	—	50%	42%	67%	10%	e

Year of publication	2019	2020	2021	2021	2021
Risk score name	Artificial intelligence (AI) model	CAS	GARFIELD-AF II	HELT-E2S2	LRAF
Risk score type	Artificial intelligence	Risk score	Formula	Risk score	Risk score
Study design	Cohort (prospective)	Cohort (prospective)	Cohort (prospective)	Cohort (prospective)	Cohort (retrospective)
Country	>10 countries	China	>10 countries	Japan	Israel
Outcome	Composite (IS, TIA, systemic embolism)	Composite (IS, systemic embolism)	Composite (IS, TIA, undefined stroke, systemic embolism)	IS	IS
Method	Neural network	Cox	Cox	Cox	Logistic
Follow-up	—	—	—	1.8 yearsa	5.5 yearsa
Prediction horizon	30–365 days	1 year	2 years	b	1 year
n events/n total (%)	b	163/6601 (2.5%)	957/52 032 (1.8%)	241/12 289 (2.0%)	304/15 621 (2.0%)
n cand. pred (EPV)	b	3 (54)	11 (87)	6 (40)	6 (51)
Internal validation	Bootstrapping	Bootstrapping	Cross-validation	Bootstrapping, cross-validation	—
Discrimination (C-statistic)	0.70 (0.56–0.83)	0.69 (0.65–0.73)	0.68 (0.66–0.70)	0.68 (0.65–0.71)h	0.65 (0.63–0.68)
Calibration	—	—	Observed vs. expected	Hosmer Lemeshow	—
Mean age (SD)	72a	67a	71a	70a	54
Male sex (%)	55	58	56	69	57
Treatment (%)
Antiplatelet	b	78%	21%	b	37%
DOAC	b	—	28%	10%	2%
VKA	100%	—	39%	64%	23%

DOAC, direct oral anticoagulant; EPV, events per variable; NA, not applicable; RCT, randomized controlled trial; SD, standard deviation; USA, United States of America; UK, United Kingdom; VKA, vitamin K antagonists.

^aMean.

^bNo information.

^cPerson years.

^d C-statistic per subgroup: CHA₂DS₂-VASc: 0.61 (0.51–0.70) in AF cohort on OAC, 0.58 (0.44–0.73) in AF cohort not on OAC.

^eNo information.

^fNot stated.

^g C-statistic per subgroup: GARFIELD-AF: 0.67 (0.64–0.71) in AF cohort on OAC, 0.69 (0.65–0.72) in AF cohort not on OAC.

^h C-statistic per subgroup: HELT-E₂S₂: 0.70 (0.65–0.76) in AF cohort on OAC, 0.69 (0.64–0.73) in AF cohort not on OAC.

Validation and update studies

Validation studies

A total of 107 validation studies were included, of which 60 validated multiple scores, resulting in a total of 327 validations. In total, 359 373 events occurred in 6 267 728 patients; 12 studies^45,49–59 did not provide population size and number of events. Most of the validations were performed on the CHA₂DS₂-VASc (n = 147) and CHADS₂ (n = 75) (Figure 3). For the newer scores, the number of external validations was limited: the IMRS-VASc (2019), ABCD (2019), AI-model (2019), CAS (2020), HELT-E₂S₂ (2021), and LRAF (2021) were not validated, and the ACTS (2019) was validated only once. Most outcome measures (n = 66) were either defined as IS or thromboembolic events (including IS, TIA, systemic embolism, and pulmonary embolism), whereas other study outcomes (n = 41) were undefined stroke or the composite of stroke and bleeding events. The majority of the studies validated a risk score in a general predominantly Caucasian AF population, with the exception of studies validating prior or post-surgery (n = 6),^60–65 in cohorts with a specific secondary disease (n = 11),^{42,44,66–74} or in cohorts with ethnicity other than predominantly Caucasian (n = 20).^{33,34,45,47,53,55,65,71,75–87} Discrimination was presented in all studies (n = 107) and indicated poor (<0.60) to reasonable (0.60–0.80) and in exceptional cases good (>0.80) discrimination; these values are visualized in the forest plots given in Supplementary material online, section ‘Random-effects meta-analysis’. For the risk scores developed after the publication of the CHA₂DS₂-VASc (2010), no discrimination lower than 0.60 was reported in the validation studies.^{32,40,43–45,47} For calibration, only 14 studies (13%) presented one or more calibration measures: 12 studies presented observed vs. expected risks,^{32,41,42,44,49,55,56,70,88–91} 4 the Hosmer–Lemeshow statistic,^40,42,56,92 calibration in the large,⁷⁰ and the Nam–D’Agostino statistic.⁴⁹ With this limited information on calibration, compared with the CHA₂DS₂-VASc, the modified-CHADS₂, ABC-score, and GARFIELD-AF showed improved calibration measures.^70,88,90,91 Detailed characteristics of the validation studies are presented in Supplementary material online, section ‘Characteristics of included validation studies’.

Update studies

All of the 40 update studies updated the CHA₂DS₂-VASc (n = 25), the CHADS₂ (n = 7), or both scores (n = 8)—no update studies were found on other risk scores. Ten studies updated these scores multiple times, resulting in a total of 55 updated scores for the CHA₂DS₂-VASc and 21 for CHADS_2.^{59,65,83,85,93–99} Except for seven scores, all scores were updated by adding one or more predictors to the original score (e.g. R₂ CHA₂DS₂-VASc: the original score with the additive predictor ‘renal failure’). New predictors included blood or urine biomarkers (e.g. D-dimer, IL-6, soluble fibrin monomer complex),^{59,74,82–84,86,92,97,98,100–111} echocardiographic characteristics (e.g. hypertrophic cardiomyopathy),^65,87,99,112 electrocardiographic markers (e.g. P-wave indices),⁹² genetics (e.g. microRNAs),¹¹³ or socio-economic status.¹¹⁴ In the remaining scores, either predictors were left out,^85,95 replaced,^80,85 or the original predictors were assessed over time (i.e. differences between the baseline and follow-up CHA₂DS₂-VASc scores).^81,94 For the CHA₂DS₂-VASc, discrimination was available for 49 updates (89%): c-statistics improved for all but three updates when compared with the original CHA₂DS₂-VASc.^57,115,116 For the CHADS₂, discrimination was presented in 17 updates (80%): c-statistics improved for all but four updates when compared with the original CHADS₂.^85,115,116 More detailed information on the new scores and their predictive performances are shown in Supplementary material online, section ‘Characteristics of included update studies’.

Pooled c-statistic

For 10 scores (CHA₂DS₂-VASc, CHADS₂, AFI, Framingham, ATRIA, SPAF, ABC, modified-CHADS₂, GARFIELD-AF, and GARFIELD-AF II) pooled c-statistics were calculated in a random-effects meta-analysis. Pooled c-statistics ranged from 0.598 [95% confidence interval (CI) 0.558–0.636, 12 validations] for the AFI to 0.715 (0.674–0.754, 10 validations) for the modified-CHADS₂ (Table 3 and Figure 4). Pooled c-statistics were 0.644 (0.635–0.653) and 0.658 (0.644–0.672) for the CHA₂DS₂-VASc and CHADS₂, respectively. All risk scores that were developed after the publication of the CHA₂DS₂-VASc (2010) showed better discriminative abilities when compared with the CHA₂DS₂-VASc (Figure 4). For the CHA₂DS₂-VASc, CHADS₂, and ATRIA, pooled c-statistics were derived from >10 studies per risk score including large sample sizes ranging from more than a half to more than 3 million patients. Of these three, the ATRIA performed superiorly compared with the CHADS_2, and CHA₂DS₂-VASc. Nevertheless, the differences between the pooled c-statistics were marginal and all corresponded with poor to reasonable model performance.

Table 3.

Results of the random-effects meta-analysis: pooled c-statistic ranging from largest to smallest validation cohort

Risk score characteristics	Development					Validation
	Original sample size	N of events	Original c-statistic	Model type	Time frame	N of studiesb (N of validations)	Sample size	N of events	Pooled c-statistic
CHA2DS2-VASc	1084	25	0.61	Risk score	1 year	82 (n = 135)	3 229 267	169 199	0.644 (0.635–0.653)
CHADS2	1733	94	0.82	Risk score	1 year	46 (n = 68)	1 479 228	71 644	0.658 (0.644–0.672)
ATRIA	10 927	685	0.73	Risk score	1 year	11 (n = 24)	562 443	45 444	0.683 (0.658–0.708)
AFI	5955	208	—	Risk score	1 year	7 (n = 12)	153 530	6879	0.598 (0.558–0.636)
GARFIELD-AF	38 935	511	0.69	Formula	1 year	4 (n = 13)	149 848	5427	0.707 (0.676–0.737)
SPAF	2012	130	—	Risk score	1 year	5 (n = 5)	116 864	3778	0.650 (0.564–0.726)
Modified-CHADS2	51 807	5526	0.72	Risk score	5 years	5 (n = 10)	109 313	10 083	0.715 (0.674–0.754)
Framingham	705	83	0.66	Risk score	5 years	6 (n = 8)	95 145	2716	0.633 (0.602–0.662)
The ABC stroke risk score	27 929 a	391	0.68	Risk score	1 year	5 (n = 11)	40 340	1441	0.678 (0.658–0.697)
GARFIELD-AF II	52 032	957	0.68	Formula	NA	1 (n = 2)	NA	NA	0.690 (0.672–0.707)

NA, data not available. Note that the number of validations slightly differs with Figure 3 since some validations needed to be excluded from the random effect meta-analysis due to studies omitting confidence intervals.

^aPerson years.

^bValidation studies were included if discrimination measures (c-statistic) together with confidence intervals were available.

Figure 4.

Discriminative performances over time. In this figure, the risk scores’ years of publication (x-axis) are plotted against the number of AF patients (y-axis 1, lower panel), the number of validation studies (y-axis 2, middle panel), and results of the pooled c-statistic (y-axis 3, upper panel). The horizontal lines and the marked red area in the upper panel indicate the c-statistic and 95% confidence interval of the CHADS₂ (first horizontal line) and CHA₂DS₂-VASc (second line) during the perioded these scores were endorsed by the relevant guidelines (resp 2006 to 2012 and 2012 to present). The pooled c-statistics were generated from the random-effect meta-analysis. Ten risk scores were included in the random-effects analysis and are therefore displayed in the figure. ^aWalraven and ACTS were externally validated, yet not included in the analysis as the yielded discrimination measures (c-statistic) did not include confidence intervals, which were required for the random-effects meta-analysis (more details in the Statistical analysis section). AF, atrial fibrillation.

Sensitivity analyses

In the 10 risk scores that were used for the random-effects meta-analysis, multiple sensitivity analyses were conducted. The outcomes of most sensitivity analyses showed no, or only marginal differences, indicating that age, anticoagulation, ethnicity, PROBAST score, prespecified IS risk, and year of publication (respectively, sensitivity analyses 1, 3–6, and 8) have no or only limited effect on most risk scores’ discriminative abilities. For sensitivity analysis 2 on the effect of different outcome definitions, the modified-CHADS₂ showed improved discriminative abilities for the outcome IS [0.749 (0.710–0.785)], when compared with the outcome of thromboembolism [0.626 (0.561–0.686)], indicating that the original pooled c-statistic of the modified-CHADS₂ [0.715 (0.674–0.754)] might be an underestimation of the scores true performance on predicting IS only. In analysis 7 on study design, overall observational studies performed marginally better [e.g. for the CHA₂DS₂-VASc, the c-statistic was 0.649 (95% CI 0.639–0.658) for observational studies and 0.619 (0.603–0.634) for RCTs]. The number of validation studies in RCTs, however, was substantially lower than for observational studies. Consequently, we expect the effect of the inclusion of RCTs in our pooled calculations to be minimal. More detailed information on the sensitivity analyses can be found in the Supplementary material online, section ‘Sensitivity analyses’.

Discussion

Summary

In this systematic review and meta-analysis, we reviewed risk scores predicting IS in AF patients using data of over 6 million individuals. We identified 19 original scores predicting IS in AF patients, which were validated a total of 327 times, and updated 76 times—nearly all on either the CHA₂DS₂-VASc or CHADS₂. Of these 19 scores, 10 were included in our meta-analysis on their discriminatory abilities in external validations. Although all risk scores showed poor to only reasonable performance, the scores published after the CHA₂DS₂-VASc (2010) tended to have slightly better discriminatory abilities when compared with the scores published before the CHA₂DS₂-VASc, with the exception of the modified-CHADS₂ (published in 2008) which showed overall the best discriminative abilities. Update studies were found for the CHA₂DS₂-VASc and CHADS₂ only, showing improved discrimination. Information on calibration was omitted in nearly all studies developing, validating or updating a risk score. We observed methodological biases in all studies.

Clinical implication

By presenting a comprehensive comparison of the pooled discriminative performances of 19 risk scores, our study may support the choice of the best discriminative risk score in clinical or research settings. Faced with a patient diagnosed with new-onset AF, the clinician using such a risk score will more correctly classify the patient as low, intermediate, or high risk for IS.^5–7 With this knowledge, one can argue that the clinician might consider the use of other risk scores than the CHA₂DS₂-VASc, for example, the modified-CHADS₂ or one of the newer risk scores instead to classify his or her patients. However, though differences were found on discriminatory abilities, conclusions on the risk score’s overall predictive performances should be taken cautiously. First, differences in discriminative abilities were only marginal and all corresponded with poor to only reasonable score performance. Secondly, not only discrimination but also calibration (i.e. the agreement between the predicted and the observed risk) is a key element in risk score assessment.^15,70 Yet, nearly all studies developing, validating or updating a risk score omitted information on this essential aspect of predictive modelling. Without knowing whether a risk score over- or underpredicts the observed risk of IS, misclassification and thus over- or undertreatment may be a serious issue but unbeknownst to the clinician in the absence of reliable data on calibration.^15,70 Finally, it should be noted that these promising newer risk scores updates have not, or only limited, been externally validated.

Comparison to literature

Our findings confirm the modest overall discriminatory abilities of commonly used risk scores regarding IS in AF patients.^8–10 This finding may be attributed to multiple factors, such as differences in study designs, measurement methods, and study case mix.^117,118 For example, the CHADS₂ is developed in a heterogeneous population of patients with AF and shows good discriminative abilities in the development cohort,⁴⁶ in contrast to moderate discrimination in the validation cohorts. When validating the CHADS₂ in less heterogeneous but clinically relevant populations, for example, in patients with AF and impaired kidney function, risk score performance drops.^{70, 95,115,119} While the components of the risk score may predict well in heterogeneous general AF cohorts, other risk factors more specific for this homogeneous high-risk population may improve the discriminative abilities.^11,15,20,70 Indeed, supporting this mechanism, probably due to the inclusion of more specific predictors, a slight improvement in discriminative ability was found in the newer risk scores and updates

Strengths and limitations

Our study comes with strengths and limitations. The main strength regards the assessment of the predictive abilities of these risk scores by performing a random-effect meta-analysis including a well-powered cohort of more than 6 million patients, yielding robust pooled c-statistics for the 10 most commonly validated risk scores. The study has, however, several limitations. First, no random-effects meta-analysis was conducted on calibration, due to the limited number of studies assessing calibration and the heterogeneity of these calibration measures. As a consequence, conclusions on overall predictive performances should be taken with caution as not only discrimination but also calibration is essential in risk score assessment. Future development and validation studies should include assessments on calibration to enable pooled calibration measures. Secondly, all studies were indicative for methodological bias based on the PROBAST, questioning the reliability of the presented discriminative abilities.¹¹⁸ High risk of bias and concerns regarding applicability were mostly observed in the outcome and analysis domain and is common in prediction research, regardless of publication year.¹²⁰ Due to the similarity in the PROBAST scores, we performed a subgroup analysis based on the median PROBAST score to gain more insight in the effect of high vs. low risk of bias. Though no substantial effects were found, our risk of bias assessment underlines the work that needs to be done in the outcome and analysis domains. In future studies, adhering to the PROBAST²⁰ and TRIPOD¹³ guidelines is warranted. Thirdly, a substantial part of the studies based the risk scores’ discriminative abilities on the prediction of composite outcomes, instead of IS prediction only. The use of composite outcomes is debated, especially if the outcomes are contradictive to each other and need different treatment strategies, for example, IS and haemorrhagic stroke. In our study, however, outcomes predominantly regarded non-contradictive composite outcomes (e.g. IS together with TIA or systemic embolism) which may be defendable from the perspective of a clinician. Moreover, we studied the effect of composite outcome usage by means of a sensitivity analysis: no substantial effects were found, indicating that the inclusion of composite outcomes did not, or only marginally, influence our study results. Next, most studies validated the risk scores in patients already on anticoagulation treatment, whereas in clinical practice the risk scores are used as a tool for therapy decision. Validating in patients already using anticoagulation might have led to bias: high risk patients are more likely to receive anticoagulation, and thus paradoxically, a reduced risk of developing IS may be observed in them, leading to suboptimal performance of stroke risk scores.^121,122 To assess whether this treatment paradox was present, we performed a sensitivity analysis on anticoagulation use, which showed only marginal differences. Another limitation regards the exclusion of studies in the random-effect meta-analysis when no confidence interval was presented next to the c-statistic, possibly leading to selection bias. Yet, due to the large number of studies that included confidence intervals (n = 95, 89%), we do not expect the exclusion of the small subset (n = 12, 11%) to have an influential impact on the average performances. Also, we excluded non-English studies, which might have had influence on the limited number of studies that were included regarding ethnicities other than Caucasian. Finally, we limited our study to IS risk prediction and did not focus on bleeding risk prediction. It should be noted that for the decision-making of anticoagulation therapy in AF patients, bleeding risk scores and their predictive performances are of importance as well.

Conclusion

We identified 19 primary risk scores regarding IS risk in patients with AF along with 327 validations and 76 updates, mostly on the CHADS₂ and CHA₂DS₂-VASc. All risk scores showed largely similar, poor to reasonable, and discriminative performance; information on calibration was not reported for most studies. Compared with the CHADS₂ and CHA₂DS₂-VASc, newer risk scores and updates showed improved discrimination and might therefore be considered for use in clinical practice. To confirm this positive trend, external validations to assess discrimination but especially calibration of these newer risk scores are needed.

Supplementary material

Supplementary material is available at Europace online.

Funding

The work on this study by J.M. and M.v.D. was supported by a grant from the Dutch Kidney Foundation (20Ok016).

Data availability

The data underlying this article will be shared on reasonable request to the corresponding author.