Comparison of Risk Stratification Schemes to Predict Thromboembolism in People With Nonvalvular Atrial Fibrillation

Abstract

Objectives

We assessed 5 risk stratification schemes for their ability to predict atrial fibrillation (AF)–related thromboembolism in a large community-based cohort.

Background

Risk schemes can help target anticoagulant therapy for patients at highest risk for AF–related thromboembolism. We tested the predictive ability of 5 risk schemes: the Atrial Fibrillation Investigators, Stroke Prevention in Atrial Fibrillation, CHADS2 (Congestive heart failure, Hypertension, Age ≥ 75 years, Diabetes mellitus, and prior Stroke or transient ischemic attack) index, Framingham score, and the 7th American College of Chest Physicians Guidelines.

Methods

We followed a cohort of 13,559 adults with AF for a median of 6.0 years. Among non-warfarin users, we identified incident thromboembolism (ischemic stroke or peripheral embolism) and risk factors from clinical databases. Each scheme was divided into low, intermediate, and high predicted risk categories and applied to the cohort. Annualized thromboembolism rates and c-statistics (to assess discrimination) were calculated for each risk scheme.

Results

We identified 685 validated thromboembolic events that occurred during 32,721 person-years off warfarin therapy. The risk schemes had only fair discriminating ability, with c-statistics ranging from 0.56 to 0.62. The proportion of patients assigned to individual risk categories varied widely across the schemes. The proportion categorized as low risk ranged from 11.7% to 37.1% across schemes, and the proportion considered high risk ranged from 16.4% to 80.4%.

Conclusions

Current risk schemes have comparable, but only limited, overall ability to predict thromboembolism in persons with AF. Recommendations for antithrombotic therapy may vary widely depending on which scheme is applied for individual patients. Better risk stratification is crucially needed to improve selection of AF patients for anticoagulant therapy.

Nonvalvular atrial fibrillation (AF) is a major risk factor for thromboembolism, causing approximately 15% of the ischemic strokes in the U.S. (1). Anticoagulant therapy with vitamin K antagonists such as warfarin sodium can substantially reduce the risk of AF–related thromboembolism, but at the risk of incurring more hemorrhagic complications (2). Accurate stratification of patients with AF by thromboembolism risk should ideally target the use of warfarin for patients at highest risk of thromboembolism and reduce the exposure of low-risk patients to the complications of warfarin (3–5).

Several prominent risk stratification schemes have been developed to help distinguish those patients with AF who are at high risk for ischemic stroke and other systemic thromboembolism from those with a risk sufficiently low that anticoagulation might not be beneficial when considering the associated bleeding risks (2,6–9). Two schemes were developed from multivariable analyses of pooled data from randomized trial participants: the Atrial Fibrillation Investigators (AFI) and the Stroke Prevention in Atrial Fibrillation (SPAF) risk schemes (2,6). The CHADS₂ (Congestive heart failure, Hypertension, Age ≥ 75 years, Diabetes mellitus, and prior Stroke or transient ischemic attack) index, named for a combination of clinical risk factors, was subsequently developed from an amalgamation of the AFI and SPAF schemes and validated using data from a registry of hospitalized Medicare beneficiaries with AF (7). A risk score based on the Framingham Heart Study cohort was developed to predict 5-year risk of stroke, but it has yet to be validated in other community settings (8). Finally, a guideline from the 7th American College of Chest Physicians Conference on Antithrombotic and Thrombolytic Therapy (ACCP) was developed through expert opinion and review of the available evidence (9).

Although many of the clinical variables overlap among the different risk schemes, there are differences in how specific variables are coded (e.g., age as a continuous variable as opposed to a dichotomous variable), combined, and weighted. Because these variations may lead to significant differences in whether patients are categorized as being at low or high risk for thromboembolism, the choice of guideline could potentially influence the recommendation of warfarin for individual patients. We assessed the predictive ability of these 5 risk stratification schemes when applied to a large, independent, community cohort of patients with AF.

Methods

The ATRIA (AnTicoagulation and Risk Factors In Atrial Fibrillation) study is a cohort of 13,559 adults with diagnosed nonvalvular AF who received care within Kaiser Permanente of Northern California, a large integrated health care delivery system. Details of the cohort assembly have been described previously (10,11). We identified patients with a diagnosis of AF between July 1, 1996, and December 31, 1997, by searching automated inpatient, outpatient, and electrocardiographic databases for physician-assigned International Classification of Diseases-Ninth Revision-Clinical Modification (ICD-9-CM) diagnosis of AF (427.31). The cohort was followed up through September 2003, a median follow-up of 6.0 years (interquartile range 3.1 to 6.7 years). To specifically address nonvalvular AF, we excluded patients with diagnoses of mitral stenosis, documented valvular repair or replacement, transient post-operative AF, or concurrent hyperthyroidism. Warfarin exposure among patients was determined from computerized records from pharmacy, laboratory, and ambulatory visits using previously described and validated methods (10). The analyses for our study were restricted to the 10,932 patients who had periods of time when they appeared not to be taking warfarin.

Patient characteristics and risk stratification schemes for thromboembolism

Each of the 5 risk stratification schemes (Table 1) was constructed to assign patients to low, intermediate, and high thromboembolism risk categories, consistent with previous studies (3–5). The specific risk factors included age, gender, history of ischemic stroke, diagnosed heart failure, diagnosed hypertension, and diabetes mellitus. Data for individual medical conditions were obtained by searching for relevant ICD-9-CM codes in automated clinical databases during the 5 years before enrollment and throughout follow-up. This search strategy has been previously validated by comparing diagnoses to a review of a subset of patients’ medical records (10). The ATRIA cohort lacked data on individual patients’ systolic blood pressure and left ventricular ejection fraction during the study period, risk factors that were used in the SPAF and Framingham risk schemes (6,8). Instead, in our study, a clinical diagnosis of hypertension was substituted in place of systolic blood pressure >160 mm Hg, and diagnosed heart failure was substituted for left ventricular fractional shortening ≤25%. The Framingham risk score used systolic blood pressure as a risk factor, with 0 to 4 points assigned to specific ranges of systolic blood pressure. For our study, we considered a diagnosis of hypertension in the ATRIA cohort to equal 3 points on the Framingham score and then conducted a sensitivity analysis of whether substituting points of 2 or 4 resulted in significantly different results.

Table 1.

5 Risk Stratification Schemes Used to Predict Thromboembolism in Persons With Nonvalvular Atrial Fibrillation^*

Risk Scheme	Low Risk	Intermediate Risk	High Risk
AFI	Age <65 yrs and no risk factors	Age >65 yrs and no other risk factors	Prior ischemic stroke or transient ischemic attack, history of hypertension, history of diabetes mellitus
SPAF	No risk factors	History of hypertension	Prior stroke, women older than 75 yrs, recent clinical heart failure, left ventricular fractional shortening ≤25% on echocardiography
CHADS2†	Score 0	Score 1 to 2	Score 3 to 6
Framingham‡	Score 0 to 7	Score 8 to 15	Score 16 to 31
7th ACCP	Age <65 yrs and no other risk factors	Age 65 to 75 yrs and no other risk factors	Prior ischemic stroke, age >75 yrs, moderate to severe left ventricular dysfunction, history of hypertension, diabetes mellitus

^*The AFI, SPAF, CHADS₂, and Framingham schemes were developed to predict atrial fibrillation–related ischemic stroke, not ischemic stroke plus peripheral embolism.

^†The CHADS₂ index is a point system that assigns 1 point each for Congestive heart failure, Hypertension, Age 75 years or older, and Diabetes mellitus and 2 points for prior Stroke or transient ischemic attack (7).

^‡The Framingham score is a point system based on the following clinical factors: age (0 to 10 points), female gender (6 points), systolic blood pressure (0 to 4 points), diabetesmellitus (5 points), and prior ischemic stroke or transient ischemic attack (6 points) (8).

ACCP = American College of Chest Physicians Conference on Antithrombotic and Thrombolytic Therapy guidelines (9); AFI = Atrial Fibrillation Investigators (2); CHADS₂ = Congestive heart failure, Hypertension, Age ≥ 75 years, Diabetes mellitus, and prior Stroke or transient ischemic attack; SPAF = Stroke Prevention in Atrial Fibrillation (6).

Identification of thromboembolic events

We searched the health plan electronic hospitalization and billing records for primary discharge diagnoses referring to incident thromboembolic events, either ischemic stroke or other peripheral embolism. The validity of potential events was then adjudicated by an outcomes committee of 3 physicians using a formal study protocol, with at least 2 physicians reviewing the medical records for each potential event and a third physician reviewing cases in which the initial reviewers disagreed. If there was no consensus on the validity of an event, an expert neurologist adjudicated the event. Valid ischemic strokes were defined as neurological deficits of sudden onset that persisted for more than 24 h and were not explained by other etiologies (e.g., trauma, infection, or vasculitis). Valid nonstroke peripheral emboli were defined as emboli identified by radiographic imaging, intraoperative examination, or pathological findings, and without underlying atherosclerotic disease in the affected artery. We excluded outcome events that occurred during hospitalization or as a complication from a diagnostic or interventional procedure. Because the goal of our study was to examine the comparative utility of thromboembolic risk schemes for the purpose of informing anticoagulant therapy decision making, we only included thromboembolic events that occurred off warfarin therapy, determined by chart review.

Statistical analyses

Analyses were restricted to patients who were not taking warfarin therapy. Patients off warfarin were assigned to a low, intermediate, or high thromboembolism risk category for each risk scheme, and thromboembolism rates for each category were calculated as the number of events per 100 person-years of follow-up off of warfarin therapy. Because patients could accumulate additional clinical risk factors over time, we updated each patient’s risk factor status throughout follow-up using weighted logistic regression models. Agreement between pairs of risk schemes was tested using weighted kappa statistics. The discriminatory ability of each risk scheme as applied to the ATRIA cohort was determined by calculating a c-statistic using logistic regression analysis. The c-statistic is analogous to the area under the receiver-operating characteristic curve and represents the probability that the predicted risk is higher for a patient with the outcome than without (12). A c-statistic of 1 indicates perfect discrimination, whereas a value of 0.5 indicates no discrimination (12,13). Because individual subjects in the ATRIA cohort could have alternating periods on and off warfarin, we weighted the analyses for the time each patient was followed up off of warfarin therapy. In addition, to address potential bias occurring because of patients having different lengths of follow-up off warfarin, we also calculated c-statistics for each risk scheme when applied to a subgroup of 5,588 patients who were not on warfarin at baseline and who had continuous follow-up off warfarin therapy for a fixed period of 12 months.

All analyses were performed using SAS software, version 9.1 (SAS Institute Inc., Cary, North Carolina). The institutional review boards of the collaborating institutions approved the study, and the need for individual patient written consent was waived because of the nature of the study.

Results

A total of 32,721 person-years of follow-up among patients with AF not taking warfarin were available for analysis. The mean age of patients at baseline was 72 years, and 78.7% had at least 1 clinical risk factor for thromboembolism (Table 2). Forty percent of patients acquired at least 1 additional clinical risk factor over the follow-up period. The proportion of cohort member person-years categorized as low risk varied considerably across the risk schemes, ranging from 11.7% of the cohort using the 7th ACCP guidelines to 37.1% using the Framingham risk scheme (Table 3). There was also substantial variation in the proportion of patients considered high risk, which ranged from 16.4% of the cohort using the Framingham risk scheme to as high as 80.4% using the 7th ACCP guidelines. Among patients who were not taking warfarin at baseline, agreement between schemes varied from poor (kappa of 0.21 for the Framingham scheme vs. the 7th ACCP guidelines) to good (kappa of 0.71 for the AFI vs. the 7th ACCP guidelines).

Table 2.

Clinical Characteristics of Persons With Nonvalvular Atrial Fibrillation Not Taking Warfarin at Baseline

Clinical Characteristic	No. of Patients (%)
Age 75 yrs or older	5,026 (46.0)
Women	4,730 (43.3)
Prior ischemic stroke	909 (8.3)
Diagnosed hypertension	5,519 (50.5)
Diagnosed heart failure	3,143 (28.8)
Diabetes mellitus	1,804 (16.5)

Table 3.

Proportion of ATRIA Cohort Categorized by 5 Risk Stratification Schemes Used to Predict Atrial Fibrillation–Related Thromboembolism and Discriminatory Ability of Risk Schemes (c-Statistics)

	Risk for Thromboembolism (%)			c-Statistic
	Low	Intermediate	High	All Patients	Subgroup*
AFI	13.1	24.7	62.3	0.56	0.61
SPAF	27.7	28.5	43.8	0.60	0.65
CHADS2	18.8	61.2	20.1	0.58	0.67
Framingham	37.1	46.6	16.4	0.62	0.69
7th ACCP	11.7	7.9	80.4	0.56	0.60

^*Subgroup of 5,588 patients not on warfarin at baseline and with continuous follow-up off of warfarin for at least 12 months.

Abbreviations as in Table 1.

We identified 685 validated thromboembolic events (643 ischemic strokes and 42 peripheral emboli) that occurred off warfarin therapy, for an overall rate of 2.1 per 100 personyears. The observed rates in all risk schemes increased monotonically from low- to high-risk categories, but the thromboembolic event rates in individual risk categories varied across the risk stratification schemes (Fig. 1). The group of patients categorized as low risk using the 7th ACCP guidelines had an observed annualized thromboembolism rate of 0.13% (95% CI 0.05% to 0.32%). In comparison, the group of patients categorized as low risk using the Framingham risk scheme had an observed thromboembolism rate of 0.81% (95% CI 0.66% to 0.99%). The high-risk patients of the 7th ACCP guidelines had an observed thromboembolism rate of 2.5% (95% CI 2.3% to 2.7%), whereas the Framingham risk scheme high-risk patients had an observed rate of 3.9% (95% CI 3.4% to 4.5%).

Figure 1. Annual TE Rates Across Risk Groups Using 5 Risk Stratification Schemes Used to Predict AF-Related TE.

The double-barred lines represent 95% confidence intervals. ACCP = American College of Chest Physicians Conference on Antithrombotic and Thrombolytic Therapy; AF = atrial fibrillation; AFI = Atrial Fibrillation Investigators; CHADS₂ = congestive heart failure, hypertension, age ≥75 years, diabetes mellitus, and prior stroke or transient ischemic attack; SPAF = stroke prevention in atrial fibrillation; TE = thromboembolism.

All risk schemes had only a fair ability to separate patients into risk categories that corresponded to different rates of thromboembolism, which was reflected in c-statistics ranging from 0.56 for the AFI and 7th ACCP risk schemes to 0.62 for the Framingham risk scheme (Table 3). These results did not change materially in the subgroup analysis of patients with continuous follow-up off warfarin for 12 months (Table 3). Patients with and without thromboembolism had highly overlapping risk category distributions; these results are illustrated for the CHADS₂ index in Figure 2. As shown in Figure 3all 5 risk schemes have roughly comparable receiver-operating characteristic curves and discriminating ability that is only moderately better than the 45° line of no information (12).

Figure 2. Proportion of ATRIA Cohort Off Warfarin and Categorized by CHADS₂ Scores, Stratified by Development of TE.

The distribution of person-years contributed by patients not sustaining a TE is in blue and the distribution of person-years contributed by patients sustaining a TE is in yellow. Abbreviations as in Figure 1.

Figure 3. ROC Curves for 5 Risk Stratification Schemes Used to Predict AF-Related Thromboembolism.

The 45° dotted line represents the line of no information. ROC = receiver-operating characteristic; other abbreviations as in Figure 1.

In additional sensitivity analyses, the c-statistic for the Framingham score was essentially unchanged when the diagnosis of hypertension was reassigned a point score of 2 or 4. A scoring system for the CHADS₂ and Framingham risk schemes that used their full range of possible values resulted in only marginally better c-statistics than when three categories of risk were used: 0.60 using a continuous score versus 0.58 using a 3-category score for the CHADS₂ index, and 0.64 versus 0.62 for the Framingham score. Finally, restricting the analyses to primary prevention patients (i.e., those without a history of prior stroke) did not change our results materially.

Discussion

Among the 5 major risk stratification schemes commonly used to predict thromboembolism in patients with nonvalvular AF, no one risk scheme seemed to be superior. Furthermore, all 5 schemes had at best only fair discriminating ability when applied to this cohort, with c-statistics ranging from 0.56 to 0.62, far from the ideal c-statistic target of 1. Although these risk categories were labeled as low, intermediate, and high, the absolute rates of thromboembolism across these ordered categories increased by only small degrees. The 7th ACCP guideline’s categories of low-risk, intermediate-risk, and high-risk categories, for example, corresponded to absolute thromboembolism rates of 0.13%, 0.89%, and 2.5% per year, only a modest revision of the absolute risks. These low absolute event rates dictate that the majority of patients, even those predicted to be at high risk for thromboembolism, will not sustain a thromboembolic event. Better ways to predict thromboembolism are clearly needed to optimize the use of anticoagulant therapy in patients with AF, both to prevent the overuse of anticoagulants in patients with low absolute risks of thromboembolism and to target the use of highly effective anticoagulant therapy to those patients who would most greatly benefit.

Additional independent risk factors for AF–related thromboembolism may not be included in current risk schemes. Women, for example, seem to have a higher risk for thromboembolism in the setting of atrial fibrillation independent of other clinical risk factors (14). Other clinical features and more novel biomarkers, including genetic factors, may emerge as independent incremental risk factors as well (15–17). However, it is unknown whether the incorporation of such risk factors will meaningfully improve the discrimination of current risk schemes. In other situations, such as in predicting coronary events or stroke, the addition of new risk factors has resulted in only small improvements in the discriminatory ability of risk schemes as compared with conventional risk factors alone (18–21). Future validation studies are clearly needed to assess the marginal utility of additional novel risk factors. Although there are concerns that the c-statistic is not an optimal summary measure of the value of a prediction scheme (22), c-statistics in the range we observed clearly indicate relatively poor discriminating ability.

Thromboembolic risk stratification for AF has some challenging features. Anticoagulant therapy is highly effective in reducing the risk of thromboembolism in AF. Because the consequences of ischemic stroke can be devastating, treatment thresholds for anticoagulation can be set at fairly low absolute risks. Yet unlike many other cardiovascular preventive therapies, warfarin is associated with potentially life-threatening complications, drug–drug interactions, and burdensome monitoring and dose adjustment (23–25). This dilemma highlights the importance of improving current methods of predicting thromboembolism. It also highlights the need for better ways to risk stratify patients for major hemorrhage, in particular intracranial hemorrhage, which leads to most of the disability and death from warfarin toxicity (26).

There are several limitations to our study. The ATRIA cohort lacked data on individual patients’ systolic blood pressure and left ventricular systolic function. However, most applications of the widely cited CHADS₂ index also do not use such information, and in the original models of the Framingham risk score, systolic blood pressure was not independently statistically significant despite its being incorporated into the final risk scheme (8). The rates of ischemic stroke and other systemic embolism observed in the ATRIA cohort are somewhat lower than those observed in the early randomized trials and in cohorts assembled from hospitalized patients with AF (2,7). The low thromboembolism rates observed in the ATRIA study may reflect the fact that our cohort was assembled primarily from ambulatory settings or perhaps because stroke risk factors were better managed in this more contemporary set of patients (27); other contemporary studies of patients with AF report thromboembolism rates similar to the ATRIA study (28,29). Aspirin use was unavailable in our study because nonprescription aspirin would not be recorded in the pharmacy database. Widespread aspirin use among nonanticoagulated patients may have contributed to somewhat lower rates of thromboembolism overall. The c-statistics reported in our study are quite consistent with the results obtained from the pooled trial populations and the Framingham cohort, and it is unlikely that our study limitations materially affect the validity of our core findings regarding the discriminating ability of standard risk schemes (8).

Conclusions

Current risk stratification schemes used to predict thromboembolism in persons with nonvalvular AF have similar discriminatory ability, but the ability is relatively poor. Until better means of risk stratification are available, a large proportion of patients with AF who would not have developed thromboembolism may be exposed to the risks associated with warfarin therapy. In addition, differences across risk schemes could potentially lead to substantial variation in whether or not individual patients are recommended warfarin therapy. Further research is needed to develop more accurate ways to identify prospectively those patients with AF who will sustain a thromboembolic event without warfarin therapy, and similarly, robust methods are needed to identify reliably those patients who will suffer complications when treated with anticoagulants.

Abbreviations and Acronyms

ACCP

American College of Chest Physicians Conference on Antithrombotic and Thrombolytic Therapy

AF

atrial fibrillation

AFI

Atrial Fibrillation Investigators

CHADS₂

Congestive heart failure, Hypertension, Age ≥75 years, Diabetes mellitus, and prior Stroke or transient ischemic attack

CI

confidence interval

ICD-9-CM

International Classification of Diseases-Ninth Revision-Clinical Modification