Lack of Concordance between Empirical Scores and Physician Assessments of Stroke and Bleeding Risk in Atrial Fibrillation: Results from the Outcomes Registry for Better Informed Treatment of Atrial Fibrillation (ORBIT-AF) Registry

Abstract

Background

Physicians treating patients with atrial fibrillation (AF) must weigh the benefits of anticoagulation in preventing stroke versus the risk of bleeding. While empirical models have been developed to predict such risks, the degree to which these coincide with clinicians’ estimates is unclear.

Methods and Results

We examined 10,094 AF patients enrolled in the Outcomes Registry for Better Informed Treatment of AF (ORBIT-AF) registry between June, 2010 and August, 2011. Empirical stroke and bleeding risks were assessed using the CHADS₂ and ATRIA scores, respectively. Separately, physicians were asked to categorize their patients’ stroke and bleeding risks: low- (<3%); intermediate- (3-6%); and high-risk (>6%). Overall, 72% (n=7251) in ORBIT-AF had high-risk CHADS₂ scores (≥2). However, only 16% were assessed as high stroke risk by physicians. While 17% (n=1749) had high ATRIA bleeding risk (score ≥5), only 7% (n=719) were considered so by physicians. The associations between empirical and physician-estimated stroke and bleeding risks were low (weighted Kappa 0.1 and 0.11, respectively). Physicians weighed hypertension, heart failure, and diabetes less significantly than empirical models in estimating stroke risk; physicians weighted anemia and dialysis less significantly than empirical models when estimating bleeding risks. Anticoagulation use was highest among patients with high stroke risk, assessed by either empirical model or physician estimates. In contrast, physician and empirical estimates of bleeding had limited impact on treatment choice.

Conclusions

There is little agreement between provider-assessed risk and empirical scores in AF. These differences may explain, in part, current divergence of anticoagulation treatment decisions from guideline recommendations.

Stroke is the major source of morbidity and mortality in patients with atrial fibrillation (AF).^{1, 2} The use of oral anticoagulants can significantly reduce this risk³, yet this therapy also conveys a risk of bleeding complications. Thus, proper treatment selection requires careful identification of patients in whom stroke prevention outweighs the bleeding risks.

Several tools have been developed to objectively assess these risks of stroke and bleeding in patients with AF,^4,5 and their predictive value has been validated across multiple populations.^{6, 7} However, several studies have demonstrated that actual anticoagulation choices in clinical practice often differ from what would be recommended based on risk scores.⁸ To date, there is limited information available to understand how clinicians assess risk or the degree to which their subjective assessments agree with objective scores.

Using data from the nation's largest clinical registry of AF, this paper will (1) describe physicians’ assessment of patient risk and factors associated with such determinations; (2) describe the level of agreement between physician-assessed risks for stroke and bleeding risk versus those obtained by validated empirical model estimates; (3) identify what patient and provider factors are most associated with discrepancy between physician-assessed risk and empirical scores; and (4) determine whether choice of treatment with OAC was driven more by physician-assigned or objective risk assessments.

Methods

We used data from the Outcomes Registry for Better Informed Treatment of Atrial Fibrillation (ORIBT-AF), which is a national US registry of outpatients with AF managed by primary care physicians, cardiologists, and/or electrophysiologists. A nationally-representative cohort of sites was invited to participate, and an adaptive design was employed to achieve heterogeneity of practice type and geography. Study coordination and site management was performed by the Duke Clinical Research Institute. Eligible patients were age 18 years or older, with electrocardiographically-documented AF that was not due to a reversible cause, and capable of following up every 6 months for at least 2 years. The present analysis included all enrolled patients who had each of the following at baseline: (1) calculable CHADS₂ score, (2) calculable ATRIA score, (3) physician assessment of stroke risk, and (4) physician assessment of bleeding risk.

Data were entered in a web-based case report form, primarily from the patient's medical record and treating physician. Components included demographic data, past medical history, AF history (including symptoms) and interventions, medications, vital signs, laboratory and echocardiographic assessments, and incident events. Additional details of the ORBIT-AF registry have been described previously.⁹

The empirical risk scores used in the current analysis were the CHADS₂ score for stroke risk in AF, and the ATRIA score for bleeding risk in patients with AF. The CHADS₂ score has been empirically validated and previously studied with scores of 0, 1, or ≥2 categorizing low, intermediate and high risk, respectively.^{4, 10}. Similarly, the ATRIA bleeding score has been empirically validated at scores of 0-3, 4, or ≥5 correlating to low, intermediate, and high, respectively (see Supplemental Material, Table S1 and Table S2).^{5, 7}

As part of the ORBIT-AF baseline case report form, the patient's physician was also asked to subjectively classify each patient's individual risk of stroke and bleeding into one of three categories: low (<3%), intermediate (3-6%), or high (>6%). These cutoffs were prospectively defined to generally correlate with observed event rates for low-, moderate-, and high-risk groups by the CHADS₂ and ATRIA risk stratification tools. The physicians could assign this estimate however they deemed appropriate.

The study population was subsequently stratified by empirical and physician-assessed risk categories for both stroke and bleeding (Low Risk, Intermediate Risk, and High Risk). This yielded 4 risk levels for each patient – physician-assessed stroke risk, CHADS₂-assigned stroke risk, physician-assessed bleeding risk, and ATRIA-assigned bleeding risk. Each of these 4 risk assignments had 3 levels – low, intermediate, and high. Baseline characteristics were calculated among these groups. Subsequently, we calculated agreement between empirical risk scores and physician-assessed risk by subtracting the risk level assigned by the scores from that assigned by the physicians, for each stroke and bleeding. This yielded ‘difference’ scores for every patient, for both stroke and bleeding, that ranged from -2 (the physician under-estimated risk by 2 levels [e.g., physician-assessed risk was low and empirically-assigned risk was high]) to +2 (where the physician overestimated risk by 2 levels). Using these difference scores as continuous variables we identified patient factors associated with mismatch between physician-assessed risk and empirically-categorized risk for each stroke and bleeding. Lastly, we assessed rates of treatment with oral anticoagulation (OAC) across these groups.

Statistical Methods

Agreement between physician-assigned (low, medium, high) and empirical (low, medium, high) risk assessment was displayed by presenting number of patients and percentage in each category, and measured as the weighted Kappa statistic.

Baseline characteristics are presented stratified by risk in terms of empirical score and physician-assigned risk (low, intermediate, high) for stroke and bleeding, separately. Continuous variables are presented as medians (Q1-Q3) and categorical variables are presented as proportions. Due to the significant overlap in patient populations of objective and physician-assigned risk strata, statistical testing was not performed between or across these groups.

To assess the factors associated with mismatch between physician-assigned and empirical risk we evaluated the difference in categorization (physician-assigned minus empirical), with possible values of −2, −1, 0, 1, 2 representing the number of categories apart. This outcome was treated as continuous and modeled by multivariable linear regression. A positive parameter estimate, 0.5 for example, implies that, per one unit increase in a covariate, the difference between physician-assigned and empirical risk is expected to be 0.5 categories higher, all else being equal. In short, positive coefficients correspond to factors that concern physicians more and negative coefficients correspond to factors that concern physicians less, relative to the empirical scores. P-values and confidence intervals are based on robust standard errors (to account for the correlation in the same site). Two separate regression models were developed for the evaluation of (1) stroke risk and (2) bleeding risk. The final regression model for each outcome was developed based on selected risk factors from candidate baseline characteristics (see Supplemental Material, Table S3) using backward selection, with an alpha for exclusion of 0.05. All continuous variables were tested for linearity, and non-linear relationships were accounted for using linear splines.

Missing covariate data in the regression analyses were handled by multiple imputation using Markov Chain Monte Carlo and regression methods. The extent of missing covariates, individually and jointly, was evaluated in the first phases of model building. The imputation method used Markov Chain Monte Carlo simulation to create a monotone missing data structure so that subsequent imputation could be performed by regression method. Imputed data for covariates was only used for predictors in the multivariable model, not for the outcome. Backward selection for the final regression model for each outcome was performed in the 1^st imputed data set. Final estimates and associated standard errors reflect the combined analysis over five imputed data sets Approval for ORBIT-AF was obtained from the Duke University institutional review board (IRB), and all sites obtained IRB approval subject to local regulations. All patients signed written, informed consent. Analyses of the aggregate, de-identified data were performed by the Duke Clinical Research Institute using SAS software (version 9.3, SAS Institute, Cary, North Carolina, USA).

Results

The overall ORBIT-AF population included 10,132 patients from 176 sites. After excluding patients missing parameters of the CHADS₂ and/or ATRIA scores (n=3) and those missing physician assessment of stroke and/or bleeding risk (n=35), this yielded a final study cohort of 10,094 patients from 176 sites.

Comparison of Physician-Assigned versus Empirical Risk Assessments

Overall, clinicians rated 1625 patients (16%) as having High Risk of stroke, versus 7251 (72%) who were considered High Risk by CHADS₂ (Table 1, Figure Legends FigureA). Similarly, clinicians rated 719 patients (7%) High Risk of bleeding on anticoagulation therapy versus 1749 (17%) considered High Risk by ATRIA score, (Table 1, Figure Legends FigureB). The overall weighted Kappa score between physician-assigned stroke risk and empirical stroke risk assessment was 0.10 ( 95% CI=0.10-0.11), indicating poor agreement.^{11, 12} Similarly, the weighted Kappa between physician-assigned bleeding risk and empirical bleeding risk assessment was 0.11 (95% CI=0.09-0.12) also indicating poor overall agreement.

Table 1.

Agreement between empirical and physician-assigned risk assessment for stroke and bleeding.

Stroke Risk				Bleeding Risk
Physician Stroke Assessment	CHADS2 = 0 (n=644)	CHADS2 = 1 (n=2199)	CHADS2 ≥2 (n=7251)	Physician Bleeding Assessment	ATRIA ≤3 (n=7440)	ATRIA = 4 (n=905)	ATRIA ≥5 (n=1749)
Low (<3%) (n=4173)	78%	62%	32%	Low (<3%) (n=5982)	63%	58%	44%
Intermediate (3-6%) (n=4296)	19%	33%	48%	Intermediate (3-6%) (n=3393)	31%	33%	43%
High (> 6%) (n=1625)	3%	5%	21%	High (>6%) (n=719)	6%	9%	13%

Agreement between empirical risk score, and physician-assigned assessment (column percentages).

Patient Characteristics

Baseline characteristics of patients, stratified by physician-assigned and empirical stroke risk assessments, are shown in Table 2. Patients subjectively labeled as Low Risk for stroke were older (median 71 vs. 63), more likely female (40% vs. 33%), with more medical comorbidity (CAD 29% vs. 14%, prior stroke/TIA 7.6% vs. 0%), compared with patients with a CHADS₂ score of 0. Those subjectively assessed as High Risk were similar age (median 78 for both), but more likely to have significant comorbidity (CAD 48% vs. 42%, prior stroke/TIA 37% vs. 21%) compared with patients with a CHADS₂ score of ≥2.

Table 2.

Baseline characteristics in patients stratified by empirical risk score and physician-risk categories.

	CHADS2 = 0 (n=644)	MD Low Risk (n=4173)	CHADS2 = 1 (n=2199)	MD Intermediate Risk (n=4296)	CHADS2 ≥2 (n=7251)	MD High Risk (n=1625)
Age (years)	63 (54-69)	71 (63-79)	68 (61-73)	76 (69-82)	78 (71-83)	78 (71-84)
Female	33	40	36	43	45	45
Race
White	91	90	89	89	89	89
Black or African American	3.4	5.7	5	4.6	5.2	4.4
Hispanic	3.3	3	4.3	5.3	4.3	4.3
Other	1.9	1.2	1.7	1.3	1.3	2.3
Hypertension	0	77	73	87	93	89
Hyperlipidemia	43	67	66	74	76	79
Diabetes	0	22	3.2	31	40	45
History of CAD	14	29	22	38	42	48
Prior stroke/TIA	0	7.6	0	14	21	37
Congestive Heart Failure
No CHF	100	77	93	66	57	48
NYHA Class I	0	10	3	9.7	13	12
NYHA Class II	0	9.1	2.4	16	20	26
NYHA Class III/IV	0	3.9	1.2	8.1	9.6	13
LVEF Type
Normal (≥50%)	81	72	77	70	67	65
Mild dysfunction (40-50%)	3.4	5.9	4.7	6.4	6.9	6.7
Moderate dysfunction (30%-40%)	2.5	6.8	3.6	9.2	11	13
Severe dysfunction (<30%)	0	2.5	1.5	4.9	5.4	6.7
Type of AF
New Onset	6.8	5.8	6.4	4.3	4.0	3.0
Paroxysmal	68	54	55	49	48	45
Persistent	13	16	18	17	17	19
Longstanding persistent	12	24	20	30	32	33
Rhythm Control Strategy at Baseline	47	36	39	30	28	27
Oral Anticoagulation
None	48	30	30	20	20	19
Warfarin	45	65	64	75	76	77
Dabigatran	6.2	4.8	6.6	5.3	4.2	3.8
CHADS2 Risk Score, mean (SD)	0 (0)	1.79 (1.2)	1 (0)	2.38 (1.14)	2.87 (1)	3.26 (1.27)
ATRIA Score (0-3, 4, ≥5)
Low (0-3)	99.7	78	82	72	69	67
Intermediate (4)	0.16	9.2	13	9	8.5	8.4
High (5 or more)	0.16	13	5.2	19	23	25
Site Provider
Cardiology	58	65	65	66	66	65
Electrophysiology	24	13	17	15	14	20
Primary Care Specialty	18	22	18	19	20	15

Values are presented as % or median (interquartile range), except where noted.

CAD: coronary artery disease; TIA: transient ischemic attach; LVEF: left-ventricular ejection fraction; AF: atrial fibrillation; SD: standard deviation.

Baseline characteristics of the patients, stratified by physician-assigned and empirical bleeding risk assessment, are shown in Table 3. Patients subjectively assessed as Low Risk of bleeding were of similar age and gender balance, with roughly equivalent rates of comorbid diseases as those with Low Risk ATRIA scores. Patients at High Risk of bleeding (subjectively or empirically) were overall older (median age 78 and 82, respectively), and had overall high rates of comorbidities contributing to stroke and bleeding risks.

Table 3.

Baseline characteristics in patients stratified by bleeding risk (risk score and physician-assigned).

	ATRIA ≤3 (n=7440)	MD Low Risk (n=5982)	ATRIA = 4 (n=905)	MD Intermediat e Risk (n=3393)	ATRIA ≥5 (n=1749)	MD High Risk (n=719)
Age (year)	73.5 (65-81)	73 (64-80)	68 (63-72)	78 (71-83)	82 (78-86)	78 (71-84)
Female	40	41	44	44	51	48
Race
White	90	88	82	91	90	89
Black or African American	4.4	5.5	11	4.2	4.6	4.9
Hispanic	4.2	4.8	5.2	3.2	3.7	4.2
Other	1.4	1.4	1.4	1.3	1.4	1.8
Hypertension	80	80	99	87	88	87
Diabetes	27	27	40	33	33	37
Prior GI Bleed	5.6	6.1	15	11	20	22
Anemia	0.9	14	54	23	73	32
Liver Disease	1.8	1.8	3.2	1.8	1.8	3.5
Alcohol Abuse	4.4	3.9	5.2	3.9	2.1	6
Estimated GFR (mL/min/1.73 m2)	69 (56-85)	70 (56-85)	65 (49-79)	64 (50-79)	52 (35-69)	60 (43-74)
Hemoglobin (g/dL)	13.8 (12.7-14.8)	13.7 (12.5-14.8)	12.6 (11.2-13.8)	13.3 (12-14.4)	12 (10.9-13.1)	12.8 (11.4-14.1)
Type of AF
New Onset	4.8	5.6	5.4	3.6	3.9	2.4
Paroxysmal	52	52	49	48	47	48
Persistent	17	16	18	17	15	21
Longstanding Persistent	26	26	28	31	34	28
Oral Anticoagulation
None	23	27	26	17	27	32
Warfarin	72	68	69	78	71	65
Dabigatran	5.3	5.2	5.2	4.7	2.7	2.8
CHADS2 Risk Score, mean (SD)	2.12(1.27)	2.03(1.24)	2.19(1.13)	2.6(1.25)	2.99(1.2)	2.87(1.31)
ATRIA Score
Low (0-3)	100	78	0	69	0	58
Intermediate (4)	0	8.8	100	8.8	0	11
High (5 or more)	0	13	0	22	100	30
Site Provider
Cardiology	64	67	70	65	69	54
Electrophysiology	17	14	12	14	10	28
Primary Care Specialty	20	19	18	21	20	18

Values are presented as % or median (interquartile range), except where noted.

GI: gastrointestinal; GFR: glomerular filtration rate; AF: atrial fibrillation; SD: standard deviation.

Factors Associated with Disagreement

Multivariable models of the difference between physician-assessed and empirically-calculated risk are shown in Figure 1. Physician-assigned risk was less influenced by hypertension (adjusted mean difference 0.61, 95% CI 0.56-0.65), heart failure (adjusted mean difference for NYHA class I vs. no heart failure, 0.23, 95% CI 0.17-0.29), increasing age (adjusted mean difference for 10-year increase from 60 to 85, 0.15, 95% CI 0.12-0.18), and diabetes (adjusted mean difference 0.14, 95% CI 0.09, 0.2). Prior stroke or TIA (adjusted mean difference 0.17, 95% CI 0.10-0.23), severe AF symptoms (adjusted estimate 0.08, 95% CI 0.02-0.14) and not living independently (0.12, 95% CI 0.06-0.18) were more strongly associated with physician-assigned risk than empirical stroke risk. Anemia was most significantly associated with physician-assigned bleeding risk being lower than empirically-calculated bleeding risk (adjusted estimate 1.36, 95% CI 1.30-1.42), whereas a variety of comorbidities were roughly equally associated with bleeding between physicians and the empirical risk score (e.g., prior GI bleed, heart failure class, concomitant atherosclerotic disease, alcohol abuse, and depressed renal function). Full model details can be found in the Supplemental Material, Table S4 and Table S5.

Figure 1.

Categorization of physician-assigned and empirical risk of stroke (Panel A) and bleeding (Panel B).

Figure 1. Factors associated with physician under- or over-estimation of empirical stroke (Panel A) and bleeding (Panel B) risk. NYHA: New York Heart Association; CHF: congestive heart failure; AF: atrial fibrillation; PCI: percutaneous coronary intervention; CABG: coronary artery bypass grafting; EHRA: European Heart Rhythm Association; TIA: transient ischemic attack; CAD: coronary artery disease; eGFR: estimated glomerular filtration rate.

Use of Systemic Oral Anticoagulation

Among patients at High Risk of stroke by CHADS₂ score (≥2), 80% were treated with oral anticoagulation (OAC), compared with 81% of those assessed as High Risk subjectively by the physician. In contrast, OAC use among did not vary much among those with High- or Low-risk ATRIA scores (73% vs. 77%) or High or Low physician bleeding risk estimates (73% vs. 68%, Figure 2).

Figure 2.

Rates of OAC use by stroke risk assessments (Panel A) and bleeding risk assessments (Panel B). OAC: oral anticoagulation.

Discussion

There are four main findings in this study of empirical and provider risk-stratification. First, physicians’ categorical assessment of stroke and bleeding risk in patients with AF was poorly correlated with empirical risk estimates. Second, physicians generally classified many fewer patients as having high risk for stroke and bleeding than validated empirical risk models. Third, physicians’ emphasis of specific risk factors differed significantly from the empirical scores (e.g., hypertension, heart failure). Finally, assessments of stroke risk by either empirical or physician assessments seemed to have a larger impact on subsequent decisions regarding use of anticoagulation than assessments of bleeding risk.

The disagreement between physician-assigned risk categorization and empirical risk categories in up to 80% of cases highlights several pitfalls of risk stratification (Table 1). Physicians may have difficulty translating empirical scores into absolute event rates. While the pre-defined categories on the ORBIT-AF case report form were designed to mirror derivation cohorts from empirical risk scores, many physicians may not correlate a score with an event rate. For example, it appears that physicians markedly underestimated the annual risk of stroke among AF patients in various CHADS₂ categories. However, the discriminatory value of the CHADS₂ scoring system remains modest.⁴

In our multivariable analysis of mismatch, it appears physicians weighted individual risk factors differently from weightings in the scores, for both stroke and bleeding. For example, patients with high-risk CHADS₂ scores due to prior stroke had higher physician-assessed risk than those who got to high CHADS₂ score with a combination of other factors. Analogously, the presence of anemia or significant renal disease was significantly associated with lower physician-assigned bleeding risk, relative to ATRIA score. One explanation is that physicians simply value certain risk factors differently from the empirical scores. Alternatively, this could be another manifestation of poor calibration of risk by clinicians.

Our study also highlights the influence of clinicians’ assessments on treatment decisions. Rates of OAC were lower among patients with low or intermediate CHADS₂ risk, compared with OAC use among patients deemed low or intermediate risk by their physicians. Similarly patients having high bleeding risk by ATRIA score were more likely to receive OAC than those deemed high risk by their provider. This demonstrates that in fact when discordant, physician subjective evaluation was a stronger driver of decisions than the objective empirical tool.

Furthermore, it appears that stroke risk, more so than bleeding risk, drives the decision to use OAC. Rates of OAC varied more across stroke risk categories than across bleeding risk strata. This appears contrary to prior data demonstrating physicians tend towards errors of omission (i.e., causing harm due to withholding therapy) versus errors of commission (i.e., causing harm due to therapy).^13-15 There may be several reasons for the low influence of bleeding risk: stroke risk is particularly emphasized in the US guidelines; there appears to be a lack of familiarity with validated bleeding scores; or, it may simply reflect the difficulty in predicting bleeding risk by any method. However, the overall effect was modest - rates of OAC use in our cohort were relatively high overall and a risk-treatment paradox persisted. Up to 70% of patients at low risk of stroke received OAC, and nearly 20% of patients at high-risk did not receive OAC. There remains room for improvement in the selection of AF patients for OAC.

Limitations

These analyses are derived from a national registry and sampling bias may exist. However ORBIT-AF was designed to be inclusive of a very broad sample of community practice types and locations and included a wide spectrum of AF patient types (incident and prevalent, paroxysmal and permanent). Additionally, the physician assessment was based on a single question and reporter biases are possible; however, these data were not linked to any rewards or comparative profiling so honest responses would be expected. While many stroke and bleeding risk assessment tools exist, this paper focuses on CHADS₂ and ATRIA risk assessments. That said, these are the most commonly-used and longstanding tools in the US and it is likely that results would be consistent had other risk scores been used. Additionally, the cut-points for low, moderate and high risks are subjective, yet ours were selected to correspond to our predefined physician risk ranges. Notably, we cannot say which risk assessment, empirical or physician-assigned, was more ‘correct’. Lastly, treatment decisions regarding OAC were made prior to the subjective risk assessment by the provider, and we cannot exclude the possibility that current treatment influenced the provider's risk assessment.

Conclusions

Concordance between provider-assessed risk and empirical scores for stroke and bleeding in patients with AF is low. While physicians rely on components of scores to estimate risk, they appear to weight them differently from the calculated score. Risk assessment at the patient level remains a challenge, for both stroke and bleeding, and such assignments have important implications for treatment decisions.

Funding Sources: The ORBIT-AF registry is sponsored by Janssen Scientific Affairs, LLC, Raritan, NJ. Dr. Steinberg was funded by NIH T-32 training grant #5 T32 HL 7101-38.