Assessing the Performance of the Framingham Stroke Risk Score in the REGARDS Cohort

Abstract

Background and Purpose

The most well-known stroke risk score is the Framingham Stroke Risk Score (FSRS), which was developed during the higher stroke risk period of the 1990’s, and has not been validated for blacks. We assessed the performance of the FSRS among participants in the Reasons for Geographic And Racial Differences in Stroke (REGARDS) study to determine whether it is useful in both blacks and whites.

Methods

Expected annualized stroke rates from the FSRS were compared to observed stroke rates overall and within strata defined by FSRS risk factors (age, sex, systolic blood pressure, use of antihypertensive medications, diabetes, smoking, atrial fibrillation, left ventricular hypertrophy and prevalent coronary heart disease).

Results

Among 27,748 participants stroke-free at baseline, 715 stroke events occurred over 5.6 years of follow-up. FSRS-estimated incidence rates of stroke were 1.6 times higher than observed for black men, 1.9 times higher for white men, 1.7 times higher for black women and 1.7 times higher for white women. This overestimation was consistent among most subgroups of FSRS factors, although the magnitude of overestimation varied by the risk factor assessed.

Conclusions

While higher FSRS was associated with higher stroke risk, the FSRS overestimated observed stroke rates in this study, particularly in certain subgroups. This may be due to temporal declines in stroke rates, secular trends in prevention treatments, or differences in populations studied. More accurate estimates of event rates are critical for planning research, including clinical trials, and targeting health-care efforts.

Introduction

Clinical tools to predict disease are important to direct treatments and to counsel individuals on behavioral changes that might modify risk. Recent reports have offered new tools for the prediction of a composite outcome of “hard atherosclerotic cardiovascular disease” (hard ASCVD) endpoint including both coronary and stroke events¹; however, these new tools do not directly address risk from the individual components of the composite outcome, and acceptance of these risk functions has been controversial.2 Several risk scores are available to predict stroke^3–8, the most well-known and well-accepted being the Framingham Stroke Risk Score (FSRS), which was developed during the high stroke risk period of the 1990’s. Factors included in the FSRS are age, sex, systolic blood pressure, use of antihypertensive medications, diabetes, smoking, atrial fibrillation, left ventricular hypertrophy and prevalent coronary heart disease⁴. The Cardiovascular Health Study (CHS) risk score includes the same risk factors, and a measure of frailty5. In the development of a risk score from the Atherosclerosis in Communities (ARIC) study, the goal was to determine non-traditional risk factors that may improve predictability of the model, however their analyses determined that the traditional risk factors performed reasonably well and that addition of non-traditional risk factors did not substantially improve prediction.⁵ As such, the FSRS remains the standard for predicting stroke risk in the general population.

Notably absent from both the Framingham Risk Score, the CHS Risk Score and the new AHA/ACC Pooled Cohort Risk Equation is the impact of race on stroke risk prediction. The Framingham Risk Score does not include a term for race or ethnicity since there were few blacks in the study. Similarly, in CHS, at the time the risk score was developed there was insufficient follow up among black participants (≈ 15% of the cohort). Framingham, CHS and other studies included in the new ACC/AHA instrument included relatively few blacks and racial differences were not the focus of these studies. In ARIC, race was considered but inclusion in modeling did not increase predictability beyond the traditional risk factors, and ARIC confounded race and geography with most of the black study participants from the single Jackson (MS) study site¹.

Currently available risk scores assume that the impact of risk factors is similar for blacks and whites. The Framingham Coronary Risk Score was validated in several racial/ethnic groups;⁹ however, this work did not assess the potential that risk factors could have a differential impact on whites and blacks. To our knowledge, the performance of the FSRS has not been assessed in different racial groups, and the possibility that risk factors could have a different role in blacks and whites has not been evaluated. We assessed the performance of the FSRS among black and white participants in the Reasons for Geographic And Racial Differences in Stroke (REGARDS) study to determine whether this tool is useful in both blacks and whites.

Methods

The REGARDS study is a national population-based cohort study that recruited 30,239 participants 45 years of age or older, with 45% male and 55% female; 42% black and 58% white. Recruitment began in January, 2003 and was completed in October, 2007. Twenty-one percent (21%) of the cohort was recruited from the buckle of the stroke belt (coastal plain region of NC, SC and GA), 35% from the stroke belt states (remainder of NC, SC, and GA, plus AL, MS, TN, AR, and LA), and the remaining 44% from the other 42 contiguous states (goal was 30% belt, 20% buckle, 50% remainder of nation). Detailed study methods are published elsewhere.¹⁰ In brief, participants were selected from commercially available lists of residents, and were recruited using a combination of mail notification and telephone contact. Verbal consent and baseline data, including demographics, smoking history, cardiovascular risk factor history were collected via computer-assisted telephone interview (CATI). Following completion of the baseline interview, an in-home visit was performed to collect physical measurements, including blood pressure, blood and urine samples, ECG, and a signed informed consent. Bloods are processed centrally at the University of Vermont, and ECGs are read centrally at Wake Forest University School of Medicine. Follow-up phone contact is made at six-month intervals for surveillance of suspected stroke events, with medical records for suspected events retrieved and adjudicated by a panel of physicians. The study is approved and monitored by the Institutional Review Boards at all participating institutions.

Framingham Stroke Risk Factors

Age, sex, use of antihypertensive medications, and smoking were all based on self-report during the initial phone interview. Age was categorized as: young (45–64 years old), middle (65–74 years old), or older (75+ years old). Heart disease was defined as self-report of myocardial infarction, bypass, angioplasty or stenting, or EGC evidence of MI. Participants were considered diabetic if their fasting glucose was >= 126 (or >=200 if they were not fasting) or if they self-reported medication use for diabetes. Systolic blood pressure was the average of two seated measurements. Determination of atrial fibrillation was by self-report of a physician diagnosis or ECG evidence, and left ventricular hypertrophy (LVH) was based on ECG evidence.

Determination of Stroke Events

Events reported during follow-up as possible stoke, transient ischemic attack, death, hospitalization or emergency department visit for brain aneurysm, brain hemorrhage, stroke symptoms, or unknown reason prompted a request for medical records. Initial review by a stroke nurse excluded obvious non-strokes; the remaining medical records were centrally adjudicated by physicians. For deaths without medical records, death certificates were examined and adjudicated, and proxy interviews undertaken. Stroke events were defined following the World Health Organization (WHO) definition¹¹. Those events not meeting the WHO definition but with symptoms lasting >24 hours and neuroimaging consistent with acute ischemia or hemorrhage were classified as clinical strokes, and were included in analyses.

Statistical Analysis

We assessed the prevalence of the FSRS factors among REGARDS participants at baseline by race-sex strata, computed the FSRS and compared the FSRS factors across race-sex strata using analysis of variance or chi-square tests of association, as appropriate. We then fit Poisson regression models to estimate the observed stroke incidence rates. We compared the observed stroke rates to those predicted by the FSRS, by computing the ratio of the predicted to observed risk, and the 95% confidence intervals around the risk ratios. We compared the observed and predicted stroke rates overall, by levels of predicted FSRS levels, and by levels of FSRS risk factors for each race-sex stratum. Because of missing stroke data due to ongoing adjudication of stroke events, as well as differential retrieval of medical records, we utilized multiple imputation techniques to classify in process stroke events using a logistic function predicting the likelihood that an attempted record retrieval would result in an adjudicated stroke.¹² Ten imputed datasets were used, and 22% of reported events were eligible to be imputed.

Results

Among 27,748 stroke-free participants at baseline followed for an average of 5.6 years, 30% were white females, 30% were white males, 25% were black females, and 15% were black males. Table 1 presents the average 10-year FSRS, and the prevalence of the FSRS risk factors by race-sex strata. All factors differed significantly across race-sex strata. Predicted stroke risk was highest in black males (11.9% ten-year risk, SD=10.0), followed by white males (11.0% ten-year risk, SD=9.6), black females (10.2% ten-year risk, SD=12.1), then white females (7.8% ten-year risk, SD=10.2). A similar pattern in the association between race-sex stratum and SBP was observed.

Table 1.

Average FSRS and prevalence of the FSRS risk factors by race-sex strata

	White		Black
	Female (n=8342)	Male (n=8198)	Female (n=7037)	Male (n=4171)
FSRS (mean % ten-year risk (SD)	7.8 (10.2)	11.0 (9.6)	10.2 (12.1)	11.9 (10.0)
SPB (mean mm Hg, SD)	123 (16)	127 (15)	130 (17)	132 (17)
Current Smoker	1096 (13%)	936 (11%)	1080 (15%)	803 (19%)
Age (mean, SD)	64.5 (9.6)	65.9 (9.3)	63.4 (9.3)	64.2 (9.2)
Heart Disease	972 (12%0	2059 (25%)	828 (12%)	726 (18%)
LVH	263 (4%)	82 (2%)	455 (8%)	109 (4%)
Atrial Fibrillation	85 (1%)	211 (3%)	21 (0.3%)	56 (1%)
Diabetes	1021 (13%)	1394 (17%)	1963 (29%)	1226 (31%)

Table 2 provides the ratio of the expected-to-observed annualized stroke risk by race-sex strata, overall (Figure 1), by quintiles of the FSRS (Figure 2), and by levels of each of the FSRS risk factors (Supplemental Figures I and II). Overall, the FSRS over-estimated the annualized predicted risk across all race-sex strata, and that the over-estimation was highest among white males, followed by black and white females, then black males. Considering the ratio of expected to observed stroke risk by quintile of the FSRS, across all race-sex strata the FSRS over-estimated the annualized predicted risk. In particular, those at highest risk for stroke (5^th quintile of the FSRS) had the largest over-estimation, and the over-estimation was significantly greater than 0 for all four race-sex strata. The magnitude of the over-estimation in the top quintile of predicted risk was about the same for white and black males, but was larger for black females than for white females.

Table 2.

Ratio of expected to observed stroke risk (95% confidence interval) by race-sex strata, overall and across different levels of FSRS and FSRS risk factors

	White		Black
	Female (n=8342)	Male (n=8198)	Female (n=7037)	Male (n=4171)
Overall	1.71 (1.49, 1.96)	1.88 (1.66, 2.11)	1.71 (1.49, 1.96)	1.60 (1.35, 1.89)
FSRS (mean, SD)
Q1 (<2.98)	1.72 (1.03, 2.86)	3.46 (0.87, 13.8)	1.05 (0.59, 1.85)	1.12 (0.28, 4.37)
Q2 (2.98–5.17)	1.70 (1.08, 2.67)	2.09 (1.30, 3.34)	1.37 (0.86, 2.19)	1.96 (0.93, 4.13)
Q3 (5.17–8.27)	1.92 (1.27, 2.92)	1.58 (1.18, 2.11)	1.02 (0.73, 1.42)	1.67 (1.07, 2.61)
Q4 (8.27–14.59)	1.26 (0.94, 1.69)	1.46 (1.17, 1.83)	1.31 (0.99, 1.73)	1.16 (0.86, 1.57)
Q5 (>=14.59)	1.85 (1.49, 2.30)	2.04 (1.70, 2.47)	2.60 (2.03, 3.33)	1.94 (1.50, 2.50)
SPB Category
Normal	1.34 (1.03, 1.76)	1.59 (1.23, 2.06)	1.47 (1.01, 2.14)	1.77 (1.09, 2.85)
Pre-Hypertension	1.78 (1.44, 2.19)	1.67 (1.42, 1.96)	1.67 (1.36, 2.06)	1.44 (1.13, 1.82)
Stage 1	2.44 (1.74, 3.43)	2.45 (1.85, 3.24)	1.60 (1.24, 2.06)	1.75 (1.30, 2.35)
Stage 2	1.11 (0.72, 1.71)	4.72 (2.23, 9.97)	2.69 (1.63, 4.43)	1.68 (1.02, 2.77)
Smoking Status
Non-Smoker	1.84 (1.57, 2.15)	.92 (1.69, 2.19)	1.67 (1.44, 1.95)	1.65 (1.37, 1.99)
Smoker	1.20 (0.87, 1.64)	1.56 (1.15, 2.12)	1.92 (1.37, 2.70)	1.41 (1.01, 1.96)
Age Category
Young (45–64 years)	1.72 (1.27, 2.35)	2.12 (1.65, 2.74)	1.66 (1.30, 2.13)	1.53 (1.17, 2.00)
Middle (65–74 years)	1.74 (1.37, 2.22)	1.76 (1.46, 2.11)	1.60 (1.28, 1.98)	1.65 (1.36, 2.16)
Old (75+ years)	1.66 (1.34, 2.05)	1.76 (1.43, 2.15)	1.90 (1.45, 2.47)	1.61 (1.17, 2.22)
Heart Disease
No Heart Disease	1.70 (1.44, 2.00)	1.71 (1.46, 2.00)	1.65 (1.40, 1.94)	1.60 (1.31, 1.94)
Heart Disease	1.73 (1.31, 2.29)	2.08 (1.70, 2.53)	1.95 (1.44, 2.65)	1.57 (1.16, 2.12)
LVH
No LVH	1.68 (1.45, 1.95)	1.84 (1.62, 2.08)	1.65 (1.40, 1.94)	1.60 (1.33, 1.93)
LVH	1.89 (1.25, 2.87)	2.16 (1.43, 3.25)	1.94 (1.44, 2.61)	1.58 (1.11, 2.26)
Atrial Fibrillation
No AFib	1.61 (1.37, 1.89)	1.87 (1.63, 2.14)	1.50 (1.30, 1.74)	1.54 (1.29, 1.84)
AFib	2.06 (1.50, 2.82)	1.82 (1.38, 2.41)	4.09 (2.48, 6.77)	2.10 (1.23, 3.56)
Diabetes
No Diabetes	1.65 (1.39, 1.92)	1.98 (1.63, 2.14)	1.52 (1.27, 1.82)	1.53 (1.22, 1.91)
Diabetes	2.00 (1.45, 2.76)	1.88 (1.47, 2.41)	2.05 (1.59, 2.66)	1.79 (1.37, 2.34)

Figure 1.

Ratio of expected to observed stroke risk and 95% confidence interval, by race (red=white, blue=black) and sex (F=female, M=male)

Figure 2.

Ratio of expected to observed stroke risk and 95% confidence interval, across the quintiles of the Framingham Stroke Risk Score by race (red=white, blue=black) and sex (F=female, M=male)

The ratio of expected to observed stroke risk by race-sex strata across levels of each of the factors contributing to the FSRS indicated that across all risk factors, at all levels, the FSRS over-estimated the stroke risk relative to that observed. The over-estimation was particularly large at higher levels of the risk factors, such as for those with Stage 2 hypertension and for those with atrial fibrillation. With respect to race and sex, there were no consistent patterns regarding for whom the over-estimation was largest. In some cases, the over-estimation was larger for blacks than for whites (e.g. for females with Stage 2 hypertension, or for female smokers), while in other cases, the over-estimation was worse for whites than for blacks (e.g. male non-smokers, or for males with LVH).

Discussion

In this large cohort of black and white adults aged 45 and older, the validity of the FSRS was confirmed by a clearly higher observed number of stroke events among those with higher FSRS scores; however, the FSRS over-estimated the observed stroke rates, particularly in certain subgroups. Generally, the FSRS suggested there should be approximately twice as many strokes occurring than we detected, and with a few exceptions, this was consistent across strata defined by the risk factors. This overestimation of stroke risk is similar to the suggested overestimation of risk that has been central to the controversy surrounding the predicted ASCVD risk in the AHA/ACC guideline instrument.²

The FSRS is not the only stroke risk score that is available. Several others have been developed, including the CHS Risk Score⁵, the ARIC Risk Score⁶ and others^{7, 8}. Although REGARDS does not have all of the variables contributing to each of these, it is expected that results would be the similar to those observed with the FSRS, given the similarities among the performance of the scores.^{6, 13}

While the magnitude of the expected-to-observed ratio of strokes was reasonably consistent, there were several exceptions. For all four race-sex strata, the expected-to-observed ratio was higher for study participants in the highest quintile of the FSRS. This finding supports the notion that preventive treatments reduced the number of events in these individuals. This is particularly the case for white men and black women with stage 2 hypertension (systolic blood pressure ≥160 mm Hg). The expected-to-observed ratio was also quite high for both black men and women with atrial fibrillation, potentially suggesting treatment of this condition has improved with time. With these exceptions, the magnitude of the expected-to-observed ratio of events was reasonably consistent, suggesting more of a more general change in risk of stroke.

With REGARDS being one of the validation studies for the new AHA/ACC guideline models that showed lower observed than predicted risk, we have suggested four potential contributors to the overestimation of risk.¹⁴ First, several of the studies used to develop the risk equation included both self-reported events with subsequent medical record adjudication and active surveillance for events in community hospitals, while in REGARDS we use only self-reported events to retrieve records for subsequent adjudication. As such, we may be systemically missing some stroke events. Second, dynamic temporal changes in the prevalence of statin use, which was not considered in the validation cohorts including REGARDS, could contribute to fewer events than expected. Third, increased used of revascularization could contribute to fewer events than expected.¹⁵ Finally, the duration of follow-up was relatively shorter in REGARDS than the studies used to develop the risk functions. All of these factors could potentially be playing a role in the overestimation of stroke events in this report. However, our follow-up contacts were quite good (76% of participants completed over 75% of anticipated follow-up contacts), and we “cast a very broad net” for suspected events (over 4 suspected events were reviewed for each event adjudicated to be a stroke). In addition, incident statins use have at most a 30% effect on stroke risk¹⁶ only in the subset of the population beginning treatment, carotid revascularization would impact a relatively small portion of the population, and our estimates are stable with over 800 stroke events.

Alternatively, with the Framingham study having a cohort assessed before 2000 and the REGARDS cohort assessed after 2000, the overestimation could be a product of a dramatic temporal decline in incident stroke. Between 1999 and 2010, data from CDC WONDER show there has been a decrease from in age-adjusted death from stroke (ICD10: I60 – I60) from 61.6/100,000 to 39.1/100,000 (a 37% decrease)¹⁷; with most believing that the decline in mortality is primarily attributable to a decline in incidence. We believe that a substantial portion of the overestimation of stroke risk is associated with these dramatic declines in stroke risk, suggesting the need for a recalibration of the risk formulas for incident stroke, and highlighting the need for on-going epidemiologic studies that can address issues such as this over time.

As with all research, this study has limitations. While the FSRS is designed to determine 10-year risk of stroke, we only have 5.6 years of follow-up thus far in REGARDS, and thus have annualized stroke rates. We have made the assumption that the rates are constant with time, and this may not be true. Further, we rely on participant report of stroke, which may introduce a bias due to underreporting of potential events, inability to obtain medical records for stroke adjudication or losses to follow-up. Although we have minimized these biases by utilizing external resources such as National Death Index searches and relying on proxy reports, as well as by imputing events that are still in process, they still may exist.

In summary, among black and white participants in the REGARDS cohort, the FSRS over-estimated the actual stroke risk, across sex and race, and at all levels of observed risk factors. The ability to accurately estimate stroke risk is important for planning research and for targeting clinical efforts towards stroke prevention. Our results might be explained by secular trends in stroke incidence since the FSRS was developed. Thus, future research should focus on estimating stroke risk more accurately across races in the context of current medical care trends.