Abstract
Background
Accurate risk assessment of atherosclerotic cardiovascular disease (ASCVD) is essential to effectively balance the risks and benefits of therapy for primary prevention.
Objective
To compare the calibration and discrimination of the new American Heart Association (AHA) and American College of Cardiology (ACC) ASCVD risk score with alternative risk scores and to explore preventive therapy as a cause of the reported risk overestimation using the AHA-ACC-ASCVD score.
Design
Prospective epidemiologic study of ASCVD.
Setting
MESA (Multi-Ethnic Study of Atherosclerosis), a community-based, sex-balanced, multiethnic cohort.
Patients
4227 MESA participants aged 50 to 74 years and without diabetes at baseline.
Measurements
Observed and expected events for the AHA-ACC-ASCVD score were compared with 4 commonly used risk scores—and their respective end points—in MESA after a 10.2-year follow-up.
Results
The new AHA-ACC-ASCVD and 3 older Framingham-based risk scores overestimated cardiovascular events by 37% to 154% in men and 8% to 67% in women. Overestimation was noted throughout the continuum of risk. In contrast, the Reynolds Risk Score overestimated risk by 9% in men but underestimated risk by 21% in women. Aspirin, lipid-lowering or antihypertensive therapy, and interim revascularization did not explain the overestimation.
Limitation
Comparability of MESA with target populations for primary prevention and possibility of missed events in MESA.
Conclusion
Of the 5 risk scores, 4, including the new AHA-ACC-ASCVD score, showed overestimation of risk (25% to 115%) in a modern, multiethnic cohort without baseline clinical ASCVD. If validated, overestimation of ASCVD risk may have substantial implications for individual patients and the health care system.
Primary Funding Source
National Heart, Lung, and Blood Institute.
For more than a decade, national guidelines have recommended the use of an objective tool to assess risk for coronary heart disease (CHD) based on the Framingham risk score (FRS) to guide therapy for primary prevention (1–4). The original Framingham risk prediction algorithm to predict CHD (known as FRS-CHD) incorporated age; sex; diabetes; systolic and diastolic blood pressures; levels of total, low-density lipoprotein, and high-density lipoprotein cholesterol; and smoking to estimate a 10-year risk for angina, myocardial infarction, or death due to CHD (4). In 2001, the Third Report of the Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III [ATPIII]) recommended a modified FRS (ATPIII-FRS-CHD) and set thresholds and goals for lipid-lowering therapy based on the 10-year CHD risk (5).
The applicability of Framingham-based algorithms to modern populations has been questioned (6–10). Framingham-based scores are based on a homogeneous, geographically limited, white male–dominated cohort from a prior generation when cardiovascular risk profiles and preventive pharmacotherapy were both less well-developed and used than in modern cohorts. Multiple studies in diverse populations suggest that Framingham-based risk-scoring systems may misclassify risk, particularly in women, and overestimate CHD risk (6, 7, 9, 11). In response, the Reynolds Risk Score (RRS) was developed in 2007 and included parental history of premature CHD and high-sensitivity C-reactive protein (12). To address concerns that the original Framingham-based scores ignored other increasingly common atherosclerotic cardiovascular disease (ASCVD) events, such as stroke, an FRS for the prediction of CVD (FRS-CVD) was developed in 2008 (11).
Most recently, the American Heart Association (AHA) and the American College of Cardiology (ACC) developed a new ASCVD risk score to guide ASCVD risk-reducing therapy (13). This new risk score was derived from 4 racially and geographically diverse prospective cohort studies (independent from MESA [Multi-Ethnic Study of Atherosclerosis]), including “applicable data” from the Framingham Original and Offspring Study cohorts (13–16). This new risk score uses the same traditional risk factors as the original FRSs and offers separate equations for white and African American men and women (Appendix Table 1, available at www.annals.org).
Appendix Table 1.
Intended Age Group, Predicted Outcome, and Variables Included Among Risk Scores Being Evaluated
| Risk Score | Reference | Target Age Group, y | Target Cardiovascular Events | Variables |
|---|---|---|---|---|
| FRS-CHD | 4 | 30–74 | Angina, MI, death from CHD, and coronary insufficiency | Age, total cholesterol level, HDL cholesterol level, BP, diabetes status, smoking status, and sex |
| FRS-CVD | 11 | 30–74 | Angina, MI, death from CHD, stroke, TIA, peripheral vascular disease, and heart failure | Age, total cholesterol level, HDL cholesterol level, BP, diabetes status, smoking status, sex, and antihypertensive medication use |
| ATPIII-FRS-CHD | 5 | >20 | MI and death from CHD | Age, total cholesterol level, HDL cholesterol level, BP, smoking status, sex, and antihypertensive medication use |
| RRS | 12, 22 | Women: 45–80 Men: 50–80 |
MI, death from CHD, stroke, and coronary revascularization | Age, total cholesterol level, HDL cholesterol level, BP, diabetes status, smoking status, sex, hs-CRP level, family history, and HbA1c level* |
| AHA-ACC-ASCVD | 13 | 40–79 | MI, death from CHD, and stroke | Age, total cholesterol level, HDL cholesterol level, BP, diabetes status, smoking status, sex, white or African American ethnicity, and antihypertensive medication use |
ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; ATPIII = Adult Treatment Panel III; BP = blood pressure; CHD = coronary heart disease; CVD = cardiovascular disease; FRS = Framingham risk score; HbA1c = hemoglobin A1c; HDL = high-density lipoprotein; hs-CRP = high-sensitivity C-reactive protein; MI = myocardial infarction; RRS = Reynolds Risk Score; TIA = transient ischemic attack.
Female participants with diabetes only.
Both the guideline writing committee and several investigator groups found that the AHA-ACC-ASCVD risk score developed in 2013 overestimated risk in independent cohorts (10, 13, 17, 18). Overestimation is of particular clinical relevance at the 10-year predicted risk threshold of 7.5% given recent guideline recommendations encouraging pharmacotherapy treatment above this risk level (19). We sought to compare the calibration and discrimination of these risk scores in an age-, sex-, and race-balanced, modern multiethnic population without clinical ASCVD at enrollment. We also sought to understand whether the use of preventive therapies (aspirin, lipid-lowering or antihypertensive therapy, and interim revascularization), common to modern cohorts, was responsible for risk score overestimation. Such information might be useful in helping to guide implementation policy and future risk score and guideline development.
Methods
MESA is a multicenter prospective community-based epidemiologic study of cardiovascular disease in an age- and sex-balanced, multiethnic cohort. The study design and methods have been previously published (20). In brief, 6814 participants aged 45 to 84 years who identified themselves as white, African American, Hispanic, or Chinese were recruited from 6 U.S. communities in 2000 to 2002. Participants were free of clinical CVD at enrollment. Medical history, anthropometric measurements, and laboratory data were assessed as previously described (20). Parental history of CHD was considered positive at the baseline visit if either parent had experienced a heart attack. Sensitivity analysis was done using the RRS definition of premature parental history of CHD (mother or father aged <60 years with myocardial infarction) on the subsample reporting detailed family history data at the second MESA visit (92% of participants).
Follow-up and Event Adjudication
New CHD and CVD events were recorded over a median 10.2-year follow-up. Participants were interviewed by telephone every 9 to 12 months to identify interim hospital admissions, outpatient diagnoses of CHD or CVD, and deaths. To verify self-reported diagnoses, copies of medical records for all hospitalizations and outpatient CHD and CVD diagnoses as well as death certificates were requested. Two physician members of the MESA mortality and morbidity review committee independently classified events; the full committee made the final classification for disagreements. Events consisted of myocardial infarction; definite or probable angina; resuscitated cardiac arrest; stroke (not transient ischemic attack); or death due to CHD, stroke, atherosclerosis, or other CVD. A detailed description of the MESA method is available at www.mesa-nhlbi.org.
In an attempt to identify any ASCVD events missed by the MESA study, we examined, at any point during follow-up, all 3175 MESA participants included in the Part A hospital claims within the Centers for Medicare & Medicaid Services (CMS) billing database for cases not adjudicated by MESA.
Risk Score Calculations
Five risk prediction scores were calculated from the baseline MESA data (2000 to 2002): FRS-CHD, FRS-CVD, ATPIII-FRS-CHD, RRS, and AHA-ACC-ASCVD. The study population was limited to participants aged 50 to 74 years to meet the target population for each score (Appendix Table 1). Because the ATPIII-FRS-CHD and RRS for men were not developed for risk calculation among people with diabetes, MESA participants with diabetes at the baseline examination (12%) were excluded from these analyses. Additional exclusion of participants with missing data required for risk score calculation (2.7%) resulted in a final sample size of 4227.
Risk scores were calculated with the equation-based method (21) for each model (4, 5, 11–13, 22). Sensitivity analysis was performed with the original FRS definition of diabetes for all risk scores including this variable. The RRS for women includes hemoglobin A1c (HbA1c) level as a continuous variable for women with diabetes only. Because HbA1c level was not available until the second MESA visit, fasting glucose level from the first visit was used to calculate an HbA1c value with the following formula: fasting glucose level = (28.7 × HbA1c level) − 46.7 (23, 24). This formula was evaluated for accuracy in our cohort by calculating HbA1c levels from fasting glucose levels at the second visit and was found to correlate well with measured HbA1c levels at this time point (r = 0.80).
Predicted and Observed Number of Events
For each risk score, we reported the number of expected events and the number of observed events specific to the target end point of that score (Appendix Table 1). Events were censored at 10-year follow-up. For each participant with fewer than 10 years of follow-up, including those who died, we lowered their 10-year risk estimate to correspond to their length of follow-up using an exponential survival function to scale the risk score. If Fr10 denotes the 10-year proportion with events according to an FRS, then the 1-year proportion is: FR1 = −ln(1 − Fr10)/10. So, for a person with 8.5 years of follow-up: FR8.5 = 1 − exp(−FR1 × 8.5). Median follow-up in MESA was 10.2 years; 2692 (62%) and 3766 (89%) participants had 10 and 9 or more years of follow-up, respectively.
Hosmer–Lemeshow plots were examined separately for men and women to evaluate how well each model fit the observed CVD events. Further, Hosmer–Lemeshow tests for goodness of fit ranked participants according to their predicted probability of having an event and divided them into 10 groups, each of which was represented by a dot on the calibration plots. An overall chi-square statistic was calculated separately for men and women for each score and outcome.
The cumulative effect of multiple treatments was evaluated by doing a sensitivity analysis that included regular aspirin use, lipid-lowering or antihypertensive medication use, and any coronary revascularization. In an effort to study revascularizations that may have prevented subsequent events, all revascularizations more than 2 days before an event or revascularizations not followed by any event were selected for modeling.
Discrimination
The discriminative capability of each risk score was compared separately for men and women using the c-statistic (25). The discrimination slope was also calculated as the absolute difference in the average predictions for those with and without the outcome.
Role of the Funding Source
The National Heart, Lung, and Blood Institute had no role in the design, conduct, analysis, or reporting of this study.
Results
Baseline characteristics are described in Table 1. Participants’ mean age was 61.5 years, and 53.5% were women. Approximately 42% were white, 26% were African American, 20% were Hispanic, and 12% were Chinese. Aspirin use, lipid-lowering or antihypertensive therapy at baseline or any time during study follow-up, and revascularizations before an event are shown in Appendix Tables 2 and 3 (available at www.annals.org). Statins were the drugs used for 92.5% to 94% of participants receiving lipid-lowering therapy (Appendix Table 2).
Table 1.
Characteristics From Baseline Examination
| Variable | Total (n = 4227) | Men (n = 1961) | Women (n = 2266) |
|---|---|---|---|
| Mean age (SD), y | 61.5 (7.1) | 61.5 (7.2) | 61.5 (7.1) |
| Ethnicity, n (%) | |||
| White | 1771 (41.9) | 844 (43.0) | 927 (40.9) |
| Chinese | 494 (11.7) | 241 (12.3) | 253 (11.2) |
| African American | 1110 (26.3) | 476 (24.3) | 634 (28.0) |
| Hispanic | 852 (20.2) | 400 (20.4) | 452 (19.9) |
| Mean total cholesterol level (SD)* | |||
| mmol/L | 5.08 (0.91) | 4.90 (0.88) | 5.24 (0.91) |
| mg/dL | 197 (35) | 190 (34) | 203 (35) |
| Mean HDL cholesterol level (SD)* | |||
| mmol/L | 1.34 (0.39) | 1.17 (0.30) | 1.49 (0.41) |
| mg/dL | 52 (15) | 45 (12) | 57 (16) |
| Mean systolic BP (SD), mm Hg | 126 (21) | 125 (19) | 126 (22) |
| Smoking history, n (%)* | |||
| Never | 2045 (48.4) | 768 (39.2) | 1277 (56.3) |
| Former | 1624 (38.4) | 918 (46.8) | 706 (31.2) |
| Current | 558 (13.2) | 275 (14.0) | 283 (12.5) |
| Mean BMI (SD), kg/m2* | 28.2 (5.3) | 27.8 (4.3) | 28.5 (6.0) |
| Median hs-CRP level (IQR), nmol/L* | 18.6 (8.2–40.9) | 13.0 (6.6–28.6) | 25.8 (11.0–54.9) |
| Parental history of heart attack, n (%)* | |||
| No | 2680 (63.4) | 1299 (66.2) | 1381 (60.9) |
| Yes | 1547 (36.6) | 662 (33.8) | 885 (39.1) |
| Mean calculated HbA1c level (proportion of total hemoglobin level) (SD) | 4.8 (0.4) | 4.7 (0.4) | 4.8 (0.4) |
BMI = body mass index; BP = blood pressure; HbA1c = hemoglobin A1c; HDL = high-density lipoprotein; hs-CRP = high-sensitivity C-reactive protein; IQR = interquartile range.
2-tailed t test for the difference between the means of men and women was significant at P < 0.01. The chi-square test was used for categorical variables.
Appendix Table 2.
Medication Use, by Examination*
| Medication | Baseline†
|
Ever Throughout Study‡
|
||
|---|---|---|---|---|
| Men | Women | Men | Women | |
| Aspirin§ | 546 (27.84) | 484 (21.36) | 1132 (57.73) | 1214 (53.57) |
|
| ||||
| Antihypertensive | 642 (32.74) | 821 (36.23) | 1142 (58.24) | 1362 (60.11) |
|
| ||||
| Lipid-lowering|| | 308 (15.71) | 348 (15.36) | 855 (43.60) | 1009 (44.53) |
|
| ||||
| Any | 1031 (52.58) | 1197 (52.82) | 1563 (79.70) | 1866 (82.35) |
|
| ||||
| None | 930 (47.42) | 1069 (47.18) | 398 (20.30) | 400 (17.65) |
Values are numbers (percentages).
Examination 1.
Observed medication use any time during the study but before an event when applicable. Includes participants with missing data on risk factors.
Regular use.
Lipid-lowering therapy included a statin in 92.5% of participants at baseline and 94.0% of participants ever throughout the study.
Appendix Table 3.
Risk Score–Specific Events Among Participants Who Had Coronary Revascularization*
| Event Status | Events, n (%) | |||
|---|---|---|---|---|
| FRS-CHD† | FRS-CVD‡ | ATPIII-FRS-CHD§ | AHA-ACC-ASCVD§ | |
| Revascularization after CHD event|| | 108 (58.70) | 114 (61.96) | 42 (22.83) | 43 (23.37) |
| Revascularization ≤48 h before CHD event¶ | 45 (24.46) | 45 (24.46) | 25 (13.59) | 25 (13.59) |
| Revascularization >48 h to ≤30 d before CHD event** | 4 (2.17) | 3 (1.63) | 5 (2.72) | 5 (2.72) |
| Revascularization >30 d before CHD event** | 5 (2.72) | 5 (2.72) | 3 (1.63) | 4 (2.17) |
| Revascularization with no CHD event** | 22 (11.96) | 17 (9.24) | 109 (59.24) | 107 (58.15) |
ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; ATPIII = Adult Treatment Panel III; CHD = coronary heart disease; CVD = cardiovascular disease; FRS = Framingham risk score.
184 participants. CHD event is specific to each risk score. The Reynolds Risk Score was excluded from this analysis because revascularization was part of the end point.
Angina pectoris, nonfatal myocardial infarction, and coronary insufficiency.
Angina pectoris, nonfatal myocardial infarction, peripheral vascular disease, and heart failure.
Nonfatal myocardial infarction.
Event occurred after censoring and could not have contributed to overestimation.
Event was considered possibly periprocedural and was not considered in the evaluation of overestimation.
Events modeled as possibly explanatory for overestimation.
The FRS-CHD, FRS-CVD, ATPIII-FRS-CHD, and AHA-ACC-ASCVD overestimated risk for the cardiovascular end points they were designed to predict by 53%, 37%, 154%, and 86%, respectively, in men and 48%, 8%, 46%, and 67%, respectively, in women (Table 2). The overall discordance between observed and predicted cardiovascular events was lowest for the RRS (−3%).
Table 2.
Predicted and Observed Events for Each Risk Score
| Risk Score | Predicted Events, n (%) | Observed Events, n (%) | Signed Absolute Difference | Discordance, %* | c-Statistic | Discrimination Slope |
|---|---|---|---|---|---|---|
| Total (n = 4227) | ||||||
|
| ||||||
| FRS-CHD† | 397.6 (9.41) | 263 (6.22) | 3.18 | 51 | 0.68 | 0.05 |
|
| ||||||
| FRS-CVD‡ | 561.3 (13.28) | 448 (10.60) | 2.68 | 25 | 0.71 | 0.09 |
|
| ||||||
| ATPIII-FRS-CHD§ | 288.7 (6.83) | 134 (3.17) | 3.66 | 115 | 0.71 | 0.06 |
|
| ||||||
| RRS|| | 314.0 (7.43) | 323 (7.64) | −0.21 | −3 | 0.72 | 0.07 |
|
| ||||||
| AHA-ACC-ASCVD¶ | 387.2 (9.16) | 218 (5.16) | 4.00 | 78 | 0.71 | 0.06 |
|
| ||||||
| Men (n = 1961) | ||||||
| FRS-CHD† | 251.1 (12.80) | 164 (8.36) | 4.44 | 53 | 0.69 | 0.05 |
|
| ||||||
| FRS-CVD‡ | 358.7 (18.29) | 261 (13.31) | 4.98 | 37 | 0.71 | 0.09 |
|
| ||||||
| ATPIII-FRS-CHD§ | 218.6 (11.15) | 86 (4.39) | 6.76 | 154 | 0.71 | 0.05 |
|
| ||||||
| RRS|| | 213.5 (10.89) | 196 (9.99) | 0.89 | 9 | 0.70 | 0.06 |
|
| ||||||
| AHA-ACC-ASCVD¶ | 232.1 (11.84) | 125 (6.37) | 5.46 | 86 | 0.71 | 0.06 |
| Women (n = 2266) | ||||||
|
| ||||||
| FRS-CHD† | 146.5 (6.47) | 99 (4.37) | 2.10 | 48 | 0.60 | 0.01 |
|
| ||||||
| FRS-CVD‡ | 202.6 (8.94) | 187 (8.25) | 0.69 | 8 | 0.70 | 0.05 |
|
| ||||||
| ATPIII-FRS-CHD§ | 70.2 (3.10) | 48 (2.12) | 0.98 | 46 | 0.67 | 0.02 |
|
| ||||||
| RRS|| | 100.5 (4.44) | 127 (5.60) | −1.17 | −21 | 0.72 | 0.05 |
|
| ||||||
| AHA-ACC-ASCVD¶ | 155.1 (6.84) | 93 (4.10) | 2.74 | 67 | 0.70 | 0.05 |
ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; ATPIII = Adult Treatment Panel III; CHD = coronary heart disease; CVD = cardiovascular disease; FRS = Framingham risk score; RRS = Reynolds Risk Score.
Percentage discordance calculation: ([{expected percentage − observed percentage}/observed percentage] × 100).
End points are myocardial infarction, death from CHD, and angina.
End points are myocardial infarction, death from CHD, angina, stroke, transient ischemic attack, peripheral vascular disease, and heart failure.
End points are myocardial infarction and death from CHD.
End points are myocardial infarction, death from CHD, stroke, and coronary revascularization.
End points are myocardial infarction, death from CHD, and stroke.
Discordance between observed and expected risk was found throughout the risk continuum, including those at moderate risk (Figures 1 and 2 and Tables 3 and 4). This is evident in the Hosmer–Lemeshow calibration plots, with a predominance of overestimation in men and women by all of the risk scores except the RRS, which showed a predominance of underestimation in women, and the FRS-CVD, which revealed little discordance in women (Figure 2 and Table 4). Participants with an AHA-ACC-ASCVD risk score greater than 7.5% are of particular relevance to the 2013 ACC-AHA Guideline on the Treatment of Blood Cholesterol to Reduce Atherosclerotic Cardiovascular Risk in Adults. In men with an AHA-ACC-ASCVD risk score of 7.5% to 10%, the actual event rate was only 3% (Table 3). Among women with an AHA-ACC-ASCVD risk score of 7.5% to 10%, the actual event rate was only 5.1% (Table 4).
Figure 1.
Risk score–specific predicted and observed events in men, by decile of calculated risk.
Hosmer–Lemeshow calibration plots for men (n = 1961). ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; ATPIII = Adult Treatment Panel III; CHD = coronary heart disease; CVD = cardiovascular disease; FRS = Framingham risk score; RRS = Reynolds Risk Score.
Figure 2.
Risk score–specific predicted and observed events in women, by decile of calculated risk.
Hosmer–Lemeshow calibration plots for women (n = 2266). ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; ATPIII = Adult Treatment Panel III; CHD = coronary heart disease; CVD = cardiovascular disease; FRS = Framingham risk score; RRS = Reynolds Risk Score.
Table 3.
Predicted and Observed Events for Each Risk Score in Clinically Relevant Risk Categories: Men
| Risk Score, % | Participants, n | Predicted Events, n (%) | Observed Events, n (%) | Signed Absolute Difference | Discordance, %* | Hosmer–Lemeshow Chi-Square Statistic |
|---|---|---|---|---|---|---|
| FRS-CHD | ||||||
|
| ||||||
| 0.0–4.9 | 203 | 6.9 (3.4) | 4 (2.0) | 1.4 | 73 | 1.26 |
|
| ||||||
| 5.0–7.4 | 304 | 19.1 (6.3) | 10 (3.3) | 3.0 | 91 | 4.59 |
|
| ||||||
| 7.5–9.9 | 343 | 29.9 (8.7) | 14 (4.1) | 4.6 | 114 | 9.27 |
|
| ||||||
| ≥10.0 | 1111 | 195.2 (17.6) | 136 (12.2) | 5.3 | 44 | 21.78 |
|
| ||||||
| Total | 1961 | 251.1 (12.8) | 164 (8.4) | 4.4 | 53 | 36.91 |
|
| ||||||
| FRS-CVD | ||||||
| 0.0–4.9 | 85 | 2.8 (3.3) | 1 (1.2) | 2.1 | 182 | 1.21 |
|
| ||||||
| 5.0–7.4 | 156 | 9.9 (6.3) | 7 (4.5) | 1.8 | 41 | 0.90 |
|
| ||||||
| 7.5–9.9 | 237 | 21.0 (8.9) | 7 (3.0) | 5.9 | 200 | 10.25 |
|
| ||||||
| ≥10.0 | 1483 | 325.0 (21.9) | 246 (16.6) | 5.3 | 32 | 24.60 |
|
| ||||||
| Total | 1961 | 358.7 (18.3) | 261 (13.3) | 5.0 | 37 | 36.96 |
| ATPIII-FRS-CHD | ||||||
|
| ||||||
| 0.0–4.9 | 327 | 11.1 (3.4) | 3 (0.9) | 2.5 | 270 | 6.12 |
|
| ||||||
| 5.0–7.4 | 309 | 19.4 (6.3) | 6 (1.9) | 4.3 | 224 | 9.89 |
|
| ||||||
| 7.5–9.9 | 318 | 27.7 (8.7) | 4 (1.3) | 7.5 | 593 | 22.25 |
|
| ||||||
| ≥10.0 | 1007 | 160.3 (15.9) | 73 (7.2) | 8.7 | 119 | 56.59 |
|
| ||||||
| Total | 1961 | 218.6 (11.1) | 86 (4.4) | 6.8 | 154 | 94.85 |
|
| ||||||
| RRS | ||||||
| 0.0–4.9 | 528 | 16.7 (3.2) | 17 (3.2) | 0.0 | −2 | 0.00 |
|
| ||||||
| 5.0–7.4 | 350 | 21.8 (6.2) | 21 (6.0) | 0.2 | 4 | 0.03 |
|
| ||||||
| 7.5–9.9 | 274 | 23.9 (8.7) | 18 (6.6) | 2.1 | 33 | 1.58 |
|
| ||||||
| ≥10.0 | 809 | 151.1 (18.7) | 140 (17.3) | 1.4 | 8 | 1.01 |
|
| ||||||
| Total | 1961 | 213.5 (10.9) | 196 (10.0) | 0.9 | 9 | 2.62 |
| AHA-ACC-ASCVD | ||||||
|
| ||||||
| 0.0–4.9 | 375 | 12.8 (3.4) | 5 (1.3) | 2.1 | 156 | 4.94 |
|
| ||||||
| 5.0–7.4 | 317 | 19.7 (6.2) | 8 (2.5) | 3.7 | 147 | 7.45 |
|
| ||||||
| 7.5–9.9 | 263 | 22.8 (8.7) | 8 (3.0) | 5.6 | 186 | 10.56 |
|
| ||||||
| ≥10.0 | 1006 | 176.7 (17.6) | 104 (10.3) | 7.2 | 70 | 36.32 |
|
| ||||||
| Total | 1961 | 232.1 (11.8) | 125 (6.4) | 5.5 | 86 | 59.27 |
ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; ATPIII = Adult Treatment Panel III; CHD = coronary heart disease; CVD = cardiovascular disease; FRS = Framingham risk score; RRS = Reynolds Risk Score.
Percentage discordance calculation: ([{expected percentage − observed percentage}/observed percentage] × 100).
Table 4.
Predicted and Observed Events for Each Risk Score in Clinically Relevant Risk Categories: Women
| Risk Score, % | Participants, n | Predicted Events, n (%) | Observed Events, n (%) | Signed Absolute Difference | Discordance, %* | Hosmer–Lemeshow Chi-Square Statistic |
|---|---|---|---|---|---|---|
| FRS-CHD | ||||||
|
| ||||||
| 0.0–4.9 | 963 | 32.2 (3.3) | 29 (3.0) | 0.3 | 11 | 0.33 |
|
| ||||||
| 5.0–7.4 | 594 | 36.3 (6.1) | 26 (4.4) | 1.7 | 40 | 3.13 |
|
| ||||||
| 7.5–9.9 | 343 | 29.5 (8.6) | 16 (4.7) | 3.9 | 85 | 6.78 |
|
| ||||||
| ≥10.0 | 366 | 48.5 (13.2) | 28 (7.7) | 5.6 | 73 | 9.98 |
|
| ||||||
| Total | 2266 | 146.5 (6.5) | 99 (4.4) | 2.1 | 48 | 20.21 |
|
| ||||||
| FRS-CVD | ||||||
| 0.0–4.9 | 724 | 22.9 (3.2) | 19 (2.6) | 0.5 | 20 | 0.68 |
|
| ||||||
| 5.0–7.4 | 477 | 29.3 (6.1) | 25 (5.2) | 0.9 | 18 | 0.67 |
|
| ||||||
| 7.5–9.9 | 325 | 28.1 (8.6) | 28 (8.6) | 0.0 | 0 | 0.00 |
|
| ||||||
| ≥10.0 | 740 | 122.3 (16.5) | 115 (15.5) | 1.0 | 6 | 0.52 |
|
| ||||||
| Total | 2266 | 202.6 (8.9) | 187 (8.3) | 0.7 | 8 | 1.87 |
| ATPIII-FRS-CHD | ||||||
|
| ||||||
| 0.0–4.9 | 1836 | 35.0 (1.9) | 29 (1.6) | 0.3 | 21 | 1.05 |
|
| ||||||
| 5.0–7.4 | 235 | 14.2 (6.0) | 8 (3.4) | 2.6 | 77 | 2.84 |
|
| ||||||
| 7.5–9.9 | 109 | 9.3 (8.6) | 5 (4.6) | 4.0 | 87 | 2.21 |
|
| ||||||
| ≥10.0 | 86 | 11.7 (13.6) | 6 (7.0) | 6.6 | 94 | 3.18 |
|
| ||||||
| Total | 2266 | 70.2 (3.1) | 48 (2.1) | 1.0 | 46 | 9.29 |
|
| ||||||
| RRS | ||||||
| 0.0–4.9 | 1608 | 33.2 (2.1) | 55 (3.4) | −1.3 | −40 | 14.60 |
|
| ||||||
| 5.0–7.4 | 262 | 16.0 (6.1) | 16 (6.1) | 0.0 | 0 | 0.00 |
|
| ||||||
| 7.5–9.9 | 158 | 13.7 (8.7) | 16 (10.1) | −1.5 | −15 | 0.43 |
|
| ||||||
| ≥10.0 | 238 | 37.6 (15.8) | 40 (16.8) | −1.0 | −6 | 0.18 |
|
| ||||||
| Total | 2266 | 100.5 (4.4) | 127 (5.6) | −1.2 | −21 | 15.21 |
| AHA-ACC-ASCVD | ||||||
|
| ||||||
| 0.0–4.9 | 1140 | 27.4 (2.4) | 19 (1.7) | 0.7 | 44 | 2.64 |
|
| ||||||
| 5.0–7.4 | 355 | 21.8 (6.1) | 16 (4.5) | 1.6 | 36 | 1.65 |
|
| ||||||
| 7.5–9.9 | 235 | 20.5 (8.7) | 12 (5.1) | 3.6 | 71 | 3.86 |
|
| ||||||
| ≥10.0 | 536 | 85.4 (15.9) | 46 (8.6) | 7.3 | 86 | 21.60 |
|
| ||||||
| Total | 2266 | 155.1 (6.8) | 93 (4.1) | 2.7 | 67 | 29.75 |
ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; ATPIII = Adult Treatment Panel III; CHD = coronary heart disease; CVD = cardiovascular disease; FRS = Framingham risk score; RRS = Reynolds Risk Score.
Percentage discordance calculation: ([{expected percentage − observed percentage}/observed percentage] × 100).
An evaluation of discrimination (how well each model discriminated between a participant having or not having an event) revealed little difference between the performance of the risk scores in men (c-statistic, 0.69 to 0.71; discrimination slope, 0.05 to 0.09). Among women, the c-statistic and discrimination slope varied little among the FRS-CVD, ATPIII-FRS-CHD, RRS, and AHA-ACC-ASCVD risk scores (c-statistic, 0.67 to 0.72; discrimination slope, 0.02 to 0.05) but were lower (less able to discriminate if a participant was to have or not have an event) for the FRS-CHD risk score (c-statistic, 0.60; discrimination slope, 0.01) (Table 2 and Appendix Figure [available at www.annals.org]).
Appendix Figure.
Area under the curve for all 5 risk prediction models.
ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; ATPIII = Adult Treatment Panel III; CHD = coronary heart disease; CVD = cardiovascular disease; FRS = Framingham risk score; RRS = Reynolds Risk Score. Left. Men (n = 1961). Right. Women (n = 2266).
The effect of medication use at baseline and follow-up and interim revascularizations on the discordance between the observed and expected event rate was evaluated in a sensitivity analysis restricted to men and women who did not receive any preventive therapy (aspirin use, lipid-lowering or antihypertensive medication use, and any coronary revascularization) as assessed at baseline and all 5 follow-up visits. Limiting our evaluation to untreated participants (aspirin or lipid-lowering or antihypertensive medication use at baseline or at any of the 5 follow-up visits or coronary revascularization) resulted in greater risk overestimation for all risk scores examined (Table 5). An analysis that censored participants at the time of cholesterol-lowering medication use or revascularization also showed greater overestimation by the AHA-ACC-ASCVD risk score (Appendix Table 4, available at www.annals.org). A sensitivity analysis using the RRS definition of premature parental history of CHD resulted in a change from a 9% overestimation to a 5% underestimation of risk via the RRS in men and greater underestimation (from −21% to −27%) via the RRS among women. The ability of the RRS to discriminate between those who did and did not have events (c-statistics) remained unchanged for men but worsened for women (c-statistic increased from 0.72 to 0.70) when the RRS definition of premature parental history of CHD was used (Appendix Table 5, www.annals.org). Analyses using the calculated HbA1c level and including participants with diabetes produced only modest changes in our results (Appendix Table 6, www.annals.org).
Table 5.
Predicted and Observed Risk for Each Risk Score Among Never-Treated Participants*
| Risk Score | Predicted Events, n (%) | Observed Events, n (%) | Signed Absolute Difference | Discordance, %† | c-Statistic | Discrimination Slope |
|---|---|---|---|---|---|---|
| Total (n = 790) | ||||||
|
| ||||||
| FRS-CHD | 57.5 (7.28) | 12 (1.52) | 5.76 | 379 | 0.75 | 0.07 |
|
| ||||||
| FRS-CVD | 68.1 (8.62) | 18 (2.28) | 6.34 | 278 | 0.77 | 0.08 |
|
| ||||||
| ATPIII-FRS-CHD | 36.6 (4.63) | 10 (1.27) | 3.37 | 266 | 0.80 | 0.07 |
|
| ||||||
| AHA-ACC-ASCVD | 44.3 (5.60) | 14 (1.77) | 3.83 | 216 | 0.79 | 0.06 |
| Men (n = 392) | ||||||
|
| ||||||
| FRS-CHD | 39.1 (9.99) | 10 (2.55) | 7.43 | 291 | 0.77 | 0.06 |
|
| ||||||
| FRS-CVD | 49.4 (12.60) | 13 (3.32) | 9.29 | 280 | 0.72 | 0.07 |
|
| ||||||
| ATPIII-FRS-CHD | 30.9 (7.88) | 9 (2.30) | 5.58 | 243 | 0.75 | 0.05 |
|
| ||||||
| AHA-ACC-ASCVD | 31.0 (7.90) | 11 (2.81) | 5.09 | 181 | 0.76 | 0.06 |
| Women (n = 398) | ||||||
|
| ||||||
| FRS-CHD | 18.4 (4.62) | 2 (0.50) | 4.12 | 820 | 0.61 | −0.01 |
|
| ||||||
| FRS-CVD | 18.7 (4.70) | 5 (1.26) | 3.44 | 274 | 0.82 | 0.04 |
|
| ||||||
| ATPIII-FRS-CHD | 5.7 (1.43) | 1 (0.25) | 1.18 | 470 | 0.57 | −0.01 |
|
| ||||||
| AHA-ACC-ASCVD | 13.3 (3.34) | 3 (0.75) | 2.59 | 344 | 0.67 | 0.04 |
ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; ATPIII = Adult Treatment Panel III; CHD = coronary heart disease; CVD = cardiovascular disease; FRS = Framingham risk score.
Never-treated participants did not receive aspirin or lipid-lowering or antihypertensive medication at baseline or at any of the 5 follow-up visits and did not have coronary revascularization. The Reynolds Risk Score was excluded from this analysis because revascularization was part of the end point.
Percentage discordance calculation: ([{expected percentage − observed percentage}/observed percentage] × 100).
Appendix Table 4.
Predicted and Observed Events Using the AHA-ACC-ASCVD Risk Score Among MESA Participants, With Censoring at Cholesterol-Lowering Medication Use or Revascularization*
| Variable | Predicted Events, n (%) | Observed Events, n (%) | Signed Absolute Difference | Discordance, %† | c-Statistic | Discrimination Slope |
|---|---|---|---|---|---|---|
| Total (n = 3248) | 224.2 (6.90) | 99 (3.05) | 3.85 | 126 | 0.76 | 0.08 |
| Men (n = 1513) | 136.8 (9.04) | 57 (3.77) | 5.27 | 140 | 0.75 | 0.08 |
| Women (n = 1735) | 87.4 (5.04) | 42 (2.42) | 2.61 | 108 | 0.74 | 0.08 |
ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; MESA = Multi-Ethnic Study of Atherosclerosis.
Because the exact day of cholesterol-lowering medication use was not known, participants were censored at the last time observed not to be receiving this medication. Participants who reported cholesterol medication use at examination 1 or 2 were therefore excluded from the analysis.
Percentage discordance calculation: ([{expected percentage − observed percentage}/observed percentage] × 100).
Appendix Table 5.
Predicted and Observed Events for Each Risk Score Using the RRS Definition of Premature Parental History of CHD*
| Risk Score | Predicted Events, n (%) | Observed Events, n (%) | Signed Absolute Difference | Discordance, %† | c-Statistic | Discrimination Slope |
|---|---|---|---|---|---|---|
| Total (n = 3869) | ||||||
| FRS-CHD | 367.8 (9.51) | 234 (6.05) | 3.46 | 57 | 0.68 | 0.05 |
| FRS-CVD | 516.9 (13.4) | 402 (10.39) | 2.97 | 29 | 0.71 | 0.08 |
| ATPIII-FRS-CHD | 261.8 (6.77) | 112 (2.89) | 3.87 | 134 | 0.71 | 0.05 |
| RRS | 248.9 (6.43) | 286 (7.39) | −0.96 | −13 | 0.72 | 0.05 |
| AHA-ACC-ASCVD | 356.3 (9.21) | 188 (4.86) | 4.35 | 90 | 0.70 | 0.05 |
| Men (n = 1803) | ||||||
| FRS-CHD | 233.3 (12.94) | 150 (8.32) | 4.62 | 56 | 0.69 | 0.05 |
| FRS-CVD | 332.9 (18.46) | 238 (13.20) | 5.26 | 40 | 0.71 | 0.09 |
| ATPIII-FRS-CHD | 199.0 (11.04) | 74 (4.10) | 6.93 | 169 | 0.72 | 0.05 |
| RRS | 170.4 (9.45) | 179 (9.93) | −0.48 | −5 | 0.70 | 0.05 |
| AHA-ACC-ASCVD | 216.1 (11.98) | 110 (6.10) | 5.88 | 96 | 0.71 | 0.06 |
| Women (n = 2066) | ||||||
| FRS-CHD | 134.6 (6.51) | 84 (4.07) | 2.45 | 60 | 0.58 | 0.01 |
| FRS-CVD | 183.9 (8.90) | 164 (7.94) | 0.97 | 12 | 0.69 | 0.04 |
| ATPIII-FRS-CHD | 62.8 (3.04) | 38 (1.84) | 1.20 | 65 | 0.64 | 0.01 |
| RRS | 78.5 (3.80) | 107 (5.18) | −1.38 | −27 | 0.70 | 0.03 |
| AHA-ACC-ASCVD | 140.2 (6.79) | 78 (3.78) | 3.01 | 80 | 0.67 | 0.03 |
ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; ATPIII = Adult Treatment Panel III; CHD = coronary heart disease; CVD = cardiovascular disease; FRS = Framingham risk score; RRS = Reynolds Risk Score.
Either parent having a myocardial infarction at age <60 y.
Percentage discordance calculation: ([{expected percentage − observed percentage}/observed percentage] × 100).
Appendix Table 6.
Predicted and Observed Events for Each Risk Score, Including Participants With Diabetes* and a Calculated Hemoglobin A1c Level From a Baseline Fasting Glucose Level
| Risk Score | Predicted Events, n (%) | Observed Events, n (%) | Signed Absolute Difference | Discordance, %† | c-Statistic | Discrimination Slope |
|---|---|---|---|---|---|---|
| Total (n = 4804) | ||||||
| FRS-CHD | 495.3 (10.31) | 347 (7.22) | 3.09 | 43 | 0.69 | 0.06 |
| FRS-CVD | 726.8 (15.13) | 584 (12.16) | 2.97 | 24 | 0.72 | 0.11 |
| ATPIII-FRS-CHD | 332.6 (6.92) | 175 (3.64) | 3.28 | 90 | 0.72 | 0.06 |
| RRS | 390.8 (8.14) | 418 (8.70) | −0.57 | −7 | 0.72 | 0.07 |
| AHA-ACC-ASCVD | 511.5 (10.65) | 281 (5.85) | 4.80 | 82 | 0.71 | 0.07 |
| Men (n = 2271) | ||||||
| FRS-CHD | 312.8 (13.77) | 220 (9.69) | 4.09 | 42 | 0.69 | 0.06 |
| FRS-CVD | 467.9 (20.60) | 346 (15.24) | 5.37 | 35 | 0.72 | 0.11 |
| ATPIII-FRS-CHD | 251.7 (11.08) | 117 (5.15) | 5.93 | 115 | 0.70 | 0.05 |
| RRS | 254.4 (11.20) | 266 (11.71) | −0.51 | −4 | 0.69 | 0.06 |
| AHA-ACC-ASCVD | 309.6 (13.63) | 172 (7.57) | 6.06 | 80 | 0.70 | 0.07 |
| Women (n = 2533) | ||||||
| FRS-CHD | 182.6 (7.21) | 127 (5.01) | 2.19 | 44 | 0.63 | 0.02 |
| FRS-CVD | 259.0 (10.22) | 238 (9.40) | 0.83 | 9 | 0.70 | 0.06 |
| ATPIII-FRS-CHD | 80.8 (3.19) | 58 (2.29) | 0.90 | 39 | 0.69 | 0.02 |
| RRS | 136.4 (5.39) | 152 (6.00) | −0.62 | −10 | 0.71 | 0.05 |
| AHA-ACC-ASCVD | 201.8 (7.97) | 109 (4.30) | 3.66 | 85 | 0.69 | 0.05 |
ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; ATPIII = Adult Treatment Panel III; CHD = coronary heart disease; CVD = cardiovascular disease; FRS = Framingham risk score; RRS = Reynolds Risk Score.
Diabetes was defined as a fasting glucose level ≥7.0 mmol/L (≥126 mg/dL) or treated diabetes; 310 of 2271 (13.7%) men and 267 of 2533 (10.5%) women were considered to be diabetic.
Percentage discordance calculation: ([{expected percentage − observed percentage}/observed percentage] × 100).
A sensitivity analysis of the 3175 MESA participants in the CMS database found 16 ASCVD events not identified as 1 of the 160 MESA-adjudicated ASCVD events (9%). Restricting the analysis to participants included in the CMS database, and including the potential 9% increase in ASCVD events suggested by CMS linkage, we observed greater overestimation by the AHA-ACC-ASCVD risk score for the total cohort, men, and women (Appendix Table 7, available at www.annals.org) and for all risk categories (Appendix Table 8, available at www.annals.org) than in our primary analysis. Of the 16 potential (not adjudicated) missed events identified in the CMS billing database, 12 occurred in MESA participants with an AHA-ACC-ASCVD risk score greater than 7.5%.
Appendix Table 7.
Predicted and Observed Events Using the AHA-ACC-ASCVD Risk Score Among MESA Participants Linked by Medicare Database
| Variable | Predicted Events, n (%) | Observed Events, Including CMS Data, n (%) | Signed Absolute Difference | Discordance, %* | c-Statistic | Discrimination Slope |
|---|---|---|---|---|---|---|
| Total (n = 3175) | 345.0 (9.16) | 176 (5.54) | 4.57 | 82 | 0.69 | 0.05 |
| Men (n = 1489) | 208.2 (13.98) | 99 (6.65) | 6.36 | 96 | 0.72 | 0.06 |
| Women (n = 1686) | 136.9 (8.12) | 77 (4.57) | 2.98 | 65 | 0.64 | 0.04 |
ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; CMS = Centers for Medicare & Medicaid Services; MESA = Multi-Ethnic Study of Atherosclerosis.
Percentage discordance calculation: ([{expected percentage − observed percentage}/observed percentage] × 100).
Appendix Table 8.
Predicted and Observed Events Using the AHA-ACC-ASCVD Risk Score in Clinically Relevant Risk Categories Among MESA Participants Linked by Medicare Database
| Risk Score, % | Participants, n | Predicted Events, n (%) | Observed Events, Including CMS Data, n (%) | Signed Absolute Difference | Discordance, %* | Hosmer–Lemeshow Chi-Square Statistic |
|---|---|---|---|---|---|---|
| 0.0–4.9 | 910 | 25.7 (2.8) | 21 (2.3) | 0.52 | 23 | 0.90 |
| 5.0–7.4 | 522 | 32.5 (6.2) | 16 (3.1) | 3.15 | 103 | 8.91 |
| 7.5–9.9 | 416 | 36.3 (8.7) | 15 (3.6) | 5.12 | 142 | 13.69 |
| ≥10.0 | 1327 | 226.5 (17.1) | 124 (9.3) | 7.72 | 83 | 55.92 |
| Total | 3175 | 321.0 (10.1) | 176 (5.5) | 4.57 | 82 | 79.41 |
ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; CMS = Centers for Medicare & Medicaid Services; MESA = Multi-Ethnic Study of Atherosclerosis.
Percentage discordance calculation: ([{expected percentage − observed percentage}/observed percentage] × 100).
Discussion
This comparative analysis done in a well-characterized; age-, sex- and race-balanced; community-based; multiethnic cohort showed that 3 commonly used cardiovascular risk scores (FRS-CHD, FRS-CVD, and ATPIII-FRS-CHD) and the newly recommended risk score (AHA-ACC-ASCVD) all overestimate the risk for their intended outcome in men by 37% to 154%. The RRS had the best calibration for men with only 9% overestimation. The FRS-CHD, ATPIII-FRS-CHD, and AHA-ACC-ASCVD risk scores also overestimated risk by 46% to 67% in women. The FRS-CVD had the best calibration among women with 8% overestimation, whereas the RRS underestimated risk among women by 21%. Overestimation predominated and was noted throughout the continuum of risk for these tools, including the intermediate risk groups in which treatment decisions balancing therapeutic risk and benefit are more challenging.
A recent study by Muntner and colleagues (18) evaluated a 5-year version of the 10-year AHA-ACC-ASCVD risk score in the REGARDS (REasons for Geographic and Racial Differences in Stroke) cohort at 5-year follow-up. Consistent with our finding, the authors found overestimation of risk and similar discrimination with the AHA-ACC-ASCVD risk score in the REGARDS participants aged 45 to 79 years without a history of CHD, stroke, heart failure (identified exclusively by use of digoxin), or atrial fibrillation at baseline. They found less overestimation in a post hoc subgroup of “clinically relevant” participants and in participants older than 65 years when using a combination of adjudicated and nonadjudicated events identified in the CMS billing database. Limitations of this study include the possible decreased specificity of events identified in CMS without further adjudication and follow-up limited to 5 years to evaluate a 10-year risk score. A recent evaluation of the AHA-ACC-ASCVD, the ATPIII-FRS-CHD, and the European Society of Cardiology risk scores in the Rotterdam study cohort found event overestimation similar to that noted in our study (17). A prior analysis by Cook and colleagues compared CVD risk scores using a multiethnic prospective cohort of more than 90 000 postmenopausal women. Consistent with our study, this study showed overestimation of risk with both the ATPIII-FRS-CHD and FRS-CVD but not the RRS (8). However, in both the Rotterdam and Cook and colleagues’ studies, evaluations were done in cohorts with limited diversity and did not account for the differences in end points for which each of the risk scores was designed. Our study compares the different risk calculators with their designed end points for each score in a sex-balanced, multiethnic, middle-aged modern cohort.
Estimation of absolute cardiovascular risk is at the core of national guidelines for the appropriate use of cholesterol-lowering medication and aspirin in the primary care setting (19, 26). Although overestimation of risk with the AHA-ACC-ASCVD risk score was acknowledged in the new risk assessment guidelines and confirmed by others (10, 13, 17, 18), our analysis further shows a lack of superior calibration or discrimination compared with older risk scores.
We noted several potential reasons for the systematic overestimation when applying these risk scores to the MESA cohort: changing significance of risk factors between older and modern cohorts, more effective therapy for preventing cardiovascular events despite persistence of risk factors, healthy cohort effect (selection bias at cohort enrollment), and incomplete capture of cardiovascular events.
The first 3 explanations may be summarized by the potentially problematic use of cohorts developed decades ago to derive risk scores designed to be used at the present time. Even the newest risk score, the AHA-ACC-ASCVD score, was developed from cohorts that are decades old (13–16, 27).
Changing significance of risk factors between older cohorts that were used to develop these risk scores and the more modern MESA cohort may explain the poor estimation of effect of individual risk factors included in the risk scores. For example, smoking is dichotomized as a “yes” or “no” variable in all risk score models evaluated in this study. However, dose (or number) of cigarettes smoked and content of cigarettes have both changed substantially over time (28, 29). Unmeasured risk factors that have changed over time, such as salt and trans fat intakes, secondhand smoke exposure, and environmental pollution, may also explain some of the residual overestimation not accounted for in our models.
Although 4 of the evaluated risk scores overestimated absolute risk in women, our analysis indicates that women were associated with less overestimation than men in these models.
Risk score overestimation has been hypothesized to be a reflection of modern cardiovascular disease therapy (aspirin, lipid-lowering or antihypertensive medication, and revascularization) resulting in reduction of incident CVD events (30). Modern therapy may directly reduce CVD events and may also change the quantitative relationship between risk factors and CVD outcomes. In our study, however, these therapies—assessed at the time of risk score calculation and at any of the 5 MESA follow-up visits—were not associated with risk score overestimation in MESA. In fact, overestimation was greater in our analysis limited to participants who were not treated with any of these preventive therapies. This finding may be because participants not treated with these methods have healthy characteristics not captured by the risk scores or, in the case of lipid-lowering therapy at baseline, the effect of the medication is accounted for by the cholesterol variables in the prediction models.
The MESA participants did not have known CVD at baseline. Participants who enroll in research studies may be more health-conscious and have better general health practices than those who do not. In this way, MESA may not be an accurate representation of the general population. This limitation is shared by all of the studies used to derive risk score models and is not unique to MESA. The RRS underestimated risk by only 3% in our cohort. Likewise, in a modern study of randomly selected adult participants enrolled between 2000 and 2003 from the metropolitan Ruhr area in Germany, the FRS overestimated risk by more than 2-fold in the 93% of participants without known coronary artery disease at baseline (31). Further, this new risk score displayed worse calibration (up to 38% overestimation) and discrimination in the “contemporary” years of follow-up of the pooled derivation cohort than the earlier years of follow-up (13). In sum, this suggests that a unique healthy cohort effect in MESA is unlikely to be solely responsible for risk overestimation.
Epidemiologic methods for ascertainment of clinical events seem to be similar in MESA with those of the studies used to derive all of the risk scores, except for 2 of the 4 cohorts used to derive the AHA-ACC-ASCVD risk score (8, 11, 15, 16, 27, 32). These 2 cohorts included additional surveillance of ASCVD events (16, 33). MESA was successful in obtaining medical records for approximately 98% of reported hospitalized CHD and CVD events and information on 95% of reported outpatient cardiovascular diagnostic encounters. Follow-up telephone interviews were completed with 92% of living participants. To further explore the possibility of missed events, we analyzed CMS data. Our analysis suggests that no more than 9% of myocardial infarctions and strokes could have been missed among the MESA participants in our analytic sample. This is a high-end estimate; some of the CMS events are false-positive because of the nature of billing data. Sensitivity analysis using CMS-identified events did not significantly curtail AHA-ACC-ASCVD risk score overestimation in MESA.
Accurate cardiac risk estimation is essential to balancing the risks and benefits of preventive therapies. For example, current cholesterol management guidelines recommend consideration of statin therapy for participants with an AHA-ACC-ASCVD risk calculation of 7.5% or more over 10 years (19). Overestimation of risk would likely result in increased use of preventive medications, potentially exposing some patients to the unnecessary risks of these drugs. In particular, aspirin use in primary prevention is known to be a delicate balance between cardiovascular risk reduction and increased incidence of bleeding. Overestimation of cardiovascular risk may result in aspirin treatment of low-risk participants who are more likely to be harmed from bleeding than to benefit from a reduction in CVD. Underestimation of risk, which was seen in women with the RRS, would be expected to result in undertreatment with appropriate therapies and more preventable CVD events.
In addition to patient safety, overestimation of risk has implications for public health planning with accompanying financial ramifications for a heavily burdened health care system. Statins, for example, although clearly cost-effective in secondary prevention or high-risk primary prevention patients, become less cost-effective as primary prevention for low-risk patients (34, 35). Overestimation of risk could result in more health care dollars spent, less health gain, and exposure to drug side effects.
The RRS for men and the ATPIII-FRS-CHD risk score were not developed for risk calculation among people with diabetes, so the exclusion of these participants may have limited the effectiveness of the non-RRS and non–ATPIII-FRS-CHD risk scores that included diabetes in their risk score calculation. Sensitivity analysis showed that inclusion of diabetes had inconsistent effect on risk prediction, which resulted in no change in our overall study conclusions.
Our results indicate that 4 of the 5 risk estimation algorithms, including the AHA-ACC-ASCVD risk score, overestimate risk by 37% to 154% in men and 8% to 67% in women. One risk score, the RRS, had superior calibration and equal discrimination compared with the other scores evaluated, but it underestimated risk in women. Medication use and interim revascularizations are not robust explanations of the overestimation.
Accurate estimation of absolute risk underpins the effort by the AHA and ACC to update the prevention guidelines and is required to effectively balance therapeutic risk and benefit of an intervention for an individual patient or entire population. Physicians treating patients similar to those in MESA may consider interpreting the absolute risk generated by the new risk score with caution.
EDITORS’ NOTES.
Context
Accurate risk prediction is an important component of guidelines about interventions to prevent cardiovascular disease.
Contribution
This study found that 4 prominent risk prediction tools overestimate the risk for cardiovascular disease, and a fifth tool underestimated risk among women. The use of preventive therapies did not seem to be the reason for the overprediction observed.
Implication
The limitations of available risk prediction tools require consideration and improvement to better guide prevention interventions.
Acknowledgments
The authors thank the other investigators, the staff, and the participants of MESA for their valuable contributions. A full list of participating MESA investigators and institutions can be found at www.mesa-nhlbi.org.
Grant Support: This research was supported by contracts N01-HC-95159 through N01-HC-95169 from the National Heart, Lung, and Blood Institute and by grants UL1-TR-000040 and UL1-TR-001079 from the National Center for Research Resources.
Footnotes
Current author addresses and author contributions are available at www.annals.org.
Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M14-1281.
Reproducible Research Statement: Study protocol: Available at www.mesa-nhlbi.org. Statistical code: Requests will be considered by Dr. Rebekah Young on a case-by-case basis. Data set: Applications for use of MESA data are available to the public at www.mesa-nhlbi.org.
Author Contributions: Conception and design: A.P. DeFilippis, R. Young, M.J. Blaha.
Analysis and interpretation of the data: A.P. DeFilippis, R. Young, C. Carrubba, R.S. Blumenthal, R.A. Kronmal, R.L. McClelland, M.J. Blaha.
Drafting of the article: A.P. DeFilippis, R. Young, C. Carrubba, R.A. Kronmal, M.J. Blaha.
Critical revision of the article for important intellectual content: A.P. DeFilippis, R. Young, J.W. McEvoy, M.J. Budoff, R.S. Blumenthal, R.A. Kronmal, R.L. McClelland, K. Nasir, M.J. Blaha.
Final approval of the article: A.P. DeFilippis, R. Young, J.W. McEvoy, M.J. Budoff, R.S. Blumenthal, R.A. Kronmal, R.L. McClelland, K. Nasir, M.J. Blaha.
Provision of study materials or patients: M.J. Budoff.
Statistical expertise: R. Young, R.A. Kronmal, R.L. McClelland, M.J. Blaha.
Administrative, technical, or logistic support: A.P. DeFilippis, R.L. McClelland, M.J. Blaha.
Collection and assembly of data: R. Young, R.A. Kronmal, R.L. McClelland, M.J. Blaha.
References
- 1.Fletcher B, Berra K, Ades P, Braun LT, Burke LE, Durstine JL, et al. Council on Cardiovascular Nursing. Managing abnormal blood lipids: a collaborative approach. Circulation. 2005;112:3184–209. doi: 10.1161/CIRCULATIONAHA.105.169180. [DOI] [PubMed] [Google Scholar]
- 2.Goldstein LB, Bushnell CD, Adams RJ, Appel LJ, Braun LT, Chaturvedi S, et al. American Heart Association Stroke Council. . Guidelines for the primary prevention of stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke. 2011;42:517–84. doi: 10.1161/STR.0b013e3181fcb238. [DOI] [PubMed] [Google Scholar]
- 3.Mosca L, Benjamin EJ, Berra K, Bezanson JL, Dolor RJ, Lloyd-Jones DM, et al. Effectiveness-based guidelines for the prevention of cardiovascular disease in women—2011 update: a guideline from the American Heart Association. Circulation. 2011;123:1243–62. doi: 10.1161/CIR.0b013e31820faaf8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Wilson PW, D’Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation. 1998;97:1837–47. doi: 10.1161/01.cir.97.18.1837. [DOI] [PubMed] [Google Scholar]
- 5.National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) final report. Circulation. 2002;106:3143–421. [PubMed] [Google Scholar]
- 6.Bastuji-Garin S, Deverly A, Moyse D, Castaigne A, Mancia G, de Leeuw PW, et al. Intervention as a Goal in Hypertension Treatment Study Group. The Framingham prediction rule is not valid in a European population of treated hypertensive patients. J Hypertens. 2002;20:1973–80. doi: 10.1097/00004872-200210000-00016. [DOI] [PubMed] [Google Scholar]
- 7.Brindle P, Emberson J, Lampe F, Walker M, Whincup P, Fahey T, et al. Predictive accuracy of the Framingham coronary risk score in British men: prospective cohort study. BMJ. 2003;327:1267. doi: 10.1136/bmj.327.7426.1267. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Cook NR, Paynter NP, Eaton CB, Manson JE, Martin LW, Robinson JG, et al. Comparison of the Framingham and Reynolds risk scores for global cardiovascular risk prediction in the multiethnic Women’s Health Initiative. Circulation. 2012;125:1748–56. S1–11. doi: 10.1161/CIRCULATIONAHA.111.075929. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Liu J, Hong Y, D’Agostino RB, Sr, Wu Z, Wang W, Sun J, et al. Predictive value for the Chinese population of the Framingham CHD risk assessment tool compared with the Chinese Multi-Provincial Cohort Study. JAMA. 2004;291:2591–9. doi: 10.1001/jama.291.21.2591. [DOI] [PubMed] [Google Scholar]
- 10.Ridker PM, Cook NR. Statins: new American guidelines for prevention of cardiovascular disease. Lancet. 2013;382:1762–5. doi: 10.1016/S0140-6736(13)62388-0. [DOI] [PubMed] [Google Scholar]
- 11.D’Agostino RB, Sr, Vasan RS, Pencina MJ, Wolf PA, Cobain M, Massaro JM, et al. General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation. 2008;117:743–53. doi: 10.1161/CIRCULATIONAHA.107.699579. [DOI] [PubMed] [Google Scholar]
- 12.Ridker PM, Buring JE, Rifai N, Cook NR. Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds Risk Score. JAMA. 2007;297:611–9. doi: 10.1001/jama.297.6.611. [DOI] [PubMed] [Google Scholar]
- 13.Goff DC, Jr, Lloyd-Jones DM, Bennett G, Coady S, D’Agostino RB, Gibbons R, et al. American College of Cardiology/American Heart Association Task Force on Practice Guidelines. 2013 ACC/ AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation. 2014;129:S49–73. doi: 10.1161/01.cir.0000437741.48606.98. [DOI] [PubMed] [Google Scholar]
- 14.Fried LP, Borhani NO, Enright P, Furberg CD, Gardin JM, Kronmal RA, et al. The Cardiovascular Health Study: design and rationale. Ann Epidemiol. 1991;1:263–76. doi: 10.1016/1047-2797(91)90005-w. [DOI] [PubMed] [Google Scholar]
- 15.Friedman GD, Cutter GR, Donahue RP, Hughes GH, Hulley SB, Jacobs DR, Jr, et al. CARDIA: study design, recruitment, and some characteristics of the examined subjects. J Clin Epidemiol. 1988;41:1105–16. doi: 10.1016/0895-4356(88)90080-7. [DOI] [PubMed] [Google Scholar]
- 16.The ARIC investigators. The Atherosclerosis Risk in Communities (ARIC) Study: design and objectives. Am J Epidemiol. 1989;129:687–702. [PubMed] [Google Scholar]
- 17.Kavousi M, Leening MJ, Nanchen D, Greenland P, Graham IM, Steyerberg EW, et al. Comparison of application of the ACC/AHA guidelines, Adult Treatment Panel III guidelines, and European Society of Cardiology guidelines for cardiovascular disease prevention in a European cohort. JAMA. 2014;311:1416–23. doi: 10.1001/jama.2014.2632. [DOI] [PubMed] [Google Scholar]
- 18.Muntner P, Colantonio LD, Cushman M, Goff DC, Jr, Howard G, Howard VJ, et al. Validation of the atherosclerotic cardiovascular disease Pooled Cohort risk equations. JAMA. 2014;311:1406–15. doi: 10.1001/jama.2014.2630. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Stone NJ, Robinson JG, Lichtenstein AH, Bairey Merz CN, Blum CB, Eckel RH, et al. American College of Cardiology/American Heart Association Task Force on Practice Guidelines. . 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation. 2014;129:S1–45. doi: 10.1161/01.cir.0000437738.63853.7a. [DOI] [PubMed] [Google Scholar]
- 20.Bild DE, Bluemke DA, Burke GL, Detrano R, Diez Roux AV, Folsom AR, et al. Multi-ethnic study of atherosclerosis: objectives and design. Am J Epidemiol. 2002;156:871–81. doi: 10.1093/aje/kwf113. [DOI] [PubMed] [Google Scholar]
- 21.Gordon WJ, Polansky JM, Boscardin WJ, Fung KZ, Steinman MA. Coronary risk assessment by point-based vs. equation-based Framingham models: significant implications for clinical care. J Gen Intern Med. 2010;25:1145–51. doi: 10.1007/s11606-010-1454-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Ridker PM, Paynter NP, Rifai N, Gaziano JM, Cook NR. C-reactive protein and parental history improve global cardiovascular risk prediction: the Reynolds Risk Score for men. Circulation. 2008;118:2243–51. doi: 10.1161/CIRCULATIONAHA.108.814251. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Crane PK, Walker R, Hubbard RA, Li G, Nathan DM, Zheng H, et al. Glucose levels and risk of dementia. N Engl J Med. 2013;369:540–8. doi: 10.1056/NEJMoa1215740. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Sacks DB. Translating hemoglobin A1c into average blood glucose: implications for clinical chemistry. Clin Chem. 2008;54:1756–8. doi: 10.1373/clinchem.2008.113282. [DOI] [PubMed] [Google Scholar]
- 25.Harrell FE, Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–87. doi: 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4. [DOI] [PubMed] [Google Scholar]
- 26.U S. Preventive Services Task Force. Aspirin for the prevention of cardiovascular disease: U.S. Preventive Services Task Force recommendation statement. Ann Intern Med. 2009;150:396–404. doi: 10.7326/0003-4819-150-6-200903170-00008. [DOI] [PubMed] [Google Scholar]
- 27.Kannel WB, Feinleib M, McNamara PM, Garrison RJ, Castelli WP. An investigation of coronary heart disease in families. The Framingham offspring study. Am J Epidemiol. 1979;110:281–90. doi: 10.1093/oxfordjournals.aje.a112813. [DOI] [PubMed] [Google Scholar]
- 28.Hoffmann D, Hoffmann I. The changing cigarette, 1950–1995. J Toxicol Environ Health. 1997;50:307–64. doi: 10.1080/009841097160393. [DOI] [PubMed] [Google Scholar]
- 29.Thun MJ, Carter BD, Feskanich D, Freedman ND, Prentice R, Lopez AD, et al. 50-year trends in smoking-related mortality in the United States. N Engl J Med. 2013;368:351–64. doi: 10.1056/NEJMsa1211127. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Ford ES, Ajani UA, Croft JB, Critchley JA, Labarthe DR, Kottke TE, et al. Explaining the decrease in U.S. deaths from coronary disease, 1980–2000. N Engl J Med. 2007;356:2388–98. doi: 10.1056/NEJMsa053935. [DOI] [PubMed] [Google Scholar]
- 31.Erbel R, Möhlenkamp S, Moebus S, Schmermund A, Lehmann N, Stang A, et al. Heinz Nixdorf Recall Study Investigative Group. Coronary risk stratification, discrimination, and reclassification improvement based on quantification of subclinical coronary atherosclerosis: the Heinz Nixdorf Recall study. J Am Coll Cardiol. 2010;56:1397–406. doi: 10.1016/j.jacc.2010.06.030. [DOI] [PubMed] [Google Scholar]
- 32.Curb JD, McTiernan A, Heckbert SR, Kooperberg C, Stanford J, Nevitt M, et al. WHI Morbidity and Mortality Committee. Outcomes ascertainment and adjudication methods in the Women’s Health Initiative. Ann Epidemiol. 2003;13:S122–8. doi: 10.1016/s1047-2797(03)00048-6. [DOI] [PubMed] [Google Scholar]
- 33.Ives DG, Fitzpatrick AL, Bild DE, Psaty BM, Kuller LH, Crowley PM, et al. Surveillance and ascertainment of cardiovascular events. The Cardiovascular Health Study. Ann Epidemiol. 1995;5:278–85. doi: 10.1016/1047-2797(94)00093-9. [DOI] [PubMed] [Google Scholar]
- 34.Brandle M, Davidson MB, Schriger DL, Lorber B, Herman WH. Cost effectiveness of statin therapy for the primary prevention of major coronary events in individuals with type 2 diabetes. Diabetes Care. 2003;26:1796–801. doi: 10.2337/diacare.26.6.1796. [DOI] [PubMed] [Google Scholar]
- 35.Mitchell AP, Simpson RJ. Statin cost effectiveness in primary prevention: a systematic review of the recent cost-effectiveness literature in the United States. BMC Res Notes. 2012;5:373. doi: 10.1186/1756-0500-5-373. [DOI] [PMC free article] [PubMed] [Google Scholar]