Abstract
Aims
To evaluate the 2013 American Heart Association (AHA)-American College of Cardiology (ACC)-Atherosclerotic Cardiovascular Disease (ASCVD) risk score among four different race/ethnic groups and to ascertain which factors are most associated with risk overestimation by the AHA-ACC-ASCVD score.
Methods and results
The Multi-Ethnic Study of Atherosclerosis (MESA), a prospective community-based cohort, was used to examine calibration and discrimination of the AHA-ACC-ASCVD risk score in 6441 White, Black, Chinese, and Hispanic Americans (aged 45–79 years and free of known ASCVD at baseline). Using univariable and multivariable absolute risk regression, we modelled the impact of individual risk factors on the discordance between observed and predicted 10-year ASCVD risk. Overestimation was observed in all race/ethnic groups in MESA and was highest among Chinese (252% for women and 314% for men) and lowest in White women (72%) and Hispanic men (67%). Higher age, Chinese race/ethnicity (when compared with White), systolic blood pressure (treated and untreated), diabetes, alcohol use, exercise, lipid-lowering medication, and aspirin use were all associated with more risk overestimation, whereas family history was associated with less risk overestimation in a multivariable model (all P < 0.05).
Conclusion
The AHA-ACC-ASCVD risk score overestimates ASCVD risk among men, women, and all four race/ethnic groups evaluated in a modern American primary prevention cohort. Clinicians treating patients similar to those in MESA, particularly older individuals and those with factors associated with more risk overestimation, may consider interpreting absolute ASCVD risk estimates with caution.
Keywords: Cardiovascular risk prediction, Prevention, Risk factors
Introduction
In November 2013, the American Heart Association (AHA) and the American College of Cardiology (ACC) published a new atherosclerotic cardiovascular disease (ASCVD) risk score to guide ASCVD risk-reducing therapy.1 This new risk score was derived from four prospective cohort studies that enrolled participants between the years of 1965 and 1995.1–5 The applicability of these new pooled cohort equations, also known as the AHA-ACC-ASCVD risk score, to modern populations has been questioned, as it has been found to overestimate risk in several independent cohorts.1,6–8 Overestimation was attenuated in one study when the cohort was restricted to subjects that met criteria consistent with guideline recommendations for the use of the AHA-ACC-ASCVD risk score to guide statin therapy [not taking lipid-lowering medication at baseline and having low-density lipoprotein cholesterol (LDL-C) between 70 and 189 mg/dL].8 We have previously reported that the 2013 AHA-ACC-ASCVD and three older Framingham-based risk scores overestimated cardiovascular events by 37–154% in men and 8–67% in women in the more modern Multi-Ethnic Study of Atherosclerosis (MESA) cohort, with overestimation noted throughout the continuum of risk.9 However, questions have been raised about the relevance of MESA, as it is multi-ethnic and may represent a healthier subset of the general US population.
We first sought to compare MESA to The National Health and Nutrition Examination Survey (NHANES) data to better understand the applicability of these results to the greater US population. While the AHA-ACC-ASCVD risk score is recommended only for individuals who identify themselves as White or Black, we sought to examine the performance of the new risk score in four different race/ethnicities and to determine if the risk factors included in the AHA-ACC-ASCVD risk score impact ASCVD risk differently in White and Black individuals. Finally, we undertook additional analyses to quantify the impact of differences in age, gender, race/ethnicity, blood pressure, serum cholesterol, family history of heart attack, socio-economic status, lifestyle variables, and preventive therapies (aspirin, lipid-lowering, anti-hypertensive, and interim revascularization) common to modern cohorts, on risk score overestimation. We reasoned that such data may be useful in the implementation of ASCVD prevention strategies as well as future risk score and guideline development.
Methods
The Multi-Ethnic Study of Atherosclerosis is a prospective community-based epidemiologic study of cardiovascular disease in an age- and gender-balanced, multi-ethnic cohort free of known ASCVD at the time of enrolment. The study design and methods have been previously published.10 The Multi-Ethnic Study of Atherosclerosis has institutional review board approval at all six enrolment sites. Atherosclerotic cardiovascular disease events during study follow-up consisted of myocardial infarction (MI), definite or probable angina, resuscitated cardiac arrest, coronary heart disease (CHD) death, stroke (not transient ischaemic attack), stroke death, other atherosclerotic death, or other CVD death. A detailed description of the MESA methodology is available at www.mesa-nhlbi.org. Race/ethnicity was self-reported. Family history of CHD was considered positive at the baseline visit if either parent had suffered a heart attack. The AHA-ACC-ASCVD risk prediction score was calculated from the baseline MESA data (2000–02) using published equations, using the equations for White individuals to calculate risk in Chinese and Hispanics.1 The Multi-Ethnic Study of Atherosclerosis included participants between the ages of 45 and 84 years; the AHA-ACC-ASCVD risk calculator is recommended for individuals aged 40–79 years. Therefore, in this study, our cohort was limited to individuals aged 45–79. Additional exclusion of participants with missing data required for risk score calculation (n = 53, <1%), or no follow-up data after baseline (n = 3, <1%), resulted in a final sample size of 6441.
To assess comparability to nationally representative samples, demographic and ASCVD risk factors were detailed for the AHA-ACC-ASCVD risk score derivation cohort, MESA participants included in this study, and the ASCVD-free, non-pregnant US population ages 40–79 as reflected in NHANES data 2013–14.11 Cumulative measure of ASCVD risk factor prevalence was assessed by calculating the percentage of participants with a calculated AHA-ACC-ASCVD risk score of <5, 5–7.4, 7.5–9.9, and ≥10% 10-year risk in the MESA cohort and the ASCVD-free, non-pregnant US population ages 40–79 in NHANES survey (2007–10).1 Rates of anti-hypertensive and lipid-lowering medication use in MESA were also compared with the US adult population estimates.12,13 For these analyses, MESA data are weighted by direct age-adjustment method to the year 2000 projected US population.13
Observed and expected events for the AHA-ACC-ASCVD score, stratified by race/ethnicity and gender, were compared in MESA. Events were censored at 10 years of follow-up. For the 6072 participants who did not have an ASCVD event, 62% had ≥10 years of follow-up and 88% had >9 years of follow-up data. For each subject with <10 years of follow-up, including those who died, we lowered their 10-year risk estimate to correspond to their length of follow-up using an exponential survival function to scale the risk score. If A10 denotes the 10-year proportion with events according to the AHA-ACC-ASCVD risk score, then the 1-year proportion is A1 = −ln(1−A10)/10. So for a person with 8.5 years of follow-up, A8.5 = 1 − exp(−A1 × 8.5).
To evaluate if ASCVD risk factors included in the AHA-ACC-ASCVD risk model impact Black individuals differently from Whites, adjusted and unadjusted Cox models for ASCVD were used to calculate hazard ratios for Black men and women compared with White men and women, and modelling the interaction between race and each of the risk factors in the ASCVD-ACC-ASCVD risk score was performed. Estimation of the AHA-ACC-ASCVD Cox model in MESA was performed to compare the magnitude of individual risk factors coefficients used in the AHA-ACC-ASCVD risk score to those obtained fitting the same model in the MESA cohort. For comparison, the same analysis was performed for the Reynolds risk score14,15 and The National Cholesterol Education Program (NCEP) Adult Treatment Panel (ATP) III16 risk score in MESA.
Univariable and multivariable absolute risk regression was used to evaluate the impact of individual risk factors on the discordance between observed ASCVD and predicted ASCVD by the AHA-ACC-ASCVD risk score. The absolute risk model is a linear regression model at the individual level, predicting a dichotomous outcome (ASCVD event yes/no within 10 years). It was fit as a generalized linear model with the AHA-ACC-ASCVD risk score entered as an offset (fixing the coefficient to 1.0), with an identity link function and using robust standard errors.17–19 To this model, we add other covariates either individually or jointly. Since almost every subset exhibited overestimation, we present the negative of the coefficients from the model—this allows us to talk about differences in ‘more’ overestimation (as opposed to negative ‘less’ overestimation). As an example, consider the model that adds gender alone. By fixing the coefficient of the AHA-ACC-ASCVD risk score to 1.0, the interpretation of the gender coefficient is the average difference in overestimation in predicted risk for men when compared with women. Both men and women are overestimated, but men are significantly more so. The multivariable model allows us to explore, for instance, whether this difference is due to men and women having different distributions of other risk factors and thus evaluate the independent contribution of various subgroups.
Our model was carefully selected to (i) include the variables that were part of the risk equation, (ii) include baseline covariates that were hypothesized to be responsible for the AHA-ACC-ASCVD risk score's poor fit in MESA, such as higher socio-economic status, and (iii) to account for time-updated medical treatments that were believed to reduce the risk of ASCVD, and therefore cause ‘overestimation’ by the risk calculator. Fit statistics were not used to modify the models, nor was statistical significance of any particular parameter. This approach allowed us to evaluate the impact of age, gender, race/ethnicity, socio-economic status, lifestyle variables, family history of acute MI, blood pressure, serum cholesterol, and individual preventive therapies (aspirin, lipid-lowering, anti-hypertensive) on the discordance between AHA-ACC-ASCVD calculated risk and the observed events. Individual preventive therapies were measured at baseline and during four follow-up visits; ASCVD event surveillance included assessment at all follow-up visits and 12 additional scheduled phone interviews. Therapies and events were treated as time-updated in the model. In an effort to study revascularizations that may have prevented subsequent events, all revascularizations >5 days prior to an event, or revascularizations not followed by any event, were selected for modelling.
Sensitivity analyses were performed on MESA participants who met clinical criteria consistent with guideline recommendations for the use of the AHA-ACC-ASCVD risk score to guide statin therapy (not taking lipid-lowering medication at baseline and having an LDL-C level between 70 and 189 mg/dL).1 Additional sensitivity analyses were performed by including potential ASCVD events identified in the Part A hospital claims Centers for Medicare & Medicaid Services (CMS) billing database, at any point during follow-up, but not adjudicated as an ASCVD event by MESA.
Results
With the exception of age, ethnic diversity, and HDL cholesterol, MESA participants included in this analysis appear to be more representative of contemporary Americans, as reflected in 2013–14 NHANES data, than subjects included in the AHA-ACC-ASCVD derivation cohort (Table 1). In addition to individual risk factors, aggregate cardiovascular risk profiles in MESA appeared similar to contemporary Americans in NHANES as evident from a comparable proportion (within 3%) of subjects with a calculated AHA-ACC-ASCVD risk score of <5, 5–7.4, 7.5–9.9, and ≥10% (Supplementary material online, Table S1). Baseline and treatment trends were also similar in MESA and NHANES. At baseline (2000–02), 31% of MESA participants (age-adjusted) were treated for hypertension when compared with ∼30% of the US population in contemporary NHANES data (2001–02).12 Cholesterol-lowering medication use was reported in 20% of the US population older than 40 years in NHANES 2003–04 and in 19% of age-adjusted MESA participants at approximately the same time (2002–04). These percentages increased to an estimated 28% of US adults in 2011–12 and 34% of participants in MESA over the same time period.13
Table 1.
Baseline characteristics of Multi-Ethnic Study of Atherosclerosis subjects used in this analysis, American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease derivation cohort, and non-pregnant Americans free of known atherosclerotic cardiovascular disease between ages 40 and 79 per National Health and Nutrition Examination Survey 2013–14
| Variables, mean (SD) | MESA | AHA-ACC-ASCVD derivation cohort | NHANES 2013–14a |
|---|---|---|---|
| Age (years) | 61.3 | 56.3 | 55.6 |
| Age range | 45–79 | 40–79 | 40–79 |
| Gender (% female) | 52.6 | 56.4 | 52.6 |
| Ethnicity, n (%) | |||
| White | 38.5 | 82.6 | 69.3 |
| Chinese | 11.9 | 0 | b |
| Black | 27.6 | 17.4 | 11.0 |
| Hispanic | 22.1 | 0 | 12.3 |
| Total cholesterol (mg/dL) | 194.4 | 215.5 | 196.4 |
| HDL cholesterol (mg/dL) | 50.9 | 52.6 | 54.3 |
| Untreated SBP (mmHg) | 121.2 | 123.1 | 121.3 |
| Treated SBP (mmHg) | 134.3 | 136.1 | 131.3 |
| Current smoker (%) | 13.5 | 25.6 | 18.7 |
| Diabetesc (%) | 11.5 | 9.1 | 12.8 |
| Baseline anti-hypertensive medication use (%) | 36.4 | 21.2 | 36.6 |
aNHANES data includes non-pregnant participants ages 40–79 without prior self-reported MI or stroke. Data are weighted using the full sample 2-year mobile exam centre weight.
bNHANES collected information about Asian ethnicity, whereas MESA collected data from Chinese participants specifically.
cFor MESA, diabetes was defined as use of insulin/oral hypoglycaemic agents or fasting glucose ≥126 mg/dL. For NHANES, diabetes status was ascertained by participant's report of diabetes diagnosis from a doctor.
In MESA, risk overestimation was similar for women (100%) and men (93%) as was discrimination: c-statistic = 0.74 for women and 0.71 for men (Table 2). That is, observed rates were roughly half of that predicted by the risk score. Overestimation was observed in all race/ethnicity groups in the MESA cohort and was highest among Chinese (252% for women and 314% for men), and lowest in White women (72%) and Hispanic men (67%). A sensitivity analysis limited to MESA participants that met criteria consistent with guideline recommendations for the use of the AHA-ACC-ASCVD risk score to guide statin therapy (not taking lipid-lowering medication at baseline with an LDL-C between 70 and 189 mg/dL) produced similar overestimation as seen in the primary analysis with the exception that the lowest discordance between observed and calculated ASCVD event rates was seen in Hispanic men (71%) and women (49%) (Table 3). Sensitivity analyses to account for the possibility of missed events by including 20 un-adjudicated MI events identified in the CMS database were consistent with overall study results (Table 4 and Supplementary material online, Table S2). In Cox models that included all ASCVD risk factors present in the AHA-ACC-ASCVD risk score, the hazard ratios for ASCVD events in Black men compared with White men and the hazard ratios for Black women compared with White women were not significant in MESA (P = 0.285 and 0.350, Supplementary material online, Table S3). The lack of statistical significance indicates that race differences with respect to impact of ASCVD risk factors on event rates were undetectable in MESA. While this finding may be limited by sample size, it is clear that the effect size, if any exists, is not large enough to be detected in the MESA cohort. Not surprisingly, modelling the interaction between race and each of the risk factors also did not suggest significant differences between White and Black individuals (Supplementary material online, Table S3).
Table 2.
Atherosclerotic cardiovascular disease events predicted via the American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease risk score and atherosclerotic cardiovascular disease events observed among Multi-Ethnic Study of Atherosclerosis participants aged 45–79, stratified by gender and race/ethnicity
| Risk score | n | Predicted events, n (%) | Observed events, n (%) | Absolute difference | Discordance (%)a | c-statistic |
|---|---|---|---|---|---|---|
| Women | 3388 | 298.1 (8.80) | 149 (4.40) | 4.40 | 100 | 0.74 |
| White | 1286 | 97.9 (7.61) | 57 (4.43) | 3.18 | 72 | 0.70 |
| Black | 979 | 108.4 (11.07) | 50 (5.11) | 5.96 | 117 | 0.75 |
| Chinese | 392 | 31.7 (8.07) | 9 (2.30) | 5.78 | 252 | 0.83 |
| Hispanic | 731 | 60.2 (8.24) | 33 (4.51) | 3.72 | 82 | 0.79 |
| Men | 3053 | 425.0 (13.92) | 220 (7.21) | 6.71 | 93 | 0.71 |
| White | 1194 | 161.1 (13.49) | 85 (7.12) | 6.37 | 90 | 0.71 |
| Black | 798 | 117.2 (14.69) | 65 (8.15) | 6.54 | 80 | 0.68 |
| Chinese | 371 | 49.7 (13.38) | 12 (3.23) | 10.15 | 314 | 0.63 |
| Hispanic | 690 | 97.0 (14.06) | 58 (8.41) | 5.66 | 67 | 0.75 |
aDiscordance is defined as ([{expected percentage − observed percentage}/observed percentage] × 100).
Table 3.
Atherosclerotic cardiovascular disease events predicted via the American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease risk score and atherosclerotic cardiovascular disease events observed among Multi-Ethnic Study of Atherosclerosis participants aged 45–79, not taking lipid-lowering medication at baseline with low-density lipoprotein cholesterol between 70 and 189 mg/dL stratified by gender and race/ethnicity
| Risk score | n | Predicted events, n (%) | Observed events, n (%) | Absolute difference | Discordance (%)a | c-statistic |
|---|---|---|---|---|---|---|
| Women | 2629 | 211.3 (8.04) | 113 (4.30) | 3.74 | 87 | 0.75 |
| White | 1003 | 69.7 (6.94) | 40 (3.99) | 2.96 | 74 | 0.70 |
| Black | 744 | 77.5 (10.41) | 38 (5.11) | 5.30 | 104 | 0.75 |
| Chinese | 308 | 22.6 (7.34) | 7 (2.27) | 5.07 | 223 | b |
| Hispanic | 574 | 41.6 (7.24) | 28 (4.88) | 2.36 | 49 | 0.81 |
| Men | 2387 | 317.1 (13.28) | 164 (6.87) | 6.41 | 93 | 0.71 |
| White | 900 | 118.3 (13.14) | 68 (7.56) | 5.59 | 74 | 0.71 |
| Black | 614 | 86.2 (14.05) | 44 (7.17) | 6.88 | 96 | 0.70 |
| Chinese | 315 | 40.7 (12.91) | 10 (3.17) | 9.73 | 307 | b |
| Hispanic | 558 | 71.9 (12.89) | 42 (7.53) | 5.36 | 71 | 0.74 |
aDiscordance is defined as ([{expected percentage − observed percentage}/observed percentage] × 100).
bToo few events for this statistic.
Table 4.
Atherosclerotic cardiovascular disease events predicted via the American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease risk score and atherosclerotic cardiovascular disease events observed among Multi-Ethnic Study of Atherosclerosis participants aged 45–79, not taking lipid-lowering medication at baseline with low-density lipoprotein cholesterol between 70 and 189 mg/dL stratified by gender and race/ethnicity, including non-adjudicated Centers for Medicare & Medicaid Services myocardial infarctions
| Risk score | n | Predicted events, n (%) | Observed events, n (%) | Absolute difference | Discordance (%)a | c-statistic |
|---|---|---|---|---|---|---|
| Women | 2629 | 211.3 (8.04) | 118 (4.49) | 3.55 | 79 | 0.77 |
| White | 1003 | 69.7 (6.94) | 40 (3.99) | 2.96 | 74 | 0.70 |
| Black | 744 | 77.5 (10.41) | 39 (5.24) | 5.17 | 99 | 0.77 |
| Chinese | 308 | 22.6 (7.34) | 8 (2.60) | 4.74 | 183 | b |
| Hispanic | 574 | 41.6 (7.24) | 31 (5.40) | 1.84 | 34 | 0.84 |
| Men | 2387 | 317.1 (13.28) | 174 (7.29) | 6.00 | 82 | 0.71 |
| White | 900 | 118.3 (13.14) | 72 (8.00) | 5.14 | 64 | 0.72 |
| Black | 614 | 86.2 (14.05) | 45 (7.33) | 6.72 | 92 | 0.70 |
| Chinese | 315 | 40.7 (12.91) | 12 (3.81) | 9.10 | 239 | b |
| Hispanic | 558 | 71.9 (12.89) | 45 (8.06) | 4.83 | 60 | 0.74 |
aDiscordance is defined as ([{expected percentage − observed percentage}/observed percentage] × 100).
bToo few events for this statistic.
Estimating the AHA-ACC-ASCVD Cox model in MESA yields very different coefficients for the same constituent risk factor variables (Supplementary material online, Tables S4 and S5). This is in contrast to the more similar coefficients for estimating the Reynolds or NCEP-ATPIII risk scores via Cox models in MESA (Supplementary material online, Table S6). These findings suggest that the relationship of risk factors used in the AHA-ACC-ASCVD risk score has a different relationship to ASCVD events in the AHA-ACC-ASCVD derivation cohort than in MESA. Modelling of the AHA-ACC-ASCVD risk score in MESA demonstrates a mean absolute risk overestimation of 5.5% (P < 0.001) (Table 5). Importantly, this corrects to a non-significant 0.34% reference group underestimation (P = 0.87) in a multivariable-adjusted model. The magnitude of model overestimation is not directly comparable; however, the fact that the multivariable model estimate (0.34%) is not statistically significantly different from zero indicates that the risk factors included in this model account for the overestimation observed with the AHA-ACC-ASCVD risk score in MESA. In this analysis, male gender, higher age, Black and Chinese race/ethnicity (when compared with White), systolic blood pressure (treated and untreated), diabetes, exercise, anti-hypertensive, lipid-lowering therapy, and aspirin therapy use were all associated with risk overestimation in the single-variable analysis (Figures 1–3 and Table 5). Significance was lost for male gender (P = 0.051), Black race/ethnicity, and anti-hypertensive therapy and gained for alcohol use in the multivariable-adjusted model (Table 5). For example, after holding all other factors in our model constant, each decade increase in age was associated with a mean 3.7% greater absolute risk overestimation. Factors examined that were not associated with risk overestimation in the multivariable-adjusted model included male gender, Black and Hispanic race/ethnicity, total cholesterol, HDL cholesterol, smoking status, body mass index, diet, education level, income, anti-hypertensive medication, and interim revascularization. Family history of heart attack was associated with less overestimation in both models. A sensitivity analysis limited to MESA participants that met clinical criteria consistent with guideline recommendations for the use of the AHA-ACC-ASCVD risk score to guide statin therapy was similar to the primary analysis with the exception that alcohol use and exercise were no longer significant predictors of discordance in the multivariable-adjusted model (Supplementary material online, Table S7). Including 20 un-adjudicated CMS MIs as ASCVD events did not meaningfully change the overall results of these analyses (Supplementary material online, Tables S8 and S9).
Table 5.
Absolute risk regression (generalized linear model assuming Gaussian error distribution with robust standard errors) to evaluate the impact of a single risk factor (β-coefficient) while adjusting for all the other risk factors in the model
| Single variable |
Multivariable |
|||||
|---|---|---|---|---|---|---|
| Risk factor | Coef. | (95% CI) | P | Coef. | (95% CI) | P |
| Intercept | 5.5 | (4.92 to 6.07) | <0.001 | −0.34 | (−3.77 to 4.44) | 0.873 |
| Male | 2.31 | (1.15 to 3.48) | <0.001 | 1.28 | (−0.01 to 2.57) | 0.051 |
| Age, per 10 years | 4.9 | (4.29 to 5.5) | <0.001 | 3.73 | (3.03 to 4.43) | <0.001 |
| Race/ethnicity | ||||||
| Black | 1.51 | (0.02 to 2.99) | 0.047 | 1.42 | (−0.18 to 3.02) | 0.082 |
| Chinese | 3.19 | (1.57 to 4.81) | <0.001 | 4.07 | (2.23 to 5.92) | <0.001 |
| Hispanic | −0.05 | (−1.61 to 1.51) | 0.946 | 1.26 | (−0.62 to 3.14) | 0.189 |
| SBP, per 10 mmHg | ||||||
| Untreated | 1.13 | (0.78 to 1.47) | <0.001 | 0.62 | (0.24 to 0.99) | 0.001 |
| Treated | 1.37 | (1.06 to 1.68) | <0.001 | 0.66 | (0.29 to 1.04) | 0.001 |
| Total cholesterol, per 10 mg/dL | −0.07 | (−0.24 to 0.09) | 0.373 | −0.03 | (−0.2 to 0.14) | 0.732 |
| HDL, per 10 mg/dL | −0.14 | (−0.52 to 0.23) | 0.457 | −0.04 | (−0.48 to 0.4) | 0.862 |
| Diabetes | 7.08 | (4.63 to 9.54) | <0.001 | 4.21 | (1.72 to 6.69) | 0.001 |
| Current smoker | −1.96 | (−3.95 to 0.04) | 0.054 | 0.24 | (−1.75 to 2.23) | 0.814 |
| Alcohol user | 0.23 | (−0.95 to 1.4) | 0.705 | 1.46 | (0.20 to 2.72) | 0.023 |
| Body mass index | −0.08 | (−0.18 to 0.02) | 0.14 | −0.09 | (−0.21 to 0.04) | 0.167 |
| Unhealthy diet | −0.17 | (−1.38 to 1.04) | 0.781 | 0.00 | (−1.24 to 1.24) | 0.998 |
| Exercise, per 100 met/min/day | 0.03 | (0.00 to 0.05) | 0.022 | 0.02 | (0.00 to 0.04) | 0.026 |
| Education | ||||||
| <High school | 1.59 | (−0.13 to 3.31) | 0.07 | −0.07 | (−1.91 to 1.76) | 0.939 |
| Bachelor's | 0.54 | (−0.95 to 2.02) | 0.477 | 0.94 | (−0.59 to 2.46) | 0.227 |
| Graduate/Prof. | 0.6 | (−0.89 to 2.1) | 0.429 | 0.62 | (−0.91 to 2.2) | 0.446 |
| Income, per $10k | −0.13 | (−0.29 to 0.02) | 0.1 | 0.12 | (−0.06 to 0.31) | 0.197 |
| Family history | −2.48 | (−3.74 to −1.22) | <0.001 | −2.40 | (−3.66 to −1.13) | <0.001 |
| Treatment | ||||||
| Anti-hypertensive | 5.92 | (4.81 to 7.03) | <0.001 | 0.93 | (−0.53 to 2.39) | 0.213 |
| Lipid-lowering | 5.68 | (4.53 to 6.83) | <0.001 | 3.17 | (1.85 to 4.48) | <0.001 |
| Aspirin | 5.58 | (4.43 to 6.73) | <0.001 | 3.03 | (1.78 to 4.27) | <0.001 |
| Revascularization | 2.9 | (−1.42 to 7.21) | 0.189 | −2.15 | (−6.4 to 2.11) | 0.323 |
Coef, coefficient [absolute percentage (risk score points)]; CI, confidence interval; HDL, high-density lipoprotein cholesterol; SBP, systolic blood pressure.
Positive values represent absolute value that AHA-ACC-ASCVD risk score overestimated risk in this model (n = 6441).
Coefficients represent absolute percentage (risk score points) contributed by each variable to the difference between observed and AHA-ACC-ASCVD predicted ASCVD events. Positive values represent overestimation, whereas negative values represent underestimation. Bold text indicates that P-values were statistically significant at the P < 0.05 level.
Absolute risk model is a linear regression model at the individual level, predicting a dichotomous outcome (ASCVD event yes/no within 10 years). It was fit as a generalized linear model with the AHA-ACC-ASCVD risk score entered as an offset (fixing the coefficient to 1.0), with an identity link function and using robust standard errors. Continuous variables are mean-centred. Missing data for diet (n = 259, 4%), income (n = 222, 3%), exercise (n = 2), and education (n = 1) were handled using multiple imputation with sequential chained regression and 10 imputed datasets; Rubin's rules were used to combine the standard errors.24 Reference category for race/ethnicity is white. Unhealthy diet has been described and validated previously.25 Exercise is total intentional physical exercise. Reference category for education is high school graduate. Treatment was measured during follow-up visits and indicates that participant ever received treatment. In an effort to study revascularizations that may have prevented subsequent events, all revascularizations >5 days prior to an event, or revascularizations not followed by any event, were selected for modelling.
Figure 1.
Observed atherosclerotic cardiovascular disease percentage and American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease risk score predicted percentage. *P-value of <0.05 for proportion test comparing American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease predicted risk (total height of each bar) with the observed risk in Multi-Ethnic Study of Atherosclerosis (grey portion of bars). For example, the American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease predicted risk for men was 13.9%, whereas the observed rate was 7.2%, and a two-tailed test of proportions showed these rates were significantly different at the 95% level. Reported P-value compares the overestimation (i.e. the black bars) for each risk factor. For example, the American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease predicted risk (total height of each bar) was higher than the observed risk (grey portion of the bar) for females and for males, and the overestimation for males was significantly greater (P < 0.001) than the overestimation for females. P-values come from an absolute risk regression model, which is a linear regression model at the individual level, predicting a dichotomous outcome (atherosclerotic cardiovascular disease event yes/no within 10 years). It was fit as a generalized linear model with the American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease risk score entered as an offset (fixing the coefficient to 1.0), with an identity link function and using robust standard errors. To this model, we added each risk factor separately. P-value for race/ethnicity categories shows joint F-test of significance.
Figure 2.
Observed atherosclerotic cardiovascular disease percentage and American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease risk score predicted percentage among factors not included in the risk score. *P-value of <0.05 for proportion test comparing American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease predicted risk (total height of each bar) with the observed risk in Multi-Ethnic Study of Atherosclerosis (grey portion of bars). Reported P-value compares the overestimation (i.e. the black bars) for each risk factor. P-values come from an absolute risk regression model, which is a linear regression model at the individual level, predicting a dichotomous outcome (atherosclerotic cardiovascular disease event yes/no within 10 years). It was fit as a generalized linear model with the American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease risk score entered as an offset (fixing the coefficient to 1.0), with an identity link function and using robust standard errors. To this model, we added each risk factor separately. P-value for education categories variables shows joint F-test of significance.
Figure 3.
Observed atherosclerotic cardiovascular disease percentage and American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease risk score predicted percentage by time-updated treatments. *P-value of <0.05 for proportion test comparing American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease predicted risk (total height of each bar) with the observed risk in Multi-Ethnic Study of Atherosclerosis (grey portion of bars). Reported P-value compares the overestimation (i.e. the black bars) for each risk factor. P-values come from an absolute risk regression model, which is a linear regression model at the individual level, predicting a dichotomous outcome (atherosclerotic cardiovascular disease event yes/no within 10 years). It was fit as a generalized linear model with the American Heart Association-American College of Cardiology-Atherosclerotic Cardiovascular Disease risk score entered as an offset (fixing the coefficient to 1.0), with an identity link function and using robust standard errors. To this model, we added each risk factor separately.
Discussion
The MESA cohort appears to be a valid representation of a modern day US-based primary prevention population as demonstrated by similar baseline mean risk factor measures, the distribution of subjects in four clinically relevant risk categories, and by similar treatment rates when compared with a representative sample of ASCVD-free, non-pregnant US adults (NHANES). The AHA-ACC-ASCVD risk score overestimates risk in men, women, and all four race/ethnic groups studied in MESA—including White and Black Americans for whom the risk score is specifically recommended. Consistent with the age-matched peers considered representative of the general American primary prevention population, ∼20% of MESA subjects were on lipid-lowering therapy at baseline. However, results were essentially unchanged in a sensitivity analysis that included only MESA participants that met clinical criteria consistent with guideline recommendations for the use of the AHA-ACC-ASCVD risk score to guide statin therapy (not taking lipid-lowering medication at baseline and having an LDL-C between 70 and 189 mg/dL). This and previous data from MESA and other cohorts demonstrate that a large proportion of risk overestimation with the AHA-ACC-ASCVD risk score is not explained by the use of lipid-lowering therapy.5,9,20 In fact, a sensitivity analysis excluding MESA subjects treated with aspirin, lipid-lowering, anti-hypertensive medication at baseline or at any of the five follow-up visits or interim coronary revascularization failed to provide evidence that these treatments explain overestimation by the AHA-ACC-ASCVD risk score.9 Prior work demonstrates overestimation throughout the continuum of low to high risk by the AHA-ACC-ASCVD score and specifically in those individuals with a calculated AHA-ACC-ASCVD risk of 7.5–10% (just over the guideline-recommended risk threshold of 7.5% for consideration of stain therapy).9
The events collection and adjudication process for MESA is robust and included successfully obtaining medical records for ∼98% of reported hospitalized CHD and CVD events and information on 95% of reported outpatient cardiovascular diagnostic encounters. Follow-up telephone interviews were completed in 92% of living participants. In comparison, in the Framingham study, one of the studies used to derive the AHA-ACC-ASCVD risk score, events were captured using a combination of self-reporting, medical record review, contact with patients' personal physicians, and examinations in the Framingham Clinic.21 Despite the rigour of event capture employed in MESA, an evaluation of the CMS database identified 20 MIs not adjudicated as events in MESA. Given the methodology for capturing clinical events in MESA appears to be similar to that of the studies used to derive the AHA-ACC-ASCVD risk score, it is unlikely that MESA has missed significantly more events than that missed in the AHA-ACC-ASCVD risk score derivation cohort. Given the small number of events missed, it is unlikely that missed events explain a substantial fraction of risk overestimation. This conclusion is supported by our sensitivity analyses, which includes the addition of 20 MIs identified in the CMS database, but not adjudicated as ASCVD events in MESA, which did not significantly change the results of this study.
We demonstrated that the relationships between risk factors used in the AHA-ACC-ASCVD risk score and ASCVD events are different in the AHA-ACC-ASCVD derivation cohort than in MESA by demonstrating significantly different coefficients when estimating the AHA-ACC-ASCVD Cox model in MESA. Additional evidence of poor risk factor calibration is evident in our generalized linear modelling that implicated several factors in the AHA-ACC-ASCVD risk score: older age, systolic blood pressure (treated and untreated), and diabetes status, as contributors to overestimation in the MESA cohort. Our modelling also identified other factors not currently part of the AHA-ACC-ASCVD risk score that were associated with AHA-ACC-ASCVD risk score overestimation in MESA: Chinese race/ethnicity, alcohol use, exercise, aspirin therapy, and lipid-lowering medication. Presence of a family history of heart attack was associated with less overestimation of risk. These data are supportive of the notion that changing significance of risk factors from older cohorts, used to develop the AHA-ACC-ASCVD risk score, to the more modern MESA cohort may explain the poor performance of the AHA-ACC-ASCVD risk score in MESA.9 For example, the treatment of diabetes has changed dramatically over the decades since the creation of the pooled cohorts used to develop the AHA-ACC-ASCVD risk score. The implications of being 70 years old on cardiovascular risk is likely different now than it was decades ago.22 Treatment with anti-hypertensive, lipid-lowering, or aspirin therapy was associated with risk overestimation in our unadjusted model. Although attenuated, lipid-lowering and aspirin therapies remained associated with risk overestimation in our multivariable model, whereas anti-hypertensive therapy, the only therapy considered in the AHA-ACC-ASCVD risk equation, was no longer predictive of risk overestimation. These findings suggest that the inclusion of additional treatment variables, such as lipid-lowering and/or aspirin therapy, may improve risk prediction and therefore should be considered in future risk score development.
More than half of the growth in the total population of the USA between 2000 and 2010 was due to the increase in the Hispanic population, and the number of Asian Americans has increased by over 40% in this same time interval.23 The need for an accurate risk assessment tool in these growing American populations is clear. We found no evidence to suggest that risk factors affect ASCVD risk differently in Black vs. White individuals and calibration of the AHA-ACC-ASCVD risk score is poor in all races examined in MESA. The need for better risk prediction among all races is clear, but the need for individual race-based risk assessment algorithms is not clear from our analysis in MESA and therefore requires further investigation.
In conclusion, MESA appears representative of a modern day US-based primary prevention population. The AHA-ACC-ASCVD risk score overestimation is observed in men, women, and the four race/ethnic groups evaluated in MESA—including White and Black men and women for whom the new risk score is specifically recommended. Clinicians may need to interpret AHA-ACC-ASCVD risk estimates with caution, especially among older individuals, Chinese Americans, and those with hypertension or diabetes. Risk prediction is an evolving science and will require continual updating through the study of well-characterized, contemporary primary prevention cohorts.
Supplementary material
Supplementary material is available at European Heart Journal online.
Authors' contributions
R.Y. performed statistical analysis. A.P.D., R.A.K., R.L.M., and M.J.B. handled funding and supervision. A.P.D., R.Y., J.W.M., E.D.M., V.S., R.A.K., R.L.M., and M.J.B. conceived and designed the research. A.P.D., R.Y., and M.J.B. drafted the manuscript. A.P.D., R.Y., J.W.M., E.D.M., V.S., R.A.K., R.L.M., and M.J.B. made critical revision of the manuscript for key intellectual content.
Funding
This research was supported by contracts N01-HC-95159, N01-HC-95160, N01-HC-95161, N01-HC-95162, N01-HC-95163, N01-HC-95164, N01-HC-95165, N01-HC-95166, N01-HC-95167, N01-HC-95168, and N01-HC-95169 from the National Heart, Lung, and Blood Institute and by grants UL1-TR-000040 and UL1-TR-001079 from NCRR.
Conflict of interest: none declared.
Supplementary Material
Acknowledgements
The authors thank the other investigators, the staff, and the participants of the MESA study for their valuable contributions. A full list of participating MESA investigators and institutions can be found at http://www.mesa-nhlbi.org. A.P.D., R.Y., and M.J.B. had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
References
- 1. Goff DC Jr, Lloyd-Jones DM, Bennett G, Coady S, D'Agostino RB, Gibbons R, Greenland P, Lackland DT, Levy D, O'Donnell CJ, Robinson JG, Schwartz JS, Shero ST, Smith SC Jr, Sorlie P, Stone NJ, Wilson PW, Jordan HS, Nevo L, Wnek J, Anderson JL, Halperin JL, Albert NM, Bozkurt B, Brindis RG, Curtis LH, DeMets D, Hochman JS, Kovacs RJ, Ohman EM, Pressler SJ, Sellke FW, Shen WK, Smith SC Jr, Tomaselli GF. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 2014;129(Suppl. 2):S49–S73. [DOI] [PubMed] [Google Scholar]
- 2. Fried LP, Borhani NO, Enright P, Furberg CD, Gardin JM, Kronmal RA, Kuller LH, Manolio TA, Mittelmark MB, Newman A, O'Leary DH, Psaty B, Rautaharju P, Tracy RP, Weiler PG. The Cardiovascular Health Study: design and rationale. Ann Epidemiol 1991;1:263–276. [DOI] [PubMed] [Google Scholar]
- 3. Friedman GD, Cutter GR, Donahue RP, Hughes GH, Hulley SB, Jacobs DR Jr, Liu K, Savage PJ. CARDIA: study design, recruitment, and some characteristics of the examined subjects. J Clin Epidemiol 1988;41:1105–1116. [DOI] [PubMed] [Google Scholar]
- 4. Investigators TA. The Atherosclerosis Risk in Communities (ARIC) Study: design and objectives. Am J Epidemiol 1989;129:687–702. [PubMed] [Google Scholar]
- 5. Cook NR, Ridker PM. Further insight into the cardiovascular risk calculator: the roles of statins, revascularizations, and underascertainment in the Women's Health Study. JAMA Intern Med 2014;174:1964–1971. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Kavousi M, Leening MJ, Nanchen D, Greenland P, Graham IM, Steyerberg EW, Ikram MA, Stricker BH, Hofman A, Franco OH. Comparison of application of the ACC/AHA guidelines, Adult Treatment Panel III guidelines, and European Society of Cardiology guidelines for cardiovascular disease prevention in a European cohort. JAMA 2014;311:1416–1423. [DOI] [PubMed] [Google Scholar]
- 7. Ridker PM, Cook NR. Statins: new American guidelines for prevention of cardiovascular disease. Lancet 2013;382:1762–1765. [DOI] [PubMed] [Google Scholar]
- 8. Muntner P, Colantonio LD, Cushman M, Goff DC Jr, Howard G, Howard VJ, Kissela B, Levitan EB, Lloyd-Jones DM, Safford MM. Validation of the atherosclerotic cardiovascular disease pooled cohort risk equations. JAMA 2014;311:1406–1415. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. DeFilippis AP, Young R, Carrubba CJ, McEvoy JW, Budoff MJ, Blumenthal RS, Kronmal RA, McClelland RL, Nasir K, Blaha MJ. An analysis of calibration and discrimination among multiple cardiovascular risk scores in a modern multiethnic cohort. Ann Intern Med 2015;162:266–275. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Bild DE, Bluemke DA, Burke GL, Detrano R, Diez Roux AV, Folsom AR, Greenland P, Jacob DR Jr, Kronmal R, Liu K, Nelson JC, O'Leary D, Saad MF, Shea S, Szklo M, Tracy RP. Multi-ethnic study of atherosclerosis: objectives and design. Am J Epidemiol 2002;156:871–881. [DOI] [PubMed] [Google Scholar]
- 11. Centers for Disease Control and Prevention, National Center for Health Statistics, Department of Health and Human Services. National Health and Nutrition Examination Survey Data [2003–2014] http://wwwncdcgov/nchs/nhanes/search/nhanes13_14aspx.
- 12. Ong KL, Cheung BM, Man YB, Lau CP, Lam KS. Prevalence, awareness, treatment, and control of hypertension among United States adults 1999–2004. Hypertension 2007;49:69–75. [DOI] [PubMed] [Google Scholar]
- 13. Qiuping GP-RR, Burt VL, Kit BK. Prescription cholesterol-lowering medication use in adults aged 40 and over: United States, 2003–2012. National Center for Health Statistics Data Brief no 177. Hyattsville, MD: US Department of Health and Human Services; 2014 www.cdc.gov/nchs/data/databriefs/db177.htm#ref10 . [PubMed]
- 14. Ridker PM, Buring JE, Rifai N, Cook NR. Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds Risk Score. JAMA 2007;297:611–619. [DOI] [PubMed] [Google Scholar]
- 15. Ridker PM, Paynter NP, Rifai N, Gaziano JM, Cook NR. C-reactive protein and parental history improve global cardiovascular risk prediction: the Reynolds risk score for men. Circulation 2008;118:2243–2251. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Executive summary of the third report of the National Cholesterol Education Program (NCEP) expert panel on detection, evaluation, and treatment of high blood cholesterol in adults (Adult Treatment Panel III). JAMA 2001;285:2486–2497. [DOI] [PubMed] [Google Scholar]
- 17. Hellevik O. Linear versus logistic regression when the dependent variable is a dichotomy. Qual Quant 2009;43:59–74. [Google Scholar]
- 18. Lumley T, Kronmal R, Ma S. Relative risk regression in medical research: models, contrasts, estimators and algorithms. University of Washington Biostatistics; Working Paper Series 2006; paper 293. [Google Scholar]
- 19. Xu Y, Cheung YB, Lam KF, Tan SH, Milligan P. A simple approach to the estimation of incidence rate difference. Am J Epidemiol 2010;172:334–343. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Rana J, Tabada G, Solomon M, Lo J, Jaffe M, Sung S, Ballantyne C, Go A. Accuracy of the atherosclerotic cardiovascular risk equation in a large contemporary, multiethnic population. J Am Coll Cardiol 2016;67:2118–2130. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. D'Agostino RB Sr, Vasan RS, Pencina MJ, Wolf PA, Cobain M, Massaro JM, Kannel WB. General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation 2008;117:743–753. [DOI] [PubMed] [Google Scholar]
- 22. Roth GA, Forouzanfar MH, Moran AE, Barber R, Nguyen G, Feigin VL, Naghavi M, Mensah GA, Murray CJ. Demographic and epidemiologic drivers of global cardiovascular mortality. N Engl J Med 2015;372:1333–1341. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Humes KR, Jones NA, Ramirez RR. Overview of race and Hispanic origin: 2010. US Census Bureau 2011; 2010 Census Briefs http://www.census.gov/prod/cen2010/briefs/c2010br-02.pdf (1 July 2015).
- 24. White IR, Royston P, Wood AM. Multiple imputation using chained equations: issues and guidance for practice. Stat Med 2011;30:377–399. [DOI] [PubMed] [Google Scholar]
- 25. Trichopoulou A, Costacou T, Bamia C, Trichopoulos D. Adherence to a Mediterranean diet and survival in a Greek population. N Engl J Med 2003;348:2599–2608. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.