Abstract
Background
The accuracy of the 2013 American College of Cardiology/American Heart Association (ACC/AHA) risk equation for atherosclerotic cardiovascular disease (ASCVD) events in contemporary and ethnically diverse populations is not well understood.
Objectives
We sought to evaluate the accuracy of the 2013 ACC/AHA risk equation within a large, multiethnic population in clinical care.
Methods
The target population for consideration of cholesterol-lowering therapy in a large, integrated health care delivery system population was identified in 2008 and followed through 2013. The main analyses excluded those with known ASCVD, diabetes mellitus, low-density lipoprotein cholesterol levels <70 or ≥190 mg/dl, prior statin use, or incomplete 5-year follow-up. Patient characteristics were obtained from electronic medical records and ASCVD events were ascertained using validated algorithms for hospitalization databases and death certificates. We compared predicted versus observed 5-year ASCVD risk, overall and by sex and race/ethnicity. We additionally examined predicted versus observed risk in patients with diabetes mellitus.
Results
Among 307,591 eligible adults without diabetes between 40 and 75 years of age, 22,283 were black, 52,917 Asian/Pacific Islander, and 18,745 Hispanic. We observed 2,061 ASCVD events during 1,515,142 person-years. In each 5-year predicted ASCVD risk category, observed 5-year ASCVD risk was substantially lower: 0.20% for predicted risk <2.50%; 0.65% for predicted risk 2.50 to 3.74%; 0.90% for predicted risk 3.75 to 4.99%; and 1.85% for predicted risk ≥5.00%, with C: 0.74. Similar ASCVD risk overestimation and poor calibration with moderate discrimination (C: 0.68 to 0.74) was observed in sex, racial/ethnic, and socioeconomic status subgroups, and in sensitivity analyses among patients receiving statins for primary prevention. Calibration among 4,242 eligible adults with diabetes was improved, but discrimination was worse (C: 0.64).
Conclusions
In a large, contemporary “real-world” population, the ACC/AHA Pooled Cohort risk equation substantially overestimated actual 5-year risk in adults without diabetes, overall and across sociodemographic subgroups.
Keywords: cardiovascular, diabetes mellitus, prediction, race/ethnicity, risk factor
Introduction
Publication of the 2013 American College of Cardiology (ACC)/American Heart Association (AHA) Pooled Cohort risk equation for estimating atherosclerotic cardiovascular disease ASCVD risk is considered an important step forward, as it estimates risk for both heart disease and stroke, and provides estimates applicable to black/African Americans (1,2).
This equation was developed from several prospective U.S.-based cohorts of enrolled volunteers, primarily conducted in the 1990s, with limited ethnic diversity and age range. Predicted ASCVD risk using the equation was reported to be systematically higher than observed risk in some highly selected cohorts (3,4), but not in others (5). Importantly, however, these studies included persons screened for participation or enrolled in clinical trials, or in much earlier time periods, with limited generalizability to contemporary and diverse populations that are more representative of eligible patients treated in typical clinical practice (6,7). Validation efforts in more contemporary cohorts have further been limited due to inclusion of analyzed participants treated with statin therapy or having a high likelihood of initiating statins during follow-up (4,6,8). Lack of comprehensive surveillance in some studies is another possible reason for overestimation by the Pooled Cohort risk equation due to underascertainment of ASCVD events (3,9).
Another major knowledge gap is the lack of accurate ASCVD risk estimation specific to persons of Asian/Pacific Islander and Hispanic ethnicities, who are currently combined with the white population in the Pooled Cohort risk equation (10). Furthermore, patterns of risk in more contemporary, community-based populations may be significantly different from the older cohorts used to derive the Pooled Cohort risk equation, which does not reflect the recent treatment era and risk factor levels (11,12). Estimates suggest that nearly half of U.S. adults and up to 65% of European adults would be eligible for statins on the basis of new ACC/AHA cholesterol guidelines using the Pooled Cohort risk equation (13,14). Thus, evaluating the accuracy of ASCVD risk prediction is essential if it is being used as a decision-making tool to determine which persons should receive statin therapy for primary prevention (13,15), and it could have far-reaching impact if applied at a population level.
To address these knowledge gaps, we examined a very large, contemporary, multiethnic, community-based “real-world” population whose clinical characteristics would typically trigger a discussion of initiation of cholesterol-lowering therapy per risk assessment by the Pooled Cohort risk equation (i.e., patients 40 to 75 years of age, without known ASCVD or diabetes, who have low-density lipoprotein cholesterol [LDL-C] levels between 70 and 189 mg/dl) and compared predicted versus observed 5-year risks of ASCVD events, overall and within sex and multiple ethnic subgroups, along with measures of calibration and discrimination. We also repeated the analyses among a cohort of eligible adults with diabetes mellitus.
Methods
Source population
Kaiser Permanente Northern California (KPNC) is a large integrated healthcare delivery system providing comprehensive inpatient and outpatient care to ∼4 million ethnically diverse persons who are highly representative of the local and statewide population (16).
Study sample
Using outpatient laboratory databases, we initially identified all members of KPNC between January 1 and December 31, 2008 who were ≥21 years of age and had LDL-C levels from 70 to 189 mg/dl within 5 years before study entry. We used the earliest LDL-C measurement in the year 2008 as the index date or, if their last available LDL-C was prior to 2008, January 1, 2008 was used as the index date. We excluded those who met any of the following criteria: unknown sex or race/ethnicity; prescribed statins or other lipid-lowering therapies within 5 years before the index date; prior hospitalization for acute myocardial infarction, ischemic stroke, or receipt of coronary artery bypass surgery or percutaneous coronary intervention, identified from health plan hospital discharge or billing claims databases using previously validated methods (17,18); <12 months of continuous membership and pharmacy benefit before the index date (to ensure more complete information on clinical characteristics); and <5 years of complete follow-up, except if due to death. On the basis of the requirements of the ASCVD risk calculator, we also excluded patients with missing systolic blood pressure, total cholesterol, or high-density lipoprotein cholesterol (HDL-C) information. We further excluded patients who received statins during follow-up if used for primary prevention of ASCVD (i.e., statin initiated before a documented ASCVD event) on the basis of filled dispensings found in pharmacy databases. The cohort was then stratified by diabetes mellitus status at index date on the basis of data from a validated regional diabetes registry (20,21).
Prediction of ASCVD risk
We calculated the 10-year ASCVD predicted risk score for cohort members between 40 and 75 years of age, as per the guideline-recommended ASCVD prevention age group (1,13). To directly compare observed versus predicted ASCVD risk at 5-year follow-up, we annualized the 10-year predicted ASCVD risk and categorized the population into 4 groups of estimated 5-year risk: <2.50%; from 2.50% to <3.75%; from 3.75% to <5.00%; and 5.00% or higher, as was previously done (8).
Ascertainment of ASCVD events
Follow-up occurred through December 31, 2013. ASCVD events were defined as acute myocardial infarction, coronary heart disease (CHD) death, or fatal or nonfatal ischemic stroke. Myocardial infarction and ischemic stroke were identified from comprehensive health plan hospital discharge or billing claims databases on the basis of previously validated algorithms (17-19) using International Classification of Diseases, Ninth Edition (ICD-9) primary discharge codes. Coronary death was on the basis of International Classification of Diseases, Tenth Edition (ICD-10) codes (I11, I20–I25) (22) obtained from state death certificate files through 2013, which was the latest available information at the time of analysis.
Covariates
We characterized comorbidity during a 5-year period before the index date on the basis of relevant ICD-9 diagnosis and procedure codes (19,23) from health plan electronic medical records and a regional diabetes registry (20,21). We collected baseline data on age, sex, self-reported race/ethnicity, cigarette smoking status, and body mass index (BMI) from electronic medical records. We ascertained ambulatory measures of baseline total cholesterol, HDL-C, LDL-C, systolic blood pressure, and use of antihypertensive medication within 5 years before the index date from comprehensive health plan laboratory, outpatient visit and pharmacy databases.
Analytic approach
Analyses were conducted using SAS, version 9.3 (Cary, North Carolina). We compared baseline characteristics across predicted ASCVD risk categories using ANOVA for continuous variables and chi-squared tests for categorical variables. We calculated the 5-year incidence of ASCVD events with 95% confidence limits using the first hospitalized acute myocardial infarction, first hospitalized ischemic stroke, or coronary death observed during follow-up. The 5-year expected incidence of ASCVD risk was on the basis of the mean ASCVD predicted risk and associated 95% confidence interval (CI) among patients whose predicted risk fell within each risk category. We then examined the observed and expected 5-year incidence of ASCVD events among nondiabetic patients between 40 and 75 years of age, overall and stratified by sex, race/ethnicity (non-Hispanic white, non-Hispanic black, Asian/Pacific Islander, and Hispanic) and measures of socioeconomic status using U.S. Census-based residential block-level data (18). We also separately examined the observed and expected 5-year ASCVD incidence among eligible diabetic patients between 40 and 75 years of age.
Accuracy was assessed by evaluating calibration and discrimination. Following the approach used by Cook and Ridker (11), calibration plots were generated comparing observed versus expected risk of ASCVD events within each estimated risk categories. Discrimination was assessed using the C statistic from a logistic regression model evaluating predicted risk categories with observed 5-year ASCVD event risks. Finally, as a sensitivity analysis, we examined the accuracy of the risk equation among eligible adults with and without diabetes who received statin therapy for primary prevention during the follow-up period.
Results
Study cohort and baseline characteristics
We identified 307,591 eligible adults aged 40 to 75 years who did not have diabetes, prior ASCVD, prior receipt of lipid-lowering therapy, and who had complete 5-year follow-up (Figure 1). In addition, we separately identified 4,242 diabetic patients who did not have known ASCVD, prior lipid-lowering therapy, or any other exclusion criteria (Figure 1).
Figure 1. Cohort Assembly.
Between January 1, 2008 and December 31, 2008, a cohort of eligible adults, 40 to 75 years of age, with or without diabetes mellitus was identified. ASCVD = atherosclerotic cardiovascular disease; HDL = high-density lipoprotein cholesterol; LDL = low-density lipoprotein
Among eligible adults without diabetes, mean age was 54.8 years and 61.6% were women, with 22,283 being black/African American, 52,917 being Asian/Pacific Islander, and 18,745 being Hispanic. Nearly one-third were former or current smokers, 28% were obese and 33% were on medication for hypertension (Table 1).
Table 1. Baseline Characteristics of the Adults 40-75 Years of Age With and Without Diabetes Mellitus, Stratified by Predicted ASCVD Risk Category.
| No Diabetes Mellitus | 10-year Predicted Risk of ASCVD Events | |||||
|---|---|---|---|---|---|---|
| Characteristics | Overall N = 307,591 | <5.00% N = 175,605 | 5.00% to <7.50% N = 36,896 | 7.50% to <10% N = 25,142 | ≥10% N = 69,948 | p Value |
| 10-year predicted risk of ASCVD | ||||||
| Mean ± SD | 6.60 ± 7.02 | 2.08 ± 1.33 | 6.16 ± 0.72 | 8.68 ± 0.72 | 17.43 ± 6.42 | <0.001 |
| Median (IQR) | 3.94 (1.56-9.22) | 1.81 (0.92-3.08) | 6.12 (5.54-6.76) | 8.64 (8.05-9.29) | 15.70 (12.37-20.88) | <0.001 |
| Range | 0.04-65.64 | 0.04-5.00 | 5.00-7.50 | 7.50-10.00 | 10.00-65.64 | |
| Age, yrs, mean ± SD | 54.8 ± 9.1 | 49.8 ± 6.1 | 56.1 ± 6.8 | 58.6 ± 7.0 | 65.1 ± 7.2 | <0.001 |
| Women, n (%) | 189,511 (61.6) | 134,990 (76.9) | 18,745 (50.8) | 11,207 (44.6) | 24,569 (35.1) | <0.001 |
| Race/ethnicity, n (%) | <0.001 | |||||
| White/European (non-Hispanic) | 212,039 (68.9) | 115,652 (65.9) | 26,303 (71.3) | 18,067 (71.9) | 52,017 (74.4) | |
| Black/African American (non-Hispanic) | 22,283 (7.2) | 10,164 (5.8) | 3,414 (9.3) | 2,427 (9.7) | 6,278 (9.0) | |
| Asian/Pacific Islander | 52,917 (17.2) | 36,831 (21.0) | 5,099 (13.8) | 3,296 (13.1) | 7,691 (11.0) | |
| Hispanic | 18,745 (6.1) | 12,035 (6.9) | 1,875 (5.1) | 1,222 (4.9) | 3,613 (5.2) | |
| Other/unknown | 1,607 (0.5) | 923 (0.5) | 205 (0.6) | 130 (0.5) | 349 (0.5) | |
| Residential block-level socioeconomic status | ||||||
| Median household income | <0.001 | |||||
| ≥$35,000 | 295,497 (96.1) | 169,441 (96.5) | 35,360 (95.8) | 23,984 (95.4) | 66,712 (95.4) | |
| <$35,000 | 11,772 (3.8) | 6,004 (3.4) | 1,493 (4.0) | 1,118 (4.4) | 3,157 (4.5) | |
| Missing | 322 (0.1) | 160 (0.1) | 43 (0.1) | 40 (0.2) | 79 (0.1) | |
| Highest level of education | <0.001 | |||||
| High school graduate or higher | 275,867 (89.7) | 157,925 (89.9) | 33,073 (89.6) | 22,461 (89.3) | 62,408 (89.2) | |
| <High school | 31,409 (10.2) | 17,524 (10.0) | 3781 (10.2) | 2,641 (10.5) | 7,463 (10.7) | |
| Missing | 315 (0.1) | 156 (0.1) | 42 (0.1) | 40 (0.2) | 77 (0.1) | |
| Tobacco use, n (%) | <0.001 | |||||
| Current | 29,537 (9.6) | 10,022 (5.7) | 5134 (13.9) | 3,929 (15.6) | 10,452 (14.9) | |
| Former | 67,611 (22.0) | 19,316 (11.0) | 9762 (26.5) | 7,964 (31.7) | 30,569 (43.7) | |
| Never | 210,443 (68.4) | 146,267 (83.3) | 22,000 (59.6) | 13,249 (52.7) | 28,927 (41.4) | |
| Systolic blood pressure, mm Hg, mean ± SD | 124.5 ± 15.0 | 120.2 ± 13.8 | 126.7 ± 13.5 | 128.2 ± 13.8 | 132.8 ± 15.1 | <0.001 |
| BMI, kg/m2, n (%) | <0.001 | |||||
| <18.5 | 3,604 (1.2) | 2,249 (1.3) | 337 (0.9) | 235 (0.9) | 783 (1.1) | |
| 18.5–24.9 | 107,422 (34.9) | 70,931 (40.4) | 10,530 (28.5) | 6,978 (27.8) | 18,983 (27.1) | |
| 25.0–29.9 | 109,104 (35.5) | 57,729 (32.9) | 13,773 (37.3) | 9,544 (38.0) | 28,058 (40.1) | |
| 30.0–39.9 | 76,401 (24.8) | 38,655 (22.0) | 10,645 (28.9) | 7,325 (29.1) | 19,776 (28.3) | |
| ≥40 | 8,620 (2.8) | 4,845 (2.8) | 1,272 (3.4) | 846 (3.4) | 1,657 (2.4) | |
| Missing | 2,440 (0.8) | 1,196 (0.7) | 339 (0.9) | 214 (0.9) | 691 (1.0) | |
| Antihypertensive medication use, n (%) | 101,089 (32.9) | 34,085 (19.4) | 13,447 (36.4) | 10,733 (42.7) | 42,824 (61.2) | <0.001 |
| Lipoproteins, mg/dl, mean ± SD | ||||||
| Total | 197.5 ± 29.7 | 194.6 ± 29.0 | 201.7 ± 29.7 | 202.0 ± 30.1 | 201.0 ± 30.5 | <0.001 |
| HDL | 55.6 ± 15.4 | 58.0 ± 15.1 | 53.4 ± 15.1 | 52.8 ± 15.2 | 51.7 ± 15.4 | <0.001 |
| LDL | 118.1 ± 24.4 | 115.2 ± 23.9 | 122.4 ± 24.4 | 122.6 ± 24.7 | 121.7 ± 24.4 | <0.001 |
| Diabetes Mellitus | 10-year Predicted Risk of ASCVD Events | |||||
| Characteristics | Overall N = 4,242 | <5.00% N = 834 | 5.00% to <7.50% N = 369 | 7.50% to <10% N = 327 | ≥10% N = 2,712 | p Value |
| 10-year predicted risk of ASCVD | ||||||
| Mean ± SD | 26.81 ± 26.19 | 2.05 ± 1.39 | 6.19 ± 0.72 | 8.72 ± 0.71 | 40.23 ± 24.33 | <0.001 |
| Median (IQR) | 17.36 (5.12-42.58) | 1.82 (0.82-3.11) | 6.13 (5.56-6.79) | 8.67 (8.12-9.32) | 34.49 (19.24-57.03) | <0.001 |
| Range | 0.00-100.00 | 0.00-5.00 | 5.01-7.49 | 7.51-9.99 | 10.01-100.00 | |
| Age, yrs, mean ± SD | 58.1 ± 9.6 | 47.6 ± 5.1 | 51.7 ± 6.1 | 54.0 ± 6.3 | 62.7 ± 8.0 | <0.001 |
| Women, n (%) | 2127 (50.1) | 663 (79.5) | 229 (62.1) | 209 (63.9) | 1026 (37.8) | <0.001 |
| Race/ethnicity, n (%) | <0.001 | |||||
| White/European (non-Hispanic) | 2,371 (55.9) | 390 (46.8) | 221 (59.9) | 184 (56.3) | 1,576 (58.1) | |
| Black/African American (non-Hispanic) | 544 (12.8) | 55 (6.6) | 32 (8.7) | 36 (11.0) | 421 (15.5) | |
| Asian/Pacific Islander | 768 (18.1) | 253 (30.3) | 77 (20.9) | 62 (19.0) | 376 (13.9) | |
| Hispanic | 529 (12.5) | 127 (15.2) | 37 (10.0) | 42 (12.8) | 323 (11.9) | |
| Other/unknown | 30 (0.7) | 9 (1.1) | 2 (0.5) | 3 (0.9) | 16 (0.6) | |
| Residential block-level socioeconomic status | ||||||
| Median household income | 0.57 | |||||
| ≥$35,000 | 3,971 (93.6) | 791 (94.8) | 349 (94.6) | 306 (93.6) | 2,525 (93.1) | |
| <$35,000 | 268 (6.3) | 42 (5.0) | 20 (5.4) | 21 (6.4) | 185 (6.8) | |
| Missing | 3 (0.1) | 1 (0.1) | 0 (0.0) | 0 (0.0) | 2 (0.1) | |
| Highest level of education | 0.76 | |||||
| High school graduate or higher | 3,534 (83.3) | 705 (84.5) | 311 (84.3) | 275 (84.1) | 2,243 (82.7) | |
| <High school | 706 (16.6) | 128 (15.3) | 58 (15.7) | 52 (15.9) | 468 (17.3) | |
| Missing | 2 (0.0) | 1 (0.1) | 0 (0.0) | 0 (0.0) | 1 (0.0) | |
| Tobacco use, n (%) | <0.001 | |||||
| Current | 481 (11.3) | 29 (3.5) | 31 (8.4) | 46 (14.1) | 375 (13.8) | |
| Former | 1,227 (28.9) | 28 (3.4) | 49 (13.3) | 69 (21.1) | 1,081 (39.9) | |
| Never | 2,534 (59.7) | 777 (93.2) | 289 (78.3) | 212 (64.8) | 1,256 (46.3) | |
| Systolic blood pressure, mm Hg, mean ± SD | 129.5 ± 16.6 | 122.3 ± 13.7 | 126.9 ± 15.5 | 128.5 ± 15.0 | 1,32.3 ± 16.9 | <0.001 |
| BMI, kg/m2, n (%) | <0.001 | |||||
| <18.5 | 50 (1.2) | 12 (1.4) | 2 (0.5) | 2 (0.6) | 34 (1.3) | |
| 18.5–24.9 | 716 (16.9) | 183 (21.9) | 55 (14.9) | 42 (12.8) | 436 (16.1) | |
| 25.0–29.9 | 1,137 (26.8) | 208 (24.9) | 84 (22.8) | 88 (26.9) | 757 (27.9) | |
| 30.0–39.9 | 1,924 (45.4) | 325 (39.0) | 191 (51.8) | 161 (49.2) | 1,247 (46.0) | |
| ≥40 | 389 (9.2) | 102 (12.2) | 36 (9.8) | 32 (9.8) | 219 (8.1) | |
| Missing | 26 (0.6) | 4 (0.5) | 1 (0.3) | 2 (0.6) | 19 (0.7) | |
| Antihypertensive medication use, n (%) | 2,924 (68.9) | 410 (49.2) | 199 (53.9) | 191 (58.4) | 2,124 (78.3) | <0.001 |
| Lipoproteins, mg/dl, mean ± SD | ||||||
| Total | 175.9 ± 31.1 | 174.9 ± 27.7 | 178.7 ± 30.0 | 180.5 ± 33.6 | 175.3 ± 31.8 | <0.01 |
| HDL | 47.9 ± 13.9 | 53.7 ± 14.4 | 48.6 ± 13.9 | 49.5 ± 14.5 | 45.8 ± 13.1 | <0.001 |
| LDL | 99.7 ± 23.4 | 97.8 ± 20.9 | 101.3 ± 23.5 | 101.9 ± 25.6 | 99.7 ± 23.8 | <0.05 |
ASCVD = atherosclerotic cardiovascular disease; BMI = body mass index; HDL = high-density lipoprotein; IQR = interquartile range; LDL = low-density lipoprotein.
Fifty-seven percent of patients were in the lowest (<5%) category of 10-year predicted ASCVD risk, with 12% having predicted risk 5.0% to <7.5%, 8% having predicted risk 7.5% to <10%, and the remaining 23% having predicted risk ≥10% (Table 1). Patients with higher predicted ASCVD risk were more likely to be older and male, be former or current smokers, have higher systolic blood pressure and BMI, lower HDL-C, and higher LDL-C (Table 1).
Accuracy of ASCVD risk equation by diabetes status
Among eligible patients 40 to 75 years of age without diabetes, 2,061 ASCVD events were observed during 1,515,142 person-years, with 1,464 (0.5%) acute myocardial infarctions, 525 (0.2%) CHD deaths, and 71 (0.02%) ischemic strokes. The cumulative risk of ASCVD events was linear during the entire 5-year follow-up period (Figure 2). Observed 5-year ASCVD incidence was substantially lower than the predicted risk in each category: 0.20% (95% CI: 0.20% to 0.25%) for predicted risk <2.50%; 0.65% (0.55% to 0.70%) for predicted risk 2.50% to <3.75%; 0.90% (0.75% to 1.00%) for predicted risk 3.75% to <5.00%: and 1.85% (1.75% to 1.95%) for predicted risk ≥5.00% (Central Illustration). Calibration between the predicted versus observed 5-year ASCVD incidence was poor (Figure 4) and the discrimination was moderate (C: 0.74).
Figure 2. Cumulative Risk of ASCVD Events During 5-Year Follow-Up Off Statin Therapy.
N represents the cumulative number of events observed during each follow-up period for calculating the cumulative risk of ASCVD events per year of follow-up among all eligible adults 40 to 75 years of age who did not have diabetes and did not receive statin therapy for primary prevention of ASCVD. ASCVD = atherosclerotic cardiovascular disease.
Central Illustration. Cardiovascular Risk Prediction in Clinical Care: Comparison of Observed Versus Expected ASCVD risks.
Observed 5-year risks of ASCVD events within each predicted risk category in eligible adults, 40 to 75 years of age, are shown stratified by diabetes status. ASCVD = atherosclerotic cardiovascular disease.
Figure 4. Calibration and Discrimination of ASCVD Risk Equation.
Calibration plots and model-based discrimination estimates (C statistic) comparing observed 5-year risk of ASCVD events in each predicted risk category in eligible adults 40 to 75 years of age are shown by diabetes status and in nondiabetic sex, racial/ethnic, and socioeconomic subgroups. ASCVD = atherosclerotic cardiovascular disease.
Among eligible 40- to 75-year-olds with diabetes, 148 events were observed during 19,196 person-years, with 84 (2%) acute myocardial infarctions, 54 (1.3%) CHD deaths, and 10 (0.2%) ischemic strokes. Observed 5-year ASCVD incidence in each predicted risk category was: 0.10% (0.00% to 0.85%) for predicted risk <2.50%; 2.55% (1.35% to 4.95%) for predicted risk 2.50% to <3.75%; 2.65% (1.30% to 5.30%) for predicted risk 3.75% to <5.00%; and 5.50% (4.60% to 6.50%) for predicted risk ≥5.00% (Central Illustration). Calibration was fair between predicted versus observed 5-year risk of ASCVD events, and discrimination was only modest (C: 0.64) (Figure 4).
Accuracy of ASCVD risk equation by sex, race/ethnicity, and socioeconomic status
Among both nondiabetic men and women, there was systematic overestimation of observed 5-year ASCVD incidence in each predicted risk category, with similarly poor calibration in both sexes (Figures 3 and 4). Discrimination in nondiabetic women (C: 0.72) was better than in men (C: 0.68).
Figure 3. Comparison of Observed Versus Expected ASCVD Risks in Subgroups.
Observed 5-year risks of ASCVD events within each predicted risk category in eligible nondiabetic adults 40 to 75 years of age are shown within sex, racial/ethnic, and socioeconomic status subgroups. ASCVD = atherosclerotic cardiovascular disease.
Given that Asian/Pacific islanders and Hispanics are considered under “white or other” in the ASCVD risk equation, we compared the observed versus predicted 5-year ASCVD incidence separately among non-Hispanic whites, non-Hispanic blacks, Asian/Pacific Islanders, and Hispanics. We found systematic overestimation of actual ASCVD risk in each of these ethnic groups (Figure 3) and similarly poor calibration, but with discrimination varying in each group (C: 0.70 to 0.74) (Figure 4). Results were also similar across measures of socioeconomic status (Figure 4).
Assessment of ASCVD risk equation in persons treated with statins for primary prevention
In sensitivity analyses, we separately examined eligible adults who did or did not have diabetes, but received statin therapy for primary prevention during follow-up. We found that in eligible adults without diabetes, there was overestimation of actual 5-year ASCVD incidence in each predicted risk category (Online Figure 1). Observed rates for this population were approximately 6× lower than predicted. In contrast to findings in untreated eligible patients with diabetes, we found that there was systematic overestimation of observed 5-year ASCVD incidence in each predicted risk category among eligible adults with diabetes who received statin therapy for primary prevention (Online Figure 1). Calibration was poor and discrimination was modest in both those with (C: 0.61) and without (C: 0.68) diabetes who received statin therapy for primary prevention (Online Figure 2).
Discussion
In a large, contemporary, “real world” cohort with broad racial/ethnic diversity, we showed that the ACC/AHA Pooled Cohort risk equation substantially overestimated actual 5-year ASCVD risk in each predicted risk category within eligible adults without diabetes (40 to 75 years of age without known ASCVD and with LDL-C 70 to 189 mg/dl) who are a recommended target population for consideration of cholesterol-lowering therapy. Importantly, this overestimation was similar in both men and women, as well as in 4 major ethnic groups, with poor calibration in each subgroup. Discrimination, as measured by the C statistic, was fair to moderate, depending on the patient group. Of interest, we found better calibration among eligible adults aged 40 to 75 years with diabetes, but worse discrimination.
Since release of the 2013 ACC/AHA cholesterol treatment guidelines, there has been an ongoing debate, especially regarding application of the new Pooled Cohort risk equation for assessing ASCVD risk within and outside the United States. In selected U.S.-based prospective cohorts not used to derive the Pooled Cohort risk equation, such as the MESA (Multi-Ethnic Study of Atherosclerosis) and the REGARDS (REasons for Geographic and Racial Differences in Stroke) studies, the estimated ASCVD risk was higher than observed risk (3,4). Similarly, in a European population 55 years of age or older, the risk equation had poor calibration (15). It has been proposed that results of these and related evaluation efforts are limited due to ∼25% of participants taking statins or having a high likelihood of initiating statin during follow-up (6,8). Underascertainment of ASCVD events in these studies has also been suggested as a possible reason for observed overestimation of risk (3,9). There is also concern as to how well the risk equation works in more contemporary patients compared with the older cohorts used to derive the equation (3,11,12). Analyses from the Women's Health Study estimated hypothetical rates if no women were prescribed statins or underwent revascularization and suggested that after accounting for intervention effects of statins and revascularization, as well as hypothetical confounding by indication or underascertainment of events, there would still be a discrepancy in the observed versus predicted ASCVD risk (11). However, because that study enrolled primarily white female nurses between 1992 and 1995, the results are not necessarily generalizable to more diverse, contemporary populations. Similarly, recent analyses from the MESA study share the important limitation of lack of generalizability, as research participants are not likely to accurately represent the general population in clinical practice and during a recent treatment era (4).
Our study addresses many of these challenges. First, we evaluated the accuracy of the Pooled Cohort risk equation in the most contemporary cohort studied to date, with study entry in 2008 and complete follow-up through 2013. Secondly, we excluded patients who had received any statin therapy during the 5 years before or anytime during follow-up, thus minimizing any confounding from statin use when evaluating performance of the risk calculator. Thirdly, as we studied patients receiving care within an integrated healthcare delivery system, we had complete ascertainment of ASCVD events. Fourthly, even after restricting to the target population of nondiabetic adults, 40 to 75 years of age, who had LDL-C between 70 and 189 mg/dl and no known ASCVD, we studied a very large population of 311,827 men and women, which is notably larger and more representative than any prior studies (e.g., 3,433 adults ≥55 years of age, including diabetics, in the Rotterdam Study (15) and 10,997 from the REGARDS study with a selected subset of 3,333 participants ≥65 years of age linked to Medicare claims data (8)). Fifthly, we examined 4 major racial/ethnic groups, including Asian/Pacific Islanders and Hispanics, which constitute a major proportion of the U.S. population (24). Present guidelines (1) caution against applying this algorithm to groups that are neither white nor black/African American, and we further demonstrated that the Pooled Cohort risk equation overestimated actual ASCVD risk across all 4 ethnic groups in our study population with variable discrimination. Finally, in contrast to previous cholesterol treatment guidelines (25), the Pooled Cohort risk equation includes diabetes as part of the scoring criteria, rather than considering it a CHD equivalent, and a recent study showed important differences in predicted risk among those with diabetes (26). We found that calibration was improved, but discrimination was poorer for untreated eligible adults with diabetes; importantly, we showed a wide range of ASCVD risk in these patients, which could inform future studies evaluating different thresholds for initiating statin therapy in patients with diabetes.
Our study also had several limitations. First, we used 5-year versus 10-year follow-up, and it is possible that incidence rates in the first 5 years of follow-up could be less than in the subsequent 5 years. However, we observed a cumulative risk of ASCVD that was linear during the 5-year follow-up (Figure 2), which directly supports extrapolation of our results to 10-year risk. Studying 5-year risk also allowed for a very contemporary estimate of ASCVD risk. Risk score overestimation may have been due to exclusion of those who were started on statins; however, in a sensitivity analysis, there was poor calibration and suboptimal discrimination of the Pooled Cohort risk equation in eligible patients, with or without diabetes, treated with statins for primary prevention during follow-up (Online Figures 1 and 2). Inclusion of death certificate data to help define fatal CHD may have led to some misclassification. Our study population may not be fully generalizable to all parts of the United States or to patients treated in different health care delivery systems; however, approximately one-third of the sample was obese, had known hypertension, and was a current or former smoker. In addition, unlike previous studies, we had a very ethnically diverse sample of patients with uniform access to care, as well as comprehensive ascertainment of clinically recognized ASCVD events, which argues for greater generalizability and validity. Given that most adults with diabetes within our health system were receiving statins for primary prevention, and were therefore not eligible for analysis, the sample we examined is not necessarily fully representative of all patients with diabetes. However, our study sample allowed comprehensive assessment of the observed 5-year ASCVD risk among adults with clinically recognized diabetes across the range of predicted risk, and we required all patients to have the full 5-year follow-up, which is critical to allow enough time to identify all fatal and nonfatal ASCVD events. We are unaware of any evidence that patients with symptoms or signs of possible ASCVD are more or less likely to remain insured and stay or leave our health system. There is a possibility that including those who died, but not those censored from disenrollment, could artificially raise the observed event rates, but given that the absolute risk of death was very low in this population, it is highly unlikely to have had a substantive impact on the results.
Insured patients may have better health habits than patients who are not able to afford insurance or choose not to obtain it. However, if one considers the rationale for risk assessment in the ACC/AHA guideline, this “real-world” population is more likely to represent typical patients in U.S. communities who are seeing a health care provider and who may be considered for preventive therapies, as compared with volunteers enrolled into prospective cohort studies or interventional clinical trials. Furthermore, if significant variation exists in regional population characteristics or outcomes, and if there are systematic differences in cardiovascular risk reduction efforts across health care settings, then it is critical that health care providers use a risk assessment tool that is calibrated to the patient population being seen by those providers.
In sum, as recently recommended (27), we conducted a critical evaluation of the ASCVD Pooled Cohort risk equation from a relevant large, diverse, “real-world” target population, overall and in key understudied subgroups. We found that this risk equation substantially overestimated actual ASCVD risk in adults not treated with statin therapy for primary prevention without diabetes (overall and across all sociodemographic subgroups), and demonstrated suboptimal accuracy in those with or without diabetes. Our study provides evidence to support recalibration of the ASCVD Pooled Cohort risk equation in adults without diabetes, especially given the individual and public health implications of widespread application of this risk calculator. Ongoing research and dialogue in this area remains crucial, and should be encouraged in order to provide more rigorous, valid evidence in contemporary, diverse populations (26-28).
Supplementary Material
Supplemental Figure A. Observed five-year risk of atherosclerotic cardiovascular events within predicted risk category in eligible 40 to 75 year old adults who received statin therapy for primary prevention, stratified by diabetes status.
Supplemental Figure B. Calibration plots comparing observed five-year risk of atherosclerotic cardiovascular events within predicted risk category in eligible 40 to 75 year old adults who received Statins for primary prevention by diabetes status.
Clinical Perspectives.
Competency in Systems-Based Practice
The 2013 ACC/AHA Pooled Cohort risk equation estimates risk for both coronary events and stroke. However, the risk equation substantially overestimates ASCVD risk in adults without diabetes mellitus and has suboptimal accuracy across sociodemographic subgroups and also in adults with diabetes who do not receive statin therapy for primary prevention.
Translational Outlook
More research is needed to develop risk assessment tools that are calibrated to diverse, contemporary populations, especially given the implications of recommending lifelong statin therapy.
Acknowledgments
The authors would like to thank Michael Pencina, PhD and Benjamin Neely from the Duke University Department of Biostatistics and Bioinformatics for assistance with ASCVD risk calculation and Alda I. Inveiss for her expert technical assistance. Drs. Rana and Go contributed to all parts of the study. All other authors contributed to the study design, data interpretation, and editing of the paper.
Disclosures: This project was supported in part by funding from the National Heart, Lung and Blood Institute (U19 HL091179) and the Kaiser Permanente Northern California Community Benefit Fund. The funding sources had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The authors had full access to the data and had final responsibility for the decision to submit for publication. Drs. Go and Lo have received a research grant through their institution from Sanofi. Dr. Solomon has received research grants through his institution from Abbott Vascular, Inc., and Genentech. Dr. Ballantyne has received research support through his institution from Abbott Diagnostic, Amarin, Amgen, Bristol-Myers Squibb, Eli Lilly, Esperion, Merck, Novartis, Pfizer, Regeneron, Roche Diagnostic, Sanofi-Synthelabo, Takeda Development Centers of America, the National Institutes of Health, American Heart Association, and American Diabetes Association. Dr. Ballantyne has been a consultant for Abbott Diagnostics, Amarin, Amgen, Astra Zeneca, Eli Lilly, Esperion, Genzyme, Matinas BioPharma, Inc., Merck, Novartis, Pfizer, Regeneron and Sanofi-Synthelabo. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose.
Abbreviations and Acronyms
- ACC
American College of Cardiology
- AHA
American Heart Association
- ASCVD
atherosclerotic cardiovascular disease
- BMI
body mass index
- CHD
coronary heart disease
- HDL-C
high-density lipoprotein cholesterol
- ICD-9
International Classification of Diseases, Ninth Edition
- LDL-C
low-density lipoprotein cholesterol
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- 1.Goff DC, Jr, Lloyd-Jones DM, Bennett G, et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol. 2014;63:2935–59. doi: 10.1016/j.jacc.2013.11.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Stone NJ, Robinson JG, Lichtenstein AH, et al. 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol. 2014;63:2889–934. doi: 10.1016/j.jacc.2013.11.002. [DOI] [PubMed] [Google Scholar]
- 3.Ridker PM, Cook NR. Statins: new American guidelines for prevention of cardiovascular disease. Lancet. 2013;382:1762–5. doi: 10.1016/S0140-6736(13)62388-0. [DOI] [PubMed] [Google Scholar]
- 4.DeFilippis AP, Young R, Carrubba CJ, et al. An analysis of calibration and discrimination among multiple cardiovascular risk scores in a modern multiethnic cohort. Ann Intern Med. 2015;162:266–75. doi: 10.7326/M14-1281. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Andersson C, Enserro D, Larson MG, et al. Implications of the US cholesterol guidelines on eligibility for statin therapy in the community: comparison of observed and predicted risks in the Framingham Heart Study Offspring Cohort. J Am Heart Assoc. 2015;4:e001888. doi: 10.1161/JAHA.115.001888. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Lloyd-Jones DM, Goff D, Stone NJ. Statins, risk assessment, and the new American prevention guidelines. Lancet. 2014;383:600–2. doi: 10.1016/S0140-6736(13)62348-X. [DOI] [PubMed] [Google Scholar]
- 7.Pursnani A, Massaro JM, D'Agostino RB, Sr, et al. Guideline-based statin eligibility, coronary artery calcification, and cardiovascular events. JAMA. 2015;314:134–41. doi: 10.1001/jama.2015.7515. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Muntner P, Colantonio LD, Cushman M, et al. Validation of the atherosclerotic cardiovascular disease Pooled Cohort risk equations. JAMA. 2014;311:1406–15. doi: 10.1001/jama.2014.2630. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Hlatky MA, Ray RM, Burwen DR, et al. Use of Medicare data to identify coronary heart disease outcomes in the Women's Health Initiative. Circ Cardiovasc Qual Outcomes. 2014;7:157–62. doi: 10.1161/CIRCOUTCOMES.113.000373. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Smith SC, Jr, Grundy SM. 2013 ACC/AHA guideline recommends fixed-dose strategies instead of targeted goals to lower blood cholesterol. J Am Coll Cardiol. 2014;64:601–12. doi: 10.1016/j.jacc.2014.06.1159. [DOI] [PubMed] [Google Scholar]
- 11.Cook NR, Ridker PM. Further insight into the cardiovascular risk calculator: the roles of statins, revascularizations, and underascertainment in the Women's Health Study. JAMA Intern Med. 2014;174:1964–71. doi: 10.1001/jamainternmed.2014.5336. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Nissen SE. Prevention guidelines: bad process, bad outcome. JAMA Intern Med. 2014;174:1972–3. doi: 10.1001/jamainternmed.2014.3278. [DOI] [PubMed] [Google Scholar]
- 13.Pencina MJ, Navar-Boggan AM, D'Agostino RB, Sr, et al. Application of new cholesterol guidelines to a population-based sample. N Engl J Med. 2014;370:1422–31. doi: 10.1056/NEJMoa1315665. [DOI] [PubMed] [Google Scholar]
- 14.Ray KK, Kastelein JJ, Boekholdt SM, et al. The ACC/AHA 2013 guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular disease risk in adults: the good the bad and the uncertain: a comparison with ESC/EAS guidelines for the management of dyslipidaemias 2011. Eur Heart J. 2014;35:960–8. doi: 10.1093/eurheartj/ehu107. [DOI] [PubMed] [Google Scholar]
- 15.Kavousi M, Leening MJ, Nanchen D, et al. Comparison of application of the ACC/AHA guidelines, Adult Treatment Panel III guidelines, and European Society of Cardiology guidelines for cardiovascular disease prevention in a European cohort. JAMA. 2014;311:1416–23. doi: 10.1001/jama.2014.2632. [DOI] [PubMed] [Google Scholar]
- 16.Gordon NP. Characteristics of Adult Health Plan Members in the Northern California Region Membership, as Estimated from the 2011 Member Health Survey. Division of Research, Kaiser Permanente Medical Care Program; Oakland, CA: 2013. [Accessed March 1, 2016]. Available at: https://www.dor.kaiser.org/external/mhs11reg/ [Google Scholar]
- 17.Yeh RW, Sidney S, Chandra M, et al. Population trends in the incidence and outcomes of acute myocardial infarction. N Engl J Med. 2010;362:2155–65. doi: 10.1056/NEJMoa0908610. [DOI] [PubMed] [Google Scholar]
- 18.Go AS, Hylek EM, Chang Y, et al. Anticoagulation therapy for stroke prevention in atrial fibrillation: how well do randomized trials translate into clinical practice? JAMA. 2003;290:2685–92. doi: 10.1001/jama.290.20.2685. [DOI] [PubMed] [Google Scholar]
- 19.Go AS, Chertow GM, Fan D, et al. Chronic kidney disease and the risks of death, cardiovascular events, and hospitalization. N Engl J Med. 2004;351:1296–305. doi: 10.1056/NEJMoa041031. [DOI] [PubMed] [Google Scholar]
- 20.Selby JV, Ray GT, Zhang D, et al. Excess costs of medical care for patients with diabetes in a managed care population. Diabetes Care. 1997;20:1396–402. doi: 10.2337/diacare.20.9.1396. [DOI] [PubMed] [Google Scholar]
- 21.Karter AJ, Schillinger D, Adams AS, et al. Elevated rates of diabetes in Pacific Islanders and Asian subgroups: The Diabetes Study of Northern California (DISTANCE) Diabetes Care. 2013;36:574–9. doi: 10.2337/dc12-0722. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Miniño AM, Klein RJ. Health E-Stats. National Center for Health Statistics; 2010. [Accessed March 1, 2016]. Health mortality from major cardiovascular diseases: United States, 2007. Available at: http://www.cdc.gov/nchs/data/hestat/cardio2007/cardio2007.pdf. [Google Scholar]
- 23.Smith DH, Thorp ML, Gurwitz JH, et al. Chronic kidney disease and outcomes in heart failure with preserved versus reduced ejection fraction: the Cardiovascular Research Network PRESERVE Study. Circ Cardiovasc Qual Outcomes. 2013;6:333–42. doi: 10.1161/CIRCOUTCOMES.113.000221. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.U.S. Census Bureau. 2010 Census Redistricting Data (Public Law 94-171) Summary File. 2011 [Google Scholar]
- 25.Expert Panel on Detection, Evaluation, And Treatment of High Blood Cholesterol In Adults. Executive Summary of The Third Report of The National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, And Treatment of High Blood Cholesterol In Adults (Adult Treatment Panel III) JAMA. 2001;285:2486–97. doi: 10.1001/jama.285.19.2486. [DOI] [PubMed] [Google Scholar]
- 26.Karmali KN, Goff DC, Jr, Ning H, et al. A systematic examination of the 2013 ACC/AHA pooled cohort risk assessment tool for atherosclerotic cardiovascular disease. J Am Coll Cardiol. 2014;64:959–68. doi: 10.1016/j.jacc.2014.06.1186. [DOI] [PubMed] [Google Scholar]
- 27.Amin NP, Martin SS, Blaha MJ, et al. Headed in the right direction but at risk for miscalculation: a critical appraisal of the 2013 ACC/AHA risk assessment guidelines. J Am Coll Cardiol. 2014;63:2789–94. doi: 10.1016/j.jacc.2014.04.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Krumholz HM. The new cholesterol and blood pressure guidelines: perspective on the path forward. JAMA. 2014;311:1403–5. doi: 10.1001/jama.2014.2634. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplemental Figure A. Observed five-year risk of atherosclerotic cardiovascular events within predicted risk category in eligible 40 to 75 year old adults who received statin therapy for primary prevention, stratified by diabetes status.
Supplemental Figure B. Calibration plots comparing observed five-year risk of atherosclerotic cardiovascular events within predicted risk category in eligible 40 to 75 year old adults who received Statins for primary prevention by diabetes status.