Key Points
Question
Do the pooled cohort equations (PCEs) accurately predict 10-year risk of atherosclerotic cardiovascular disease (ASCVD) among participants in the Women’s Health Initiative?
Findings
In this cohort study of 19 995 women, observed risks were generally lower than predicted by PCE; adjustment for changes in statin and aspirin use resulted in small increases in observed risks. However, there was a considerable impact of adding events identified by Centers for Medicare and Medicaid Services (CMS) surveillance but not self-reported, and the PCE models discriminated risk well.
Meaning
Observed and predicted ASCVD risks were better aligned after including CMS events.
This study evaluates the pooled cohort equation’s accuracy for predicting 10-year risk of atherosclerotic cardiovascular disease among participants in the Women’s Health Initiative.
Abstract
Importance
Atherosclerotic cardiovascular disease (ASCVD) kills approximately 1 in every 3 US women. Current cholesterol, hypertension, and aspirin guidelines recommend calculating 10-year risk of ASCVD using the 2013 Pooled Cohort Equations (PCE). However, numerous studies have reported apparent overestimation of risk with the PCE, and reasons for overestimation are unclear.
Objective
We evaluated the predictive accuracy of the PCE in the Women’s Health Initiative (WHI), a multiethnic cohort of contemporary US postmenopausal women. We evaluated the effects of time-varying treatments such as aspirin and statins, and ascertainment of additional ASCVD events by linkage with the Centers for Medicare and Medicaid Services (CMS) claims.
Design, Setting, and Participants
The WHI recruited the largest number of US women (n = 161 808) with the racial/ethnic, geographic, and age diversity of the general population (1993-1998). For this study, we included women aged 50 to 79 (n = 19 995) participating in the WHI with data on the risk equation variables at baseline and who met the guideline inclusion and exclusion criteria. Median follow-up was 10 years.
Main Outcomes and Measures
For this study, ASCVD was defined as myocardial infarction, stroke, or cardiovascular death.
Results
Among the 19 995 women (mean [SD] age, 64 [7.3] years; 8305 [41.5%] white, 7688 [38.5%] black, 3491 [17.5%] Hispanic, 103 [0.5%] American Indian, 321 [1.6%] Asian/Pacific Islander, and 87 [0.4%] other/unknown), a total of 1236 ASCVD events occurred in 10 years and were adjudicated through medical record review by WHI investigators. The WHI-adjudicated observed risks were lower than predicted. The observed (predicted) risks for baseline 10-year risk categories less than 5%, 5% to less than 7.5%, 7.5% to less than 10%, and 10% or more were 1.7 (2.8), 4.4 (6.2), 5.3 (8.7), and 12.4 (18.2), respectively. Small changes were noted after adjusting for time-dependent changes in statin and aspirin use. Among women 65 years or older enrolled in Medicare, WHI-adjudicated risks were also lower than predicted, but observed (predicted) risks became aligned after including events ascertained by linkage with CMS for additional surveillance for events: 3.8 (4.3), 7.1 (6.4), 8.3 (8.7), and 18.9 (18.7), respectively. Similar results were seen across ethnic/racial groups. Overall, the equations discriminated risk well (C statistic, 0.726; 95% CI, 0.714-0.738).
Conclusions and Relevance
Without including surveillance for ASCVD events using CMS, observed risks in the WHI were lower than predicted by PCE as noted in several other US cohorts, but risks were better aligned after including CMS events.
Trial Registration
ClinicalTrials.gov identifier: NCT00000611
Introduction
Atherosclerotic cardiovascular disease (ASCVD) kills approximately 1 in every 3 US women, exceeding deaths from all forms of cancer combined.1 Guidelines recommend assessing ASCVD risk, and targeting the intensity of preventive interventions to the risk level.2 In 2013, the American College of Cardiology (ACC) and American Heart Association (AHA) risk assessment guidelines developed new Pooled Cohort Equations (PCE) that estimate 10-year risk of ASCVD, with 4 separate equations for African American and white women and men.1 The PCE were derived from the Atherosclerosis Risk in Communities (ARIC) study, the Cardiovascular Health Study (CHS), the Coronary Artery Risk Development in Young Adults (CARDIA) study, and the Framingham Original and Offspring cohorts.1 The 2013 ACC/AHA cholesterol guidelines recommended using these equations to inform discussions about initiating statin treatment for adults without clinical ASCVD,2 and more recently the 2017 ACC/AHA hypertension guidelines necessitate them for blood pressure targets and treatment recommendations. Since the publication of the PCE, numerous studies have reported apparent overestimation of risk in both US and European populations.3,4,5,6,7 Critics have attributed risk overestimation in these validation studies to the study populations being highly selected and nongeneralizable, increasing use of concomitant preventive therapies such as aspirin and statins, decreasing ASCVD event rates, and underascertainment of event rates in more recent cohorts, which usually rely on participant self-report for identifying potential events.
To date, the Women’s Health Initiative (WHI) has recruited the largest number of US women (n = 161 808) with the racial/ethnic, geographic, and age diversity of the general population, with the caveat that women had to meet the study eligibility criteria and be willing to participate in a long-term study. Therefore, we evaluated the accuracy of the PCE for predicting 10-year risk of ASCVD in 19 995 women from the WHI who had complete data on the risk equation variables, and weighted results to the age and race/ethnicity distribution of the overall WHI (n = 161 808).8 We evaluated the effects of time-varying treatments such as aspirin and statins, and ascertainment of additional ASCVD events by linkage with the Centers for Medicare and Medicaid Services (CMS) claims.
Methods
Study Design
Study participants were drawn from the WHI that included postmenopausal women aged 50 to 79 years who were enrolled (1993-1998) at 1 of 40 WHI clinical centers nationwide into either a clinical trial (CT; n = 68 132) or an observational study (OS; n = 93 676).8,9 The WHI enrolled racial/ethnic minority groups proportionate to the total general US population in these age groups. Recruitment and baseline data collection have been previously reported (eMethods in the Supplement).8,10 Institutional review board approval was obtained at each center and all participants provided written informed consent.
Statin and aspirin use was collected at baseline and year 3 for OS participants, and at baseline, years 1, 3, 6, and 9 for CT participants, and at study close-out for all participants.11 Of the total 161 808 women in WHI, a subset of 23 146 had baseline lipid measurements (eFigures 1 and 2 in the Supplement). Of these women, a total of 20 431 women had complete information on the components of the PCE. Because the exclusion/inclusion criteria that were used to develop the PCE (ACC/AHA risk assessment guidelines1) differed from the criteria used to determine statin eligibility (ACC/AHA cholesterol guidelines2), we identified a total of 19 995 WHI participants in 2 subcohorts (eFigure 1 in the Supplement) based on the criteria from these 2 guidelines1,2: (1) WHI risk validation subcohort (n = 19 581) that met the criteria for the risk assessment guidelines,1 and (2) WHI treatment eligibility subcohort (n = 13 439) that met the criteria for the cholesterol guidelines.2 As per the guidelines, the risk validation subcohort excluded women with a medical history of heart failure or atrial fibrillation,1 and the treatment eligibility subcohort excluded women with baseline statin use, LDL cholesterol levels less than 70 mg/dL, or those for whom statins were recommended2: clinical ASCVD (history of MI, stroke, peripheral arterial disease, arterial revascularization, angina, transient ischemic attack), LDL cholesterol levels of 190 mg/dL or higher, or diabetes (to convert LDL cholesterol measure to millimoles per liter, multiply by 0.0259). Women older than 75 years were excluded from the treatment eligibility cohort to be consistent with the cholesterol guidelines.2 All analyses were run separately in each subcohort.
Outcomes
The first ASCVD event was defined as the first occurrence of nonfatal myocardial infarction (MI), coronary heart disease (CHD) death, or fatal or nonfatal stroke.
WHI-Adjudicated Events
Annual questionnaires were completed by participants or their proxies who were asked if they were hospitalized since their last report, and medical records were obtained and adjudicated by trained physicians using standard criteria as described previously.12,13 In addition, even if an MI or stroke was not specifically reported or if the hospitalization was for a noncardiovascular reason, WHI inquired about and reviewed all hospitalization records for potential cardiovascular-related outcomes (eMethods in the Supplement). The WHI also ascertained deaths obtained from periodic searches of the National Death Index (NDI), to complement routine follow-up of reports of deaths by next of kin and postal authorities.
Claims-Based Events
Data from WHI have been linked to Medicare files from the CMS, expanding the data available on WHI participants 65 years and older, and this approach has been previously used for MI14 and stroke13 in the WHI. These claims-based events were included in the specified analyses for women 65 years or older enrolled in Medicare A or A/B (eMethods in the Supplement).
Statistical Analyses
Predicted rates were calculated based on the mean predicted ASCVD rates from the PCE, using the recommended race-specific equations.1 Participants were censored at the time of first occurrence of ASCVD, death, time of last follow-up, or at 12 years of follow-up, unless otherwise specified. Observed rates of ASCVD were adjusted for variable follow-up time using Kaplan-Meier survival analysis. We also plotted the average predicted risk within the clinical risk categories based on cut points of 5.0%, 7.5%, and 10%. Results were reported first as unweighted risks, and to reflect the overall WHI population, results were also weighted using inverse probability weighting to the racial/ethnic and age distribution of the overall WHI cohort (n = 161 808; eTable 1A in the Supplement). Model calibration and discrimination were assessed (eMethods in the Supplement).15,16 We assessed the impact of time-dependent statin and aspirin use during follow-up (eMethods in the Supplement).4
Furthermore, as a subset of the CT participants were randomly allocated to the active hormone therapy arm (conjugated equine estrogens [CEE] 0.625 mg/d alone in women with prior hysterectomy for a median intervention phase of 7.2 years, or CEE plus medroxyprogesterone acetate 2.5 mg/d in women with intact uterus for a median intervention phase of 5.6 years) after the baseline assessment of lipids and other risk factors, we prespecified conducting analyses that excluded them owing to potential alterations in lipids and risk factors following initiation of hormone therapy.
Finally, to assess the impact of more complete outcome ascertainment on the observed risks, we also conducted prespecified analyses that incorporated the CMS claims outcomes data available on participants 65 years or older at study entry who were enrolled in traditional fee-for-service Medicare Part A or A/B (eMethods in the Supplement).13,14
Results
Compared with the overall WHI, current study participants were of similar age, with more than 1 in 5 women 70 years or older, but were more racially diverse owing to oversampling by design of nonwhite participants (eTable 1B in the Supplement). Among the 19 995 women in the analytic sample (either the risk validation or treatment eligibility subcohorts), a total of 1236 WHI-adjudicated first ASCVD events (including 35 NDI deaths) occurred in 10 years (eMethods in the Supplement).
In unweighted analyses, the average 10-year WHI-adjudicated observed risks were lower than the predicted risks (Table 1) (Figure 1A) (eTable 2 in the Supplement). The observed (predicted) incident rates of events for women in the risk validation subcohort with baseline 10-year predicted risk less than 5%, 5% to less than 7.5%, 7.5% to less than 10%, and 10% or higher were: 1.7 (2.8), 4.4 (6.2), 5.3 (8.7), and 12.4 (18.2), respectively. The overall C statistic was 0.726 (95% CI, 0.714-0.738), Hosmer-Lemeshow calibration χ2 was 250.42 (P < .001). After weighting to the age and racial/ethnic distribution of the overall WHI, predicted rates remained higher than observed (Figure 1B): 1.4 (2.7), 4.7 (6.2), 5.5 (8.7), and 12.7 (17.4), respectively. Generally similar results were obtained among the treatment eligibility subcohort (Table 1) (eTable 2 in the Supplement).
Table 1. 2013 ACC/AHA Pooled Cohort (n = 19 995) Equations Predicted and Observed Risks, Before and After Weighting to the Overall WHI Cohort (N = 161 808)a.
| 10-Year Predicted Risk of ASCVD, % | No. | Unweighted | Weighteda | Discrimination C-Index, (95% CI) |
|||||
|---|---|---|---|---|---|---|---|---|---|
| Events/Person-Years | Events in 10 Years | 10-Year Incident Rate | 10-Year Incident Rate | ||||||
| KM-Adjusted Observed | Predicted | KM-Adjusted Observed, (95% CI) |
Predicted | KM-Adjusted Observed | Predicted | ||||
| WHI risk validation subcohort | |||||||||
| All women | 19581 | 1502/213 554 | 0.726 (0.714-0.738) |
||||||
| <5 | 6297 | 131/71 025 | 109 | 178 | 0.017 (0.016-0.018) |
0.028 | 0.014 | 0.027 | |
| 5 to <7.5 | 3276 | 172/36376 | 143 | 204 | 0.044 (0.041-0.046) |
0.062 | 0.047 | 0.062 | |
| 7.5 to <10 | 2626 | 172/29 027 | 140 | 228 | 0.053 (0.050-0.056) |
0.087 | 0.055 | 0.087 | |
| ≥10 | 7382 | 1027/77 126 | 913 | 1341 | 0.124 (0.121-0.126) |
0.182 | 0.127 | 0.174 | |
| WHI treatment eligibility subcohort | |||||||||
| All women | 13439 | 770/148 299 | 0.705 (0.688-0.723) |
||||||
| <5 | 5393 | 114/60 919 | 95 | 149 | 0.018 (0.016-0.019) |
0.028 | 0.014 | 0.027 | |
| 5 to <7.5 | 2498 | 129/27 729 | 108 | 155 | 0.043 (0.040-0.046) |
0.062 | 0.048 | 0.062 | |
| 7.5 to <10 | 1898 | 109/20 990 | 91 | 165 | 0.048 (0.045-0.052) |
0.087 | 0.050 | 0.087 | |
| ≥10 | 3650 | 418/38 661 | 364 | 554 | 0.100 (0.096-0.103) |
0.152 | 0.107 | 0.150 | |
Abbreviations: ACC, American College of Cardiology; AHA, American Heart Association; ASCV, atherosclerotic cardiovascular disease; KM, Kaplan-Meier; WHI, Women's Health Initiative; NDI, National Death Index.
Based on WHI-adjudicated events (including NDI); unweighted and weighted by race/ethnicity (white, Black, Hispanic, American Indian Asian/Pacific Islander, unknown) and age (50-59, 60-69, 70-79 years).
Figure 1. Observed vs Predicted Risk by Baseline 10-Year Risk Categories.
Kaplan-Meier observed unadjusted (dark blue bars) and predicted (light blue bars) 10-year risks of atherosclerotic cardiovascular disease events by baseline 10-year risk categories of less than 5%, 5% to less than 7.5%, 7.5% to less than 10%, and 10% or more before (A) and after (B) weighting to the age and racial/ethnic distribution of the overall WHI cohort. Observed events were WHI-adjudicated events.
At baseline, statin and aspirin use in the WHI risk validation subcohort was low (6.7% and 16.6%, respectively), differed by baseline risk category, and increased to 20% to greater than 40% during follow-up among the various risk categories (eTable 1B and eFigure 3 in the Supplement). Adjusting for the time-varying change in statin and aspirin use resulted in a slight increase in observed risks (Figure 2A) (eTable 3 in the Supplement). In another prespecified analysis, we excluded women who were assigned to the active hormone therapy arm, which only minimally lowered the observed risks (Figure 2B) (eTable 3 in the Supplement).
Figure 2. Observed vs Predicted Risk by Baseline 10-Year Risk Categories, Weighted to the Age and Racial Distribution of the Overall Women's Health Initiative (WHI) Cohort.
Kaplan-Meier observed unadjusted (blue bars), observed adjusted for statin and aspirin use during follow-up (light blue bars), and predicted (white bars) 10-year risks of atherosclerotic cardiovascular disease events by baseline 10-year risk categories of less than 5%, 5% to less than 7.5%, 7.5% to less than 10%, and 10% or more, weighted to the age and racial/ethnic distribution of the overall WHI cohort. Panel A is for all participants and panel B is after excluding active hormone arm. Observed events were WHI-adjudicated events.
Among women 65 years or older who were enrolled in Medicare A or A/B, CMS claims data identified an additional 64 cases of MI and 99 cases of stroke that were not identified by self-report (eMethods in the Supplement). After including 85% of the additional CMS cases that were missed by self-report (eMethods in the Supplement), predicted risks became closely aligned with observed risks (Figure 3) (eTable 4A in the Supplement). For example, among the risk validation subcohort with the additional CMS events, the observed (predicted) risks excluding active hormone arm and adjusting for statin/aspirin use were: 3.8 (4.3), 7.1 (6.4), 8.3 (8.7), and 18.9 (18.7). Results differed minimally in a sensitivity analysis assuming 60% positive predictive value and only the primary CMS position, or when restricting analyses to the subgroup of women randomized to placebo in the hormone therapy CT (eTable 4B in the Supplement).
Figure 3. Observed vs Predicted Risk by Baseline 10-Year Risk Categories in Women 65 Years and Older Enrolled in Medicare A or A/B (n = 6071).
In women 65 years and older enrolled in Medicare A or A/B, Kaplan-Meier observed unadjusted (dark blue bars), observed adjusted for statin and aspirin use during follow-up (light blue bars), and predicted (white bars) 10-year risks of atherosclerotic cardiovascular disease events by baseline 10-year risk categories of less than 5%, 5% to less than 7.5%, 7.5% to less than 10%, and 10% or more. Panels A and B are before including Centers for Medicare and Medicaid (CMS) events (A, participants enrolled in Medicare A or A/B fee for service; and B after excluding active hormone therapy arm); panels C and D also excluded active hormone therapy arm but included events ascertained from the CMS, assuming a positive predictive value (ie, proportion of true cases) of 60% for principal discharge diagnosis (C) or 85% for principal or secondary diagnosis (D).
aExcludes hormone therapy.
Adding the CMS events also improved both model calibrations: Hosmer-Lemeshow calibration χ2 improved from 91.870 (P < .001 [not calibrated]) to 3.32 (P = .19 [well calibrated]) (eTable 5 in the Supplement). Similar findings were observed for the treatment eligibility subcohort (Table 2).
Table 2. Predicted and Observed Risk in Women 65 Years and Older Enrolled in CMS by Racial/Ethnic and Age Subgroups Before and After Accounting for CMS Eventsa.
| 10-Year Predicted Risk of ASCVD, % | No. | Events/Person-Years | KM-Adjusted Observed Events in 10 Years | KM-Adjusted Observed, 10-Year Incidence Rate (95% CI) | Calibration Hosmer-Lemeshow Goodness of Fit (P Valueb) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Without CMS | With CMS | Without CMS | With CMS | Predicted | Without CMS | With CMS | Predicted | Without CMS | With CMS | ||
| WHI risk validation subcohort | 6071 | ||||||||||
| All | 91.870 (<.001) |
3.316 (.19) |
|||||||||
| <7.5 | 1097 | 59/12 282 | 66/9743 | 51 | 71 | 66 | 0.046 (0.042-0.051) |
0.065 (0.059-0.071) |
0.060 | ||
| 7.5 to <10 | 1098 | 82/12 111 | 99/9966 | 66 | 98 | 96 | 0.060 (0.055-0.065) |
0.089 (0.083-0.096) |
0.088 | ||
| ≥10 | 3876 | 564/40 734 | 659/33 964 | 503 | 676 | 716 | 0.130 (0.126-0.134) |
0.174 (0.170-0.179) |
0.185 | ||
| White | 37.802 (<.001) |
2.214 (.33) |
|||||||||
| <7.5 | 867 | 50/9716 | 54/7933 | 42 | 54 | 52 | 0.049 (0.044-0.054) |
0.063 (0.057-0.069) |
0.060 | ||
| 7.5 to <10 | 840 | 63/9292 | 72/7824 | 49 | 65 | 73 | 0.059 (0.053-0.065) |
0.077 (0.071-0.085) |
0.087 | ||
| ≥10 | 2352 | 361/24 963 | 412/21 494 | 314 | 399 | 410 | 0.133 (0.129-0.138) |
0.170 (0.164-0.176) |
0.174 | ||
| Black | 49.542 (<.001) |
5.828 (.05) |
|||||||||
| <7.5 | 93 | 3/1016 | 3/796 | 3 | 4 | 6 | 0.033 (0.022-0.048) |
0.038 (0.026-0.056) |
0.064 | ||
| 7.5 to <10 | 144 | 11/1592 | 16/1215 | 9 | 18 | 13 | 0.060 (0.048-0.076) |
0.128 (0.107-0.153) |
0.088 | ||
| ≥10 | 1282 | 176/13 239 | 212/10 444 | 166 | 243 | 265 | 0.129 (0.123-0.136) |
0.190 (0.181-0.198) |
0.207 | ||
| Hispanic | 12.414 (.002) |
5.948 (.05) |
|||||||||
| <7.5 | 123 | 6/1404 | 8/902 | 5 | 13 | 7 | 0.044 (0.033-0.059) |
0.104 (0.081-0.132) |
0.059 | ||
| 7.5 to <10 | 96 | 5/1051 | 7/796 | 6 | 10 | 8 | 0.058 (0.043-0.078) |
0.106 (0.083-0.135) |
0.088 | ||
| ≥10 | 193 | 20/2037 | 27/1610 | 16 | 27 | 32 | 0.084 (0.071-0.100) |
0.140 (0.121-0.161) |
0.167 | ||
| Age 65-75 y | 86.097 (<.001) |
3.772 (.15) |
|||||||||
| <7.5 | 1097 | 59/12 282 | 66/9743 | 51 | 71 | 66 | 0.046 (0.042-0.051) |
0.065 (0.059-0.071) |
0.060 | ||
| 7.5 to <10 | 1098 | 82/12 111 | 99/9966 | 66 | 98 | 96 | 0.060 (0.055-0.065) |
0.089 (0.083-0.096) |
0.088 | ||
| ≥10 | 3319 | 442/35 242 | 519/29 102 | 388 | 534 | 573 | 0.117 (0.113-0.121) |
0.161 (0.156-0.166) |
0.173 | ||
| Age >75 y | 6.762 (.03) |
0.086 (.96) |
|||||||||
| ≥10 | 557 | 122/5492 | 140/4862 | 116 | 140 | 143 | 0.208 (0.196-0.221) |
0.251 (0.237-0.265) |
0.256 | ||
| WHI treatment eligibility subcohort | 3802 | ||||||||||
| All | 55.466 (<.001) |
5.462 (.06) |
|||||||||
| <7.5 | 914 | 50/10 202 | 55/8063 | 44 | 60 | 54 | 0.048 (0.043-0.053) |
0.065 (0.059-0.072) |
0.059 | ||
| 7.5 to <10 | 842 | 57/9316 | 68/7644 | 48 | 69 | 74 | 0.057 (0.051-0.063) |
0.082 (0.075-0.089) |
0.087 | ||
| ≥10 | 2046 | 231/21 734 | 274/17 975 | 209 | 286 | 316 | 0.102 (0.097-0.107) |
0.140 (0.134-0.146) |
0.155 | ||
| White | 25.744 (<.001) |
6.375 (.04) |
|||||||||
| <7.5 | 720 | 44/8028 | 46/6548 | 37 | 46 | 42 | 0.052 (0.046-0.058) |
0.064 (0.057-0.071) |
0.059 | ||
| 7.5 to <10 | 638 | 43/7085 | 50/5913 | 35 | 47 | 56 | 0.055 (0.049-0.061) |
0.073 (0.066-0.082) |
0.087 | ||
| ≥10 | 1248 | 159/13 471 | 179/11 576 | 139 | 172 | 187 | 0.111 (0.105-0.118) |
0.138 (0.131-0.146) |
0.150 | ||
| Black | 28.748 (<.001) |
2.641 (.27) |
|||||||||
| <7.5 | 76 | 2/852 | 2/667 | 2 | 3 | 5 | 0.028 (0.017-0.045) |
0.035 (0.022-0.057) |
0.064 | ||
| 7.5 to <10 | 117 | 9/1300 | 11/1042 | 7 | 12 | 10 | 0.064 (0.050-0.082) |
0.104 (0.084-0.129) |
0.088 | ||
| ≥10 | 665 | 60/6851 | 78/5292 | 60 | 98 | 110 | 0.090 (0.083-0.099) |
0.147 (0.136-0.159) |
0.165 | ||
| Hispanic | 12.174 (.002) |
7.617 (.02) |
|||||||||
| <7.5 | 109 | 4/1236 | 6/785 | 4 | 11 | 6 | 0.040 (0.029-0.056) |
0.102 (0.079-0.133) |
0.059 | ||
| 7.5 to <10 | 73 | 4/794 | 5/591 | 4 | 7 | 6 | 0.061 (0.044-0.084) |
0.102 (0.076-0.136) |
0.088 | ||
| ≥10 | 107 | 7/1155 | 11/895 | 4 | 10 | 15 | 0.040 (0.028-0.055) |
0.091 (0.071-0.116) |
0.142 | ||
| Age 65-75 y | 55.466 (<.001) |
5.462 (.06) |
|||||||||
| <7.5 | 914 | 50/10 202 | 55/8063 | 44 | 60 | 54 | 0.048 (0.043-0.053) |
0.065 (0.059-0.072) |
0.059 | ||
| 7.5 to <10 | 842 | 57/9316 | 68/7644 | 48 | 69 | 74 | 0.057 (0.051-0.063) |
0.082 (0.075-0.089) |
0.087 | ||
| ≥10 | 2046 | 231/21 734 | 274/17 975 | 209 | 286 | 316 | 0.102 (0.097-0.107) |
0.140 (0.134-0.146) |
0.155 | ||
Abbreviations: ACC, American College of Cardiology; AHA, American Heart Association; ASCV, atherosclerotic cardiovascular disease; CMS, Centers for Medicare and Medicaid; KM, Kaplan-Meier; WHI, Women's Health Initiative; NDI, National Death Index.
Includes CMS events (eMethods in the Supplement) and using 85% PPV and any position CMS events.
P≥.05 indicates a well-calibrated model.
Among these women, we also analyzed risks by racial/ethnic groups (Table 2) (eFigure 4 in the Supplement) and age categories (Table 2). In all subgroups, predicted risks were initially higher than observed when excluding CMS events, but predicted risks became aligned with observed risks among non-Hispanic whites after also including the CMS events. The effects of adding CMS events were more variable among minorities.
eTable 5 in the Supplement shows the proportion of WHI women by racial/ethnic groups who would be potentially eligible for statin treatment based on the 2013 ACC/AHA cholesterol guidelines.2
Discussion
Without including surveillance for ASCVD events using CMS linkage, observed risks of ASCVD were lower than predicted in the WHI, as noted in several other US cohorts. Adjustment for time-dependent changes in statin or aspirin treatment during the study follow-up resulted in small increases in observed risks. In contrast, there was a considerable impact of adding events that were identified by CMS surveillance but not self-reported, and the observed risks became better aligned with predicted risks after including CMS events. The PCE models discriminated risk well.
The external validation studies that evaluated the PCE have generated considerable controversy around the predictive accuracy of these equations in contemporary cohorts. At the same time, these recent studies have been criticized for having overall event rates that were much lower than noted in the original PCE cohorts. Overestimation of risk (ie, predicted > observed risk) was noted in studies that enrolled lower-risk health care professionals in clinical trials of aspirin and did not include CMS events.17 Overestimation of risk was also noted in the Rotterdam study,3 the Multi-Ethnic Study of Atherosclerosis,5 and a selected subpopulation of Kaiser Permanente,6 among others.7 How much of the discrepancy can be attributed to changes in ASCVD rates over time (eg, the decreasing ASCVD death rates since the 1970s) vs other factors discussed below has been debated.7
Risk scores may not perform as well in a new setting as in the original setting, which may be owing to differences in disease prevalence, or to the risk equation not reliably discriminating cases from noncases.18 We found reasonably good discrimination with the PCE in the WHI (C statistic, 0.726), consistent with other studies (C statistics of 0.71-0.82 in the original Pooled Cohorts,1 0.68-0.74 in the Contemporary Pooled Cohorts,10 0.71-0.72 in the Reasons for Geographic and Racial Differences in Stroke [REGARDS] study,19 and 0.71 in the MESA study5), and comparable to other scores.20,21,22
Because discrimination was not the issue affecting the performance of the equations, the differences in the predicted:observed rates may result from apparent or real differences in disease prevalence.18 This issue is well-recognized in the literature, where it is recommended that scores should be recalibrated if incidence rates differ substantially in the new settings.23 It is well-known that CVD incidence rates have been declining, which suggests that it would be appropriate to align the predicted rates with more contemporary cohorts as long as they have comprehensive surveillance for events. In the WHI, we found that the predicted:observed rates were generally well-aligned after accounting for outcomes ascertained by linkage with CMS for capture of additional events. Similarly, in the REGARDS study, the PCE were well calibrated in the group of participants for whom use of the PCE are recommended when CMS events were included.19 Neither the WHI nor REGARDS studies sought records for events identified only through CMS.
The intensity of surveillance for events directly determines observed event rates.24 Data from studies with comprehensive surveillance methods were used to create the PCE, as several PCE cohorts were designed to monitor incidence rates over time. The original PCE cohorts used self-reporting, but also actively searched for events using multiple labor-intensive methods to detect events that were not self-reported. These methods included actively searching hospitalization and emergency department records of hospitals in their communities (eg, Framingham, ARIC), linkage with CMS (eg, CHS), and additional surveillance methods (eg, familial and social contacts, obituary surveillance, review of emergency department records, hospital death records, and local field center investigations). Event ascertainment in these studies also included a review of medical records for these events that were identified through other surveillance mechanisms. For example, ARIC implemented a community surveillance component that had near complete (almost 100%) event ascertainment.25 Studies have found that self-report of ASCVD captures approximately 60% to 75% of events.24,26,27 Hence, relying on self-report alone without more complete surveillance for events could miss a sizeable proportion of events.
Furthermore, the Privacy Rule of the Health Insurance Portability and Accountability Act (HIPAA), which was passed in 1996 and implemented in 2003 (and hence would not have affected the original PCE cohorts), would have impacted more contemporary studies.28 Post-HIPAA, hospitals and institutions have many deterrents to providing medical records to research studies (eg, fear of litigation, extra labor involved for retrieval of medical records, greater number of research studies). Future research and policy efforts should address HIPAA implications on contemporaneous event ascertainment and cost of conducting research.29
Volunteers for prevention or screening trials are healthier and have lower mortality than the general population. Hence it is important that clinicians are aware of the baseline prevalence of ASCVD in their population and other relevant characteristics that may influence the pretest probability (eg, socioeconomic status, lifestyle). Overprediction of risk may be more pronounced in income strata such as individuals with higher socioeconomic status.30 Much of the difference by cohort should be captured in the risk factors included in the equations; if not,18 this suggests that the equations may be improved by including additional factors. A recently proposed revision of the pooled cohort equations using more contemporary cohort data and new derivation methods appeared to improve the accuracy of ASCVD risk estimates31 but will require additional validation, including assessment of accuracy with and without inclusion of CMS-identified events.
We also assessed the influence of therapies, in particular statin and aspirin use, on the predicted:observed rates, as more widespread use of these therapies in more contemporary populations has been proposed as a possible explanation for the overestimation of risk noted in other studies. However, we did not find a considerable impact of statin or aspirin use, consistent with prior reports.4
Strengths and Limitations
Strengths of the study include the large sample size and number of ASCVD events that occurred in this racially and ethnically diverse population that was recruited to represent the racial/ethnic distribution of US postmenopausal women. Whereas the PCE cohorts included a total of 13 881 women (1192 ASCVD),1 the current study included a greater number (19 995; 1236 events), with more ethnic and geographic diversity. Additional strengths include the long follow-up and the well-characterized risk factor data, including clinical measurements of blood pressure and detailed information regarding medication use. The complementary surveillance of MI and stroke events from CMS claims allowed for more complete ascertainment of events that were not self-reported. For example, this analysis differed from the previously reported results from the WHI Observational Study17 and the other studies mentioned herein by including a larger number of women, incorporating CMS events, and using the PCE inclusion/exclusion criteria. The CMS analyses depend on assumptions about the reliability of CMS reported endpoints, especially among those without matching medical records. Whereas predictive values may be reasonable overall, they may be less or more reliable for CMS events that were not self reported or lacked medical records sufficient for adjudication.
The current results may not be generalizable to men or other patient groups. In the risk validation subcohort, 29% of women were randomized to the active hormone arm, which would not have been reflected in their baseline risk factor measurements. However, our prespecified analyses that excluded these women showed no significant impact on the results. We cannot rule out a healthy volunteer effect, though this holds for other cohorts.
Conclusions
Without including surveillance for ASCVD events using CMS, observed risks of ASCVD were generally lower than predicted by the PCE in this large multiethnic cohort, but observed and predicted risks were better aligned after including CMS outcomes.
Supplementary Methods
eTable S1a. Race/ethnicity distribution and age distribution before and after weighting
eTable S1b. Baseline cardiovascular risk profile characteristics for WHI participants in the risk validation or treatment eligibility subcohorts
eTable S2. Predicted and observed risks excluding NDI events, before and after weighting to the overall WHI cohort
eTable S3. Unweighted and weighted predicted and observed risks before and after adjusting for statin and aspirin use during follow-up, in all women and after excluding women in the active hormone arm
eTable S4a. 10-year predicted and observed risks in women 65 years and older before and after including CMS events
eTable S4b. 10-year predicted and observed risks in women 65 years and older who were enrolled in the placebo arm of the hormone therapy clinical trial, after including CMS events
eTable S5. Treatment recommendations based on the 2013 ACC/AHA cholesterol guidelines according to racial/ethnic groups in WHI women ages 50 to 75 years old and with complete data on the risk equations
Supplementary Figure Legends
eReferences
eFigure S1. Study flow diagram
eFigure S2. Derivation of the current analytical sample
eFigure S3. Statin and aspirin use at baseline and during follow-up by baseline risk category
eFigure S4. Predicted and observed risks in women 65 years and older before and after adjusting for statin use, aspirin use, and CMS events, according to race/ethnicity
List of WHI investigators
References
- 1.Goff DC Jr, Lloyd-Jones DM, Bennett G, et al. ; American College of Cardiology/American Heart Association Task Force on Practice Guidelines . 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation. 2014;129(25)(suppl 2):S49-S73. [DOI] [PubMed] [Google Scholar]
- 2.Stone NJ, Robinson JG, Lichtenstein AH, et al. ; American College of Cardiology/American Heart Association Task Force on Practice Guidelines . 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation. 2014;129(25)(suppl 2):S1-S45. [DOI] [PubMed] [Google Scholar]
- 3.Kavousi M, Leening MJ, Nanchen D, et al. . Comparison of application of the ACC/AHA guidelines, Adult Treatment Panel III guidelines, and European Society of Cardiology guidelines for cardiovascular disease prevention in a European cohort. JAMA. 2014;311(14):1416-1423. [DOI] [PubMed] [Google Scholar]
- 4.Cook NR, Ridker PM. Further insight into the cardiovascular risk calculator: the roles of statins, revascularizations, and underascertainment in the Women’s Health Study. JAMA Intern Med. 2014;174(12):1964-1971. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.DeFilippis AP, Young R, Carrubba CJ, et al. . An analysis of calibration and discrimination among multiple cardiovascular risk scores in a modern multiethnic cohort. Ann Intern Med. 2015;162(4):266-275. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Rana JS, Tabada GH, Solomon MD, et al. . Accuracy of the atherosclerotic cardiovascular risk equation in a large contemporary, multiethnic population. J Am Coll Cardiol. 2016;67(18):2118-2130. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Cook NR, Ridker PM. Calibration of the pooled cohort equations for atherosclerotic cardiovascular disease: An update. Ann Intern Med. 2016;165(11):786-794. [DOI] [PubMed] [Google Scholar]
- 8.The Women’s Health Initiative Study Group Design of the Women’s Health Initiative clinical trial and observational study. Control Clin Trials. 1998;19(1):61-109. [DOI] [PubMed] [Google Scholar]
- 9.Manson JE, Chlebowski RT, Stefanick ML, et al. . Menopausal hormone therapy and health outcomes during the intervention and extended poststopping phases of the Women’s Health Initiative randomized trials. JAMA. 2013;310(13):1353-1368. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Hays J, Hunt JR, Hubbell FA, et al. . The Women’s Health Initiative recruitment methods and results. Ann Epidemiol. 2003;13(9)(suppl):S18-S77. [DOI] [PubMed] [Google Scholar]
- 11.Cook NR, Ridker PM. Response to Comment on the reports of over-estimation of ASCVD risk using the 2013 AHA/ACC risk equation. Circulation. 2014;129(2):268-269. [DOI] [PubMed] [Google Scholar]
- 12.Curb JD, McTiernan A, Heckbert SR, et al. ; WHI Morbidity and Mortality Committee . Outcomes ascertainment and adjudication methods in the Women’s Health Initiative. Ann Epidemiol. 2003;13(9)(suppl):S122-S128. [DOI] [PubMed] [Google Scholar]
- 13.Lakshminarayan K, Larson JC, Virnig B, et al. . Comparison of Medicare claims versus physician adjudication for identifying stroke outcomes in the Women’s Health Initiative. Stroke. 2014;45(3):815-821. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Hlatky MA, Ray RM, Burwen DR, et al. . Use of Medicare data to identify coronary heart disease outcomes in the Women’s Health Initiative. Circ Cardiovasc Qual Outcomes. 2014;7(1):157-162. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Lemeshow S, Hosmer DW Jr. A review of goodness of fit statistics for use in the development of logistic regression models. Am J Epidemiol. 1982;115(1):92-106. [DOI] [PubMed] [Google Scholar]
- 16.Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15(4):361-387. [DOI] [PubMed] [Google Scholar]
- 17.Ridker PM, Cook NR. Statins: new American guidelines for prevention of cardiovascular disease. Lancet. 2013;382(9907):1762-1765. [DOI] [PubMed] [Google Scholar]
- 18.Poses RM, Cebul RD, Collins M, Fager SS. The importance of disease prevalence in transporting clinical prediction rules. The case of streptococcal pharyngitis. Ann Intern Med. 1986;105(4):586-591. [DOI] [PubMed] [Google Scholar]
- 19.Muntner P, Colantonio LD, Cushman M, et al. . Validation of the atherosclerotic cardiovascular disease Pooled Cohort risk equations. JAMA. 2014;311(14):1406-1415. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Ridker PM, Buring JE, Rifai N, Cook NR. Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds Risk Score. JAMA. 2007;297(6):611-619. [DOI] [PubMed] [Google Scholar]
- 21.D’Agostino RB Sr, Vasan RS, Pencina MJ, et al. . General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation. 2008;117(6):743-753. [DOI] [PubMed] [Google Scholar]
- 22.D’Agostino RB Sr, Grundy S, Sullivan LM, Wilson P; CHD Risk Prediction Group . Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA. 2001;286(2):180-187. [DOI] [PubMed] [Google Scholar]
- 23.Steyerberg EW, Borsboom GJ, van Houwelingen HC, Eijkemans MJ, Habbema JD. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med. 2004;23(16):2567-2586. [DOI] [PubMed] [Google Scholar]
- 24.Psaty BM, Delaney JA, Arnold AM, et al. . Study of cardiovascular health outcomes in the era of claims data: The Cardiovascular Health Study. Circulation. 2016;133(2):156-164. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.White AD, Folsom AR, Chambless LE, et al. . Community surveillance of coronary heart disease in the Atherosclerosis Risk in Communities (ARIC) Study: methods and initial two years’ experience. J Clin Epidemiol. 1996;49(2):223-233. [DOI] [PubMed] [Google Scholar]
- 26.Heckbert SR, Kooperberg C, Safford MM, et al. . Comparison of self-report, hospital discharge codes, and adjudication of cardiovascular events in the Women’s Health Initiative. Am J Epidemiol. 2004;160(12):1152-1158. [DOI] [PubMed] [Google Scholar]
- 27.Yasaitis LC, Berkman LF, Chandra A. Comparison of self-reported and Medicare claims-identified acute myocardial infarction. Circulation. 2015;131(17):1477-1485. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Ness RB; Joint Policy Committee, Societies of Epidemiology . Influence of the HIPAA Privacy Rule on health research. JAMA. 2007;298(18):2164-2170. [DOI] [PubMed] [Google Scholar]
- 29.Houser SH, Howard VJ, Hovater MK, Safford MM. Obtaining medical records from healthcare facilities under the HIPAA Privacy Rule: the experience of a national longitudinal cohort study. Neuroepidemiology. 2007;28(3):162-168. [DOI] [PubMed] [Google Scholar]
- 30.Colantonio LD, Richman JS, Carson AP, et al. . Performance of the atherosclerotic cardiovascular disease Pooled Cohort risk equations by social deprivation status. J Am Heart Assoc. 2017;6(3):e005676. doi: 10.1161/JAHA.117.005676 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Yadlowsky S, Hayward RA, Sussman JB, et al. . Clinical implications of revised pooled cohort equations for estimating atherosclerotic cardiovascular disease risk [published online June 5, 2018]. Ann Intern Med. doi: 10.7326/M17-3011. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplementary Methods
eTable S1a. Race/ethnicity distribution and age distribution before and after weighting
eTable S1b. Baseline cardiovascular risk profile characteristics for WHI participants in the risk validation or treatment eligibility subcohorts
eTable S2. Predicted and observed risks excluding NDI events, before and after weighting to the overall WHI cohort
eTable S3. Unweighted and weighted predicted and observed risks before and after adjusting for statin and aspirin use during follow-up, in all women and after excluding women in the active hormone arm
eTable S4a. 10-year predicted and observed risks in women 65 years and older before and after including CMS events
eTable S4b. 10-year predicted and observed risks in women 65 years and older who were enrolled in the placebo arm of the hormone therapy clinical trial, after including CMS events
eTable S5. Treatment recommendations based on the 2013 ACC/AHA cholesterol guidelines according to racial/ethnic groups in WHI women ages 50 to 75 years old and with complete data on the risk equations
Supplementary Figure Legends
eReferences
eFigure S1. Study flow diagram
eFigure S2. Derivation of the current analytical sample
eFigure S3. Statin and aspirin use at baseline and during follow-up by baseline risk category
eFigure S4. Predicted and observed risks in women 65 years and older before and after adjusting for statin use, aspirin use, and CMS events, according to race/ethnicity
List of WHI investigators