Developing an Internally Validated Veterans Affairs Women Cardiovascular Disease Risk Score Using Veterans Affairs National Electronic Health Records

Abstract

Background

The current American College of Cardiology/American Heart Association women cardiovascular disease (CVD) risk score suboptimally estimates CVD risk for young and minority women in the military. The current study developed an internally validated CVD risk score for women military service members and veterans using the Veterans Affairs (VA) national electronic health records data.

Methods and Results

The study cohort included 69 574 White, Black, and Hispanic women service members and veterans aged 30 to 79 years in 2007 treated in the VA Health Care System between January 1, 2007 and December 31, 2017 (henceforth, VA women). Stratified by race and ethnicity, the new VA women CVD risk model estimated risk coefficients and 10‐year CVD risk using a time‐variant covariate Cox model. Harrell C‐statistics, calibration plots, and net classification index were used to assess accuracy and prognostic performance of the new VA women CVD risk model. The new internally validated VA women CVD risk score performed better in predicting VA women 10‐year atherosclerosis cardiovascular disease risk than the pooled cohort American College of Cardiology/American Heart Association risk score in both accuracy (White Harrell C‐statistics, 70% versus 61%; Black, 68% versus 63%) and prognostic performance (White net classification index, 0.31; 95% CI, 0.26–0.33; Black net classification index, 0.06; 95% CI, 0.03–0.09).

Conclusions

The proposed VA women CVD risk score improves accuracy of the existing American College of Cardiology/American Heart Association CVD risk assessment tool in predicting long‐term CVD risk for VA women, particularly in young and racial/ethnic minority women.

Nonstandard Abbreviations and Acronyms

ACC/AHA

American College of Cardiology/American Heart Association

DBP

diastolic blood pressure

DM

diabetes mellitus

NRI

net reclassification index

SBP

systolic blood pressure

VA

Veterans Affairs

Clinical Perspective

What Is New?

• This study developed a new validated risk score to assess cardiovascular disease risk for the women veteran population.

What Are the Clinical Implications?

• This new risk score tool predicts long‐term cardiovascular disease risk better than the existing American College of Cardiology/American Heart Association cardiovascular disease risk score for women veterans.

• The new risk score tool is especially valuable for young women veterans aged under 40 years who are not accounted for by the existing American College of Cardiology/American Heart Association cardiovascular disease risk score tool and those of racial and ethnic minorities.

Despite increased awareness of cardiovascular disease (CVD) risk among women, CVD still remains the leading cause of death among women in the United States. ¹ , ² The burden of CVD in women varies by demographic characteristics, of which veteran status disproportionally increases the burden of CVD. ³ Although the proportion of women in the military services has increased to 15% of the military population in 2018, ⁴ women are still underrepresented in veteran CVD research and care, albeit being a population at risk. ⁵

Women in the military have more traditional CVD risk factors, such as hypertension, hyperlipidemia, diabetes mellitus (DM), and cigarette smoking, than the general population. ⁵ , ⁶ Additionally, military women have higher prevalence of nontraditional CVD risk factors such as major depression than their civilian peers ⁷ and male counterparts. ⁸ , ⁹

The widely used American College of Cardiology/American Heart Association (ACC/AHA) atherosclerosis cardiovascular disease (ASCVD) model was based on data from women aged 40 to 79 years; however, it did not include young women aged under 40 years. A previous study, ¹⁰ wherein we rederived (ie, recalibrated) the ACC/AHA ASCVD model using data from women service members, showed that the ACC/AHA models underestimated 10‐year ASCVD risk in Veterans Affairs (VA) women, especially in younger VA women aged under 40 years. The accuracy and fit of the rederived ACC/AHA model fell short of being acceptable, with Harrell C‐statistics of 0.61 and 0.63 in White and Black VA women, respectively. Furthermore, risk factors for ASCVD events change with aging, which are referred to as time‐variant covariates. The current ACC/AHA risk assessment tool does not take into consideration the aging trajectory of risk factors over the 10 years and its effects on CVD events. We aimed to incorporate effects of time‐variant risk factors in predicting 10‐year CVD risk for women in the military.

The current study developed a new internally validated CVD risk score for women in the military and veterans using VA electronic health records (EHR) data. Capitalizing on computerized, longitudinal, and integrated VA national EHR data, the new risk score was calculated by applying a time‐variant covariate Cox regression model to VA women EHR data. The VA women EHR data contained all available person‐level visits during the study period from non‐Hispanic White, non‐Hispanic Black, and Hispanic VA women. Stratified by race/ethnicity, the new CVD score calculates 10‐year risk of the first incidence of CVD events, including nonfatal heart failure and cardiac arrest, in addition to ASCVD events such as nonfatal myocardial infarction (MI), nonfatal stroke, and cardiac death in female military service members and veterans.

Methods

VA Women Cohort

The outpatient and inpatient visit records of women treated at the VA Health Care System between January 1, 2007 and December 31, 2017 were extracted from VA national EHR data. The study selected non‐Hispanic White, non‐Hispanic Black, and Hispanic VA women who were aged 30 to 79 years and alive on January 1, 2007 and had no known prior CVD events as a cohort. The study cohort was then refined to VA women who had complete data on blood pressure measurements and lipid panel tests at baseline visits.

For VA women who were deceased during the study period, death incidences were verified, and cause of death data were obtained separately from the Centers for Disease Control and Prevention National Death Index to define cardiac death. ¹¹ Figure 1 depicts procedures of inclusion and exclusion of the study data, yielding a final sample size of 69 574 female service members and veterans: non‐Hispanic White, 36 172 (52%); non‐Hispanic Black, 29 231 (42%); and Hispanic, 4171 (6%). This closely represents the current distribution of the racial and ethnic minority backgrounds of women in the military. ⁴

Figure 1. Procedures of inclusion and exclusion to create a study cohort.

VA indicates Veterans Affairs.

The study used a 2‐step procedure to construct longitudinal data of the VA women cohort with all available person‐level visits for application of a time‐variant covariate Cox model.

First, the study constructed balanced longitudinal data with an equal number of visits for all VA women during the entire study period of January 1, 2007 to December 31, 2017. The first step of creating the balanced data is necessary to solve inherent problems using EHR data, such as unequal visit‐to‐visit intervals and different dates of lipid panel orders from dates of tests and results. The study defined a visit interval as semiannual (6 months), considering a minimum interval length to capture time variance in factors such as refills of antihypertensive medications.

Each record then represented a visit with a semiannual interval. For example, the baseline visit was defined as the first 6‐month interval, January 1, 2007 to June 30, 2007, and the next available semiannual outpatient or inpatient visit as the next visit, and so forth. Every woman in the cohort had the same number of semiannual visits. If there were multiple visit records with the semiannual visit, the visit records were then consolidated into one data point per semiannual visit. Multiple visit records of dichotomized risk factors, such as DM, current smoking status, antihypertensive medication intake, and any CVD events, were consolidated into a maximum value of 1 (yes) and 0 (no). For continuous risk factors such as systolic blood pressure (SBP), total cholesterol, and high‐density lipoprotein cholesterol (HDL‐C), the average values of the factors over the multiple visit records were selected as a visit data point. If there were no visit records within the semiannual visit, the semiannual visit had missing data on risk factors. The balanced data set contained all patients with the same equal‐length follow‐up period of January 1, 2007 to December 31, 2017, with some missing data when there were no visit records. When the cohort patient died from diseases or causes other than cardiovascular diseases, the follow‐up data of the patient were left as missing (ie, censored).

Second, the balanced longitudinal data were reconstructed to fit the time‐variant covariate Cox model estimation. The second step created time intervals between 2 consecutive nonmissing semiannual visits per patient. The number and length of the time intervals for each patient reflected the number of available semiannual visits and their durations. The first time interval was set as the baseline. The current study assumed no loss to follow‐up; thus, the last time interval was set to be either a semiannual visit of death occurrence, or the last semiannual visit of the study period.

Variable Construct

CVD events, DM, and major depression study variables were constructed using International Classification of Diseases (ICD), Ninth Revision (ICD‐9) and Tenth Revision (ICD‐10) diagnosis and procedure codes extracted from the VA EHR data. The ICD codes to define each variable were validated using available documentations from the VA Phenomics Library. The VA Phenomics Library compiled VA EHR‐based phenotype algorithms including ICD codes to define diseases. Chen et al. ¹⁰ described detailed ICD codes used to construct each risk factor included in the CVD risk score calculation and validation of ICD codes used to construct CVD events with providers' notes. Over 96% and 92% of MI and stroke events, respectively, in the current VA women cohort data were validated with providers' notes. ¹⁰ , ¹²

Model Estimation

Stratified by race and ethnicity, the new VA women CVD risk model estimated risk coefficients and 10‐year CVD risk using a time‐variant covariate Cox model. The factors included in the model were age, untreated and treated SBP (in millimeters of mercury), presence of DM (yes versus no), current smoking status (yes versus no), total cholesterol (in milligrams per deciliter), HDL‐C (in milligrams per deciliter), and presence of major depressive episodes/diagnosis (yes versus no). Treated SBP was defined as an interaction between on antihypertensive medication and SBP. Data on factors were repeatedly measured from multiple visits during the study period. Age, SBP, total cholesterol, and HDL‐C were natural log transformed to follow log normal distributions. Each VA woman had CVD risk factors measured repeatedly from multiple visits during the study period. The time‐variant covariate Cox model assumes that effects of time‐varying factors on CVD risk follow step functions.

The study tested the new model's accuracy and performance in predicting the first incidence of CVD event and its validity. Model accuracy in discriminating CVD events from no events was examined using time‐dependent Harrell C‐statistics over 10 years, and model reliability was examined by calibration plots. Harrell C‐statistics for the time‐variant covariate Cox model was presented at each specific point of time along with mean and standard error. The C‐statistic measures concordance of the model, $\frac{d+1}{2}$, where d represents Somers d. ¹³ C‐statistics ≥0.7 represent good model discrimination, and a 45° line in a calibration plot represents perfect agreement between predicted and observed CVD event probabilities.

Internal validation of the model predictability was conducted to evaluate the stability of the new CVD risk model coefficient estimates by applying 2 internal validation methods, bootstrap and cross‐validation. The study used a bootstrap method to resample from the VA women cohort data with replacement and same sample size. We repeated this 100 times and averaged all estimates (bias and standard error deviation). The study also conducted a 10‐fold cross‐validation. This method randomly draws 90% of VA women from the cohort for model development (training data and 10% for a model validation), testing data, and to repeat this procedure 10 times and for average estimates. The optimism of the performance (value of C‐statistics bias) was used to evaluate internal and cross‐validation. Ideally, optimism should be close to 0; thus, optimism‐adjusted C‐statistics (C‐statistics optimism) should be identical with the original C‐statistics.

The study used the net reclassification index (NRI), closely following Pencina et al., ¹⁴ to compare the new VA women CVD risk model against both the rederived ACC/AHA women model using VA women data and the original general population pooled cohort ACC/AHA model in predicting accuracy of 10‐year ASCVD risk. ¹⁰ The ASCVD risk was classified into low, moderate, and high, and corresponding risk ranges were <7.5%, 7.5% to 19.9%, and 20% and higher, respectively. ¹⁵ Upward reclassification represents raising a grade of ASCVD risk, for example, from low risk reclassified to moderate or from low or moderate risk reclassified to high risk, whereas downward reclassification represents lowering a ASCVD risk grade. The NRI summarized a net reclassification improvement in prognostic performance of the VA women CVD risk model to the original pooled cohort ASCVD score by calculating the number of the cohort correctly reclassified. The NRI is a sum of the NRI for those who experienced ASCVD events (NRI_event) and those with no events (NRI_{no event}). NRI_event is calculated as P (predicted risk estimated upward/ASCVD event)−P (predicted risk estimated downward/ASCVD event) and NRI_{no event} as P (predicted risk estimated downward/no event)−P (predicted risk estimated upward/no event). ¹⁴ Means and 95% CIs of the NRI were calculated using a bootstrap method (resamples n=100). Absolute NRI was also calculated based actual numbers of patients reclassified upward and downward stratified by event and no‐event group for each race and ethnic group. ¹⁶ A positive value of NRI indicates improvement of risk prediction performance.

The study has been approved by both the VA North Texas Health Care System Institutional Review Board and University of Texas Southwestern Medical Center Institutional Review Board. Obtaining informed consent from subjects was waived.

Because of the sensitive nature of the data collected for this study, requests to access the data set are limited to qualified VA‐affiliated researchers trained in human subject confidentiality. Protocols may be sent to the VA North Texas Health Care System Institutional Review Board at NTXIRBAdmin@va.gov. Structured query language, SAS (SAS Institute, Cary, NC), and R (R Foundation for Statistical Computing, Vienna, Austria) programming code used in the analysis of this study are available from the corresponding author upon reasonable request. Additional methods can be found in Data S1.

Results

The current VA women study cohort included over 40% minority women and 30% young women aged under 40 years. Table 1 depicts the VA women cohort used to develop the new CVD risk score for women in the military and veterans. Race and ethnic groups in the cohort were significantly different from each other in baseline CVD risk factors. On average, non‐Hispanic White VA women were older than minority VA women at baseline, more likely to be current smokers, and had significantly higher total cholesterol values (P<0.01). Black VA women had significantly higher baseline SBP and HDL‐C values (P<0.01), and were more likely to present with DM than White and Hispanic VA women (P<0.01). Prevalence of major depression was highest among Hispanic VA women, followed by White and Black VA women (P<0.01).

Table 1.

Baseline Risk Factors Stratified by Race and Ethnic Group (Total n=69 574)^*

	White, n=36 172, 52%	Black, n=29 231, 42%	Hispanic, n=4171, 6%
Age, mean±SD, y †	47.27±8.71	45.49±7.87	44.64±8.54
SBP, mean±SD, mm Hg ‡	124.69±14.78	128.02±15.57	123.39±14.36
Diabetes mellitus, n (%) §	8253 (22.82%)	9396 (32.14%)	1043 (25.01%)
Current smoking, n (%) ||	10 846 (29.98%)	5106 (17.47%)	990 (23.74%)
Major depression, n (%) #	19 190 (47.98%)	13 771 (43.05%)	2269 (49.60%)
Total cholesterol, mean±SD, mg/dL ||	198.63±41.50	192.09±39.66	195.47±38.63
HDL‐C, mean±SD, mg/dL ‡	53.48±16.80	56.69±17.73	53.85±15.56

HDL‐C indicates high‐density lipoprotein cholesterol; SBP, systolic blood pressure; and SD, Standard deviation.

^*Chi‐squared statistics was used to describe race and ethnic groups association with categorical covariates, and post‐hoc pairwise Tukey tests were used to compare means in the continuous covariates such as age, SBP, total cholesterol, and HDL‐C levels at baseline when overall group differences were statistically significant using Analysis of Variance (ANOVA) test. The current study found that age, SBP, total cholesterol and HDL significantly differ across all three race and ethnic groups (P<0.0001).

^†White > Black ≥ Hispanic.

^‡Black > White ≈Hispanic.

^§Black > Hispanic > White.

^||White > Hispanic > Black.

^#Hispanic > White > Black.

Of all 69 574 VA women, 2176 died during the study period. The incidence of any ASCVD events, nonfatal MI, nonfatal stroke, and cardiac death among the VA women cohort was 5.3/1000 person‐years, whereas any CVD events, ASCVD events plus nonfatal heart failure, and nonfatal cardiac arrest was 5.8/1000 person‐years. The most common CVD event among VA women was nonfatal MI (4.1/1000 person‐years), followed by nonfatal stroke (1.7/1000 person‐years) with a significant racial difference in incidence (Table 2; P<0.01). Black VA women had by far a higher incidence of nonfatal stroke events than other races and ethnicities (Table 2).

Table 2.

Cardiovascular Events by Race/Ethnicity: Number and Incidence Per 1000 Person‐Years

ASCVD	White†	Black†	Hispanic†
Nonfatal myocardial infarction	1515 (4.3)	1148 (4.0)	148 (3.6)
Nonfatal stroke	538 (1.5)	592 (2.1)‡	61 (1.5)
Cardiac death*	245 (0.7)	144 (0.5)	16 (0.4)
Heart failure	175 (0.5)	151 (0.5)	18 (0.4)
Cardiac arrest	9 (0.025)	6 (0.02)	2 (0.048)

The same patient can experience multiple cardiovascular disease events. ASCVD indicates atherosclerosis cardiovascular disease.

^*Validated by National Death Index data.

^†Incidence is presented in parentheses. The incidence is per 1000 person‐years on the basis of new cases during the study period (10‐year follow‐up for all alive in the study cohort and 5‐year follow‐up for those deceased). We assumed no loss to follow‐up except death.

^‡Non‐Hispanic Black women > White and Hispanic women, P=0.04.

Tables 3 and 4 depict estimated risk coefficients of the new VA women ASCVD and composite CVD risk models, respectively, stratified by non‐Hispanic White, non‐Hispanic Black, and Hispanic VA women.

Table 3.

Risk Coefficient Estimates of Cox Time‐Variant Model Using Veterans Affairs Women Data: Nonfatal Myocardial Infarction, Nonfatal Stroke, and Cardiac Death Events

	Non‐Hispanic White		Black		Hispanic
	Estimate	SE	Estimate	SE	Estimate	SE
Ln age	2.399	0.114	2.058	0.137	2.191	0.385
Untreated SBP	1.008	0.233	0.411	0.326	0.653	0.814
Treated SBP	−0.208	0.318	1.246	0.391	−3.714	1.196
Diabetes mellitus	0.425	0.042	0.276	0.047	0.315	0.202
Current smoking	0.072	0.038	−0.020	0.048	0.356	0.196
Major depression	0.244	0.045	0.231	0.076	0.311	0.150
Ln total cholesterol	0.024	0.086	0.180	0.104	0.099	0.321
Ln HDL‐C	−1.350	0.064	−1.339	0.076	−1.225	0.245
Antihypertensive treatment	1.263	1.544	−5.795	1.90	18.290	5.771
Average C‐statistics	0.700	0.009	0.680	0.01	0.660	0.033

HDL indicates high‐density lipoprotein cholesterol; Ln, natural log transformed; and SBP, systolic blood pressure.

Table 4.

Risk Coefficient Estimates of Cox Time‐Variant Model Using Veterans Affairs Women Data: Nonfatal Myocardial Infarction, Nonfatal Stroke, Nonfatal Heart Failure, Cardiac Arrest, and Cardiac Death Event

	Non‐Hispanic White		Non‐Hispanic Black		Hispanic
	Estimate	SE	Estimate	SE	Estimate	SE
Ln age	2.493	0.110	2.074	0.131	2.291	0.366
Untreated SBP	1.018	0.228	0.526	0.317	0.812	0.778
Treated SBP	0.463	0.305	1.664	0.375	−1.391	1.131
Diabetes mellitus	0.457	0.040	0.350	0.045	0.301	0.141
Current smoking	0.073	0.037	−0.004	0.045	0.413	0.130
Major depression	0.254	0.044	0.282	0.052	0.393	0.140
Ln total cholesterol	0.167	0.082	0.096	0.099	0.107	0.303
Ln HDL‐C	−1.295	0.062	−1.276	0.073	−1.208	0.231
Antihypertensive treatment	−1.966	1.481	−7.698	0.375	7.061	5.477
Average C‐statistics	0.710	0.008	0.683	0.009	0.671	0.030

HDL‐C indicates high‐density lipoprotein cholesterol; Ln, natural log transformed; and SBP, systolic blood pressure.

Harrell C‐statistics for the non‐Hispanic White women ASCVD risk model ranged between 0.7 and 0.8 over 10 years (Figure S1A), with an average value of 0.70 and a standard error of 0.009. The non‐Hispanic Black women model C‐statistics ranged from 0.67 to 0.74 (Figure S1B), with average C‐statistics of 0.68 (standard error, 0.010), and the Hispanic women model ranged from 0.62 to 0.88 (Figure S1C), with average C‐statistics of 0.66 (standard error, 0.033). Figure 2A shows calibration plots and the plots approximated along a 45° line across all race and ethnic groups.

Figure 2. Calibration plots for Veterans Affairs women cardiovascular disease risk model by race/ethnicity.

A, ASCVD events. B, Composite CVD events. *The vertical axis represents observed ASCVD event probabilities, and the horizontal axis represents predicted CVD event probabilities. ^†A 45° line in the calibration plot represents perfect agreement between predicted and observed CVD event probabilities. ASCVD indicates atherosclerosis cardiovascular disease; and CVD, cardiovascular disease.

Heart failure and cardiac arrest events were added to create a composite CVD event including the first event of any of nonfatal MI, nonfatal stroke, nonfatal heart failure, cardiac arrest, and cardiac death, and estimated 10‐year composite CVD risk, stratified by non‐Hispanic White, non‐Hispanic Black, and Hispanic VA women (Table 4). Mean values of Harrell C‐statistics of the VA women composite CVD risk model were 0.71 (range, 0.71–0.79; standard error, 0.008), 0.68 (range, 0.68–0.75; standard error, 0.009), and 0.67 (range, 0.64–0.89; standard error, 0.030) in non‐Hispanic White, non‐Hispanic Black, and Hispanic VA women, respectively (Figures S2 and S3). The calibration plots of the VA women composite CVD risk model are depicted in Figure 2B, and plots are approximated along the 45° line across all race and ethnic groups.

Table 5 presents calculated 10‐year ASCVD and composite CVD risk scores given values of risk factors: total cholesterol 213 mg/dL, HDL 50 mg/dL, SBP 120 mm Hg, no DM, no current smoking status, and no major depression. The new VA women ASCVD risk model estimated 10‐year ASCVD risk as 3.6%, 4.4%, and 3.5%, in non‐Hispanic White, non‐Hispanic Black, and Hispanic VA women aged 38 years, respectively. At age 55 years, the estimated ASCVD risk were 8.4%, 9.2%, and 7.7%, respectively. If major depression was present, the ASCVD risk increased by about 2% (range, 1.3%–2.7%) across all race and ethnicity groups (Table 5).

Table 5.

Ten‐Year Cardiovascular Disease Event Risk in White, Black, and Hispanic Veterans Affairs Women by Race/Ethnicity

	White	Black	Hispanic
Age
S(10)*	0.9438	0.9442	0.9542
Age 38 y
ASCVD risk (%) †	3.6% (4.5%*)	4.4% (5.7%*)	3.5% (4.8%*)
Composite CVD risk (%) ‡	3.8% (6.0%*)	4.9% (6.8%*)	4.6% (5.6%*)
Age 55 y
ASCVD risk (%) †	8.4% (10.6%)	9.2% (11.4%)	7.7% (10.4%)
Composite CVD risk (%) ‡	9.3% (11.9%*)	10.2% (13.4%*)	8.6% (12.5%*)

ASCVD indicates atherosclerosis cardiovascular disease; and CVD, cardiovascular disease.

^*S (.) is the survival probability function and S(10) is the 10‐year CVD event‐free survival probability. Risk was calculated following $1‐\mathrm{S}{(10)}^{{\mathrm{e}}^{\left(x\beta ‐\overline{x}\beta \right)}}$, where x a vector of covariates in the model and $\overline{x}$ is the mean value of corresponding covariates, and β is a vector of risk coefficients corresponding covariates, x, at age 38 and 55 years. Specific values of x are total cholesterol 213 mg/dL, high‐density lipoprotein 50 mg/dL, systolic blood pressure 120 mm Hg, no diabetes mellitus, no current smoking status, and no major depression. In parenthesis is the risk when major depressive symptoms are present.

^†ASCVD events are the first incidences of any events of nonfatal myocardial infarction, nonfatal stroke, and cardiac death.

^‡Composite CVD events are the first incidences of any events of nonfatal myocardial infarction, nonfatal stroke, heart failure, cardiac arrest, and cardiac death.

The new VA women composite CVD risk model estimated risk of having any CVD events within 10 years as 3.8%, 4.9%, and 4.6% at age 38 years in non‐Hispanic White, non‐Hispanic Black, and Hispanic VA women, respectively. The estimated composite CVD risk of VA women aged 55 years was 9.3%, 10.2%, and 8.6%, in non‐Hispanic White, non‐Hispanic Black, and Hispanic VA women, respectively. When a VA woman presented with a major depression diagnosis, her risk of having CVD events in 10 years increased by 1.5% on average, with a range between 1% and 2.2% at age 38 years, and over 3% on average (range, 2.6%–3.9%) at age 55 years, across all race and ethnicity groups (Table 5).

Internal and Cross‐Validation

Table 6 shows internal validation of the new VA women ASCVD risk models. The bias and standard error deviation are presented for all risk factors. The optimism of the performance and values of C‐statistics bias were 0.009 for both White and Black VA women models and 0.0008 for Hispanic VA women (Table 6). The optimism‐adjusted C‐statistics are almost identical to the original C‐statistics of the models; thus, the new VA women CVD risk models were concluded to be internally validated.

Table 6.

Internal Validation of Veterans Affairs Women Atherosclerosis Cardiovascular Disease Risk Model*

Variable	White Women		Black Women		Hispanic Women
	Bias	RMSE	Bias	RMSE	Bias	RMSE
Ln age	0.01102	0.11540	−0.00132	0.15233	0.01987	0.42692
Ln untreated SBP	0.00718	0.25151	0.03687	0.35780	0.08465	0.84077
DM	0.00279	0.04409	0.00021	0.04982	−0.02337	0.15055
Current smoker	0.00080	0.03728	0.00133	0.04505	−0.01952	0.14192
Ln total cholesterol	0.00002	0.09113	0.02667	0.11835	0.00591	0.37230
Ln HDL‐C	0.00486	0.06958	0.00198	0.08548	0.00787	0.27818
Major depression	−0.00913	0.04576	−0.00121	0.05187	0.00934	0.15420
SBP treatment	−0.14845	1.65215	0.18876	2.27947	0.75390	7.07361
Treated Ln SBP	0.02994	0.33894	−0.03828	0.46873	−0.15628	1.47123
C‐statistics	0.00090	0.00541	0.00093	0.00669	0.00806	0.01998

DM indicates diabetes mellitus; HDL‐C, high‐density lipoprotein cholesterol; Ln, natural log transformed; RMSE, root mean squared error; and SBP, systolic blood pressure.

^*On the basis of 100 times bootstrap.

Table 7 shows cross‐validation results. The study conducted 10‐fold cross‐validation. The optimism of the performance and values of C‐statistics bias were −0.004, −0.008, and −0.063 for the White, Black, and Hispanic VA women models, respectively (Table 7). The optimism‐adjusted C‐statistics are almost identical with the VA women CVD risk models, which indicates the current model estimation was not sensitive to outliers of the study cohort data, particularly the first and last tertiles. Thus, the VA women CVD risk model was cross‐validated.

Table 7.

Cross‐Validation (10‐Fold) of Veterans Affairs Women Atherosclerosis Cardiovascular Disease Risk Model

Variable	White Women		Black Women		Hispanic Women
	Bias	RMSE	Bias	RMSE	Bias	RMSE
Ln age	−0.05585	0.34578	−0.02178	0.46978	−0.17092	1.42376
Ln SBP	0.13784	0.69663	0.19792	1.14355	−0.04432	2.83077
DM	−0.01839	0.15176	0.00644	0.15451	0.05639	0.47152
Current smoker	−0.01445	0.12700	−0.01148	0.15123	0.03914	0.44957
Ln total cholesterol	−0.01848	0.24347	0.05901	0.38637	−0.17278	0.89220
Ln HDL‐C	−0.02389	0.22465	−0.00344	0.23311	−0.06432	0.80753
Major depression	0.02003	0.13724	0.00109	0.17042	−0.08892	0.44108
SBP treatment	0.72086	5.06492	0.93181	6.74508	−0.48571	21.84194
Treated SBP	−0.14869	1.04515	−0.19271	1.38398	0.11053	4.52849
C‐statistics	−0.00369	0.01569	−0.00782	0.01769	−0.06293	0.05579

DM indicates diabetes mellitus; HDL‐C, high‐density lipoprotein cholesterol; Ln, natural log transformed; RMSE, root mean squared error; and SBP, systolic blood pressure.

Net Reclassification Index

The NRI of the new VA women ASCVD risk model compared with the original pooled cohort ACC/AHA model ¹⁷ was positive at 0.31 (95% CI, 0.26–0.33) and 0.06 (95% CI, 0.03–0.09) in White and Black VA women, respectively. Absolute NRI values were positive in both White and Black VA women at 0.04 (White: ${\text{NRI}}_{\text{event}}^{\mathrm{W}}$ 0.15, ${\text{NRI}}_{\text{no event}}^{\mathrm{W}}$ −0.11), and 0.025 (Black: ${\text{NRI}}_{\text{event}}^{\mathrm{B}}$ 0.026, ${\text{NRI}}_{\text{no event}}^{\mathrm{B}}$ −0.001), respectively (Table 8).

Table 8.

Net Reclassification Index of VA Women ASCVD Risk Model Compared With the ACC/AHA ASCVD Risk Model Using VA Women Cohort Data

ACC/AHA Model		VA Women ASCVD Risk Model
		<7.5%	7.5%–19.9%	≥20%
White	<7.5%	19 572	6962	3763
	7.5%–19.9%	50	3108	2636
	≥20%	0	0	80
Black	<7.5%	20 185	1485	0
	7.5%–19.9%	4531	2828	147
	≥20%	46	0	9

The study used the net reclassification index closely following Pencina et al. ¹⁴ ACC/AHA indicates American College of Cardiology/American Heart Association; ASCVD, atherosclerosis cardiovascular disease; and VA, Veterans Affairs.

Shaded value indicated a diagonal line, where NRIs are concordant.

Discussion

This study capitalized on a large‐scale, longitudinal VA EHR database and developed an internally validated CVD risk model tailored to the unique demographics and risk factor profile in female service members and veterans. The current VA women study cohort, composed of 48% racial and ethnic minority women, represents the ethnic distribution of women in the military, which makes the proposed VA women CVD risk score adequate and reliable to assess a long‐term CVD risk for these women. The proposed VA women CVD risk score included multiple traditional CVD risk factors, including age, SBP, DM, current smoking, total cholesterol, and HDL‐C, and a new nontraditional risk factor, major depression. The VA women CVD risk model better performs in predicting 10‐year ASCVD risk for VA women than the pooled cohort ACC/AHA model according to both Harrell C‐ statistics and the NRI. Accuracy of the proposed VA women CVD risk score in predicting 10‐year ASCVD risk of VA women was substantially improved (Harrell C‐statistics White, 0.70; Black, 0.68) compared with the ACC/AHA ASCVD model ¹⁰ (Harrell C‐statistics, White 0.61; Black, 0.63). The positive NRI values also support that the new VA women CVD risk model has a better prognostic performance in predicting 10‐year ASCVD risk of VA women than the pool cohort ASCVD score.

The study uses all available VA EHR visit data on CVD risk factors and events in calculating predictive CVD risk score for women in the military and veterans by applying a time‐variant Cox regression model. This VA women CVD risk score is the first score that applies a time‐variant covariate Cox model to estimate effects of age‐related changes in CVD risk factors on 10‐year CVD risk using the VA women cohort data.

With a 3‐fold increase in number of active‐duty female service members between 1973 and 2010, ¹⁸ female service members constituted ≈16% of active‐duty service members in 2018. ¹⁹ As a result, an increased demand for CVD care for younger female service members is highly anticipated. This proposed VA women CVD risk score, derived from the cohort including a substantial number of younger military women, provides accurate and reliable 10‐year CVD risk score for younger military women that can be used as part of the assessment of CVD risk in younger women who have served or are currently serving in the military.

The new CVD risk score includes a more comprehensive list of CVD events such as heart failure and cardiac arrest in addition to ASCVD events. With more types of CVD events, heart failure, and cardiac arrest, the model accuracy in discriminating CVD events has improved across all race and ethnic groups while still remaining reliable (Figure 2, a 45° approximation of calibration plots).

The newly proposed VA women CVD risk score can improve potential under‐ or overtreatment of VA women by more accurately calculating 10‐year CVD risk (NRI>0). Using the ACC/AHA ASCVD risk calculator derived from general population pooled cohort data, ¹⁷ a 55‐year‐old woman with total cholesterol of 213 mg/dL, HDL‐C 50 mg/dL, untreated SBP 120 mm Hg, no DM, no current smoking status, and no major depression would have low (<5%) 10‐year ASCVD risk (2.1% in White women, 3.0% in Black women ¹⁷ ). Thus, she would not be considered for cholesterol‐lowering treatment despite elevated cholesterol level on the basis of the 2018 cholesterol management and treatment guideline. ¹⁵ On the contrary, the proposed VA women CVD score calculates that the same 55‐year‐old woman would have moderate risk (>7.5%) of ASCVD (White, 8.4%; Black, 9.2%) and should be considered for cholesterol‐lowering therapy.

The current model is parsimonious but powerful, with good accuracy and reliability in its prediction of long‐term CVD risk among female service members and veterans. Despite the current literature on many CVD biomarkers, overfitting the predictive model with inclusion of biomarkers generates a model that is vulnerable to type I error as well as difficult to implement in the general clinical settings. Ultimately, body mass index (BMI), diastolic blood pressure (DBP), CVD biomarkers, menopause status and symptoms, and pregnancy‐related hypertension or preeclampsia were excluded from the model on the basis of multiple criteria, specifically Akaike information criteria, time‐dependent Harrell C‐statistics, calibration plots, and statistical significance. Inclusion of BMI and menopausal status as independent risk factors increased discrimination of the model but negatively associated with increased CVD risk, which was unexpected. This suggested multicollinearity with other traditional CVD risk factors such as cholesterol and SBP. To prevent multicollinearity problems, inclusion of BMI and menopausal status would have required omitting one of the traditional CVD factors. However, inclusion of blood cholesterol in the risk prediction model resulted in better discrimination and predictability than inclusion of BMI without consideration for blood cholesterol levels. In addition, BMI data from the VA EHR were prone to data entry errors in weight and height records. Values of BMI were sensitive to different data‐cleaning methods of weights and heights despite large‐scale data. Thus, the current VA women CVD risk score included cholesterol risk factors but omitted BMI.

Inclusion of CVD biomarkers, such as hemoglobin A1C, ²⁰ troponin, ²¹ , ²² , ²³ , ²⁴ and fibrinogen, ²⁵ , ²⁶ in the final CVD risk model was not feasible because of the very small proportion of the VA women cohort with complete data on CVD biomarkers. Other biomarkers, such as ankle‐brachial index, ²⁷ high‐sensitivity C‐reactive protein, ²⁸ and coronary artery calcium ²⁸ were not included in the model, which is supported by US Preventative Taskforce recommendations. ²⁹ However, exclusion of CVD biomarkers from the final model may account for unexpected signs of current cigarette smoking risk estimates in the Black VA women model, albeit not significant (P value close to 0.9). For example, high‐sensitivity C‐reactive protein level was higher among current cigarette smokers than former smokers ³⁰ , ³¹ and was also found to be different by race and ethnicity among women. ³² In addition, these additional nontraditional risk markers are not routinely available on most patients.

A history of pregnancy complication, as a potential independent CVD risk factor, ³³ , ³⁴ was not included in the final model partly because of the low proportion (<10%) of the VA women cohort with pregnancy complication history from the VA EHR records. Exclusion of a history of pregnancy complication is also supported in the previous literature. ³⁵ , ³⁶

Overall, the Black women model performance with inclusion of DBP was poorer compared with the current study model with SBP (increased Akaike information criteria by 57.48) despite findings in previous studies. ³⁷ , ³⁸ Although inclusion of DBP in place of SBP showed similar model performance in C‐statistics and Akaike information criteria for White and Hispanic women, risk coefficients of untreated and treated DBP were unexpectedly negative (non‐Hispanic White, −0.276 in ASCVD and −0.723 in composite CVD models; Hispanic, −1.049 in ASCVD and −2.906 in composite CVD models), thus suggesting SBP is a better predictor than DBP.

The new VA women CVD risk score included major depression as an independent risk factors for CVD. Despite recent evidence supporting major depression as an independent CVD risk factor, ⁹ , ³⁹ , ⁴⁰ and American Heart Association's CVD treatment guidelines and recommendations for screening and management of depressive symptoms, ⁴¹ many have cautioned a causal inference of depression on CVD ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ drawn from application of naïve models to cross‐sectional data. The current study has addressed this by using longitudinal VA EHR data with an application of a time‐variant covariate Cox model. ⁴⁶

Under the current study design, 10% (n=7056) of the VA women cohort had <10 years of follow‐up data because of administrative censoring of data collection in 2017. VA women with censored data were slightly younger (by 0.4 years, P<0.001), had lower SBP (by 0.59 mm Hg, P=0.002), had lower total cholesterol (by 1.1 mg/dL, P=0.031), and had a lower prevalence rate of DM (by 2.9%, P<0.001) at baseline than those with a full 10 years of follow‐up data (Table S1). Under the study's assumption of no loss to follow‐up, this indicates that healthier women visited the VA Health Care System less frequently than those with more CVD risk factors. Cox model estimates have been found unbiased as long as right censoring is independent from CVD events. ⁴⁷ , ⁴⁸ The current study's right censoring is administrative and thus independent from CVD events.

There are important limitations to note. The current study excluded Asian and other‐race/ethnic VA women, which comprised about 8% of the VA women cohort. End points of the study (MI, stroke, heart failure, cardiac arrest, and cardiac death) were based on ICD codes from the structured EHR; thus, the quality of accuracy of events was vulnerable to medical records errors. However, previous studies conducted on the quality of ICD codes found high concordance rates in MI (96%) and stroke (92%) between ICD codes entered into the VA EHR system and providers' notes. ¹⁰ , ¹² Concordance rate of heart failure events between ICD codes and providers' notes was slightly lower at 86% than MI and stroke, but 100% in cardiac arrest, with one patient suffering cardiac arrest secondary to drug overdose. The NDI, whose data sources are state death certificates, is the gold standard for death ascertainment. State death certificates are limited in cause of death because they include multiple underlying causes of death. Finally, despite popular use of the NRI in comparative analysis of a new predictive model to an existing model, the NRI has many limitations. Algorithms for calculating NRI values using either a bootstrap method ¹⁴ , ⁴⁹ , ⁵⁰ or absolute numbers ¹⁶ are currently in debate. ⁵¹ Thus, the use of the NRI should not be an alternative to other model performance measures such as C‐statistics and calibration; it should be conjunctive.

Future studies are warranted. The proposed VA women CVD risk score needs to be externally validated with data from women service members and veterans who seek health care outside of the VA system to be used as a clinical decision tool in CVD treatment and prevention for all women in the military. The current model may underestimate CVD risk for VA women with a history of gestational complications or other chronic conditions. A future study developing new statistical methods and approaches that tailor the VA women CVD risk model to capture incremental CVD risk by a history of gestational complications and other chronic conditions is also warranted.

In conclusion, the current study developed a parsimonious model equipped with good prediction accuracy and reliability to assess VA women long‐term CVD risk. The proposed VA women CVD risk score improves the accuracy of the existing ACC/AHA CVD risk assessment tool in predicting long‐term CVD risk for VA women, particularly in young and racial/ethnic minority women. Use of the proposed VA women CVD risk score in assessing women veterans' cardiovascular health may have significant clinical implications.

Sources of Funding

The study was funded by the U.S. Department of Defense Peer‐Reviewed Medical Research Discovery Award (W18XWH1810159); Haekyung Jeon‐Slaughter, PhD, is the study's principal investigator. The views, opinions, and/or findings contained in this article are those of the authors and should not be construed as an official Department of the Army position, policy, or decision unless so designated by other documentation.

Disclosures

None.

Supporting information

Data S1

Table S1

Figures S1–S3

(J Am Heart Assoc. 2021;10:e019217. DOI: 10.1161/JAHA.120.019217.)