Cardiovascular Risk Prediction Functions Underestimate Risk in HIV Infection

Abstract

Background

Cardiovascular disease (CVD) risk is elevated in HIV-infected individuals, with contributions from both traditional and non-traditional risk factors. The accuracy of established CVD risk prediction functions in HIV is uncertain. We sought to assess the performance of three established CVD risk prediction functions in a longitudinal cohort of HIV-infected men.

Methods

Framingham Heart Study (Framingham) functions for hard coronary heart disease (Framingham CHD) and atherosclerotic CVD (Framingham ASCVD) and the American College of Cardiology/American Heart Association (ACC/AHA ASCVD) function were applied to the Partners HIV cohort. Risk scores were calculated between 1/1/2006 and 12/31/2008. Outcomes included CHD (myocardial infarction [MI] or coronary death) for the Framingham CHD function and ASCVD (MI, stroke or coronary death) for the Framingham ASCVD and ACC/AHA ASCVD functions. We investigated the accuracy of CVD risk prediction for each function when applied to the HIV cohort using comparison of Cox regression coefficients, discrimination, and calibration.

Results

The HIV cohort was comprised of 1280 men followed for a median of 4.4 years. There were 80 (6.3%) ASCVD events; 5-year incidence rate was 16.7 per 1000 person years. Discrimination was moderate to poor as indicated by low c statistic (0.68 for Framingham CHD, 0.65 for ACC/AHA, and 0.67 for Framingham ASCVD). Observed CVD risk exceeded predicted risk for each of the functions in most deciles of predicted risk. Calibration, or goodness-of-fit of the models, was consistently poor, with significant chi-square p values for all functions. Recalibration did not significantly improve model fit.

Conclusions

Cardiovascular risk prediction functions developed for use in the general population are inaccurate in HIV infection and systematically underestimate risk in a cohort of HIV-infected men. Development of tailored CVD risk prediction functions incorporating traditional CVD risk factors and HIV-specific factors is likely to result in more accurate risk estimation to guide preventative CVD care.

INTRODUCTION

Cardiovascular disease (CVD) risk prediction functions are widely used to predict CVD risk and prevent disease through identification of the highest risk individuals who merit intensive risk factor modification. HIV-infected individuals face an increased CVD risk, with multiple studies demonstrating a 1.5 to 2-fold increased risk of myocardial infarction (MI) or stroke compared with non HIV-infected individuals.^1–3 While traditional CVD risk factors such as smoking, dyslipidemia, diabetes, and hypertension are prevalent in HIV,^{2, 4–10} emerging data support a prominent role of novel CVD risk factors related to chronic inflammation and immune activation in HIV-related CVD risk.^11–15 Moreover, specific HIV medications have been shown to alter CVD risk.^{16, 17} Established CVD risk functions were developed using risk factors common in the general population and do not reflect these unique mechanistic factors thought to drive HIV-related CVD. Consequently, existing functions may not accurately estimate risk in the setting of HIV.¹⁸

The most widely applied CVD risk functions^19–26 enable clinicians to generate an overall risk score based on a composite of traditional CVD risk factors. We assessed the performance of several of the most commonly used CVD risk functions, including the Framingham functions for hard coronary heart disease (Framingham CHD) and atherosclerotic cardiovascular disease (Framingham ASCVD) and the American College of Cardiology/American Heart Association (ACC/AHA) Pooled Cohorts Equations (ACC/AHA ASCVD). To assess the functions in a cohort of HIV-infected men, we compared regression coefficients and calculated calibration and discrimination of the established functions. We hypothesized that the established CVD risk functions would underestimate risk in HIV-infected individuals.

METHODS

The data, analytic methods, and study materials will be made available to other researchers for purposes of reproducing the results one year after the dataset is locked through Dataverse (https://dataverse.harvard.edu/).

Framingham and ACC/AHA functions

Three established CVD risk functions were evaluated: the Framingham hard CHD function (Framingham CHD),²⁰ the 2008 Framingham global CVD function (Framingham ASCVD)²¹ and the 2013 ACC/AHA function (ACC/AHA ASCVD).²⁵ The Framingham global CVD function was developed for a composite CVD outcome and was modified for the present study to ASCVD using the algorithm described below and in the original 2008 manuscript.¹⁹ Risk factors included in the Framingham functions are age, diabetes, smoking, blood pressure category, total cholesterol, and HDL. Risk factors included in the ACC/AHA function are age, diabetes, smoking, systolic blood pressure, total cholesterol, HDL, and anti-hypertensive treatment. We adapted the 10-year risk prediction functions to 5-year functions to align with the study follow-up period. Given that proportional hazards assumption was met for variables in the Cox models, we developed a 5-year risk model for each function by (a) applying the Cox regression coefficients from the 10-year risk models to the 5-year risk models; and (b) replacing, in the model, 10-year baseline survival (event-free) estimates at the average values of the risk factors with 5-year baseline survival (event-free) estimates at the average values of the risk factors.^{19, 20} The 5-year functions are included in the Appendix.

HIV cohort

The HIV population was identified from the Partners HIV cohort, an observational clinical care cohort of HIV-infected individuals receiving longitudinal HIV care through the Partners HealthCare System (PHS) in Boston, Massachusetts. Data are derived from the Research Patient Data Registry (RPDR), a data registry which includes comprehensive clinical information for all patients from PHS. HIV-infected patients were initially identified from the RPDR by the presence of at least one ICD-9-CM code for HIV (042 or V08). The HIV cohort was then limited to patients with four or more ICD codes for HIV based on a validation study that demonstrated this definition of HIV status to be 88.9% sensitive and 98.6% specific for HIV infection based on physician medical record review. We limited the analyses to men because of low event numbers among women. Patients selected for inclusion were required to have at least one clinical encounter in calendar years 2006-2008, a blood pressure measurement available in 2006-2008, lipid laboratory values available in calendar years 2004-2008, smoking status available at baseline (2006-2008), and first HIV code that occurred prior to the start of observation for each individual. Exclusion criteria included history of a CVD event prior to the start of observation, age <30 or >74 for the Framingham CHD and ASCVD functions, and age <40 or >79 for the ACC/AHA ASCVD function. The medical records of several pre-specified subgroups in whom participants did not meet eligibility criteria by automated review were manually reviewed by physician or research nurse to determine whether patients met inclusion criteria for the study; these included patients with exactly three ICD codes for HIV, patients with the first HIV diagnosis after the observation start date, and patients missing one component of the CVD risk functions. The start of observation occurred on the date of risk score calculation, during a window period from 1/1/2006 to 12/31/2008. Participants who did not experience the outcome were censored at the earliest of non-cardiovascular death, last follow-up visit, or five years after an individual’s start of observation. The study was approved by the Partners Human Research Committee and no informed consent was required.

Calculation of predicted CVD risk

The baseline visit (visit of the risk score calculation) was the participant’s first visit that occurred between 1/1/2006 and 12/31/2008 that had non-missing values for the risk factors necessary to generate a risk score. Age used was at baseline for each individual patient. Blood pressure data were obtained through electronic health record (EHR) health maintenance coded field data and by query of free text notes. Total and HDL cholesterol were obtained from laboratory data. Diabetes diagnosis was validated for this study and was defined as the presence of three or more ICD-9-CM codes of 250 and all subtypes (sensitivity 86.2%, specificity 93.0%). Smoking status was obtained through application of an NLP-based algorithm developed and validated in the Partners HIV cohort.²⁷ For the ACC/AHA function, white coefficients were employed as this was the most common race/ethnicity and there was no evidence of an interaction between race and traditional CVD risk factors on the outcomes, as discussed below.

Outcomes

Hard CHD was defined as MI (ICD-9-CM code 410 and all subtypes) or coronary death. ASCVD was defined as MI (ICD-9-CM code 410 and all subtypes), stroke (ICD-9-CM codes 433-434 and all subtypes) or coronary death. The published Framingham (2008) function is for global CVD. Before applying it to the HIV sample, the function was adapted for the outcome of ASCVD for this analysis by using the adjustment factor of the ratio of the number of patients in the cohort with ASCVD to the number of patients with global CVD, following the methodology as outlined in the 2008 Framingham global CVD publication.²¹ Global CVD is defined as MI (ICD-9-CM code 410 and all subtypes), stroke (ICD-9-CM codes 433-434 and all subtypes), coronary death, coronary insufficiency (ICD-9-CM codes 411, 412, 414 and all subtypes), angina (inpatient ICD-9-CM code 411.1x), TIA (ICD-9-CM code 435 and all subtypes), PAD (ICD-9-CM codes 440.2x, 440.3x, and 440.4x), or heart failure (ICD-9-CM code 428 and all subtypes). Outcomes were identified by the presence of an ICD code for the relevant outcome that occurred after the start of observation; only the first code was included. ICD-based ascertainment of outcome events was validated by reviewing a randomly-selected subset of patients with and without ICD codes for MI and stroke. Medical record review by a trained clinical nurse as the gold standard. For MI, sensitivity was 81.5% and specificity was 86.3%. For stroke, sensitivity was 86.7% and specificity was 78.8%. Coronary death data were obtained from the Partners RPDR; data on inpatient deaths are provided by the hospital, and data on out-of-hospital deaths are obtained from the Social Security Administration’s Death Master File (DMF). The DMF does not contain cause-of -death, so for the present study medical records for all patients who died were individually reviewed by a physician to determine whether the cause of death was coronary.

Statistical Analysis

Analyses described below were carried out separately for each of the three risk functions (Framingham CHD, Framingham ASCVD, and ACC/AHA ASCVD) for men (we limited the analyses to men because of low event numbers among women). Race/ethnicity groups were combined in the analyses because there were no significant interactions between race and the other risk factors on the outcomes at the 0.10 level of significance. We used white coefficients for the ACC/AHA function in the primary analysis. Framingham functions were developed on mostly white participants and there are no separate models for other races. In sensitivity analyses, we repeated all analyses 1) applying black coefficients for the ACC/AHA function to the entire HIV cohort and 2) stratifying the analysis by race. For the race-stratified analyses, we categorized patients as Black only, White only, or Non-black (White, Asian, Hispanic, other or unknown) in separate analyses, using white coefficients for the White only and Non-black groups and black coefficients for the Black only group.

Comparison of coefficients

Cox proportional-hazards modeling was used to generate regression coefficients for each of the risk factors using data from the HIV cohort. Regression coefficients generated for the HIV cohort were descriptively compared to those previously generated for each function from Framingham data. The coefficients generated from the HIV data were used to generate an “HIV function” that represents the best prediction function based on risk factor and outcome data from the HIV cohort, using the same traditional CVD risk factors as those included in the original functions and not including HIV-related risk factors.²⁰ We did not directly compare relative risks between the ACC/AHA ASCVD and HIV functions because of the difficulty of interpretation of relative risks for the interaction terms included in the ACC/AHA function.

In supplemental analyses, we generated a new model based on risk factor and outcome data from the HIV cohort for men and women combined. For each function, we included all original risk factors and interaction terms with sex for each risk factor. We then used backwards selection to remove non-significant interaction terms one at a time using a 0.10 level of significance (the interaction term with the largest p-value >0.10 was removed in the backward approach and we reran the regressions; the process was repeated until all remaining interaction terms had p<0.10). Original risk factors were retained in the model regardless of their significance. Analyses were conducted using SAS version 9.4.

Discrimination

To assess the ability of each prediction function to separate individuals who experience an event from individuals who do not, we calculated the c statistic. For each outcome, two c statistics were calculated. The first applied the established function (Framingham CHD, ACC/AHA ASCVD, or Framingham ASCVD) to the HIV data and the second applied the “HIV function,” developed using risk factor and outcome data from the HIV cohort itself rather than using Framingham or ACC/AHA cohort data, to the HIV data.

Calibration

For each of the three risk functions and for the “HIV function”, calibration (goodness-of-fit of predicted risk to observed risk) was assessed using the Demler et al modification of the D’Agostino-Nam χ² statistic.²⁸ The HIV cohort was divided into deciles of predicted 5-year risk of CVD according to each risk function. For calculation of the χ² statistic, we collapsed deciles with small numbers of categories until all resulting groups contained at least 5 events as required by Demler et al. This strategy was employed to ensure a stable χ² statistic. For each resulting group, the mean predicted risk and the observed Kaplan-Meier estimate of event rate were calculated and compared using the Demler/D’Agostino-Nam χ² statistic; a non-significant χ² (p value >0.05) indicates good calibration (i.e., good fit of predicted risk to observed risk). In the event of poor calibration of the published risk models to the HIV data, recalibration was performed by replacing the mean values of risk factors and average incidence rates used from the established functions with those derived from the HIV cohort as discussed in D’Agostino et al.²⁰ Calibration was then recalculated in a similar manner where participants were grouped in the following three categories of predicted risk instead of deciles: <5%, 5-7.5%, >7.5%.

RESULTS

Baseline characteristics

Table 1 shows demographic and clinical characteristics of the HIV cohort among men age 30-74 for all individuals and according to CHD event status. The cohort was comprised of 1,272 men, with median follow-up time of 4.4 years. Mean HDL was 43.7 mg/dL among patients without CHD events versus 40.7 mg/dL among those with events, and the proportion of patients with optimal or normal blood pressure was 60.9% among patients without CHD events versus 54.2% among those with events. More than 45% of patients were smokers; among patients with CHD events, 60.4% were smokers (vs. 44.9% among those without events). Diabetes was present in 13.5%, with a rate of 27.1% among patients with CHD events (vs. 13.0% among patients without events). Among all patients, 94.6% were receiving antiretroviral therapy (ART). Median CD4 cell count was 457; among patients without CHD events, median CD4 count was 457 (vs. 348 among those with events). The rate of viral suppression was 77.1% overall, 77.5% among patients without CHD events, and 66.7% among patients with CHD events.

Table 1.

Baseline characteristics of HIV-infected men age 30-74

	All (N=1272)	No CHD Event (N=1224)	CHD Event (N=48)
Age (yrs) – mean+/−SD	51.2±8.7 (1272)	51.1±8.7 (1224)	55.2±8.1 (48)
Age (yrs) – (min, max)	(30.2,73.9)	(30.2,73.9)	(34.9,72.3)
Race/ethnicity – % (n)
White	60.6 (771/1272)	60.5 (741/1224)	62.5 (30/48)
Black	19.7 (250/1272)	19.7 (241/1224)	18.8 (9/48)
Hispanic	13.8 (176/1272)	13.7 (168/1224)	16.7 (8/48)
Other/Unknown	5.9 (75/1272)	6.0 (74/1224)	2.1 (1/48)
Total cholesterol (mg/dL) – mean+/−SD	179.0±45.7 (1272)	178.9±45.6 (1224)	182.8±47.8 (48)
Total cholesterol (mg/dL) categories – % (n)
< 160	33.8 (430/1272)	33.5 (410/1224)	41.7 (20/48)
160-199	36.9 (470/1272)	37.4 (458/1224)	25.0 (12/48)
200-239	20.9 (266/1272)	21.0 (257/1224)	18.8 (9/48)
240-279	6.2 (79/1272)	6.0 (74/1224)	10.4 (5/48)
>= 280	2.1 (27/1272)	2.0 (25/1224)	4.2 (2/48)
HDL cholesterol (mg/dL) – mean+/−SD	43.6±14.2 (1272)	43.7±14.3 (1224)	40.7±11.3 (48)
HDL cholesterol (mg/dL) categories – % (n)
< 35	25.9 (329/1272)	25.6 (313/1224)	33.3 (16/48)
35-44	31.6 (402/1272)	31.7 (388/1224)	29.2 (14/48)
45-49	14.4 (183/1272)	14.1 (173/1224)	20.8 (10/48)
50-59	16.7 (212/1272)	16.8 (206/1224)	12.5 (6/48)
>= 60	11.5 (146/1272)	11.8 (144/1224)	4.2 (2/48)
Systolic blood pressure – mean+/−SD	123.6±14.8	123.5±14.6 (1224)	127.2±17.0 (48)
Treated systolic blood pressure – mean+/−SD (n)	127.5±16.7 (370)	127.5±16.6 (346)	127.5±19.3 (24)
Untreated systolic blood pressure – mean+/−SD (n)	122.1±13.6 (902)	121.9±13.5 (878)	127.0±14.8 (24)
Blood pressure categories – % (n)
Optimal (SBP<120, DBP<80)	31.0 (394/1272)	31.2 (382/1224)	25.0 (12/48)
Normal (SBP<130, DBP<85)	29.6 (377/1272)	29.7 (363/1224)	29.2 (14/48)
High Normal (SBP<140, DBP<90)	17.0 (216/1272)	16.8 (206/1224)	20.8 (10/48)
Stage I HTN (SBP<160, DBP<100)	17.4 (221/1272)	17.4 (213/1224)	16.7 (8/48)
Stage II-IV HTN (SBP>=160, DBP>=100)	5.0 (64/1272)	4.9 (60/1224)	8.3 (4/48)
Antihypertensive medication – % (n)	29.1 (370/1272)	28.3 (346/1224)	50.0 (24/48)
Smoking – % (n)	45.4 (578/1272)	44.9 (549/1224)	60.4 (29/48)
Diabetes – % (n)	13.5 (172/1272)	13.0 (159/1224)	27.1 (13/48)
CD4 count – median (Q1, Q3)	457.0 (277.0,672.0)	457 (281,672)	348 (222,687)
CD4 count < 200 cells/mm3 – % (n/N)	14.9 (176/1183)	14.7 (168/1140)	18.6 (8/43)
HIV viral load < 400 copies/mL – % (n/N)	77.1 (804/1043)	77.5 (778/1004)	66.7 (26/39)
Log HIV viral load – median (Q1, Q3)	4.2 (3.5, 4.7)	4.2 (3.5, 4.7)	4.5 (3.9, 4.9)
ART use – % (n)	94.6 (1203/1272)	94.5 (1157/1224)	95.8 (46/48)
Duration follow-up (yrs) – median (Q1, Q3)	4.4 (3.0,5.0)	4.5 (3.1,5.0)	1.7 (0.6,3.8)
5-year hard CHD rate – % (n)	3.8 (48)	-	-
5-year hard CHD incidence rate (per 1000 PY)	10.0	-	-
5-year ASCVD rate – % (n)	6.1 (78)	-	-
5-year ASCVD incidence rate (per 1000 PY)	16.4	-	-

SD indicates standard deviation; mg, milligrams; dL, deciliter; HDL, high-density lipoprotein; SBP, systolic blood pressure; DBP, diastolic blood pressure; HTN, hypertension; Q1, quarter 1; Q3, quarter 3; ART, antiretroviral therapy; CHD, coronary heart disease; PY, person years; ASCVD, atherosclerotic cardiovascular disease.

HIV viral load is represented as the median log viral load among individuals with detectable viral loads. Hard CHD is defined as MI or coronary death. ASCVD is defined as MI, stroke, or coronary death.

The 5-year hard CHD event rate was 3.8% (48/1272) and the 5-year ASCVD event rates was 6.1% (78/1272). Incidence rates for hard CHD and ASCVD are shown in Table 1. Median risk scores for each function are shown in Supplemental Table 1 and include predicted risk scores based on established functions, observed risk, and predicted risk scores based on the Cox models developed from the HIV cohort (“HIV function”).

Supplemental Table 2 shows demographic and clinical characteristics of the HIV cohort among women age 30-74 overall and according to CHD event status.

Comparison of coefficients

Table 2 shows coefficients and two-sided 95% confidence intervals for risk factors from the HIV cohort’s regression model and also displays published coefficients derived from the original cohort’s (Framingham or ACC/AHA) model for each function among men. Qualitative comparison of coefficients was performed, taking into consideration that interpretation of results is limited by a low number of events. The majority of the coefficients developed from the “HIV function” did not achieve statistical significance (confidence interval of the coefficient contained 0) for each of the three established functions evaluated. For hard CHD, increasing total cholesterol category was associated with higher hard CHD risk in both the Framingham and the HIV-developed models. Being in the lowest HDL category (<35) yielded the largest hard CHD risk and being in the highest HDL category (≥60) was associated with lowest risk in both hard CHD models. HTN was associated with higher risk of hard CHD in both models. As anticipated, age, diabetes, and smoking were associated with higher hard CHD risk in both models.

Table 2.

Coefficients for coronary heart disease risk factors among men

Framingham CHD
Risk Factor	Original Function	HIV Function (95% CI)
Age	0.05	0.048 (0.013, 0.084)
Age Squared	−	−
Blood Pressure
Optimal (SBP<120, DBP<80)	0.09	−0.209 (−0.986, 0.568)
Normal (SBP<130, DBP<85)	Reference	Reference
High Normal (SBP<140, DBP<90)	0.42	−0.027 (−0.85, 0.797)
Stage I HTN (SBP<160, DBP<100)	0.66	−0.271 (−1.154, 0.612)
Stage II-IV HTN (SBP≥160, DBP≥100)	0.9	0.406 (−0.719, 1.532)
Total Cholesterol
< 160	−0.38	0.603 (−0.13, 1.335)
160-199	Reference	Reference
200-239	0.57	0.278 (−0.598, 1.153)
240-279	0.74	0.901 (−0.149, 1.952)
>= 280	0.83	1.427 (−0.095, 2.95)
HDL
<35	0.61	0.045 (−0.781, 0.871)
35-44	0.37	−0.435 (−1.255, 0.385)
45-49	Reference	Reference
50-59	0	−0.726 (−1.748, 0.297)
>= 60	−0.46	−1.345 (−2.875, 0.185)
Diabetes	0.53	0.71 (0.05, 1.369)
Smoking	0.73	0.804 (0.213, 1.395)
ACC/AHA ASCVD
Risk Factor	Original Function	HIV Function (95% CI)
Ln Age	12.344	1.855 (−34.058, 37.768)
Ln Total Cholesterol	11.853	5.641 (−23.64, 34.921)
Ln HDL Cholesterol	−7.990	−9.173 (−29.337, 10.991)
Ln Treated SBP	1.797	0.463 (−1.357, 2.283)
Ln Untreated SBP	1.764	0.417 (−1.418, 2.252)
Smoking	7.837	1.384 (−11.186, 13.954)
Diabetes	0.658	0.406 (−0.128, 0.939)
Ln Age × Ln Total Cholesterol	−2.664	−1.303 (−8.6, 5.994)
Ln Age × Ln HDL Cholesterol	1.769	2.102 (−2.921, 7.125)
Ln Age × Smoking	−1.795	−0.236 (−3.359, 2.887)
Ln Age × Ln Age	NA	NA
Framingham ASCVD
Risk Factor	Original Function	HIV Function (95% CI)
Ln Age	3.061	2.886 (1.352, 4.419)
Ln Total Cholesterol	1.124	0.471 (−0.491, 1.434)
Ln HDL Cholesterol	−0.933	−0.761 (−1.471, −0.051)
Ln Treated SBP	1.999	0.775 (−1.057, 2.606)
Ln Untreated SBP	1.933	0.713 (−1.133, 2.559)
Smoking	0.655	0.474 (0.026, 0.921)
Diabetes	0.574	0.38 (−0.161, 0.921)

CHD indicates coronary heart disease; CI, confidence interval; SBP, systolic blood pressure; DBP, diastolic blood pressure; HDL, high-density lipoprotein; ACC indicates American College of Cardiology; AHA, American Heart Association; ASCVD indicates atherosclerotic cardiovascular disease.

For the ACC/AHA ASCVD function, age, total cholesterol, systolic blood pressure, smoking, and diabetes were associated with higher ASCVD risk in both the ACC/AHA and the HIV models. For both models, as expected, increased HDL was associated with lower ASCVD risk.

For the Framingham ASCVD function, all risk factors had similar relative effects in the Framingham and the HIV models, with increased age, total cholesterol, systolic blood pressure, smoking, and diabetes associated with higher risk and HDL associated with lower risk.

Discrimination

Table 3 shows the c statistic, a measurement of discrimination, and its two-sided 95% confidence interval for each function applied to the HIV cohort. In the table, the “Original Function” columns indicate the c statistic derived from applying the Framingham CHD, ACC/AHA ASCVD, or Framingham ASCVD function to the HIV cohort’s data, respectively. The “HIV Function” column indicates the c statistic derived from a Cox regression model developed from the HIV cohort itself, using the same variables as contained in the relevant Framingham or ACC/AHA function but using risk factor values and outcome data measured on the HIV cohort (“HIV function”). Discrimination of all published functions when applied to the HIV cohort was suboptimal (Original Function columns), with c statistics of 0.68 for Framingham CHD, 0.65 for ACC/AHA ASCVD, and 0.67 for Framingham ASCVD. Application of the Cox regression model developed from the HIV cohort to the HIV cohort (“HIV Function” columns) resulted in improved but not optimized discrimination for most groups (c statistics of 0.73 for Framingham CHD, 0.66 for ACC/AHA ASCVD, and 0.67 for Framingham ASCVD).

Table 3.

Discrimination of the Framingham CHD, ACC/AHA ASCVD, Framingham ASCVD and “HIV function” applied to the HIV cohort data (c statistics and 95% CIs) among men

	Framingham CHD	ACC/AHA ASCVD	Framingham ASCVD
Original Function	0.68 (0.61, 0.75)	0.65 (0.59, 0.71)	0.67 (0.61, 0.73)
HIV Function	0.73 (0.67, 0.81)	0.66 (0.60, 0.73)	0.67 (0.61, 0.73)

CHD indicates coronary heart disease; ACC, American College of Cardiology; AHA, American Heart Association; ASCVD, atherosclerotic cardiovascular disease.

Calibration and observed versus predicted risk

Figure 1 shows observed and predicted 5-year risk for each of the functions by decile of predicted risk for all participants. For all functions, observed risk exceeded predicted risk for most deciles of predicted risk. Figure 2 shows observed and predicted 5-year risk for each of the functions by predicted risk group (<5%, 5-7.5%, and >7.5%). Observed risk exceed predicted risk for all categories in all functions except for >7.5% predicted risk for the Framingham hard CHD function.

Figure 1.

Observed and predicted 5-year risk by decile of predicted risk. Panel A is for Framingham CHD; Panel B is for ACC/AHA; Panel C is for Framingham ASCVD.

Figure 2.

Observed and predicted 5-year risk by predicted risk group (<5%, 5-7.5%, and >7.5%). Panel A is for Framingham CHD; Panel B is for ACC/AHA; Panel C is for Framingham ASCVD.

Supplemental Table 1 shows median risk scores by function and compares predicted versus observed risk. For all three functions, observed risk exceeds predicted risk, whether predicted risk is calculated using the traditional functions or the “HIV function”. All predicted risk scores, however, are greater and closer to the observed risk when the “HIV function” is applied versus when the traditional functions are applied.

The calibration, or goodness-of-fit, of the Framingham and ACC/AHA functions applied to the HIV cohort was assessed using the calibration χ² statistic. For the three functions assessed, calibration was poor and reflects inadequate fit of the general population functions to the HIV cohort. For the Framingham CHD function, the calibration χ² statistics was 13.6 (P=0.019, 5 degrees of freedom). For the ACC/AHA function, the calibration χ² statistics was 23.9 (P=0.001, 7 degrees of freedom). For the Framingham ASCVD function, the calibration χ² statistic was 24.6 (P=0.0004, 6 degrees of freedom).

We recalibrated the Framingham or ACC/AHA model in order to attempt to improve model fit of these functions by using baseline survival and mean risk factor values from the HIV cohort instead of the Framingham or ACC/AHA cohorts’ values. After recalibration, goodness-of-fit remained poor for all functions and model performance did not improve (data not shown).

In the primary analysis for the ACC/AHA ASCVD function, white coefficients were employed given that there was no evidence of any interaction between race and traditional CVD risk factors in the HIV cohort. We repeated all analyses using black coefficients for the ACC/AHA function and generated similar results. To further confirm that each function poorly discriminated and underestimated risk in the HIV cohort, we conducted analyses stratified by race and showed that discrimination remained moderate and calibration remained poor (Supplemental Tables 3 and 4).

In supplemental analyses, we generated a new model (“HIV Function”) among men and women combined including significant interaction terms with sex for each risk factor. Results of this analysis are shown in Supplemental Table 5.

DISCUSSION

We evaluated three established cardiovascular risk prediction functions in an HIV cohort and found that the functions systematically underestimate CVD risk in HIV. Discrimination and calibration were both suboptimal when applying the functions to a cohort of largely ART-treated men engaged in HIV care, indicating that risk prediction algorithms developed for use in the general population may underperform and may be inadequate in their ability to appropriately order CVD risk and accurately calculate predicted risk. Our findings suggest that established CVD risk functions do not provide an accurate estimation of risk in the setting of HIV disease and may fail to identify patients at elevated CVD risk who would benefit from aggressive risk reduction. Adaptation of current CVD risk functions with the incorporation of new risk factors is likely to be needed for safe and accurate risk estimation in HIV.

The question of transportability of CVD risk prediction functions to populations which differ from those in which they are developed has received considerable attention.²⁰ To date, the Framingham functions have been the most widely used to quantify the absolute risk of developing a specific CVD outcome for an individual patient over a given time period.^{20, 29} The release of ACC/AHA guidelines on CVD risk assessment and cholesterol treatment represents a turning point in the approach to CVD prevention in the general population^{25, 26, 30} with the development of a new CVD risk prediction function derived from a more representative population and utilizing the broader outcome of ASCVD. The applicability of established functions in the setting of HIV, however, remains incompletely understood. The ACC/AHA guidelines specifically call for validation in diverse settings,²⁵ and in this analysis, we find established prediction functions to be inadequate when applied to individuals with HIV.

Developing accurate CVD risk functions in HIV-infected individuals has been identified as a priority by both HIV and cardiovascular advocacy groups.^{25, 26, 31} The hypothesis that conventional CVD risk prediction functions may not perform accurately in HIV is based on proposed mechanistic factors. While traditional CVD risk factors are prevalent in HIV and contribute to CVD risk,⁶ extensive data now suggest that HIV-specific inflammation and immune activation play prominent roles in mediating CVD risk.^{11, 13, 14} Even in virally suppressed patients, inflammation and immune activation continue to be abnormal,³² and persistent chronic inflammation may increase long-term CVD risk and severity.^{33, 34} Adding to the complexity of the mechanism of HIV-associated CVD, several individual HIV medications have been shown to be associated with increased CVD risk,^{16, 17} although whether they mediate risk via effects on traditional risk factors versus novel mechanisms remains unclear and is likely to differ by drug. Established CVD risk algorithms were modeled after risk factors common in the general population and do not reflect these unique mechanistic factors thought to drive HIV-related CVD.

Prior studies in HIV have evaluated the correlation between CVD risk scores and subclinical atherosclerosis³⁵ and suggest that Framingham functions underestimate risk of AMI³⁶ and stroke.³⁷ Data from the HIV Outpatient Study³⁸ demonstrated accurate risk estimation but suboptimal discrimination for the Framingham global CVD function, underestimation and reasonable discrimination by the ACC/AHA and Data collection on Adverse events of Anti-HIV Drugs (D:A:D) functions,^{39, 40} and overestimation with poor discrimination for the Systematic Coronary Risk Evaluation (SCORE) function.⁴¹ A recent study from the Centers for AIDS Research Network of Integrated Clinical Systems (CNICS) cohort found the ACC/AHA function to adequately discriminate MI risk and to be moderately calibrated, with variability in performance by demographic subgroups.⁴² Additionally, incorporation of HIV-specific variables did not improve the risk score.⁴² Our study builds on prior knowledge by providing a formal assessment of several established functions’ performance in terms of discrimination, calibration, and comparison of regression coefficients and provides new knowledge by evaluating the ACC/AHA and two Framingham functions in a cohort of HIV-infected individuals with the ability to evaluate the outcome of ASCVD, including MI and stroke.

Our data support the hypothesis that established CVD risk functions underestimate risk for HIV-infected men. Our assessment of the models’ discriminatory capacity indicated that all the functions sub-optimally ordered risk in the HIV cohort; the c statistics, which represent the probability that patients who develop the outcome will be assigned a higher risk than those who do not, were lower than the standard threshold for all three functions when applied to the HIV cohort. Observed risk was greater than predicted risk across all three risk functions, and for most deciles and groups of predicted risk. The degree of agreement between predicted and observed risk was formally studied through assessment of calibration, with large χ² statistics and significant p values indicating poor model fit for all three functions. Comparison of regression coefficients, applying the original function versus the “HIV function,”, indicate different magnitude and at times direction of effect of traditional risk factors on cardiovascular outcomes, although qualitative comparisons are limited by a relatively low number of events.

To attempt to improve model performance, we generated an “HIV function” which represents the best function that can be developed based on risk factor and outcome data from the HIV cohort itself, but using risk factors identical to those included in the established function. We evaluated these “best HIV functions” – developed from the three established functions – in terms of traditional risk factors, discrimination, and calibration. For these newly-developed functions, we found that few of the regression coefficients for traditional CVD risk factors achieved significance. Discrimination, as indicated by the c statistic, improved for the HIV functions but remained inadequate overall, and recalibration only modestly improved model fit for each HIV function. These findings indicate that the best model we can develop, directly computed from the HIV data, only marginally improves risk prediction and that additional research is needed – likely including the addition of new variables – to develop an adequate risk model in the HIV population.

Together, these data suggest that cardiovascular risk prediction using established algorithms developed for the general population remains suboptimal in the setting of HIV disease, and that as a first step, the discriminatory capacity of the models needs to be improved. Our findings of the models’ moderate to poor discriminatory capacity are consistent with the possibility of an unaccounted-for risk factor that is relevant to HIV-infected individuals but not to the populations from which the functions were developed. This “missing” variable is likely to be related to HIV-associated inflammation and immune activation, which have been shown to directly impact cardiovascular risk in this population.^43–45 While HIV-related, novel CVD risk factors have been demonstrated to be significantly associated with CVD outcomes, whether the incorporation of such factors into established risk prediction functions will improve the performance of the functions remains unknown. Moreover, risk factors considered for inclusion should be clinically relevant to facilitate widespread implementation of a new HIV-specific function. While the primary purpose of this study was to investigate the role of previously studied, traditional CVD risk factors in risk prediction in HIV, future work is planned to evaluate the incorporation of HIV-specific variables, including HIV disease parameters and medications as well as HIV itself, into tailored risk prediction algorithms.

Our study has several limitations, including the possibility of inaccurate outcome ascertainment due to missing data or incorrect classification. CVD outcomes were validated for the current study; as 10-year outcome data become available, events are planned to be adjudicated. The follow-up time of 5 years (versus 10) was selected to balance and optimize completeness and accuracy of EHR data (which has improved over time in the RPDR) with longer duration of follow-up. While the lower number of events limits interpretation of current results, we are prospectively collecting 10-year data and anticipate an increasing number of events as more follow-up time is accumulated. Additionally, we did not include women in the analysis because of difficulty interpreting results due to low event numbers; findings are thus generalizable to HIV-infected men. Future analyses of 10-year events including women are planned.

In conclusion, cardiovascular risk prediction using established general population algorithms was found to be inaccurate in HIV infection, with consistent underestimation of risk. In this study employing a longitudinal HIV clinical care cohort, we assessed the performance of two established Framingham functions and the ACC/AHA function and found that all functions resulted in poor model fit when applied to the HIV population. Our findings suggest that current application of established CVD risk functions in clinical practice may not result in accurate CVD risk assessment and that reliance on such measures may fail to identify high-risk HIV-infected individuals who would benefit from aggressive cardiovascular risk modification. Refining current CVD risk functions through addition of novel HIV-related factors – including HIV itself –may improve CVD risk function performance. The lack of transportability of established CVD functions to HIV underscores the need for tailored CVD risk prediction strategies for this at-risk population to manage and prevent one of the most serious complications of long-term HIV infection.

CLINICAL PERSPECTIVE.

What is new?

• In this study evaluating cardiovascular disease (CVD) risk prediction in HIV populations, established CVD risk prediction functions were shown to underestimate risk in HIV-infected men and poorly discriminate between individuals who experience an outcome from those who do not.

• The study adds new knowledge by evaluating several risk functions in parallel for the first time, including the Framingham function for hard coronary heart disease (CHD) and two functions that predict atherosclerotic CVD (ASCVD), the American College of Cardiology/American Heart Association function and an adapted Framingham global CVD function.

What are the clinical implications?

• Our findings suggest that established CVD risk functions may not accurately estimate risk in the setting of HIV disease and may fail to identify men at elevated CVD risk who would benefit from aggressive risk reduction.

• The incorporation of HIV-specific variables into current CVD risk prediction algorithms may be necessary to accurately predict risk for this population that is already at heightened risk.

• Tailored CVD risk prediction strategies may apply to other high-risk population with chronic inflammatory conditions.