Reevaluating the Use of Race/Ethnicity in the MESA Risk Score

Abstract

Background

Cardiovascular disease risk scores are widespread in research and clinical settings. The MESA (Multi‐Ethnic Study of Atherosclerosis) risk score estimates the 10‐year risk of coronary heart disease and was novel in its use of coronary artery calcium alongside traditional risk factors, including race/ethnicity. However, including race/ethnicity in a risk score model requires careful consideration. We developed a race‐free MESA risk score and compared it with the original.

Methods

We used data from the MESA cohort, a community‐based cohort of individuals aged 45 to 84 years who identified as White, Hispanic/Latino, Black, or Chinese and were free of prevalent cardiovascular disease upon study entry, to develop a race‐free risk score estimating 10‐year coronary heart disease risk. We used the same 2‐step regularization procedure as that for the original MESA risk score: (1) least absolute shrinkage and selection operator for variable selection, and (2) ridge regression to prevent overfitting. Neither race/ethnicity nor any interaction term with race/ethnicity was included as a candidate predictor.

Results

When excluding race/ethnicity, several interaction terms were retained that were not retained in the original MESA risk score. The area under the receiver operating characteristic curve for the race‐free risk score was 0.820 (95% CI, 0.801–0.840), and the difference in area under the receiver operating characteristic curve was not significant (estimate, 0.0032 [95% CI, −0.0008 to 0.0073]). The discrimination slope in the race‐free model was 0.101, while that of the original MESA risk score was 0.0937.

Conclusions

We developed a race‐free MESA risk score that performs comparably to the MESA risk score in both calibration and discrimination.

Nonstandard Abbreviations and Acronyms

CHCN‐BTH

Cohort Study on Chronic Disease of Communities Natural Population in Beijing, Tianjin, and Hebei

LASSO

least absolute shrinkage and selection operator

MESA

Multi‐Ethnic Study of Atherosclerosis

MOSAAIC

Multi‐Ethnic Observational Study in American Asian and Pacific Islander Communities

NRI

net reclassification index

PREVENT

Predicting Risk of Cardiovascular Disease Events

Clinical Perspective.

What Is New?

• A race‐free risk score for 10‐year risk of coronary heart disease was developed in the MESA (Multi‐Ethnic Study of Atherosclerosis) cohort and compared with the original MESA risk score, which includes race/ethnicity; we find that the race‐free risk score performs similarly to the original MESA risk score, considering both overall performance and performance stratified by race/ethnicity.

What Are the Clinical Implications?

• The race‐free risk score is publicly available and allows for calculation of an individual's 10‐year risk of coronary heart disease without use of their race/ethnicity.

• These findings suggest that in this setting, race/ethnicity do not explain meaningful variation in coronary heart disease risk beyond what is explained by the other predictors in the model, contributing to the growing literature on the development of race‐free risk scores.

Cardiovascular disease risk scores are important tools for guiding prevention strategies and helping clinicians identify individuals at high risk. The MESA (Multi‐Ethnic Study of Atherosclerosis) risk score was published in 2015 for the prediction of the 10‐year risk of coronary heart disease (CHD) and was also made available at this time as an online calculator to facilitate its use in clinical settings. The novelty of the MESA risk score was its inclusion of coronary artery calcium (CAC). CAC, measured from routine cardiac‐gated noncontrast computed tomography scans, improves risk prediction for CHD events. ¹ , ² Other than CAC, the MESA risk score includes age, sex, high‐density lipoprotein cholesterol, total cholesterol, systolic blood pressure, antihypertensive medication use, current smoking, diabetes, family history of heart attack, lipid‐lowering medication use, and, notably, race/ethnicity.

Since its publication, the MESA risk score has been cited in guidelines for primary prevention of cardiovascular disease (CVD), ³ guidelines on the management of blood cholesterol, ⁴ as well as literature comparing multiple risk scores. ⁵ Recently, broader conversations about the ways in which the use of race/ethnicity in biomedical research can cause harm ⁶ , ⁷ , ⁸ have raised questions about the inclusion of race/ethnicity as a predictor in risk prediction models.

Cardiovascular health outcomes show clear differences by race/ethnicity. In this way, we can think of race/ethnicity as a risk marker, but care is needed to emphasize that this does not mean it is a causal risk factor. ⁹ That is, race/ethnicity may act as a proxy for social determinants of health and structural racism and may not reflect intrinsic differences in cardiovascular biology. ⁸

When we consider that race/ethnicity act as a proxy for social determinants of health and are often a predictor of risk, the question remains as to whether including race/ethnicity alongside traditional risk factors will impact cardiovascular risk prediction. The use of race/ethnicity in these models can lead to tangible differences in the care an individual receives. One well‐known example is the race‐based estimated glomerular filtration rate equations, which included an adjustment factor that increases the estimates for Black individuals, which has downstream effects on their eligibility for kidney disease therapies. ¹⁰ In some cases, including race/ethnicity can improve the accuracy of predictions among racial/ethnic groups. ¹¹ , ¹² Yet this is not always true; for example, the removal of race stratification from the pooled cohort equations for atherosclerotic CVD showed no meaningful difference in performance. ¹³

These discussions have spurred recent research into the implications of race/ethnicity in risk prediction. For example, recently Khan and colleagues ¹⁴ released novel Predicting Risk of Cardiovascular Disease Events (PREVENT) equations for the prediction of risk of an atherosclerotic CVD and heart failure end point, which are race‐free, but included neighborhood socioeconomic status instead.

In this work, we aim to understand the implications of removing race/ethnicity from the MESA risk score by creating a race‐free risk score and comparing its performance to the original MESA risk score.

Methods

Study Participants

MESA is a prospective, community‐based cohort study initiated in 2000 with the aim of studying the prevalence and progression of subclinical markers of CVD. Between 2000 and 2002, 6814 participants aged 45 to 84 years without known clinical CVD who self‐identified as White, Black, Hispanic/Latino, or Chinese were enrolled at 6 US academic medical centers. Any participant who indicated Hispanic/Latino was included in that group regardless of what other group they may have selected; this is why we use the term race/ethnicity rather than race alone. Greater detail on the study design and recruitment is available in the work of Bild et al. ¹⁵ The study was approved by the institutional review board of each field center, and all participants gave written informed consent. Data are available upon request via the MESA website at https://www.mesa‐nhlbi.org/.

CAC Measurement

CAC was quantified via chest computed tomography, more specifically, ECG‐gated electron‐beam computed tomography at 3 field centers (Chicago, IL; Los Angeles, CA; and New York, NY) and multidetector computed tomography at the 3 other field centers (Baltimore, MD; Forsyth County, NC; and St. Paul, MN). The resulting images were studied at the MESA computed tomography reading center (Los Angeles Biomedical Research Institute at Harbor–University of California Los Angeles Medical Center, Torrance, CA) and the quantity of CAC determined using the Agatston scoring method. ¹⁶ More detail on measurement is available in the published protocol. ¹⁷

Covariates

Smoking status was defined as currently or not currently smoking. Systolic blood pressure was measured following a 5‐minute rest period in a seated position using a Dinamap model Pro 100 monitor (Critikon, Tampa, FL). For each participant, 3 blood pressure measurements were taken per visit, and the reported value was taken to be the average of the second and third measurements. Diabetes was defined by having fasting glucose ≥126 mg/dL or using glucose‐lowering medication. Lipid profiles, which included high‐density lipoprotein cholesterol and total cholesterol, were analyzed at the Central Lipid Laboratory. Known family history of heart disease was an indicator of reporting a parent, sibling, or child who had a heart attack.

Ascertainment of Events

Participants were contacted every 9 to 12 months by phone to inquire about hospital admissions and cardiovascular diagnoses and procedures. Death certificates were obtained for all deaths, and next‐of‐kin interviews were sought but not often obtained for possible CVD deaths outside of the hospital setting. Trained staff went through participant medical records to assess possible cardiovascular events and recorded any relevant corresponding clinical information. At this point, 2 physicians independently classified any such event and assigned the incidence date.

The definition of a CHD death depended on the lack of a known noncardiac or nonatherosclerotic cause of death and included those who had a myocardial infarction in the previous 28 days, chest pain within 72 hours, or a history of CHD.

End Point

CHD events were defined as myocardial infarction, resuscitated cardiac arrest, fatal CHD, and coronary revascularization only if the participant also had prior or concurrent adjudicated angina. Myocardial infarction was defined on the basis of symptoms, ECG, and cardiac biomarker levels. For a classification of myocardial infarction, a participant must have had cardiac biomarkers twice the upper limit of normal, regardless of pain or ECG findings; Q waves, regardless of pain or biomarker levels; or a combination of chest pain, ST‐T evolution or new left bundle‐branch block, and biomarker levels between once and twice the upper limits of normal.

If an individual had multiple events included in the end point, we took the time to event to be time to the earliest event. We used a cutoff of 10 years due to our interest in the estimation of 10‐year risk.

Statistical Analysis

Of note, our model development process was identical to that for the original MESA risk score, with the exception that for the updated score, race/ethnicity nor any interaction with race/ethnicity were included among the set of candidate predictors.

Our model development included 2 stages of regularization. First, we used least absolute shrinkage and selection operator (LASSO) regression for variable selection. In the LASSO penalization step, we considered the continuous variables age, log(CAC+1), total cholesterol (mg/dL), high‐density lipoprotein cholesterol (mg/dL), seated systolic blood pressure (mm Hg), and the binary variables sex, diabetes, known family history of heart disease, hypertension medication use, current smoking status, and use of lipid‐lowering medication. Additionally, interactions were considered between the variables age, sex, and CAC with all other predictors; antihypertensive medications with systolic blood pressure; and lipid‐lowering medications with total cholesterol.

Next, we implemented a ridge regression step, where the coefficients for the variables retained from the LASSO step were shrunk to reduce overfitting. LASSO and ridge tuning parameters were selected via 10‐fold cross‐validation, using the partial likelihood as the performance criterion. Models were fit and cross‐validation performed using the implementation of the penalized R package version 0.9‐52 (R Foundation for Statistical Computing, Vienna, Austria).

The proportional hazards assumption was assessed by considering the Schoenfeld residuals of the unpenalized model including the same predictors. The Grambsch–Therneau test indicated that none of the models considered showed a global violation in the assumption.

We repeated the same model development process without the CAC variable to investigate the implications of including race/ethnicity in the absence of knowledge about CAC.

Comparison of Performance

We evaluated both the discrimination and calibration of the fitted models and compared these metrics for the updated risk score model, excluding race/ethnicity, with the original MESA risk score. It is worth noting that the published MESA risk score without CAC used the set of covariates that were selected in the LASSO step with CAC, where no interactions were retained. For the updated MESA risk score, we included the interactions that were selected in the corresponding LASSO step for both the versions with and without CAC.

The metrics considered were as follows: For discrimination, we considered the area under the time‐dependent receiver operating characteristic curve (AUC), ¹⁸ Harrell's concordance index, and the discrimination slope, where the discrimination slope is the average predicted 10‐year probability of an event among those who experienced an event minus the average predicted 10‐year probability of an event among those who did not experience an event. To characterize the change in discrimination between the models with and without race/ethnicity among those experiencing an event and those who did not, we also calculated the event and nonevent continuous net reclassification index (NRI). In this context, the event NRI can be thought of as the proportion of participants who had an event who were correctly assigned a higher 10‐year risk by the model without race/ethnicity than they were by the model with race/ethnicity minus the proportion who were incorrectly assigned a lower risk by the model without race/ethnicity than they were by the model with race/ethnicity (accounting for right‐censoring). The nonevent NRI is the proportion of participants who did not have an event who were correctly assigned a lower 10‐year risk by the model without race/ethnicity compared with the model with race/ethnicity, minus the proportion who were incorrectly assigned a higher 10‐year risk by the model without race/ethnicity.

For calibration, we considered the calibration plot, Nam and D'Agostino's calibration test, ¹⁹ and the Cox intercept model. ²⁰ The calibration plots were constructed by splitting participants into groups by their decile of 10‐year predicted probability of having an event, and, for each group, plotting the Kaplan–Meier estimate of the observed risk over 10 years versus the average 10‐year predicted risk for that group. The Cox intercept model is a logistic regression model of the 10‐year event status on the logit of the risk score's predicted probabilities of an event. For comparison of overall predictive accuracy, we considered the difference in Brier scores.

Bootstrap optimism correction with 200 replicates was performed for the time‐dependent AUC, Harrell's concordance index, the discrimination slope, Cox intercept slope and intercept, and the Brier scores using the location‐shifted approach to adjust CIs. ²¹

Beyond comparison of performance, we compared the predicted risk values to see the extent to which risk categorization changes between the original and race‐free MESA risk score.

Results

The baseline characteristics of the study participants stratified by race/ethnicity are summarized in Table 1. Of the 6814 total participants, 29 participants were excluded due to loss of follow‐up, and 67 participants were excluded for missing covariates. Among the 67 participants, 26 were missing high‐density lipoprotein cholesterol, 15 were missing lipid‐lowering medication, 19 were missing smoking status, 5 were missing diabetes status, and 2 were missing systolic blood pressure.

Table 1.

Baseline Characteristics at Examination 1

Characteristic	White (N=2592)	Black (N=1842)	Chinese (N=798)	Hispanic/Latino (N=1486)
Age, y	63±10	62±10	62±10	61±10
Male	1241 (48)	826 (45)	386 (48)	718 (48)
Log(mean of Agatston Calcium Score +1)	2.60±2.63	1.89±2.43	2.08±2.36	1.93±2.42
On lipid‐lowering medication	478 (18)	309 (17)	116 (15)	194 (13)
Diabetes	156 (6.0)	323 (18)	103 (13)	261 (18)
Known family history of heart disease	1258 (49)	729 (40)	146 (18)	560 (38)
On hypertension medication	860 (33)	927 (50)	229 (29)	486 (33)
Currently smokes	299 (12)	330 (18)	45 (5.6)	201 (14)
Total cholesterol, mg/dL	196±35	190±36	193±32	198±37
High‐density lipoprotein cholesterol, mg/dL	52±16	52±15	50±13	48±13
Seated systolic blood pressure, mm Hg	123±20	132±22	125±22	127±22

Values are reported as mean±SD for continuous variables and n (%) for categorical variables.

Over the 10‐year follow‐up period considered, there were 430 events, including 49 CHD deaths, 205 nonfatal myocardial infarctions, 21 resuscitated cardiac arrests, and 155 revascularizations with prior or concurrent angina (Table 2).

Table 2.

Event Counts for Events Defining the End Point

Event	White	Black	Chinese	Hispanic/Latino	Total
Atherosclerotic CHD death	16	20	4	9	49
Nonfatal myocardial infarction	87	52	16	50	205
Resuscitated cardiac arrest	7	11	0	3	21
Revascularization with prior or concurrent angina	75	36	15	29	155

Event counts contributing to the end point. Only the first event is counted. For example, if someone had a resuscitated cardiac arrest before they died of atherosclerotic CHD, their first event is considered to be a resuscitated cardiac arrest.

CHD indicates coronary heart disease.

MESA Risk Score With CAC

When race/ethnicity were included in the model, no interaction terms were retained (Figure 1A). However, when race/ethnicity were not included, several interaction terms were selected, including several with CAC. After variable selection, the ratio of events to parameters was 23.89. The inclusion of these interaction terms did contribute to a statistically significant difference in discrimination, with an improvement of 0.00423 (95% CI, 0.00038–0.00807) in time‐dependent AUC, though with optimism correction this difference was not statistically significant (estimate, 0.00227 [95% CI, −0.00159 to 0.00611]). Stratifying by race/ethnicity, the difference in time‐dependent AUC is only statistically significant among the Black participants, where the model with interactions had a time‐dependent AUC 0.0102 higher (95% CI, 0.0012–0.0193) than the model without interactions, though, again, this significance was not retained when optimism correction was performed (estimate, 0.00860 [95% CI, −0.00087 to 0.01808]). The overall difference in Brier scores comparing the model with and without interaction terms also showed a significant improvement in overall predictive accuracy when interaction terms were included (difference in Brier scores −0.000713 [95% CI, −0.001020 to −0.000407], optimism‐corrected, −0.000495 [95% CI, −0.00080 to −0.000217]). When stratifying by race/ethnicity, we find that the model with interactions had a statistically significant improvement in overall predictive accuracy in all subgroups except the Chinese, in which case the difference in Brier scores was not statistically significant, as is evident in Table S1. When correcting for optimism, there was only a statistically significant improvement in overall prediction accuracy in the Black and Hispanic/Latino subgroups (estimates, −0.000860 [95% CI, −0.001430 to −0.000302]; and −0.000737 [95% CI, −0.001386 to −0.000090], respectively).

Figure 1. Variables selected in models with and without race/ethnicity.

Variables selected in the LASSO step in the models with and without race/ethnicity, in the setting with CAC (A) and without CAC (B). CAC indicates coronary artery calcium; HDL, high‐density lipoprotein; LASSO, least absolute shrinkage and selection operator; and SBP, systolic blood pressure.

Hazard ratios (HRs) for the updated model and instructions for calculation of the risk score are presented in Table 3.

Table 3.

Centered and Uncentered Coefficients for the Risk Score

	Variable	Centered		Uncentered
		Coefficient	HR	Coefficient	HR
Main	Age, y	0.028	1.029	0.028	1.029
	Sex (male)	0.377	1.458	−0.195	0.823
	Log(CAC+1)	0.338	1.402	0.506	1.659
	Lipid‐lowering medication	0.368	1.445	1.658	5.247
	Diabetes	0.389	1.476	0.389	1.476
	Known family history of heart disease	0.394	1.483	1.522	4.580
	Hypertension medication	0.241	1.272	0.944	2.571
	Current smoking status	0.685	1.983	0.928	2.531
	Total cholesterol	0.002	1.002	0.002	1.002
	High‐density lipoprotein cholesterol	−0.012	0.988	−0.012	0.988
	Systolic blood pressure	0.014	1.014	0.017	1.017
Interaction	Age×systolic blood pressure	−0.018	0.982	−0.018	0.982
	Age×lipid‐lowering medication	−0.019	0.981	−0.019	0.981
	Total cholesterol×sex	0.003	1.003	0.003	1.003
	Systolic blood pressure×hypertension Medication	−0.006	0.994	−0.006	0.994
	Log(CAC+1)×lipid‐lowering medication	−0.054	0.948	−0.054	0.948
	Log(CAC+1)×systolic blood pressure	−0.001	0.999	−0.001	0.999
	Log(CAC+1)×current smoking status	−0.111	0.895	−0.111	0.895
	Baseline survival at year 10	0.9780195287		0.9997573857

Coefficients and HRs for the centered and uncentered coefficients. The centered version is presented for ease of interpretation of the main effects. This version corresponds to the coefficients when continuous covariates are centered at their mean values while, categorical covariates are at their reference values. The uncentered version is included to predict an individual's risk. To estimate an individual's 10‐year risk of a coronary heart disease event, multiply the values of the risk factors by the corresponding uncentered β coefficient and sum these values to yield a value, denoted B. Then compute 1—(uncentered baseline hazard)^(exp(B)).

CAC indicates coronary artery calcium; and HR, hazard ratio.

The overall predicted event rates were similar between the models with and without race/ethnicity, with a difference of 0.012% (95% CI, −0.033 to 0.057]), as shown in Table 4. When stratifying by race/ethnicity, we see the most notable difference in predicted event rates in the Chinese participants, where the difference between the models with and without race/ethnicity is 0.94% (95% CI, 0.82–1.10), as shown in Table 5. In this case, the model without race/ethnicity tends to overestimate the risk, failing to fully capture the lower risk in our Chinese participants. For further comparison of the predicted 10‐year risks between the models in this population, the number of individuals with their 10‐year predicted risk <5%, 5% to 7.5%, 7.5% to 20%, and >20% for the models with and without race/ethnicity, both stratified by race/ethnicity and overall, are presented in Tables S2 and S3. While there is broad agreement in the assignment of risk categories, some additional events are (correctly) classified as higher risk by the model without race/ethnicity, and some nonevents are (incorrectly) classified as higher risk by the model without race/ethnicity. From Figure 2A, it is evident that the overestimation occurs primarily for those in the lowest quantiles of predicted risk, and similar overestimation occurs in the model with race/ethnicity. From this figure, the calibration in the Chinese participants appears comparable between the models with and without race/ethnicity.

Table 4.

Overall Expected and Observed Event Rates

CAC	Observed event rate, %	Without race/ethnicity, %	With race/ethnicity, %	95% CI for difference: without race/ethnicity—with race/ethnicity
With CAC	6.90 (6.27 to 7.53)	7.20 (7.00 to 7.40)	7.19 (6.99 to 7.38)	0.012 (−0.033 to 0.057)
Without CAC		7.15 (6.99 to 7.31)	7.12 (6.97 to 7.27)	0.029 (−0.009 to 0.067)

Observed and expected event rates and their 95% CIs, where the observed event rates correspond to Kaplan–Meier estimates and the expected event rates correspond to those predicted by the models with and without race/ethnicity.

CAC indicates coronary artery calcium.

Table 5.

Expected and Observed Event Rates, Stratified by Race/Ethnicity

CAC	Race/ethnicity	Observed event rate	Without race/ethnicity	With race/ethnicity	95% CI for difference: without race/ethnicity—with race/ethnicity
With CAC	White	7.53 (6.48 to 8.57)	7.94 (7.59 to 8.28)	7.81 (7.49 to 8.13)	0.13 (0.05 to 0.20)
	Black	7.07 (5.84 to 8.3)	6.91 (6.55 to 7.27)	7.39 (7.01 to 7.77)	−0.48 (−0.56 to −0.40)
	Chinese	4.79 (3.22 to 6.33)	5.84 (5.36 to 6.31)	4.89 (4.52 to 5.27)	0.94 (0.82 to 1.10)
	Hispanic/Latino	6.7 (5.35 to 8.02)	7 (6.55 to 7.45)	7.08 (6.65 to 7.51)	−0.079 (−0.17 to 0.01)
Without CAC	White	7.53 (6.48 to 8.57)	7.02 (6.77 to 7.27)	7.65 (7.4 to 7.9)	−0.63 (−0.68 to −0.58)
	Black	7.07 (5.84 to 8.3)	7.58 (7.28 to 7.89)	7.34 (7.05 to 7.62)	0.24 (0.18 to 0.31)
	Chinese	4.79 (3.22 to 6.33)	6.11 (5.71 to 6.51)	4.92 (4.61 to 5.22)	1.20 (1.10 to 1.30)
	Hispanic/Latino	6.70 (5.35 to 8.02)	7.38 (7.03 to 7.74)	7.1 (6.77 to 7.42)	0.29 (0.21 to 0.36)

Observed and expected event rates along with 95% CIs, stratified by race/ethnicity. Again, observed event rates correspond to Kaplan–Meier estimates, and the expected event rates correspond to those predicted by the models with and without race/ethnicity.

CAC indicates coronary artery calcium.

Figure 2. Calibration plots, stratified by race/ethnicity.

Hosmer–Lemeshow calibration plots for the models with CAC (A) and without CAC (B), stratified by race/ethnicity. Participants in each racial and ethnic group were divided into quantile groups based on quantile of predicted risk. For each quantile group of participants, the Kaplan–Meier estimate of the observed risk was calculated. The line in red is the line of unity. CAC indicates coronary artery calcium.

The AUCs for each model of interest are shown in Table 6. The AUC for the model without race/ethnicity was 0.820 (95% CI, 0.801–0.840), while with race/ethnicity it was 0.817 (95% CI, 0.797–0.836). With optimism correction, the AUC for the model without race/ethnicity was 0.813 (95% CI, 0.794–0.833) and that for the model with race/ethnicity was 0.812 (95% CI, 0.792–0.831). The overall difference in AUC between the models with and without race/ethnicity was not significant (estimate, 0.0032 [95% CI, −0.0008 to 0.0073]; optimism‐corrected, 0.0010 [95% CI, −0.0030 to 0.0050]). When stratifying by race/ethnicity, the difference in AUC between the models with and without race/ethnicity was not significant for any racial/ethnic group except for the Black participants, where the model without race/ethnicity was actually slightly better, showing a difference in AUC of 0.0103 (95% CI, 0.0008–0.0198), although this statistical significance was not retained when correcting for optimism (estimate, 0.0087 [95% CI, −0.0008 to 0.0182]).

Table 6.

AUC Comparison: Race‐Free Versus With Race (With CAC)

Race/Ethnicity	AUC in model without race/ethnicity	AUC in model with race/ethnicity	Difference: without race/ethnicity—with race/ethnicity
Overall	0.820 (0.801 to 0.840)	0.817 (0.797 to 0.836)	0.0032 (−0.0008 to 0.0073)
White	0.799 (0.767 to 0.831)	0.797 (0.765 to 0.828)	0.0026 (−0.0036 to 0.0088)
Black	0.811 (0.772 to 0.850)	0.800 (0.760 to 0.841)	0.0103 (0.0008 to 0.0198)
Chinese	0.860 (0.812 to 0.908)	0.863 (0.814 to 0.912)	−0.0030 (−0.0130 to 0.0070)
Hispanic/Latino	0.851 (0.813 to 0.888)	0.847 (0.810 to 0.884)	0.0036 (−0.0032 to 0.0103)

AUC for each model of interest, with CAC. The second and third columns present the AUCs and corresponding 95% CIs for the models without and with race/ethnicity, both stratified by race/ethnicity and overall. The fourth column shows the difference in AUC between the 2 models, calculated as (without race/ethnicity)—(with race/ethnicity).

AUC indicates area under the time‐dependent receiver operating characteristic curve; and CAC, coronary artery calcium.

Harrell's concordance index demonstrated a similar pattern, with an estimated value of 0.800 (95% CI, 0.781–0.819) and 0.797 (95% CI, 0.778–0.816) in the models without and with race/ethnicity, respectively (optimism‐corrected, 0.793 [95% CI, 0.774–0.812]; and 0.791 [95% CI, 0.772–0.810], respectively). The difference in Harrell's concordance index was 0.00322 and was not statistically significant (95% CI, −0.00070 to 0.00713; optimism‐corrected, 0.00237 [95% CI, −0.00154 to 0.00628]).

We also calculated the discrimination slope, which describes the difference in risk between events and nonevents, on average. In the model without race/ethnicity, the discrimination slope was 0.101, indicating that the predicted risk among events was 10.1% higher than nonevents, on average (optimism‐corrected, 0.094). With race/ethnicity, the discrimination slope was 0.0937 (optimism‐corrected, 0.088).

The event NRI was 0.18 (95% CI, 0.084–0.267), where it was estimated that 59.3% of participants experiencing an event were correctly assigned a higher 10‐year risk and 41.3% were incorrectly assigned a lower 10‐year risk. The nonevent NRI was 0.32 (95% CI, 0.303–0.349), where an estimated 66.3% of participants not experiencing an event were correctly assigned a lower 10‐year risk and 33.6% were incorrectly assigned a higher 10‐year risk. That is, both the event and nonevent NRI values indicate that the model without race/ethnicity was, more often than not, beneficially assigning a higher 10‐year risk to those experiencing events than the model with race/ethnicity and a lower 10‐year risk to those not experiencing events than the model with race/ethnicity.

With regard to calibration, the calibration plots presented in Figure 3A show that both the models with and without race/ethnicity appear adequately well calibrated. This was also true when stratifying by race/ethnicity (Figure 2A) or by sex (Figure S1). This was further substantiated by Nam and D'Agostino's calibration test, which was not statistically significant for both the models with and without race/ethnicity (test statistic, 14.70; P=0.065; and test statistic, 6.73; P=0.57, respectively). Nam and D'Agostino's calibration test was also performed separately by race/ethnicity for each model and did not indicate statistically significant miscalibration in any racial/ethnic group. In the Chinese population, in the first 5 deciles of predicted risk, the observed risk is near 0. Consequently, in both the models with and without race/ethnicity, we overestimate the risk for individuals in those deciles, which is an issue not observed with the other racial/ethnic groups. For the higher‐risk deciles in the Chinese population, the model without race/ethnicity tends to overestimate risk, while the model with race/ethnicity tends to underestimate it.

Figure 3. Calibration plots for models with and without race/ethnicity.

Hosmer–Lemeshow calibration plots for the models with CAC (A) and without CAC (B). Participants were divided into groups based on quantile of predicted risk. For each quantile group of participants, the Kaplan–Meier estimate of the observed risk was calculated. The line in red is the line of unity. CAC indicates coronary artery calcium.

To further assess the calibration, we also fit a Cox intercept model. The null hypothesis is that the intercept is 0 and the slope is 1, ²⁰ which would indicate perfect calibration, where the predicted probabilities are exactly equal to the true underlying probabilities. The intercept for the model without race/ethnicity was −0.227 (95% CI, −0.448 to −0.007), indicating slight overestimation of the risk of an event (optimism‐corrected, −0.152 [95% CI, −0.531 to −0.090]). There was less overestimation for the model with race/ethnicity, where the intercept was −0.152 (95% CI, −0.380 to 0.075) (optimism‐corrected, −0.218 [95% CI, −0.446 to 0.008]). The CIs for the slope of the models both included 1, and hence we fail to reject the null (estimate, 0.993 [95% CI, 0.896–1.090]; optimism‐corrected, 0.960 [95% CI, 0.863–1.057]; and estimate, 0.957 [95% CI, 0.862–1.050]; optimism‐corrected, 0.915 [95% CI, 0.820–1.008] with and without race/ethnicity, respectively). Values of the slope significantly >1 would indicate a pattern of overestimation of probabilities less than one half and underestimation of probabilities greater than one half, whereas values of the slope significantly <1 would indicate a pattern of underestimation of probabilities less than one half and overestimation of probabilities greater than one half.

The overall prediction accuracy, as measured by the Brier score, was higher in the model without race/ethnicity (difference in Brier scores, −0.00072 [95% CI, −0.00103 to −0.00041]; optimism‐corrected, −0.00058 [95% CI, −0.00089 to −0.00027]), with a Brier score of 0.0490 (95% CI, 0.0443–0.0536; optimism‐corrected, 0.0495 [95% CI, 0.0448–0.0541]) in the model without race/ethnicity; and 0.0497 [95% CI, 0.0450–0.0544]; optimism‐corrected, 0.0501 [95% CI, 0.0454–0.0548]) in the model with race/ethnicity. When stratifying by race/ethnicity, we find that the overall predictive accuracy was significantly better in the model without race/ethnicity among the White, Hispanic/Latino, and Chinese subgroups (difference of −0.00085 [95% CI, −0.00138 to −0.00031]; optimism‐corrected, −0.00066 [95% CI, −0.00119 to −0.00012]; difference, −0.00089 [95% CI, −0.00153 to −0.00024]; optimism‐corrected, −0.00092 [95% CI, −0.00157 to −0.00028]; and difference, −0.00203 [95% CI, −0.00153 to −0.00024]; optimism‐corrected, −0.00185 [95% CI, −0.00278 to −0.00092], respectively), and was not significantly different among the Black subgroup (difference, 0.00167 [95% CI, −0.000349 to 0.000683]; optimism‐corrected, 0.00024 [95% CI, −0.00027 to 0.00076]). When stratifying by sex, we found that the model without race/ethnicity had a higher overall predictive accuracy in both men and women (difference, −0.00119 [95% CI, −0.00177 to −0.000616] and −3×10⁻⁴ [95% CI, −0.000581 to −1.83×10⁻⁵], respectively), as shown in Table S4.

MESA Risk Score Without CAC

When excluding race/ethnicity in the setting without CAC, additional interaction terms were selected (Figure 1B). After variable selection, the ratio of events to parameters was 26.88, similar to that for the model with CAC. However, in this case, when comparing the model with the interaction terms with that without, we find that the additional interaction terms do not yield a statistically significant difference in AUC (estimate, 0.0037 [95% CI, −0.0001 to 0.0076]; optimism‐corrected, 0.0016 [95% CI, −0.0022 to 0.0055]).

The aforementioned challenge in capturing the lower risk of our Chinese population is exacerbated without CAC, in which case the difference in predicted event rates between the models with and without race/ethnicity is 1.20% (95% CI, 1.10–1.30), as shown in Table 5. Again, the model without race/ethnicity tends to overestimate the risk in the Chinese participants. Both models, however, have the worst calibration for the Chinese compared with the other racial/ethnic groups, which is evident in Figure 2B. Also of note, in the models without CAC, the issues with calibration are not limited to the groups with the lowest risk: We see miscalibration in the intermediate risk range, which is more impactful for treatment decisions. Risk score categorizations (10‐year predicted risk <5%, 5%–7.5%, 7.5%, and >20%) for the models with and without race/ethnicity are presented both overall and stratified by race/ethnicity in Tables S5 and S6. Despite the miscalibration we see in Figure 2B, the models with and without race/ethnicity largely agree on the assignment of risk score categories, and the model without race/ethnicity correctly assigned several nonevents to lower‐risk categories than the model with race/ethnicity.

Without CAC, as shown in Table 7, we see an AUC of 0.770 (95% CI, 0.749–0.791) and 0.772 (95% CI, 0.752–0.793) for the models with and without race/ethnicity, respectively (optimism‐corrected, 0.762 [95% CI, 0.741–0.783]; and 0.764 [95% CI, 0.744–0.785]). The overall difference in AUC was not statistically significant (estimate, 0.0021 [95% CI, −0.0027 to 0.0070]; optimism‐corrected, 0.0015 [95% CI, −0.0033 to 0.0064]). Furthermore, when stratifying by race/ethnicity, the difference in AUC was not statistically significant for any racial/ethnic group, with or without optimism correction. Similarly, Harrell's concordance index was 0.750 (95% CI, 0.729–0.770; optimism‐corrected, 0.742 [95% CI, 0.721–0.762]) in the model without race/ethnicity and 0.749 (95% CI, 0.728–0.769; optimism‐corrected, 0.7416 [95% CI, 0.7206–0.7616]), and the difference was not significant (estimate, 0.000986 [95% CI, −0.00371 to 0.00568]; optimism‐corrected, 0.00031 [95% CI, −0.00439 to 0.00500]). The discrimination slopes were also comparable, at 0.0587 (optimism‐corrected, 0.0508) for the model without race/ethnicity and 0.0547 (optimism‐corrected, 0.0473) for that with race/ethnicity. However, inspection of the event NRI values reveals that, more often than not, the model without race/ethnicity predicted lower 10‐year risk values for those experiencing an event than the model with race/ethnicity (event NRI, −0.243 [95% CI, −0.336 to −0.154]). Similarly, compared with the model with race/ethnicity, the model without race/ethnicity more frequently predicted higher 10‐year risk among nonevents.

Table 7.

AUC Comparison: Race‐Free Versus With Race (Without CAC)

Race/ethnicity	AUC in model without race/ethnicity	AUC in model with race/ethnicity	95% CI (without race/ethnicity—with race/ethnicity)
Overall	0.772 (0.752 to 0.793)	0.770 (0.749 to 0.791)	0.0021 (−0.0027 to 0.0070)
White	0.754 (0.720 to 0.788)	0.750 (0.716 to 0.785)	−0.00048 (−0.00468 to 0.00373)
Black	0.754 (0.713 to 0.795)	0.748 (0.706 to 0.790)	−0.0042 (−0.01563 to 0.00723)
Chinese	0.794 (0.738 to 0.851)	0.801 (0.745 to 0.857)	0.00564 (−0.00359 to 0.01487)
Hispanic/Latino	0.818 (0.778 to 0.857)	0.820 (0.781 to 0.859)	0.00227 (−0.00784 to 0.01237)

AUC for each model of interest, without CAC. The second and third columns present the AUCs and corresponding 95% CIs for the models without and with race/ethnicity, both stratified by race/ethnicity and overall. The fourth column shows the difference in AUC between the 2 models, calculated as (without race/ethnicity)—(with race/ethnicity).

AUC indicates area under the time‐dependent receiver operating characteristic curve; and CAC, coronary artery calcium.

That said, a comparison of overall prediction accuracy, assessed with the Brier score, showed that the model without race/ethnicity shows a statistically significant improvement over the model with race/ethnicity, although this difference is very small (difference in Brier scores, −0.00046 [95% CI, −0.00072 to −0.00020]; optimism‐corrected, −0.00036 [95% CI, −0.00062 to −0.00010]). When stratifying by race/ethnicity, there was a statistically significant improvement (with and without optimism correction) in overall prediction accuracy in the model without race/ethnicity for each subgroup except among the White participants, where the predictive accuracy was lower in the model without race/ethnicity than that with race/ethnicity (difference in Brier scores, 0.000774 [95% CI, 0.00035–0.00120]; optimism‐corrected, 0.00091 [95% CI, 0.00048–0.00133]).

Putting these results together, we find that the discrimination and overall prediction accuracy remain comparable between the models with and without race/ethnicity when CAC is excluded. However, in contrast with the findings on the removal of race/ethnicity when CAC is included, the exclusion of CAC does not appear to allow for comparable performance without race/ethnicity. While the overall difference in AUC remains not significant when excluding CAC, removing race/ethnicity when CAC is not included has a noteworthy impact on the calibration. In particular, Nam and D'Agostino's calibration test indicates that there is statistically significant miscalibration in the model that excludes CAC and race/ethnicity (test statistic, 17.10; P=0.029), while there is no evidence of miscalibration for the model that excludes CAC but does include race/ethnicity (test statistic, 13.4; P=0.099). This is also apparent in Figure 3B and Figure S2, where the points stray farther from the line of unity for the model without race/ethnicity than that with race/ethnicity. In particular, comparing panels (A) and (B) of Figures 2 and 3, when race/ethnicity is not included, we see noticeably better calibration in the model with CAC. That said, the slope and intercept of the Cox intercept model showed no evidence of miscalibration for the models with and without race/ethnicity, which may be due to averaging out of the more granular trends in the over‐ and underprediction that are evident in the calibration plots (Figures 2 and 3).

Discussion

Our results indicate that, with CAC, the race‐free MESA risk score performs comparably to the MESA risk score with regard to both calibration and discrimination. Without CAC, the race‐free score has similar discrimination but has slightly worse calibration. That is, in the MESA cohort, when we have CAC scores, we are able to predict the risk of CHD just as well without adjusting for race/ethnicity, and without CAC, we see only a small drop in calibration. Our findings build on previous research assessing the role of race in risk prediction.

Our analysis with the MESA cohort provides particular insight into the extent to which race/ethnicity affects predictions in the presence and absence of a covariate that is highly predictive of the end point. In the context of the MESA risk score, when we included CAC, a strong predictor of the CHD risk, in the risk score, the inclusion of race/ethnicity did not make a meaningful difference in performance. However, when we performed the same analysis without including CAC in the risk score, the model without race/ethnicity showed evidence of significant miscalibration. This indicates that if a risk score includes a strong predictor of the outcome of interest, race/ethnicity may not make a meaningful difference in its predictions. However, in the absence of a strong predictor like CAC, more work may be needed to get comparable performance. Indeed, other work has shown that, in other contexts, excluding race can worsen a risk score's measures of algorithmic fairness. ²² This points to the nuance in the discussion of how we can adapt risk scores to be race‐free. There are important disparities by race/ethnicity in CVD, ⁶ and it is important that risk scores can reflect this. However, our results suggest that these differences are adequately captured by the cardiovascular risk factors included in the race‐free model.

The comparable performance of the race‐free MESA risk score indicates that it could serve as a viable alternative to the previous version. This is particularly relevant given the potential harms of including race in risk prediction models: There are numerous examples of race‐based equations creating harm in marginalized populations ⁸ One example is a score that was intended to be used for referral to cardiology services classifying Black patients as lower risk of death, which could reduce access to care. ²² Another pertinent example is how race‐based estimated glomerular filtration rate equations result in Black patients getting higher values, indicating better kidney function, which has had adverse consequences to the care received and ultimately on health outcomes. ²³

The possibility of such consequences has motivated the development of race‐free versions of several other risk scores. In 2021, race‐free estimated glomerular filtration rate equations were released, ²⁴ and these equations are now recommended for clinical practice by the National Kidney Foundation and the American Society of Nephrology. ²⁵ Additionally, in 2024, updated race‐free PREVENT equations were published for the prediction of CVD. ¹⁴ Of note, both the updated American Heart Association PREVENT equations and updated MESA risk score shared the goal of producing a race‐free version of a previously published risk score. However, the recently updated American Heart Association PREVENT equations were developed using a new model incorporating measures of kidney function, metabolic health, and socioeconomic variables that were not included in the pooled cohort equations. By contrast, the focus of the present study was whether removing race/ethnicity from the model development of the MESA risk score materially altered its performance.

The PREVENT equations are also relevant for our understanding of the challenges faced for risk prediction in the Chinese participants of the MESA cohort. The race‐free PREVENT equations had the worst calibration in the Asian population of the racial and ethnic groups considered, with a calibration slope of 0.87 (interquartile range [IQR], 0.73–0.97), compared with 1.11 (IQR, 0.79–1.24) in Black individuals, 1.01 (IQR, 0.82–1.14) in non‐Hispanic White individuals, and 0.94 (IQR, 0.80–1.05) in Hispanic individuals, where the IQR is calculated as the 25th to 75th percentile of cohorts. A calibration slope <1 for the Asian population reflects overpredicting risk, as was the case for the MESA risk score for our Chinese participants, which we saw in both the models with and the models without CAC (Figure 2). Also of note was that in the MESA risk score without CAC, the issues with calibration were not limited to the groups with the lowest risk: There was miscalibration in the intermediate‐risk range, which is more impactful for treatment decisions.

The overprediction of risk in the Chinese participants in the MESA risk score may be a result of unmeasured variables contributing to the lower risk among these participants, which underscores the importance of future research to adequately understand the lower risk of these participants. One way to capture this lower risk may be the inclusion of additional covariates. One that may be relevant is the number of years living in the United States, as prevalence and mortality rates from CHD are higher among Chinese immigrants than those living in mainland China. ²⁶ Another is C‐reactive protein, which may be related to the higher atherosclerotic CVD risk in South Asian individuals when compared with Chinese individuals. ²⁷ In the CHCN‐BTH (Cohort Study on Chronic Disease of Communities Natural Population in Beijing, Tianjin, and Hebei), a large cohort study including participants from mainland China, elevated lipoprotein(a) was associated with increased risk of CHD, a trend that also held in individuals without dyslipidemia. ²⁸ It is also important to recognize that the challenge of capturing risk in the Chinese may evolve over time: Nguyen and colleagues found that, in the United States, CVD is increasing more quickly among Chinese individuals when compared with non‐Hispanic White individuals. ²⁹ The recently launched MOSAAIC (Multi‐Ethnic Observational Study in American Asian and Pacific Islander Communities) may also shed more light on cardiovascular risk prediction in this population. It is important to emphasize that both the models with and without race/ethnicity had challenges in capturing the lower risk in the Chinese participants, and this is a crucial area for future work.

Moving forward, there are several additional factors that motivate the use of the race‐free MESA risk score. First, it avoids the conflation of race as a biological risk factor like systolic blood pressure or total cholesterol. This may help to avoid perpetuating myths around race/ethnicity in medicine, facilitating a shift away from race‐based medicine. We point interested readers to the National Academies of Sciences, Engineering, and Medicine's report on this issue ³⁰ for a more in‐depth discussion on the pitfalls of race‐based medicine and suggestions moving forward than we are able to provide here. Second, the race‐free risk score is more inclusive in the sense that an individual's risk can still be calculated for those who do not fit cleanly into 1 of the racial and ethnic groups considered in MESA or who do not wish to disclose their race/ethnicity. That said, the risk score was estimated using the MESA cohort, and consequently, caution should be taken if interpreting the risk score for someone outside the population from which the MESA study participants were drawn.

We wish to emphasize that the removal of race/ethnicity from a risk score is a nuanced methodological and ethical issue. If we had found, for example, that the performance of the race‐free score was significantly worse overall or in a particular subgroup, this would not mean that the pursuit of a race‐free score should be abandoned. Such a finding could reflect that the current set of predictors is insufficient to capture variation due to variables for which race/ethnicity was acting as a proxy. In our setting, we found that the simple removal of race/ethnicity from our model development did not worsen performance and, by some measures, even improved performance.

That is, the findings of our comparative study challenge the commonly held assumption that including race/ethnicity in model development will improve performance. Our results demonstrate that, in a setting with strong predictors of the outcome of interest, it is possible that race/ethnicity can be removed from model development without worsening overall or race/ethnicity–specific performance. This supports the conclusion that ethics‐oriented model modifications need not come at the expense of predictive accuracy when available predictors adequately capture variation in risk. Our performance assessment reflects the view that fairness cannot be equated with the removal of race/ethnicity from the risk score and requires careful evaluation of model performance (both overall and stratified by race/ethnicity) and continued efforts to improve measurement of the drivers of risk that race may proxy. Taken together, our analyses suggest that the extent to which a risk score needs to be modified to work successfully without race/ethnicity may depend on the strength of the relationship among other candidate predictors and the outcome of interest; this can inform ongoing discussions by the American Heart Association and the National Academies on removing race from clinical algorithms while maintaining fairness.

Limitations

The generalizability of our findings is limited by the characteristics of those included in MESA. In particular, all participants in MESA fall into 1 of the 4 racial/ethnic groups White, Black, Chinese, and Hispanic/Latino. Additionally, all individuals were within the ages of 45 to 85 years and were free of preexisting CVD. Because, in clinical settings, CAC is typically measured only for high‐risk patients, the population to which this score is applied in clinical settings may be at higher risk than the individuals in MESA. Given that external validation was not performed in an independent cohort, while our analyses give insight into the performance of these models within MESA, the generalizability of findings to populations with different demographic compositions and risk factor distributions is not yet clear. Evaluating the performance of these models in external cohorts is an important avenue for future research.

Second, formal testing for comparison of performance addressed overall prediction accuracy, by considering the difference in Brier scores, and discrimination, by considering the difference in AUC scores and that in Harrell's concordance index. For a time‐to‐event end point, at the time of publication, we were not aware of a computationally feasible approach to formally test for a statistically significant difference in calibration between the models.

It is also worth noting that the HRs presented in Table 3 should not be interpreted causally. For example, one may note that from Table 3, the estimated risk is higher among those taking lipid‐lowering medication than those who do not. However, this reflects the fact that those taking lipid‐lowering medication have dyslipidemia, not that the lipid‐lowering medication itself elevated their risk.

Finally, in the present study, we conducted a thorough comparison of performance when making the sole change of including or excluding race/ethnicity. Future work should assess the extent to which we could improve upon the race‐free model's performance by including additional predictors.

Conclusions

We have created a race‐free MESA risk score that, with CAC, has performance on par with the original MESA risk score, with regard to both calibration and discrimination. While we also updated the CAC‐free MESA risk score, we found a small increase in the miscalibration in the model excluding race/ethnicity. However, due to the aforementioned advantages of race‐free risk prediction, we still recommend using the race‐free score. The updated risk score can be a clinically useful tool for the interpretation of patients' CAC scores in the context of other relevant clinical characteristics. This race‐free MESA risk score is more inclusive, since risk can now be calculated for individuals who do not fit into 1 of the 4 racial/ethnic groups included in MESA or who do not wish to disclose their race/ethnicity. Our work adds to a growing literature on the inclusion of race/ethnicity in risk prediction. The updated risk score is available on the MESA website.

Sources of Funding

This work is funded by the American Heart Association data grant “Debiasing Clinical Care Algorithms,” awarded in 2024. The MESA study was supported by contracts 75N92020D00001, HHSN268201500003I, N01‐HC‐95159, 75N92020D00005, N01‐HC‐95160, 75N92020D00002, N01‐HC‐95161, 75N92020D00003, N01‐HC‐95162, 75N92020D00006, N01‐HC‐95163, 75N92020D00004, N01‐HC‐95164, 75N92020D00007, N01‐HC‐95165, N01‐HC‐95166, N01‐HC‐95167, N01‐HC‐95168, and N01‐HC‐95169 from the National Heart, Lung, and Blood Institute; and by grants UL1‐TR‐000040, UL1‐TR‐001079, and UL1‐TR‐001420 from the National Center for Advancing Translational Sciences.

Disclosures

None.

Supporting information

Data S1

STROBE Statement