Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Mar 1.
Published in final edited form as: Atherosclerosis. 2018 Dec 15;282:202–209. doi: 10.1016/j.atherosclerosis.2018.12.005

Multisite atherosclerosis in subjects with metabolic syndrome and diabetes and relation to cardiovascular events: The Multi-Ethnic Study of Atherosclerosis

Yanglu Zhao a,b, Marcella A Evans c, Matthew A Allison d, Alain G Bertoni e, Matthew J Budoff f, Michael H Criqui d, Shaista Malik b, Pamela Ouyang g, Joseph F Polak h, Nathan D Wong a,b,c
PMCID: PMC6401246  NIHMSID: NIHMS1517702  PMID: 30600075

Abstract

Background and aims:

The extent and relation of multisite atherosclerosis to cardiovascular disease (CVD) in metabolic syndrome (MetS) and diabetes (DM) are not well documented. We examined the extent of multisite atherosclerosis and its prognostic value for CVD events in MetS and DM.

Methods:

In CVD-free subjects from the Multi-Ethnic Study of Atherosclerosis, multisite atherosclerosis was measured as: (1) the number of arterial beds involved (coronary calcium>0, abdominal aortic calcium>0, carotid intima-media thickness ≥1mm and ankle brachial index<1 or ≥1.4); (2) a composite score summing the quartile rank for each atherosclerosis measure. Hazard ratios (HRs) and c-statistics were calculated for incident CVD and coronary heart disease (CHD) over 10.6 years.

Results:

Of 1,675 individuals (mean age 64 years, 51% male), 33.4% had MetS and 15.9% had DM. The number of atherosclerotic sites was higher in those with DM (mean±SD=1.67±1.15) and MetS (1.49±1.12) versus neither MetS/DM (1.09±1.09) (p<0.0001). CVD rates per 1000 person-years ranged from 3.5, 8.2, and 10.0 in those with 0 sites positive to 35.1, 79.6 and 103.4 in those with 4 sites positive among neither DM/MetS, MetS and DM groups, respectively. HRs (95% CI) for CVD comparing those with 4 vs. 0 atherosclerotic sites were 4.0 (0.8–19.1), 4.9 (2.0–12.0), and 14.4 (3.6–57.6), respectively. C-statistics adding multisite atherosclerosis measures increased over models without the measures and with CIMT or ABI but not CAC.

Conclusions:

Multisite atherosclerosis is greater with MetS or DM, and predicts CVD and CHD events. Risk prediction is improved over CIMT and ABI but not CAC.

Keywords: Multisite atherosclerosis, metabolic syndrome, diabetes, cardiovascular disease

Graphical Abstract

graphic file with name nihms-1517702-f0001.jpg

Introduction

Individuals with metabolic syndrome (MetS) and/or diabetes mellitus (DM) are at increased risk of cardiovascular disease (CVD) events and mortality13. In addition, subclinical atherosclerosis is more common based on greater levels of coronary artery calcium (CAC) and carotid intimal-medial thickness (CIMT) 4,5. We previously showed in the Multiethnic Study of Atherosclerosis (MESA) that CHD risk in those with DM and MetS can be stratified by CAC (and to a lesser extent by CIMT), showing a 10-fold difference in CHD risk between the CAC=0 group and the CAC =400+ group 6; more recently, the role of CAC in improving long-term risk reclassification, even in those with longer duration DM was demonstrated7. It is known that increased levels of CAC, a lower ankle brachial index (ABI), and a higher CIMT predict an increased risk of CHD events and mortality,812 and more recently abdominal aortic calcium (AAC) has been shown to predict higher levels of CHD and CVD events.13 Moreover, in another MESA investigation, the number of calcified extra-coronary sites was also shown to be associated in a graded fashion with the risk of CHD events and mortality and total mortality.14

Not well documented is the distribution of atherosclerosis within different vascular beds in people with MetS or DM and how this adds to CHD and CVD event prediction, and whether there are differences compared to those without MetS or DM. In this study, we aimed at examining the burden of multisite atherosclerosis and its association as well as prognostic significance for future CVD and CHD events in these individuals using the MESA data.

MATERIALS AND METHODS

Study population

MESA is a prospective community-based study of CVD among 6,814 asymptomatic men and women aged 45–84 recruited from 6 field centers (Baltimore, MD; Chicago, IL; Los Angeles, CA; New York, NY; St Paul, MN; and Winston-Salem, NC) between 2000–2002. The study design and methods have previously been presented elsewhere.15 Subjects were free of known CVD and from one of four race/ethnic groups: Caucasian, African-American, Caucasian, Chinese-American, and Hispanic. Exams 1, 2 and 3 were conducted during 2000–2002, 2003–2004 and 2005–2006, respectively. MESA was approved by the institutional review boards at all participating centers, and all participants provided written informed consent at all study visits.

AAC and CAC were measured in 1,793 participants in either exam 2 or exam 3. ABI was measured in exam 1 and exam 3. CIMT was measured in exam 1. The baseline exam in this study was either exam 2 or exam 3 according to the time of AAC and CAC scanning. For participants followed up from exam 2, ABI and CIMT at exam 1 were used; for participants followed up from exam 3, ABI from exam 3 (no ABI was done at Exam 2) and CIMT from exam 1 (a complete CIMT was only done in exam 1; Exam 2 only had right sided CIMT and no CIMT was done in Exam 3) were used. We finally included 1,675 MESA participants who had valid Exam 2 or 3 data on AAC, CAC, ABI and CIMT measures, as well as follow-up for CVD events. Participants were excluded if they had incident cardiovascular events or revascularization procedures prior to their Exam 2 or 3 CT examination.

Study measurements

Information about participant demographics (including socioeconomic status measured by educational and income level), medical history, current medication use, and family history was collected using standardized questionnaires. Resting blood pressure was measured three times with the average of the last two blood pressures used. Glucose, total cholesterol and high-density lipoprotein cholesterol (HDL-C) measurements were obtained after a 12-hour fast. The Friedewald equation was used to estimate LDL-C.15

DM was defined as physician diagnosed DM, or fasting glucose ≥ 126 mg/dL, or taking insulin or taking oral hypoglycemic medications as previously used in MESA.7 Severity of DM was examined separately according to the following: (1) DM duration < 5 years vs. ≥ 5 years; (2) HbA1c < 7% vs. ≥ 7%; (3) 10-year ASCVD risk score (PCE) < 7.5% vs. ≥ 7.5%; (4) DM + MetS vs. DM only. MetS without DM was defined based on the AHA/NHLBI 2005 definition of ≥ 3 risk factors based on: waist circumference (> 88cm in men or > 102 cm in women), HDL-cholesterol (< 40 mg/dl in men or < 50 mg/dl in women), blood pressure (≥ 130/85 mmHg or on antihypertensive medication), fasting triglycerides (≥ 150 mg/dl), and fasting glucose ≥ 100 mg/dl. Participants were stratified by these disease states based on data taken from MESA Exam 2 or 3 concomitant to their CT examination date.

Subclinical atherosclerosis measurements

CAC and AAC were detected with either an electron-beam CT scanner (Chicago, Los Angeles, and New York) or a multi-detector CT system (Baltimore, St Paul, and Winston-Salem), with calcium scores calculated using the Agatston method.16 CIMT was assessed using B-mode ultrasound (Logiq 700 ultrasound device; General Electric Medical Systems, Waukesha, WI) and calculated as the mean of common carotid IMT and inner carotid IMT (from Exam 1). 17 Systolic blood pressure measurements in the bilateral brachial, dorsalis pedis, and posterior tibial arteries were obtained in the supine position using a handheld Doppler instrument with a 5-mHz probe. The higher of the brachial artery pressures was used as the denominator. For each lower extremity, the ABI numerator used was the highest pressure (dorsalis pedis or posterior tibial) from that leg. A borderline or abnormal ABI, which has been shown to be associated with increased mortality, was defined as < 1.0 or ≥ 1.4.18

We defined multisite atherosclerosis in two different ways: (1) number of involved vascular beds (ranged 0–4): the primary multisite atherosclerosis index consisted of the number of vascular beds positive for disease defined as follows: 1) CAC >0, 2) AAC >0, 3) CIMT ≥1mm, 4) ABI < 1.0 or ≥ 1.4; (2) multisite atherosclerosis score: we assigned a score of 0 to 4 to each measure as: 1) CAC (scored as 0 if absent, or 1–4 according to gender-specific quartiles of positive score); 2) AAC (scored as 0 if absent, or 1–4 according to gender-specific quartiles of positive score); 3) CIMT (0 if in the first gender-specific quintile, 1–4 according to subsequent 2nd-5th quintiles); 4) ABI (scored as 0 if 1.0 ≤ ABI <1.4, 1–4 for the highest to the lowest gender-specific quartiles of ABI <1.0, and 1 if ABI ≥ 1.4). The multisite atherosclerosis score is the sum of the above four scores, with a range of 0 −16. We further divided the multisite atherosclerosis score into quartiles. In the sensitivity analysis, we excluded 16 subjects with ABI ≥1.4 and defined a positive ABI as <1.0.

Ascertainment of CVD and CHD events

After the baseline exam (either exam 2 or 3), we utilized follow-up data for CVD and CHD events through December 2015. The mean follow-up time was 10.6 years. At intervals of 9–12 months, a telephone interview was conducted to inquire about interim hospital admissions, cardiovascular diagnoses, and deaths. An adjudication committee received copies of all death certificates and medical records for hospitalizations and outpatient cardiovascular diagnoses and conducted next-of-kin interviews for out-of-hospital cardiovascular deaths for verification. Two physicians independently classified and assigned incidence dates. For disagreements, a full mortality and morbidity review committee made the final classification. Follow-up of each subject continued to first event, death, loss to follow-up, or the last follow-up call during 2015, whichever occurred first. Incident CHD included myocardial infarction, resuscitated cardiac arrest, angina, or coronary heart disease death, revascularization, percutaneous coronary intervention and coronary artery bypass grafting; incident CVD included CHD (from above) plus stroke, heart failure, transient ischemic attack, and peripheral vascular disease.

Statistical analysis

Descriptive data are presented as mean ± standard deviation (SD) for continuous variables and frequencies for categorical variables. ANOVA tested for continuous variables and Chi-squared tests for categorical variables across disease states. CVD and CHD event rates per 1000 person-years were calculated according to the two different measures of multisite atherosclerosis. Cox regression examined the relationship between both the number of vascular beds positive and the quartiles of the multisite atherosclerosis score in relation to incident CVD and CHD events in the total sample and within each disease group. All models were adjusted for the 2013 AHA/ACC ASCVD Pooled Cohort Equation (PCE) risk score, race/ethnicity and family history of premature CVD.19 In sensitivity analysis, we adjusted risk factors in the PCE instead of risk scores. We included interaction terms for MetS and DM with each multisite atherosclerosis measure to examine the possible heterogeneous association in each disease group. Single atherosclerosis measures (CAC, AAC, CIMT and ABI) were examined in relation to CVD and CHD events adjusted for each other and other risk factors. In addition, we compared the number of atherosclerotic sites and multisite atherosclerosis score with the measures of atherosclerosis at single site, namely CAC score, AAC score, CIMT and ABI abnormality (Yes/No) regarding their additional predictive value beyond the traditional risk factors using C-statistics for survival data. SAS 9.4 (North Carolina, US) were used for statistical analysis. A p value <0.05 (and p<0.1 for interaction test) was considered as statistically significant.

RESULTS

Out of 1,675 eligible MESA subjects (mean age 64.5 years, 51.2% male, 38.8% Caucasian, 25.6% Hispanic, 21.1% African American, 14.6% Chinese), 560 (33.4%) had MetS and 266 (15.9%) had DM. Participant characteristics in demographics and cardiovascular risk factors significantly differed among the disease groups, except for smoking status (Table 1). Compared to those with neither disease, those with MetS and DM had poorer risk profiles. Those with MetS were less likely male (42.7%), while those with DM were more likely male (57.1%); Caucasians had a lower proportion with DM compared to other races, while Asians had a lower proportion of MetS. A family history of CVD was also less prevalent in those with DM. Compared to those with neither condition, those with MetS and DM were successively more likely to have positive subclinical atherosclerosis measures.

Table 1.

Baseline characteristics across disease groups

All
n=1,675
Neither disease
n=849
(50.7%)
Metabolic syndrome
n=560
(33.4%)
Diabetes
n=266
(15.9%)
p value
b
Baseline age (year) 64.5 ± 9.7 63.4 ± 9.8 65.5 ± 9.4 65.8 ± 9.3 <0.01
Male 857 (51.2%) 466 (54.9%) 239 (42.7%) 152 (57.1%) <0.01
Ethnicity <0.01
 Caucasian 650 (38.8%) 358 (42.2%) 229 (40.9%) 63 (23.7%)
 Chinese American 244 (14.6%) 147 (17.3%) 62 (11.1%) 35 (13.2%)
 African American 353 (21.1%) 175 (20.6%) 105 (18.8%) 73 (27.4%)
 Hispanic 428 (25.6%) 169 (19.9%) 164 (29.3%) 95 (35.7%)
Current smoker 182 (10.9%) 97 (11.4%) 54 (9.6%) 31 (11.7%) 0.52
SBP (mm Hg) 123.8 ± 20.6 118.6 ± 19.2 128.8 ± 20.1 129.8 ± 21.5 <0.01
DBP (mm Hg) 70.2 ± 9.90 69.5 ± 9.5 71.4 ± 10.4 70.0 ± 9.9 <0.01
On HTN medication 703 (42.0%) 220 (25.9%) 310 (55.4%) 173 (65.0%) <0.01
LDL-C (mg/dL) 112.4 ± 31.1 114.3 ± 29.2 113.6 ± 33.1 103.6 ± 32. 0 <0.01
HDL-C (mg/dL) 51.7 ± 15.3 57.6 ± 15.9 45.3 ± 11.1 46.1 ± 13.2 <0.01
On lipid medication 403 (24.1%) 153 (18.0%) 152 (27.1%) 98 (36.8%) <0.01
Triglycerides (mg/dL) 134.8 ± 98.2 102.1 ± 53.6 167.8 ± 86.0 169.4 ± 171.1 <0.01
Fasting glucose (mg/dL) 98.1 ± 27.5 88.2 ± 8.0 95.2 ± 10.6 135.6 ± 51.1 <0.01
Waist circumference (cm) 97.7 ± 13.9 91.9 ± 12.1 103.8 ± 12.0 103.6 ± 15.2 <0.01
BMI (kg/m2) 27.9 ± 5.11 25.9 ± 4.3 30.0 ± 4.6 30.2 ± 6.0 <0.s0 1
Family history of CVD 899 (53.7%) 439 (51.7%) 334 (59.6%) 126 (47.4%) <0.01
           
Prevalence of single-site atherosclerosis
CAC 952 (56.8%) 422 (49.7%) 349 (62.3%) 181 (68.1%) <0.01
AAC 560 (33.4%) 236 (27.8%) 208 (37.1%) 116 (43.6%) <0.01
CIMT 523 (31.2%) 203 (23.9%) 209 (37.3%) 111 (41.7%) <0.01
Abnormal ABI 164 (9.8%) 63 (7.4%) 66 (11.8%) 35 (13.2%) <0.01
           
Incident events
CVD 175 (10.5%) 91 (6.1%) 108 (13.4%) 64 (18.1%) <0.01
CHD 263 (15.7%) 52 (10.7%) 75 (19.3%) 48 (24.1%) <0.01
a

AAC – presence of abdominal aortic calcium on CT scan; ABI – ankle brachial index < 1.0 or ≥ 1.4; BMI -body mass index; CAC – presence of coronary artery calcium on CT scan; CIMT – carotid intimal medial thickness ≥ 1.0 mm (maximal IMT of internal and common carotids); CVD – cardiovascular disease; DBP – diastolic blood pressure; HDL-C – high density lipoprotein-cholesterol; HTN – hypertension; LDL-C – low density lipoprotein-cholesterol; SBP – systolic blood pressure.

Conversion factors into SI units: LDL-C and HDL-C divide by 38.5, triglycerides divide by 88.5, and glucose divide by 18.

b

Values shown are n (%) or mean ± SD.

c

Indicates p value across disease groups.

d

Some participants had missing values for some variables: LDL-C n=30; HDL-C n=6; triglycerides n=5; fasting glucose n= 5.

The mean number of sites positive for atherosclerosis was significantly higher in those with DM (mean ± SD =1.67±1.15) and MetS (1.49±1.12) compared to neither condition (1.09±1.09) (Table 2). Those with 0 or 1 vascular beds positive for atherosclerosis were the most common in the non-disease group while those with 3 or 4 beds involved were most common in those with DM. The multisite atherosclerosis score showed a similar distribution pattern. 44.5% of those with neither MetS nor DM were in the lowest quartile of multisite atherosclerosis score while those with DM had the highest percentage of subjects (29.3%) in the highest quartiles. Among those with 476 subjects with only one involved vascular bed, 65.8% had CAC, followed by CIMT, AAC and ABI (Supplemental Fig. 1). Among 723 of those with CAC=0, an average of 70% had no other atherosclerotic vascular sites, with a lower percentage seen in those with DM (56%) and higher in the neither disease group (78%) (Supplemental Fig. 2). In contrast, among those with coronary calcification only 32% had no other positive atherosclerosis vascular beds. Of the four measures of DM severity, PCE ≥ 7.5% and DM duration ≥ 5 years were significantly associated with the extent of multisite atherosclerosis while HbA1c ≥7% and MetS status were not (Supplemental Table 1).

Table 2.

Distribution of multisite atherosclerosis measures overall and each disease group

All
n=1,675
Neither disease
n=849
(50.7%)
Metabolic syndrome
n=560
(33.4%)
Diabetes
n=266
(15.9%)
P value a
Number of vascular beds positive for atherosclerosis
Mean number of involved vascular beds 1.31 ± 1.13 1.09 ± 1.09 1.49 ± 1.12 1.67 ± 1.15 <0. 01
0 bed 509 (30.4%) 334 (39.3%) 127 (22.7%) 48 (18.1%) <0.01
1 bed 476 (28.4%) 227 (26.7%) 169 (30.2%) 80 (30.1%)
2 beds 391 (23.3%) 179 (21.1%) 149 (26.6%) 63 (23.7%)
3 beds 255 (15.2%) 97 (11.4%) 95 (17.0%) 63 (23.7%)
4 beds 44 (2.6%) 12 (1.4%) 20 (3.6%) 12 (4.5%)
Multisite atherosclerosis score
Mean score 4.49 ± 3.54 3.76 ± 3.28 5.02 ± 3.54 5.67 ± 3.79 <0.01
Median [inter quartile range] 4 [2–7] 3 [1–6] 4 [2–7] 5 [3–8]
1st quartile (score 0–2) 593 (35.4%) 378 (44.5%) 155 (27.7%) 60 (22.6%) <0.01
2nd quartile (score 3–4) 360 (21.5%) 170 (20.0%) 127 (22.7%) 63 (23.7%)
3rd quartile (score 5–7) 384 (22.9%) 177 (20.9%) 142 (52.4%) 65 (24.4%)
4th quartile (score 8–16) 338 (20.2%) 124 (14.6%) 136 (24.3%) 78 (29.3%)
a

Indicates p value across disease groups

During the mean follow-up time of 10.6 years, 263 CVD and 175 CHD events occurred, among which 59 were myocardial infarction, 5 were resuscitated cardiac arrest, 61 were angina, 41 were percutaneous coronary intervention, 9 were coronary artery bypass grafting, 13 were other revascularization, 16 were CHD death, 58 were stroke, 41 were heart failure, 17 were transient ischemic attack, 12 were peripheral vascular disease and 3 were other CVD deaths.

We examined CVD and CHD event rates in the total sample, as well as in the three disease groups during a mean follow-up of 10.6 years (Fig. 1). Overall, those with 4 positive atherosclerotic vascular sites had a CVD rate as high as 66.9 per 1,000 person-years, nearly 13 times that of those with 0 atherosclerotic vascular sites (5.2 per 1,000 person-years). Corresponding CHD event rates were 54.5 vs. 2.7 per 1,000 person-years. CVD event rates were 5.8, 9.2, 20.7 and 42.3 per 1000 person-years for those in the first, second, third and fourth quartile of multisite atherosclerosis score, respectively. In each disease group, those with more atherosclerotic vascular sites or with higher multisite atherosclerosis scores had higher CVD/CHD event rates. The DM group had the highest CVD/CHD event rates within the same level of extent of multisite atherosclerosis while the group with neither DM/MetS had the lowest event rates. In those whose DM duration ≥ 5 years, CVD/CHD event rates were similar by multisite atherosclerosis measures (Supplemental Fig. 3).

Figure 1.

Figure 1.

CVD (A and B) and CHD (C and D) event rates per 1,000 person-years, stratified by disease group and multisite atherosclerosis measures. Persons with more atherosclerotic vascular beds and higher multisite atherosclerosis score showed higher CVD and CHD event rates.

Compared to those with 0 involved vascular beds, HRs for CVD events were incrementally higher for those with 1, 2, 3 or 4 involved vascular beds (Table 3). The HR for those with all 4 atherosclerotic vascular beds vs. 0 involved vascular beds was 6.79 (95% CI: 3.63–12.71) overall and ranged from 3.99 to 14.40 in disease groups. For CHD events, corresponding HR was 10.67 (4.97–22.89) overall. Adjusted HRs and 95% CI for total CVD events per 1 unit increase of multisite atherosclerosis score ranged 1.12 to 1.21 among disease groups. HR for CVD and CHD events comparing those in the 4th vs. 1st quartile of the multisite atherosclerosis score was higher in neither disease group than in subjects with MetS and DM. This difference of HRs among groups was statistically significant for CHD events (p=0.03 for interaction test). All other interaction tests were not significant. When the PCE score was replaced with the risk factors in PCE, relationships were only slightly attenuated (Supplemental Table 2). After excluding 16 subjects with ABI ≥ 1.4, HRs remained similar to the main results in Table 3. In the subgroup of 723 subjects with CAC=0, neither the number of vascular beds nor the multisite atherosclerosis score was associated with CVD or CHD risk. Among those with DM, HRs of CVD/CHD events and number of mutisite atherosclerosis measures were significantly heterogenous according to DM duration (< 5 years vs. ≥ 5 years) and HbA1c (<7% vs. ≥ 7%): HR for CVD per 1 number of vascular bed was 2.63 for those with ≥ 5 years of DM, and was only 1.31 among those with < 5 years DM; corresponding HR was 2.70 among those with HbA1c < 7% and was 1.46 among the HbA1c ≥ 7%. HRs for CHD events were similar (p value <0.05 for interaction test). In a sensitivity analysis, we also showed our Exam 1 CIMT used to similarly predict CHD and CVD events in those whose index exam (based on CAC) was Exam 2 as compared to Exam 3. In addition, the average of exam 1 ABI and exam 3 ABI (1.12±0.12) was similar to that from Exam 1 ABI (1.11±0.12) and HRs for CVD and CHD events in these subgroups were similar to the total sample.

Table 3.

Adjusted hazard ratio and 95% confidence interval of number of vascular beds and multisite atherosclerosis score for CVD and CHD events

All
n=1,675
Neither disease
n=849 (50.7%)
Metabolic syndrome
n=560 (33.4%)
Diabetes
n=266 (15.9%)
p value for interaction test
CVD
Number of involved vascular beds
0 bed Reference Reference Reference Reference 0.71
1 bed 1.87 (1.19–2.93) 2.70 (1.02–4.20) * 1.32 (0.64–2.72) 1.54 (0.55–4.38)
2 beds 3.04 (1.1.95–4.74) § 3.60 (1.76–7.37) 2.03 (1.01–4.11) * 3.32 (1.18–9.37) *
3 beds 4.67 (2.95–7.40) § 6.06 (2.90–12.68) § 2.54 (1.20–5.36) * 6.27 (223–17.68)
4 beds 6.79 (3.63–12.71) § 3.99 (0.83–19.09) 4.86 (1.97–12.00) 14.40 (3.60–57.55)
Multisite atherosclerosis score
1st quartile Reference Reference Reference Reference 0.15
2nd quartile 1.35 (0.85–2.15) 1.89 (0.88–4.06) 1.12 (0.52–2.40) 0.69 (0.27–1.78)
3rd quartile 2.83 (1.88–4.25) § 4.38 (2.23–8.60) § 2.10 (1.07–4.10) * 1.71 (0.73–3.99)
4th quartile 4.72 (3.11–7.16) § 7.05 (3.43–14.50) § 3.38 (1.73–6.61) 3.57 (1.68–8.36)
CHD
Number of involved vascular beds
0 bed Reference Reference Reference Reference 0.23
1 bed 2.25 (1.22–4.13) 4.21 (1.35–13.18) * 1.58 (0.61–4.09) 0.31–2.94)
2 beds 3.99 (2.21–7.22) § 9.74 (3.16–30.00) § 2.01 (0.91–5.84) 0.88–7.68)
3 beds 5.66 (3.07–10.45) § 12.44 (3.81–40.65) § 3.18 (1.22–8.26) * 1.26–10.83) *
4 beds 10.67 (4.97–22.89) § 18.16 (2.98–110.63) 5.72 (1.85–17.73) 13.89 (3.25–59.29)
Multisite atherosclerosis score
1st quartile Reference Reference Reference Reference 0.03
2nd quartile 1.69 (0.92–3.09) 2.70 (0.82–8.89) 1.47 (0.56–3.90) 0.80 (0.29–2.25)
3rd quartile 3.41 (2.00–5.82)§ 9.52 (3.44–26.33) § 2.33 (0.96–5.64) 1.39 (0.53–3.67)
4th quartile 5.71 (3.32–9.83) § 16.14 (5.49–47.44) § 3.98 (1.66–9.54) 2.78 (1.12–6.90) *
a

Hazard ratios were adjusted for Pooled Cohort Equation, ethnicity, lipid medications and family history of CVD

b

 

*

p <0.05

p <0.01

p <0.001

§

p <0.0001

Single atherosclerosis measures (CAC, AAC, CIMT and ABI) were examined in relation to CVD and CHD events adjusted for each other and other risk factors (Supplemental Table 3). Log-transformed CAC scores showed strong associations with CVD and CHD events adjusted for AAC, CIMT, ABI and other risk factors. However, AAC, CIMT and ABI showed non-significant associations with endpoints after adjustment of CAC and other factors, indicating the predictive value of CAC in the presence of other single atherosclerosis measures but not vice versa.

We then compared the C-statistics of risk prediction models containing multisite atherosclerosis measures vs. each single site atherosclerosis measure (Table 4). The base model included only traditional risk factors. Models 1–4 each included one more single site atherosclerosis measure (CAC,AAC,CIMT or ABI) in addition to traditional risk factors. Model 5 and 6 included one of the multisite atherosclerosis measures (number of atheroclerotic vascular beds or mulisite atherosclerosis score) in addition to traditional risk factors. In the total population, prediction models for CVD, including number of atheroclerotic vascular beds (Model 5), had significantly higher C-statistics than models with traditional risk factors, or traditional risk factors plus CIMT or ABI, while there was non-siginificant improvement over risk models with AAC or CAC; prediction models for CVD events including multisite atherosclerosis score (Model 6) had significantly higher C-statistics than base models, or traditional risk factors plus AAC, CIMT, or ABI while there was non-siginificant improvement over risk models with CAC. Within each disease group, the improvement of Model 5 compared to all other models was not significant among those with MetS. Other comparisons of C-statistics were similar in each disease group. Improvement of C-statistics were similar for CHD events overall and in those with neither disease but was not significant among those with MetS or DM. Meanwhile the two multisite atherosclerosis measures had similar incremental prediction ability as CAC score for both CVD and CHD event. Further sensitivity analysis examining C-statistics of above models in the CAC=0 subgroup showed that none of the subclinical measures (AAC, CIMT, ABI, number of involved vascular beds, or multisite atherosclerosis scores) had significant improved C-statistics over the base model.

Table 4.

C-statistics for prediction of CVD and CHD events comparing models with multisite atherosclerosis measures and single site measures

All
n=1,675
Neither disease
n=849 (50.7%)
Metabolic syndrome
n=560 (33.4%)
Diabetes
n=266 (15.9%)
CVD
Model C--statistic p value* p value C-statistic p vSalue* p vSalue C-statistics p value* p value C-statistics p value* p value
Model 0: RFs 0.72 0.002 <0.001 0.75 0.020 0.007 SS 0.128 0.064 0.66 0.035 0.020
Model 1: RF + CAC 0.76 0.244 0.608 0.78 0.668 0.809 0.73 0.298 0.563 0.72 0.738 0.567
Model 2: RF + AAC 0.74 0.110 0.011 0.76 0.087 0.069 0.71 0.105 0.035 0.72 0.402 0.152
Model 3: RF + CIMT 0.73 0.005 <0.001 0.75 0.025 0.005 0.71 0.136 0.037 0.68 0.050 0.028
Model 4: RF + ABI 0.72 0.005 0.001 0.75 0.031 0.009 0.71 0.314 0.166 0.67 0.036 0.022
Model 5: RF + number of atherosclerotic vascular beds 0.75 / / 0.78 / / 0.72 / / 0.73 / /
Model 6: RF + multisite 0.75 / / 0.78 / / 0.73 / / 0.74 / /
CHD
Model C-statistic p value* p value C-statistic p value* p value C-statistics p value* p value C-statistics p value* p value
Model 0: RFs 0.73 0.001 <0.001 0.76 0.047 0.001 0.SS74 0.169 0.108 0.69 0.046 0.042
Model 1: RF + CAC 0.77 0.386 0.995 0.81 0.445 0.941 0.76 0.417 0.592 0.73 0.303 0.550
Model 2: RF + AAC 0.75 0.050 0.002 0.78 0.273 0.023 0.74 0.163 0.108 0.74 0.601 0.835
Model 3: RF + CIMT 0.74 0.002 <0.001 0.77 0.080 <0.001 0.74 0.306 0.133 0.71 0.120 0.167
Model 4: RF + ABI 0.73 0.003 <0.001 0.75 0.033 <0.001 0.74 0.344 0.250 0.71 0.143 0.149
Model 5: RF + number of atherosclerotic vascular beds 0.77 / / 0.80 / / 0.75 / / 0.75 / /
Model 6: RF + multisite atherosclerosis score 0.77 / / 0.82 / / 0.76 / / 0.74 / /
a

AAC – abdominal aortic calcium; ABI – ankle brachial index; CAC – coronary artery calcium; CIMT –carotid intimal medial thickness; RF-risk factor. RFs include age, gender, ethnicity, smoking, SBP, HTN medication, HDL-C, lipid medications and family history of CVD.

b

 

*

p value comparing Model 5 vs. Model 1–4

p value comparing Model 6 vs. Model 1–4.

DISCUSSION

Numerous studies have established positive associations between single-site atherosclerosis measures and future CVD or CHD risk.6,813 Among them, CAC has normally been found to be the single strongest predictor beyond traditional risk factors.20 Several studies have examined atherosclerosis at two or more sites and their association with future mortality or events13,14,2124. These studies have limitations: some used atherosclerosis measures with similar features, i.e. measured by calcification13,14, or plaques23,24. Some have failed to included coronary artery atherosclerosis measures22, and none specifically examined multisite atherosclerosis in the DM and MetS populations in comparison to those free of DM/MetS.

Our study created multisites atherosclerosis scores that summarized four specific subclinical atherosclerosis measures: AAC, CAC, CIMT and ABI, representing various features of atherosclerotic disease in four different sites of the body, and examining how the quantity of this multisite or “systemic” atherosclerosis varies according to those with DM, MetS or neither condition and predicts subsequent CHD and CVD events in a long follow-up period. As expected, the extent of multisite atherosclerosis was the highest in those with DM and the lowest in the those with neither condition, consistent with the CVD risk distribution among the three groups. Among those with DM, DM duration and their 10-year ASCVD risk score were also related to the extent of multisite atherosclerosis. CAC was found to be the most prevalent atherosclerotic site among the four examined vascular beds and comprised 65% of those with only one atherosclerotic vascular bed. In addition, the absence of coronary calcification was related to less atherosclerosis in other vascular sites.

We also showed a graded relationship between the number of arterial atherosclerosis sites and the multisite atherosclerosis composite score with CHD and CVD event risk in those with and without MetS and DM. While the HRs for incident CHD and CVD events comparing the 4th vs. 1st quartile of multisite atherosclerosis score were the greatest in those with neither MetS/DM than in those with MetS or DM, absolute event rates were higher in the reference groups among those with MetS and DM compared to neither condition. Prior work from MESA by Tison and colleages has examined the distribution and relation to CHD events and mortality in those with extracoronary calcification (ECC) in the aortic valve, aortic root, mitral valve, and thoracic aorta, noting a graded relationship of risk for CHD events, CHD mortality, and total mortality associated with the number of ECC sites positive.14

When added to tranditional risk factors, the two measures of multisite atherosclerosis, namely the number of atherotic vascular beds and the multisite atherosclerosis score, showed better predictive ability for future CVD and CHD events compared to the risk models with CIMT and ABI and sometimes AAC but not CAC. CAC was previously found to be the strongest pedictor for CVD beyond traditonal risk scores or individual risk factors; our study shows that intergreting other subclinical atherosclerosis measures with CAC as multisite atherosclerosis measures did not further improve the risk discrimination beyond CAC. However, others have noted the extent to which CAC is concentrated in one vessel was found to improve prediction over total CAC scores 25. Wong et al. found AAC positively correlated with CAC, CIMT and ABI.26 We find CAC to be the most prevalent atherosclerosis measure and the greatest contributor to the positive atherosclerosis sites. These reasons may explain the positive correlation between the number of atherosclerotic vascular beds and CAC score and such collinearity with CAC leads to similar predictive ability between CAC and multisite atheroslerosis measures. Our findings suggest that screening subclinical atherosclerosis at other sites may provide limited utility for risk stratification beyond CAC, including those with MetS and DM.

Current AHA/ACC guidelines for CVD risk assessment indicate CAC and ABI screening as a class IIb level of evidence B recommendation when the treatment decision is uncertain after global risk estimation such as from the PCE.19 Concerns about population-wide CAC scanning are the potential risk of radation exposure and issues of cost, although radiation is quite low and the cost of CAC scans at most centers now ranges from under $100 to $250 USD. CIMT is not recommended alone due to limited risk reclaassification potential; however, data are stronger to show its risk-reclassification ability in combination with identification of carotid plaques. In the DM population, the role of CAC score in risk stratification has also been shown17,2730 and recent American Diabetes Association guidelines have stated that in adults with diabetes ≥40 years of age, measurement of CAC is reasonable for cardiovascular risk assessment.31 In addition, in a review of algorithms to screen for subclinical atherosclerosis in those with diabetes, CAC was most frequently used in early stages of evaluation to assist risk classification.32 Our study demostrated that although the multisite atherosclerosis measures are associated future CVD and CHD risks independent of traditional risk factors, they did not have additional prediction value beyond CAC.

The standardized data collection, including measurement of risk factors and subclinical disease measures across sites is an important strength of MESA, as is the systematic adjudication process for CVD and CHD events. Limitations of our study include the modest sample sizes in certain subgroups, precluding the ability to examine gender or ethnic group differences. Also CAC and AAC were measured using Agataston scores but there may be other measures, e.g. calcium volume score and density score that could offer potential additional information.33

We show the extent of multisite atheroscerosis is greater in those with MetS and DM than in those without these conditions. Also, those with more extensive subclinical atheroscerosis in multiple sites suffer higher CVD and CHD risk. However, clinical utility of these measures is limited beyond assessment of CAC in those with MetS and DM.

Supplementary Material

1
2
3
4
5
6

Highlights – ATH-D-18-01306.

  • Persons with diabetes or metabolic syndrome have more atherosclerotic sites.

  • Cardiovascular disease rates are greater the number of atherosclerotic sites.

  • Those with 4 vs. 0 atherosclerotic sites have a 4 to 14-fold greater risk of CVD events.

  • Multisite atherosclerosis adds to CVD event prediction, except over coronary calcium.

ACKNOWLEDGEMENTS:

Presented in part at the American Heart Association Council on Epidemiology and Prevention, San Francisco, CA on March 20, 2014.

FINANCIAL SUPPORT:

The original MESA study was supported by contracts HHSN268201500003I, N01-HC-95159, N01-HC-95160, N01-HC-95161, UL1-TR-000040, N01-HC-95162, UL1-TR-001079, N01-HC-5163, N01-HC-95164, N01-HC-95165, UL1-TR-001420, N01-HC-95166, N01-HC-95167, N01-HC-95168 and N01-HC-95169 from the National Heart, L ung, and Blood Institute, and by grants UL1-TR-000040, UL1-TR-001079, and UL1-TR-001420 from NCATS. The authors thank the other investigators, the staff, and the participants of the MESA study for their valuable contributions. A full list of participating MESA investigators and institutions can be found at http://www.mesa-nhlbi.org.

ABBREVIATIONS

AAC

abdominal aortic calcium

CAC

coronary artery calcium

CHD

coronary heart disease

CIMT

carotid intimal-medial thickness

CVD

cardiovascular disease

DM

diabetes mellitus

MetS

metabolic syndrome

PCE

pooled cohort equation

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

CONFLICT OF INTEREST:

Dr. Matthew J. Budoff receives grant support from General Electric. The other authors have no conflict of interests.

REFERENCES

  • 1.Isomaa BO, Almgren P, Tuomi T, Forsén B, Lahti K, Nissen M, Taskinen MR, Groop L. Cardiovascular morbidity and mortality associated with the metabolic syndrome. Diabetes care 2001;24:683–689. [DOI] [PubMed] [Google Scholar]
  • 2.Lakka HM, Laaksonen DE, Lakka TA, Niskanen LK, Kumpusalo E, Tuomilehto J, Salonen JT. The metabolic syndrome and total and cardiovascular disease mortality in middle-aged men. JAMA 2002;288:2709–2716. [DOI] [PubMed] [Google Scholar]
  • 3.Malik S, Wong ND, Franklin SS, Kamath TV, Gilbert JL, Pio JR, Williams GR. Impact of the metabolic syndrome on mortality from coronary heart disease, cardiovascular disease, and all causes in United States adults. Circulation 2004;110:1245–1250. [DOI] [PubMed] [Google Scholar]
  • 4.McNeill AM, Rosamond WD, Girman CJ, Heiss G, Golden SH, Duncan BB, East HE, Ballantyne C. Prevalence of coronary heart disease and carotid arterial thickening in patients with the metabolic syndrome (The ARIC Study). Am J Cardiol 2004;94:1249–1254. [DOI] [PubMed] [Google Scholar]
  • 5.Wong ND, Sciammarella MG, Polk D, Gallagher A, Miranda-Peats L, Whitcomb B, Hachamovitch R, Friedman JD, Hayes S, Berman DS. The metabolic syndrome, diabetes, and subclinical atherosclerosis assessed by coronary calcium. J Am Coll Cardiol 2003;41:1547–1553. [DOI] [PubMed] [Google Scholar]
  • 6.Malik S, Budoff MJ, Katz R, Blumenthal RS, Bertoni AG, Nasir K, Szklo M, Barr RG, Wong ND. Impact of subclinical atherosclerosis on cardiovascular disease events in individuals with metabolic syndrome and diabetes: the multi-ethnic study of atherosclerosis. Diab Care 2011;34:2285–2290. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Malik S, Zhao Y, Budoff M, Nasir K, Blumenthal RS, Bertoni AG, Wong ND. Coronary artery calcium score for long-term risk classification in individuals with type 2 diabetes and metabolic syndrome from the Multi-Ethnic Study of Atherosclerosis. JAMA Cardiol 2017; 2:1332–1340. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Wong ND, Hsu JC, Detrano RC, Diamond G, Eisenberg H, Gardin JM. Coronary artery calcium evaluation by electron beam computed tomography and its relation to new cardiovascular events. Am J Cardiol 2000;86:495–498. [DOI] [PubMed] [Google Scholar]
  • 9.Kondos GT, Hoff JA, Sevrukov A, Daviglus ML, Garside DB, Devries SS, Chomka EV, Liu K. Electron-beam tomography coronary artery calcium and cardiac events: a 37-month follow-up of 5635 initially asymptomatic low-to intermediate-risk adults. Circulation 2003;107:2571–2576. [DOI] [PubMed] [Google Scholar]
  • 10.Shaw LJ, Raggi P, Schisterman E, Berman DS, Callister TQ. Prognostic value of cardiac risk factors and coronary artery calcium screening for all-cause mortality. Radiology 2003;228:826–833. [DOI] [PubMed] [Google Scholar]
  • 11.O’Leary DH, Polak JF, Kronmal RA, Manolio TA, Burke GL, Wolfson SK, Jr. Carotid-artery intima and media thickness as a risk factor for myocardial infarction and stroke in older adults. Cardiovascular Health Study Collaborative Research Group. N Engl J Med 1999;340:14–22. [DOI] [PubMed] [Google Scholar]
  • 12.Criqui MH, Langer RD, Fronek A, Feigelson HS, Klauber MR, McCann TJ, Browner D. Mortality over a period of 10 years in patients with peripheral arterial disease. N Engl J Med 1992;326:381–386. [DOI] [PubMed] [Google Scholar]
  • 13.Criqui MH, Denenberg JO, McClelland R, Allison MA, Ix JH, Guerci A, Cohoon KP, Srikanthan P, Watson KE, Wong ND. Abdominal aortic calcium, coronary artery calcium, and cardiovascular morbidity and mortality in the Multi-Ethnic Study of Atherosclerosis. Arterioscler Thromb Vasc Biol 2014. July;34(7):1574–9.. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Tison GH, Guo M, Blaha MJ, McClelland RL, Allison MA, Szklo M, Wong ND, Blumenthal RS, Budoff MJ, Nasir K. Multisite extracoronary calcification indicates increased risk of coronary heart disease and all-cause mortality: the Multiethnic Study of Atherosclerosis. J Cardiovasc Comput Tomog 2015; 9: 406–414. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Bild DE, Bluemke DA, Burke GL, Detrano R, Diez Roux AV, Folsom AR, Greenland P, JacobsJr DR, Kronmal R, Liu K, Nelson JC. Multi-ethnic study of atherosclerosis: objectives and design. Am J Epid 2002; 156:871–881. [DOI] [PubMed] [Google Scholar]
  • 16.Carr JJ, Nelson JC, Wong ND, McNitt-Gray M, Arad Y, Jacobs DR Jr, Sidney S, Bild DE, Williams OD, Detrano RC. Calcified coronary artery plaque measurement with cardiac CT in population-based studies: standardized protocol of Multi-Ethnic Study of Atherosclerosis (MESA) and Coronary Artery Risk Development in Young Adults (CARDIA) study. Radiology 2005;234:35–43. [DOI] [PubMed] [Google Scholar]
  • 17.Malik S, Budoff MJ, Katz R, Blumenthal RS, Bertoni AG, Nasir K, Szklo M, Barr RG, Wong ND. Impact of subclinical atherosclerosis on cardiovascular disease events in individuals with metabolic syndrome and diabetes: the multi-ethnic study of atherosclerosis. Diabetes care 2011;34:2285–2290. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Criqui MH, McClelland RL, McDermott MM, Allison MA, Blumenthal RS, Aboyans V, Ix JH, Burke GL, Liu K, Shea S. The ankle-brachial index and incident cardiovascular events in the MESA (Multi-Ethnic Study of Atherosclerosis). J Am Coll Cardiol 2010; 56:1506–1512. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Goff DC, Lloyd-Jones DM, Bennett G, Coady S, D’Agostino RB, Gibbons R, Greenland P, Lackland DT, Levy D, O’Donnell CJ, Robinson JG. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol 2014; 63:2935–2959. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Sathiyakumar V, Blumenthal RS, Nasir K, Martin SS. Addressing Knowledge Gaps in the 2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk: a Review of Recent Coronary Artery Calcium Literature. Curr Athero Rep 2017;19:7. [DOI] [PubMed] [Google Scholar]
  • 21.Prevalence, Vascular Distribution, and Multiterritorial Extent of Subclinical Atherosclerosis in a Middle-Aged Cohort: The PESA (Progression of Early Subclinical Atherosclerosis) Study. Circulation 2015;131(24):2104–13. [DOI] [PubMed] [Google Scholar]
  • 22.Laclaustra M, Casasnovas JA, Fernández-Ortiz A, Fuster V, León-Latre M, Jiménez-Borreguero LJ, Pocovi M, Hurtado-Roca Y, Ordovas JM, Jarauta E, Guallar E. Femoral and carotid subclinical atherosclerosis association with risk factors and coronary calcium: the AWHS study. Journal of the American College of Cardiology 2016;67(11):1263–74. [DOI] [PubMed] [Google Scholar]
  • 23.López-Melgar B, Fernández-Friera L, Oliva B, García-Ruiz JM, Peñalvo JL, Gómez-Talavera S, Sánchez-González J, Mendiguren JM, Ibáñez B, Fernández-Ortiz A, Sanz J. Subclinical atherosclerosis burden by 3D ultrasound in mid-life: the PESA Study. Journal of the American College of Cardiology 2017; 70(3):301–13. [DOI] [PubMed] [Google Scholar]
  • 24.Weckbach S, Findeisen HM, Schoenberg SO, Kramer H, Stark R, Clevert DA, Reiser MF, Parhofer KG. Systemic cardiovascular complications in patients with long-standing diabetes mellitus: comprehensive assessment with whole-body magnetic resonance imaging/magnetic resonance angiography. Investigative Radiol 2009;44(4):242–50. [DOI] [PubMed] [Google Scholar]
  • 25.Blaha MJ, Budoff MJ, Tota-Maharaj R, Dardari ZA, Wong ND, Kronmal RA, Eng J, Post WS, Blumenthal RS, Nasir K. Improving the CAC score by addition of regional measures of calcium distribution: multi-ethnic study of atherosclerosis. JACC: Cardiovascular Imaging 2016;9(12):1407–16. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Wong ND, Lopez VA, Allison M, Detrano RC, Blumenthal RS, Folsom AR, Ouyang P, Criqui MH. Abdominal aortic calcium and multisite atherosclerosis: the Multiethnic Study of Atherosclerosis. Atherosclerosis 2011;214:436–441. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Raggi P, Shaw LJ, Berman DS, Callister TQ. Prognostic value of coronary artery calcium screening in subjects with and without diabetes. J Am Coll Cardiol 2004;43:1663–1669. [DOI] [PubMed] [Google Scholar]
  • 28.Won KB, Chang HJ, Niinuma H, Sung J, Shin S, Cho IJ, Shim CY, Hong GR, Kim YJ, Choi BW, Chung N. Evaluation of the predictive value of coronary artery calcium score for obstructive coronary artery disease in asymptomatic Korean patients with type 2 diabetes mellitus. Coron Artery Dis 2015;26:150–156. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Anand DV, Lim E, Hopkins D, Corder R, Shaw LJ, Sharp P, Lipkin D, Lahiri A. Risk stratification in uncomplicated type 2 diabetes: prospective evaluation of the combined use of coronary artery calcium imaging and selective myocardial perfusion scintigraphy. Eur Heart J 2006;27;713–721. [DOI] [PubMed] [Google Scholar]
  • 30.Elkeles RS, Godsland IF, Feher MD, Rubens MB, Roughton M, Nugara F, Humphries SE, Richmond W, Flather MD. Coronary calcium measurement improves prediction of cardiovascular events in asymptomatic patients with type 2 diabetes: the PREDICT study. Eur Heart J 2008;29:2244–2251. [DOI] [PubMed] [Google Scholar]
  • 31.American Diabetes Association 8. Cardiovascular disease and risk management. Diabetes care 2016;39(Supplement 1):S60–71. [DOI] [PubMed] [Google Scholar]
  • 32.Zhao Y, Wong ND. Screening subclinical coronary artery disease with noninvasive modalities in patients with diabetes. Cardiovasc Endocrinol 2015. ;4:120–126. [Google Scholar]
  • 33.Criqui MH, Denenberg JO, Ix JH, McClelland RL, Wassel CL, Rifkin DE, Carr JJ, Budoff MJ, Allison MA. Calcium density of coronary artery plaque and risk of incident cardiovascular events. JAMA 2014; 311:271–278. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1
2
3
4
5
6

RESOURCES