Cardiovascular/stroke risk predictive calculators: a comparison between statistical and machine learning models

Abstract

Background

Statistically derived cardiovascular risk calculators (CVRC) that use conventional risk factors, generally underestimate or overestimate the risk of cardiovascular disease (CVD) or stroke events primarily due to lack of integration of plaque burden. This study investigates the role of machine learning (ML)-based CVD/stroke risk calculators (CVRC_ML) and compares against statistically derived CVRC (CVRC_Stat) based on (I) conventional factors or (II) combined conventional with plaque burden (integrated factors).

Methods

The proposed study is divided into 3 parts: (I) statistical calculator: initially, the 10-year CVD/stroke risk was computed using 13 types of CVRC_Stat (without and with plaque burden) and binary risk stratification of the patients was performed using the predefined thresholds and risk classes; (II) ML calculator: using the same risk factors (without and with plaque burden), as adopted in 13 different CVRC_Stat, the patients were again risk-stratified using CVRC_ML based on support vector machine (SVM) and finally; (III) both types of calculators were evaluated using AUC based on ROC analysis, which was computed using combination of predicted class and endpoint equivalent to CVD/stroke events.

Results

An Institutional Review Board approved 202 patients (156 males and 46 females) of Japanese ethnicity were recruited for this study with a mean age of 69±11 years. The AUC for 13 different types of CVRC_Stat calculators were: AECRS2.0 (AUC 0.83, P<0.001), QRISK3 (AUC 0.72, P<0.001), WHO (AUC 0.70, P<0.001), ASCVD (AUC 0.67, P<0.001), FRS_cardio (AUC 0.67, P<0.01), FRS_stroke (AUC 0.64, P<0.001), MSRC (AUC 0.63, P=0.03), UKPDS56 (AUC 0.63, P<0.001), NIPPON (AUC 0.63, P<0.001), PROCAM (AUC 0.59, P<0.001), RRS (AUC 0.57, P<0.001), UKPDS60 (AUC 0.53, P<0.001), and SCORE (AUC 0.45, P<0.001), while the AUC for the CVRC_ML with integrated risk factors (AUC 0.88, P<0.001), a 42% increase in performance. The overall risk-stratification accuracy for the CVRC_ML with integrated risk factors was 92.52% which was higher compared all the other CVRC_Stat.

Conclusions

ML-based CVD/stroke risk calculator provided a higher predictive ability of 10-year CVD/stroke compared to the 13 different types of statistically derived risk calculators including integrated model AECRS 2.0.

Introduction

Cardiovascular disease (CVD) is the leading cause of mortalities around the world which has accounted for the global deaths of 17.9 million people (1). Out of these 17.9 million, 85% of the deaths were attributed to myocardial infarction and stroke (1). Prevention of CVD/stroke events is an important challenge in front of the medical community, which is causing a higher global economic burden (2,3). Thus, there an increasing demand to provide accurate and time-efficient preventive tools that can provide a long-term prognosis of CVD/stroke events at an affordable cost to the patients. Atherosclerosis is one of the dominant causes of CVD/stroke events (4-10). One way to prevent the growth of atherosclerosis is to treat the risk factors which are responsible for the initiation and progression of atherosclerotic plaque. Besides controlling the risk factors, treating the arteries by measuring the atherosclerotic plaque extent and making the plans for its regression is also an established way of reducing the risk of CVD (11).

According to INTERHEART* and INTERSTROKE* studies [*as it is taken from the Yusuf et al. and O’Donnell et al. (12-14)], conventional cardiovascular risk factors (CCVRF) such as diabetes mellitus, hypertension, hyperlipidemia, smoking, alcohol consumption, physical inactivity, and body mass index are attributed to ~90% risk of CVD/stroke events (12-14). Current guidelines also suggest the treatment of all these modifiable risk factors for CVD prevention (15-19). Most of these guidelines take the help of statistically derived conventional cardiovascular risk calculators (CVRC) for initiation of CVD/stroke treatment plans (e.g., for initiation of statins) (15,17-21). Some commonly used statistically derived CVRC (CVRC_Stat) are the Framingham Risk Score (FRS), the UK Prospective Diabetes Score (UKPDS), the Reynold Risk Score (RRS), the Systematic Coronary Risk Evaluation Chart, the QRISK3 calculator, and the World Health Organization (WHO) are the commonly used CVD risk prediction models. All such CVRC_Stat reported a modest performance while providing a 10-year risk assessment especially in patients suffering from diabetes mellitus (22,23) and rheumatoid arthritis (24,25). Furthermore, it has been observed that most of CVRC_Stat provide overestimation or underestimation of CVD risk (26,27).

The reason for such sub-optimal performance of CVRC_Stat lies in their design. These CVRC_Stat were designed using statistical regression-based techniques that consider only a limited number of risk factors (28). These regression-based methods assume a linear relationship between the input risk factors and the outcome of interest (for example the CVD events) (26,29,30). However, in practice, the risk due to CVD depends upon the complex interaction between conventional cardiovascular risk factors (CCVRF) (31). Furthermore, these risk prediction models assume a limited or no interaction between the CCVRF itself, thus, this assumption ends-up in providing an oversimplified and approximate risk assessment model (26,29,30). Another important factor that may lead to the poor performance of CVRC_Stat is the exclusion of image-based phenotypes. The CCVRF may note provide the complete information about the atherosclerotic plaque components, which, however, can easily be captured using imaging modalities (32,33). Thus, there is a need to refine the search of risk prediction models that can provide more robust risk estimation by considering the non-linear interaction between a large number of a diverse set of input risk factors (33). Lastly, these models for CVRC_Stat never use cohort knowledge in their persistence layer (database layer) or to embed intelligence in their designs, which is so important for an accurate prediction of 10-year risk (34). Machine learning (ML)-has been actively pursued in medical imaging recently covering several anatomic areas such as liver cancer (35,36), lung cancer (37,38), prostate cancer (39,40) more recently in carotid plaque characterization (32-36). These techniques are generally the data-driven algorithms, which identify the complex interactions among the CCVRF to provide accurate CVD risk estimation (26,33). The ML-based techniques have plenty amount of applications in the medical domain including the CVD/stroke risk assessment (41-47). However, there were a few attempts were made to investigate the performance of ML-based CVD/Stroke risk stratification against to the CVRC_Stat (26,30,48-52). In this study, a Support Vector Machine (SVM) algorithm was adapted for ML-based CVD/stroke risk calculator (CVRC_ML). The proposed study had two main hypotheses for this study: (I) CVRC_ML can provide a better risk stratification compared to CVRC_Stat and (II) the inclusion of carotid ultrasound image-based phenotypes (CUSIP) into the CCVRF (so-called integrated systems) can provide a better predictive ability compared to the CCVRF alone, irrespective of the type of risk calculator (CVRC_ML or CVRC_Stat).

In order to prove these hypotheses, this study has three unique objectives. These are:
To compare the performance of the 12 types of CVRC_{Stat (conventional)} and CVRC_{ML (conventional)};
To compare the performance of the CVRC_{Stat (integrated)} and the CVRC_{ML (integrated)};
To compare the performance of the CVRC_{ML (conventional)} and the CVRC_{ML (integrated)}.

Definitions of CVRC_{Stat (conventional),} CVRC_{Stat (integrated),} CVRC_{ML (conventional),} and CVRC_{ML (integrated)}

It should be noted that (I) 12 types of CVRC_{Stat (conventional)} means statistically derived CVRC_Stat that used 14 CCVRF. These 12 types of CVRC_{Stat (conventional)} are: (i) the FRS, (ii) the UKPDS56, (iii) the UKPDS60, (iv) the RRS, (v) Atherosclerosis CVD (ASCVD) calculator, (vi) the NIPPON risk chart, (vii) the SCORE risk chart, (viii) the QRISK3, (ix) the WHO risk chart, (x) the PROCAM, (xi) FRS (Stroke), and (xii) my_stroke risk calculator (MSRC). (II) CVRC_{Stat (integrated)} means CVRC_Stat that used integrated risk factors, which were the combination of 14 CCVRF and 5 CUSIP. AtheroEdge Composite Risk Score version 1 (AECRS 1.0) (53-55) was the recently developed type of CVRC_{Stat (integrated)}. In this study, we extended the AECRS 1.0 to AECRS 2.0 by further adding the status of chronic kidney disease (CKD). This will allow the physicians to perform the CVD/stroke risk stratification of patients suffering from CKD. In this study, AECRS 2.0 will be referred as CVRC_{Stat (integrated)}. (III) CVRC_{ML (conventional)} means CVRC_ML that used 14 CCVRF, and (IV) CVRC_{ML (integrated)} means CVRC_ML that used integrated risk factors. Hereafter, throughout this study, the above definitions will be followed. The definition various 14 CCVRF and 5 CUSIP are provided in the “Method” section of this article.

Figure 1 shows the overall architecture of the proposed system that compares the CVD/stroke risk stratification performed using the ML-based approach against the risk stratification performed using 13 different types of statistically derived CVRC (12 types of CVRC_{Stat (conventional)} and CVRC_{Stat (integrated)}]. In this study, a combination of glycated hemoglobin (HbA1c) and maximum carotid intima-media thickness has been used as the event-equivalence gold standard (EEGS). Description of such type of combinational EEGS has been given in the “Discussion” section of this study.

Figure 1.

The overall architecture of the proposed system that compares the CVD/stroke risk stratification performed using the CVRC_ML and the 13 types of statistically derived CVRC_Stat. CVRC_ML, ML-based cardiovascular risk calculator; CVRC_Stat, 13 types of different statistically derived cardiovascular risk calculators; CVD, cardiovascular disease; CUSIP, carotid ultrasound image-based phenotypes; EEGS, event-equivalence gold standard; AUC, area under the curve.

Methods

Study participants

This retrospective study recruited a Japanese ethnicity cohort of 202 patients between July 2009 and December 2010. All the patients were approved by the institutional review board of the Ohashi Medical Center, Toho University Japan, and informed consent was obtained from all the study participants. The proposed study is unique in its concept compared to all our previous studies that used a similar Japanese cohort (53-60). In total, 404 CUS scans (2 CUS scans × 202 patients) were collected from both the left and right common carotid arteries of 202 patients, out of which nine scans were excluded from this study due to the poor visualization. Thus, a total of 395 CUS scans were used in this study. All the CUS scans were retrospectively analyzed by two experts who had 15 years of experience in the field of radiology.

Carotid ultrasound image acquisition

The ultrasound image of the common carotid artery (CCA) was obtained by using a B-mode ultrasound scanner (Aplio XG, Xario, Aplio XV, Toshiba Inc., Tokyo, Japan) equipped with 7.5 MHz linear arrays of the transducer. In order to capture the ultrasound image and to measure the image-based phenotypes, a standardized protocol of the American Society of Echocardiography (ASE) was followed (61). Initially, the patients were examined in a supine position with their head tilted backward. At first, the carotid artery of the patients was identified by the transverse scan and then, once identified; the probe was rotated by 90º to capture the longitudinal image of walls of the carotid artery. The complete protocol for image acquisition was discussed in detail in our previous studies (53-60). The average image resolution was 0.0529 mm-per-pixel.

Conventional cardiovascular risk factors

In this study, 14 CCVRF including demographics and blood biomarkers were collected for each patient. These 14 CCVRF are (I) age, (II) gender, (III) systolic blood pressure, (IV) diastolic blood pressure, (V) status of hypertension, (VI) glycated hemoglobin (HbA1c), (VII) low-density lipoprotein cholesterol (LDL-C), (VIII) high-density lipoprotein cholesterol (HDL-C), (IX) total cholesterol (TC), (X) triglyceride (TG), (XI) a ratio of total cholesterol and high-density lipoprotein cholesterol (TC/HDL), (XII) smoking status, (XIII) family history of CVD in first-degree relative, and (XIV) estimated glomerular filtration rate (eGFR). The baseline characteristics of these conventional cardiovascular risk factors have been presented in the “Baseline characteristics” section.

Current image phenotype measurements and 10-year CUSIP prediction

Current CUSIP measurements

Besides the CCVRF, this study also measures five CUSIP such as (I) average cIMT (cIMTave_curr) (62), (II) maximum cIMT (cIMTmax_curr) (62), (III) minimum cIMT (cIMTmin_curr) (62), (IV) variability in cIMT (cIMTV_curr) (63,64), and (V) carotid total plaque area (cTPA_curr) (56,60,62,65-67). The subscript “curr” indicates the measurement of the corresponding CUSIP at the baseline. Two operators (experienced and novice) each having 15 years of experience in radiology measured all these five CUSIP from the 395 CUS scans using automated software (AtheroEdge, AtheroPoint^TM, Roseville, California, USA) (62,63,66,68,69). This software works in three stages: (I) automatic detection of the region-of-interest, (II) identification of far wall of the CCA followed by a delineation of the lumen-intima (LI) and media-adventitia (MA) interfaces, and (III) measurement of current CUSIP throughout the length of CUS scan. Such type of measurement is as the full-length measurement (62,63,66,68-70). Note that our cIMT measurements consider both basic cIMT and plaque burden if any.

The cIMTave_curr was then measured as the mean polyline distance between the 100 connected set of points taken on LI and MA interfaces (71,72). Polyline distance metric was used to measure the perpendicular distance between the vertex on the LI interface and the corresponding polyline segment on the ML interfaces. Since the plaque growth in bidirectional [so-called Glagov’s Phenomenon (73)], the distance was also measured between the vertex on the MA interface and the corresponding polyline segment on the LI interface (71,72). All the distances were then averaged together to obtain the cIMTave_curr. Similarly, cIMTmax_curr and cIMTmin_curr were computed as the maximum and minimum distance, respectively, between LI and MA borders. The cIMTV_curr is a recently developed marker of variations cIMT (63,64). The procedure for measuring cIMTV_curr has been discussed in our recent study (63,64). In this study, we automatically measured the carotid plaque area as the sum of all pixels encapsulated within the envelope of LI and MA interfaces that spanned all along the length of the CUS scan (56,60). Note that since this plaque follows the morphology of the atherosclerotic plaque variation, it was also termed as morphological total plaque area (mTPA) (60) or cTPA_curr (56,60). It must be noted that the cTPA_curr is comprised of the focal thickening region which is more than 50% of the surrounding IMT (61,74,75) or the region where cIMTmax >1.5 mm (61,74-78).

10-year CUSIP measurements

The five types of 10-year CUSIP (cIMTave_10-year, cIMTmax_10-year, cIMTmin_10-year, cIMTV_10-year, and cTPA_10-year) are predicted using the standardized annual progression rates of cIMT and carotid plaque area (53,54,58,79) when applied on the original five types of image-based risk factors. The subscript ‘10-year’ indicates the 10-year prediction of the phenotype. These five types of 10-year CUSIP are computed by integrating the five types of current CUSIP (cIMTave_curr, cIMTmax_curr, cIMTmin_curr, cIMTV_curr, and cITPA_curr) with the 11 CCVRF (age, ethnicity, gender, carotid artery type, HbA1c, LDL-C, total cholesterol (TC), SBP, smoking, BMI, and estimated glomerular filtration rate or eGFR). Given the annual progression rates of cIMT due to all 11 CCVRF, an overall increase in cIMT from the current value, in 10-years, was computed. In the current study, the eGFR blood biomarker was used for the 10-year CVD/stroke risk prediction.

Statistically derived 10-year CVD/stroke risk calculators

In this study, the 10-year CVD/stroke risk was computed using 13 types of CVRC_Stat. These 13 types of CVRC_Stat are: (I) FRS (80), (II) UKPDS56 (81), (III) UKPDS60 (82), (IV) RRS (83), (V) Pooled Cohort Risk Score (PCRS) (15), (VI) NIPPON risk chart (84), (VII) SCORE risk chart (85), (VIII) QRISK3 (86), (IX) World Health Organization (WHO) risk chart (87,88)), (X) PROCAM (89), (XI) FRS (Stroke) (90), (XII) MSRC (91), and (XIII) AECRS2.0. Out of 13, the first 12 types of CVRC_{Stat (conventional)} are purely based on the 14 CCVRF and they do not consider image-based biomarkers of CVD/stroke (for example cIMT and cTPA) into their risk estimation model. Recently, Khanna et al. (54,55) made a first-ever attempt to bridge this gap between CCVRF and image-based phenotypes by developing an integrated 10-year risk calculator called AECRS1.0. Khanna et al. (53) further evaluated the performance of AECRS1.0 against the 10 types of CVRC_{Stat (conventional)} and reported a higher predictive ability (AUC 0.927, P<0.001) of the AECRS1.0 compared to other 10 types of CVRC_{Stat (conventional)}.

In our proposed study, the AECRS1.0 was extended by adding an eGFR into the risk prediction model (so-called AECRS2.0). Earlier studies have reported a strong link between CVD/stroke and CKD (92,93). Hence, the inclusion of eGFR in the risk prediction model of AECRS2.0 provided an additional benefit to the physicians, while evaluating the 10-year CVD/stroke risk in patients suffering from CKD. The AECRS2.0 is considered as the 13^th type of CVRC_Stat. Since the AECRS2.0 considers the integrated risk factors (CCVRF + CUSIP), we refer to it as CVRC_{Stat (integrated)}. Although a set of input risk factors used for the development of all these 13 types of CVRC_Stat (12 types of CVRC_{Stat (conventional)} and one type of CVRC_{Stat (integrated)}] is different, a common thread links all these CVRC_Stat under one category. This common thread is the use of statistics or the statistical methods for the development of this CVRC_Stat. In our proposed study we are comparing the CVD/stroke risk stratification of patients performed using 13 different types of CVRC_Stat and the automated CVRC_ML. In short, this study compared the ML-based method vs. the statistical-based methods of CVD/stroke risk stratification.

Machine learning-based risk stratification

The generalized architecture of the ML-based risk stratification system is shown in Figure 2. The overall system architecture is divided into three parts: (I) offline system (or training model), (II) online system (testing model), and (III) performance evaluation. At first, the database was partitioned using a 10-fold cross-validation protocol (so-called K10). The details about the working of the K10 protocol have already been discussed in our previous studies (47,94-96).

Figure 2.

The generalized architecture of machine learning-based CVD/stroke risk stratification. CUSIP, carotid ultrasound image-based phenotypes; EEGS, event-equivalence gold standard; AUC, area under the curve.

The offline ML-based system then extracts 14 conventional features (gender, age, HbA1c, LDL, HDL, TC, TG, LR, HT, SBP, DBP, smoking FH, and eGFR) from the training database. It should be noted that in order to investigate the effect of current CUSIP on the 10-year risk stratification of patients, a dotted plug-and-play block of CUSIP has been shown in Figure 2. The training features along with EEGS will then be used to train the SVM-based ML classifier. Once the SVM-based ML classifier is trained, the offline training coefficients are then used to transform the online test features into the high-risk or low-risk category for the patients using the online SVM classifier. Such an online ML system can then be utilized in routine preventive CVD/stroke prognosis systems without the requirement of EEGS. The performance of the proposed ML-based risk stratification system can be evaluated using the area-under-the-curve (AUC) metric, which is most commonly employed in medical studies (97-99).

Experimental protocol

The hypothesis of the proposed study is based on three types of experiments shown in Figure 3.

Figure 3.

The three types of experimental protocols followed in this proposed study.

Experiment 1: 12 types of CVRC_Stat against CVRC_ML utilizing 14 types of CCVRF

In this experiment (Figure 3A), the performance of the ML-based calculator [CVRC_{ML (conventional)}] was compared against the 12 types of CVRC_{Stat (conventional)} using 14 CCVRF. Here, we hypothesize that, given the same 14 CCVRF, ML-based calculator [CVRC_{ML (conventional)}] provides a better CVD/stroke risk stratification compared to the 12 types of statistically derived conventional calculators [CVRC_{Stat (conventional)}].

Experiment 2: CVRC_{Stat (integrated)} vs. CVRC_{ML (integrated)} with integrated risk factors

In this experiment (Figure 3B), the performance of ML-based calculator [CVRC_{ML (integrated)}] was compared against the CVRC_{Stat (integrated)} using integrated risk factors (14 CCVRF + 5 CUSIP). Here we hypothesize that, given the same integrated risk factors, ML-based calculator [CVRC_{ML (integrated)}] provides a better CVD/stroke risk stratification compared to the statistically derived CVRC _{Stat (integrated)}. Note that as explained in the “Introduction” section, CVRC_{Stat (integrated)} means the AECRS2.0 calculator.

Experiment 3: CVRC_{ML (conventional)} with 14 CCVRF vs. CVRC_{ML (integrated)} with 19 integrated risk factors

In this experiment (Figure 3C), we hypothesize that ML-based calculator with integrated risk factors CVRC_{ML (integrated)} can provide a better CVD/stroke risk compared to the ML-based calculator with 14 CCVRF [CVRC_{ML (conventional)}]. This experiment showed the higher predictive ability of the integrated risk factors compared to the CCVRF alone.

Statistical analysis

All types of statistical analyses were performed using SPSS23.0 and MATLAB2017b. The baseline characteristics shown in Table 1 are expressed in mean ± standard deviation for continuous variable and as a percentage for the categorical variable. Two sample students’ t-test and chi-square tests were used for continuous variable and categorical variables, respectively. The receiver operating characteristics (ROC) curve was used to compare the performance of CVD/stroke risk stratification performed using the CVRC_ML and the 13 types of CVRC_Stat [12 types of CVRC_{Stat (conventional)} and one CVRC_{Stat (integrated)}]. Area-under-the-curve (AUC) was computed for all the 13 types of CVRC_Stat and the ML-based system. In order to obtain the ROC curve and to compute the AUC value, a combination of HbA1c and the cIMTmax_curr was used as a response variable. Such type of response variable, which is also called an event-equivalence gold standard (EEGS), has already been studied in our previous publications (33,53,55,58,59,70,79). This is because this EEGS is associated with the atherosclerosis process and mimics the characteristics of the true endpoint such as CVD/stroke events (70,79). Although AUC was used as a primary performance evaluation (PE) metric, additional PE metrics like sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy were also computed using the same EEGS. The level of statistical significance was tested for P<0.05.

Table 1. Baseline characteristics of the patients divided into low-risk and high-risk classes.

Parameters	Overall	Low-risk#	High-risk#	P
Total (n)	202	22 (10.89%)	180 (89.11%)	–
Male, n (%)	156 (77.23%)	16 (10.26%)	140 (89.74%)	0.596
Age (years)	69±11	70.77±9.21	68.74±11.15	0.414
HbA1c (%)†	6.28±1.11	7.56±0.94	6.12±1.03	<0.0001
LDL-C (mg/dL)	100.75±31.48	97.50±31.49	101.14±31.55	0.610
HDL-C (mg/dL)	50.49±14.97	48.27±14.34	50.76±15.06	0.464
TC (mg/dL)	174.33±36.73	168.32±34.13	175.07±37.05	0.417
TG (mg/dL)	174.33±36.73	168.32±34.13	175.07±37.05	0.425
TC/HDL	3.65±1.01	3.74±1.19	3.64±0.99	0.658
SBP (mmHg)	134.55±8.92	136.36±7.90	134.33±9.04	0.315
DBP (mmHg)	87.28±4.46	88.18±3.95	87.17±4.52	0.315
HT, n (%)	147 (72.77%)	18 (12.24%)	129 (87.76%)	0.315
Smoking, n (%)	81 (40.10%)	9 (11.11%)	72 (88.89%)	0.935
eGFR (mL/min/1.73 m2)	45.06±20.92	44.14±22.51	45.18±20.78	0.826
FH, n (%)	24 (11.88%)	3 (12.50%)	21 (87.50%)	0.789

^†, significant confounding factors; ^#, a combination of HbA1c and cIMTmax was used for risk stratification. HbA1c, glycated hemoglobin; LDL-C, low-density lipoprotein cholesterol; HDL-C, high-density lipoprotein cholesterol; TC, total cholesterol; TG, triglyceride; SBP, systolic blood pressure; DBP, diastolic blood pressure; HT, hypertension; eGFR, estimated glomerular filtration rate; FH, family history.

Results

Baseline characteristics

The baseline characteristics of the study participants are shown in Table 1. A cohort of 202 Japanese ethnicity patients was recruited in this study with a mean age of 69±11years (range, 29–88 years). Out of 202 patients, 156 (77.23%) were males and 46 (22.77%) were females. The average value of HbA1c was 6.28±1.11% (range, 4.80–13%), LDL-C was 100.75±31.48 mg/dL (range, 24–193 mg/dL), HDL-C was 50.49±14.97 mg/dL (range, 18–115 mg/dL), TC was 174.33±36.73 mg/dL (range, 61–255 mg/dL), TG was 117.10±56.69 mg/dL (range, 32–255 mg/dL), and eGFR was 45.06±20.92 mL/min/1.73 m² (range, 3–110 mL/min/1.73 m²). Out of 202 patients, 49 (24.26%) were suffering from type 2 diabetes mellitus, 162 (80.20%) were suffering from chronic kidney disease, 147 (72.77%) were hypertensive, 81 (40.10%) were smokers, and 24 (11.88%) patients had a family history of coronary heart disease in the first degree relatives. The criteria for type 2 diabetes mellitus was HbA1c ≥6.5% or treatment for hyperglycemia (100), criteria for hypertension was SBP ≥140 mmHg or DBP ≥90 mmHg or treatment with antihypertensive medications (101), and the criteria for chronic kidney disease was EGFR <60 mL/min/1.73 m² (102). As shown in Table 1, all the patients were further classified into two risk categories using an event equivalence gold standard (a combination of HbA1c ≥6.5% and cIMTmax ≥1.5 mm). The risk of category-based baseline characteristics is presented in Table 1.

10-year CVD/stroke risk stratification: 12 types of CVRC_{Stat (conventional)} vs. CVRC_{ML (conventional)}

The ROC curves are shown in Figure 4A compared to the performance of the 12 types of CVRC_{Stat (conventional)} and CVRC_{ML (conventional)} using the 14 types of CCVRF. The CVD/stroke risk stratification performed using CVRC_{ML (conventional)} reported the higher AUC value (AUC 0.70, P=0.11) compared to the mean AUC value (AUC 0.62, P<0.05) computed from 12 types of CVRC_{Stat (conventional)}, indicating an overall improvement by ~13%. From Figure 4A, it is clear that the CVRC_{ML (conventional)} provides higher predictive power and a better or comparable risk stratification compared to the 12 types of CVRC_{Stat (conventional)}.

Figure 4.

Comparing the performance of ML-based risk calculator and the statistically derived risk factors for (A) conventional risk factors and (B) integrated risk factors. (A) Given the 14 CCVRF, the performance of CVRC_{ML (conventional)} was compared against the 12 types of CVRC_{Stat (conventional)}. (B) Given the integrated risk factors, the performance of CVRC_{ML (integrated)} was compared against the CVRC_{Stat (integrated)} (AECRS2.0). The black arrow in Figure 4A indicates the dotted black colored, receiver operating characteristics curve of the CVRC_{ML (conventional)} with 14 CCVRF. CVRC_ML, ML-based cardiovascular risk calculator; CVRC_Stat, statistically derived cardiovascular risk calculator; FRS, Framingham Risk Score; UKPDS56, United Kingdom Prospective Diabetes Study 56; RRS, Reynold Risk Score; ASCVD, atherosclerosis CVD calculator by ACC/AHA; SCORE, Systematic Coronary Risk Evaluation chart; WHO, World Health Organization chart; FRS (Stroke), Framingham Stroke Risk Calculator; MSRC, My_Stroke Risk Calculator; AECRS2.0, AtheroEdge Composite Risk Score version 2.

10-year CVD/stroke risk stratification: CVRC_{Stat (integrated)} vs. CVRC_{ML (integrated)}

The ROC curves are shown in Figure 4B compared to the performance of the CVRC_{Stat (integrated)} against CVRC_{ML (integrated)} using the integrated risk factors (14 CCVRF + 5 CUSIP). The CVD/stroke risk stratification performed using CVRC_{ML (integrated)} reported the higher AUC value (AUC 0.88, P<0.001) compared to the AUC of CVRC_{Stat (integrated)} (AUC 0.83, P<0.001), indicating an overall improvement by ~6%. It should be noted that the CVRC_{Stat (integrated)} with integrated risk factors is nothing but the AECRS2.0 calculator. Figure 4B further validated our hypothesis of better risk stratification using CVRC_{ML (integrated)} compared to the CVRC_{Stat (integrated)}.

Comparison between CVRC_{ML (conventional)} and CVRC_{ML (integrated)}: effect of image-phenotypes

Figure 5 explicitly compares the performance of two types of ML-based CVRC: (I) CVRC_{ML (conventional)} and (II) CVRC_{ML (integrated)}. The inclusion of integrated risk factors into the ML-based risk calculator has resulted in a higher AUC value (AUC 0.88, P<0.001) compared to the ML-based risk calculator with conventional risk factors alone (AUC 0.70, P=0.11), indicating an overall improvement of ~26%. The improvement in AUC value is in line with a recent study presented by our group (79). This experiment showed that the integrated risk factors have higher predictive ability, not only in statistically derived risk calculators but also with the ML-based risk calculators. The reason for the higher predictive ability of the integrated risk factors is due to the integration of CUSIP with the CCVRF. The CUSIP truly reflect the carotid atherosclerotic plaque burden of the patients and thus, provide better risk estimation of the patients.

Figure 5.

Comparing the performance of CVRC_{ML (conventional)} with 14 CCVRF with the performance of CVRC_{ML (integrated)} with integrated risk factors. CVRC_{ML (conventional)}, CVRC_ML with 14 conventional risk factors; CVRC_{ML (integrated)}, CVRC_ML with integrated risk factors.

Sensitivity analysis and performance evaluation

In this study, the 10-year CVD/stroke risk was computed using 13 types of CVRC_Stat which were either derived using proportional hazard or regression-based methods (15,80-84,86,89,91,103), it is important to investigate the sensitivity of 10-year risk if the coefficients are varied. In our study, we have varied the coefficients of these risk calculators by ±2% and recorded the variation in AUC values. It should be noted that all 13 types of CVRC_Stat showed a variation of less than 5% in the overall AUC value. The average sensitivity, specificity, PPV, NPV, and accuracy for the 10-fold cross-validation (I) for the 12 types of CVRC_{Stat (conventional)} were 34%, 80.47%, 14.72%, 93.38%, and 76.74%, respectively, (II) for the CVRC_{ML (conventional)} were 23.86%, 93.01%, 23.08%, 93.29%, and 87.43%, respectively, (III) for the CVRC_{Stat (integrated)} were 69.89%, 75.60%, 20.10%, 96.62%, and 75.14%, respectively, and (IV) for the CVRC_{ML (integrated)} were 10.60%, 99.72%, 76.59%, 92.70%, and 92.52%, respectively. It should be noted sensitivity and PPV for all the CVRC is less than the specificity and NPV. This is because of the unbalanced nature of the database with only ~8% of samples were categorized into high-risk class. Due to the lower sensitivity values, it can be interpreted that the predictive ability of all the CVRC to correctly identify low-risk patients is more than the high-risk patients. The similar kind of observations of lower sensitivity values compared to septicity values was previously reported in multiple studies (26,51,104). Another observation from these PE metrics is that the overall accuracy for CVRC_{ML (integrated)} is highest amongst all the other CVRC (CVRC_{Stat (conventional)} or CVRC_{Stat (integrated)} or CVRC_{ML (conventional)}]. This has again validated our hypothesis of higher predictive ability of the integrated risk calculators compared to the CCVRF alone.

Discussion

Claims and summary

There were two hypotheses for this study: (I) the ML-based CVD/stroke risk calculator (CVRC_ML) can provide a better CVD/stroke risk stratification compared to statistically derived risk calculators (CVRC_Stat) and (II) the inclusion of CUSIP into the CCVRF can provide a higher predictive ability compared to the CCVRF alone, irrespective of the type of risk calculator (CVRC_ML or CVRC_Stat). The results are shown in Figure 4A,B, confirmed the first hypothesis of this proposed study. Given the 14 CCVRF (Figure 4A), CVRC_{ML (conventional)} provided a higher AUC value (AUC 0.70, P=0.11) compared to mean AUC of all the 12 types of CVRC_{Stat (conventional),} with an overall improvement of ~13%. Similarly, given the integrated risk factors (Figure 4B), CVRC_{ML (integrated)} provided a higher AUC value (AUC 0.88, P<0.001) compared to the AUC value of CVRC_{Stat (integrated)} (AUC 0.83, P<0.001), with an overall improvement of ~6%. The reason for this improved performance in ML-based risk calculators is their ability to consider a non-linear and complex relationship between the risk factors, which is, however, ignored by the conventional CVRC_{Stat (conventional)} (28,33). The results shown in Figure 5, confirmed the second hypothesis of this proposed study. The ML-based CVD/stroke risk calculator with integrated risk factors provided a higher AUC value (AUC 0.88, P<0.001) compared to an ML-based risk calculator with CCVRF (AUC 0.70, P=0.11)_, demonstrating an overall improvement of ~26%. This is because integrated risk factors combine the 5 CUSIP that provides additional information about the morphological variations in atherosclerotic plaque, which is, however, not well-captured by the CCVRF, alone (32,33,53,54,57-59,79). Such type of comparison between CVRC_ML with two different sets of input risk factors has been recently studied by our group (79). In conclusion, there is an improvement in risk stratification with CVRC_{ML (integrated)} (AUC 0.88, P<0.001) compared to all the CVRC_{Stat (conventional)} (AUC 0.62, P<0.05) with an improvement in an AUC value of ~42%. The improvement in CVD/stroke risk stratification is depicted in Figure 6. The overall sensitivity, specificity, PPV, NPV, and accuracy for the CVRC_{ML (integrated)} were 10.60%, 99.72%, 76.59%, 92.70%, and 92.52%, respectively.

Figure 6.

Incremental risk stratification performance from (I) CVRCStat(Conventional), (II) CVRCStat(Integrated) to CVRCML(Integrated), showing an improvement by ~42% between conventional statistical calculators and SVM-based calculator with integrated risk factors. The black arrow indicates the ROC curve of the SVM-based ML calculator with integrated risk factors.

Benchmarking

Table 2 compares eight studies that compare the ML-based risk stratification against statistical calculator-based CVD/stroke risk assessment. As shown in Table 2, there are 12 attributes from column C1 to column C12 for each of the 8 studies (row R1 to row R8). Out of the eight studies, only three studies (row R4, row R7, and row R8) consider a combination of CCVRF and the CUSIP as input phenotypes. Two studies out of these three have used the CUS-based image phenotypes. The SVM is the most commonly used ML algorithm. It should be noted that, among all the eight studies, only the proposed study (row R8) compares the ML-based risk stratification results against the 13 types of statistically derived CVRC. The remaining seven studies (row R1 to row R7) either use FRS or ASCVD or UKPDS risk calculator for the comparison. The proposed study uses the integrated risk factors as the input to the ML-based algorithm. The inclusion of integrated risk factors is the main cause of an increase in the AUC value from 0.93 to 0.99, much stronger to other studies.

Table 2. Comparison of the studies that performed the ML vs. non-ML CVD/stroke risk assessment.

Author (year)	N	ML algorithm	CVRCStat	Imaging modality	CCVRF + image phenotypes	#F	Types of features	Ground truth	Training protocol	FU (years)	PE
Narain et al. (105), 2016	689	QNN	FRS	×	×	7	CCVRF	From physician	K3	–	ACC: 98.57%
Unnikrishnan et al. (50), 2016	2,406	SVM	FRS	×	×	9	CCVRF	CVD events	K5	10	AUC: 0.71
Weng et al. (26), 2017	378,256	RF, LR, GBM, & ANN	ASCVD	×	×	30	CCVRF	CVD events	K4	10	AUC: 0.764
Venkatesh et al. (49), 2017	6,814	RF	FRS	MRI + CUS	√	735	CCVRF, image phenotypes, & serum biomarkers	CVD events	K3	12	C-index: 0.81
Zarkogianni et al. (51), 2017	560	HWNN	UKPDS	×	×	15	CCVRF	CVD events	K10	5	AUC: 0.71
Kakadiaris et al. (48), 2018	6,459	SVM	ASCVD	×	×	9	CCVRF	CVD events	K2	13	AUC: 0.92
Han et al. (106), 2019	86,155	Boosted ensemble	FRS & ASCVD	CT	√	70	Clinical + laboratory + CAC	CVD events	Hold-out	4.6	AUC: 0.82
Proposed, 2019	202	SVM	13 CVRC	CUS	√	19	CCVRF + CUSIP	EEGS	K10	–	AUC: 0.70 (CCVRF); AUC: 0.88 (CCVRF + CUSIP)

CVD, cardiovascular diseases; QNN, quantum neural network; SVM, support vector machine; RF, random forest; LR, logistic regression; GBM, gradient boosting machines; ANN, artificial neural network; HWNN, hybrid wavelet neural networks; CVRC, cardiovascular risk calculators; CUS, carotid ultrasound; CT, computed tomography; MRI, magnetic resonance imaging; CCVRF, conventional cardiovascular risk factors; ASCVD, atherosclerosis cardiovascular disease; FRS, Framingham risk score; FU, follow-up; PE, performance evaluation; CUSIP, carotid ultrasound image-based phenotypes; AUC, area under the curve.

Use of event-equivalence gold standard for the risk stratification

Cerebrovascular and cardiovascular events are generally considered as the primary endpoints or gold standards for the CVD/stroke risk assessment and these are expensive and time-consuming (107-109). In order to provide an affordable CVD/stroke risk assessment solution to the physicians, low-cost preventive surrogate biomarkers need to be explored and refined (79). Surrogate biomarkers mimic the characteristics of the primary endpoints and thus can be considered as alternative gold standards. Furthermore, surrogate biomarkers (I) exhibit a link with the primary endpoints, (II) are clinically relevant, and (III) reproducible (110). Since such surrogate biomarkers provide equivalence to the primary gold standards, they are termed as EEGS (70,79). This EEGS needs to be evaluated using a small sample size, with lower-cost, and for short duration (107,108). The Food and Drug Administration of the USA has defined some of the commonly used surrogate biomarkers of CVD/stroke (111). Mancini et al. (112) also presented a study that reported a list of surrogate biomarkers of CVD. Since the patients are diabetic and have a moderate-to-high CVD risk profile, we have selected a combination of HbA1c and cIMTmax as the EEGS. Both the HbA1c and cIMTmax have a strong association with CVD/stroke events. Such types of EEGS were already been tested and evaluated in recent studies (53,54,58,59,79).

Therapeutic implications of risk stratification

Risk stratification assists the physicians while making the prognostic decision about the prevention of CVD/stroke events. The value of screening using AECRS2.0 or ML-based calculators can be more effective when computing 10-year risk, unlike the current risk as one can compute the contribution of individual risk factors and their joint associations which influences the overall CVD/stroke risk. For example, if the patients are suffering from diabetes, hyperlipidemia, hypertension, or CKD, then an appropriate combination of medications need to be prescribed for preventing the onset of CVD/stroke events. This is purely based on the contribution of individual risk category in which the patients belong. This is the point where the CVRC comes into the picture. These CVRC estimate the 10-year risk of CVD/stroke of the patients and further categorize them into appropriate risk profiles using predefined thresholds. Certain guidelines also take the assistance of these CVRC to recommend the initiation of statin therapy. The recent guidelines for the assessment of cardiovascular risk by the American College of Cardiology (ACC) and American Heart Association (AHA) recommended the use of low-to-moderate intensity statins if the 10-year risk computed using atherosclerotic CVD (ASCVD) calculator is ≥7.5% (15,113). The National Institute for Health and Care Excellence (NICE) guidelines recommended the use of 20 mg Atorvastatin if the 10-year risk computed using the QRISK2 calculator is ≥10% (20,21,113). Canadian guidelines recommended the statin initiation if the FRS is ≥10% (113-115). Since the CVRC_ML provides a better risk assessment of the patients compared to the statistically derived CVRC_Stat, they can accurately determine the statin eligibility of the patients (48,116). The ML-based risk stratification further prevents the excessive or unnecessary statins recommended by the CVRC_Stat to patients (48). This has been recently observed by the study presented by Kakadiaris et al. (48) who reported the use of statins to 46% (AUC 0.76) of the study population using the ACC/AHA calculator, while the SVM-based CVRC_ML recommended the statins to only 11.1% of their study population (AUC 0.92). Thus, compared to the statistically derived CVRC, the ML-based risk calculators can be employed for routine CVD/stroke risk assessment in clinical settings (116).

From the baseline characteristics, it is clear that the selected Japanese cohort belongs to the low-to-moderate risk category, with 49 patients (24.26%) was suffering from type 2 diabetes mellitus, 162 (80.20%) were suffering from chronic kidney disease, 147 (72.77%) were hypertensive. As discussed earlier, NICE (20,21), Canadian (115), and the recent ACC/AHA guidelines (15,16) favor the use of low-to-moderate intensity of statins for high-risk patients. This is also applicable for the patients stratified into the high-risk category using the ML-based CVD risk calculator. In cohorts with similar baseline characteristics, a treatment plan can further be modified based on the presence of risk factors like type 2 diabetes mellitus, hypertensive, dyslipidemia, and chronic kidney disease. For example besides statins therapy, high-risk patients suffering from type 2 diabetes mellitus can be treated with Metformin, Sulfonylureas (117,118). Similarly, high-risk patients suffering from hypertension can be treated angiotensin-converting enzyme (AEC), Angiotensin II receptor blockers (119), Renin-Angiotensin-Aldosterone System (RAAS) Blockade (120). Initial lipid-lowering medications like Atorvastatin, Simvastatin, and Pitavastatin can be used for controlling hyperlipidemia (121-124). However, the decision of an appropriate choice of medications is strictly based on the risk category of patients and the judgment made by physicians.

A note on sample size

In general, the selected data should not be identical. In our proposed study, samples were taken from both the left and right carotid artery of the same person. Although both left and right carotid arteries follow a similar genetic makeup and functionality of supplying oxygen-rich blood to the brain tissues, they work independently in two different pathways. Furthermore, due to the multi-focal and random nature of the atherosclerotic disease, the deposition of the plaque can be different in both the left and right carotid arteries. Thus, initially, total samples of 404 (202 patients × 2 CUS scans) non-identical samples were selected to perform various investigations in this study. Due to the poor image quality, nine CUS scans were omitted from this study and thus a total of 395 ultrasound B-mode carotid scans were finally selected for further statistical analysis. To investigate the validity of the selected sample size (i.e., 395 CUS scans), a power analysis was performed with confidence level of 95%, the margin of error (MoE) of 5%, and data proportion () of 0.5. The power analysis results in the desired sample size of 384. This means the selected sample size 395 samples is ~3% higher than the required sample size of 384. This validated the claim that the sample size of 395 was enough to perform this study. The detailed discussion on power analysis provided in the Supplementary files.

Strength, limitations, and future scope

Although we reported a robust and superior performance of the SVM-based CVRC_ML to risk stratify the patients, we intend to improvise the study in the following areas: (I) the proposed study needs to be validated in the database with varying ethnicities. This will help the physicians to understand the effect of ethnicity on the ML-based risk stratification of the patients. Except for QRISK3, nearly all the 13 types of CVRC_Stat were developed for specific types of ethnic groups. Thus, it would be interesting to know the behavior of the ML-based system and CVRC_Stat on the diverse ethnic cohorts. Besides ethnicity, it is also important to consider inflammatory markers such as erythrocyte sedimentation rate and the high-sensitivity C-reactive protein into the risk prediction algorithm (125-130). This is because these inflammatory markers have a strong association with CVD/stroke (125-130). Besides these intended improvisations, the proposed study is novel in its concept and perhaps the first study that compares the ML-based risk stratification with 13 types of CVRC_Stat. An influential effect of integrated risk factors has also been investigated in this study. Due to the inclusion of CUS scans and limited (but most common) number of risk factors, the proposed ML-based can be deployed in routine preventive CVD/stroke risk stratification applications at low-cost (79). The proposed study is a pilot investigation of the ML system in an environment of 13 types of CVRC_Stat. Further validation of this study can be tried to upgrade the current system having more robustness and independent of ethnicity. This retrospective pilot study can also be tested on a longitudinal dataset that actual cardiovascular or cerebrovascular events instead of EEGS. Lastly, hybrid classifiers can be built for better training design (35,131,132).

Conclusions

This study is a first-ever attempt for comparing the ML-based risk calculator against the 13 types of statistical CVRCs. The study hypothesized that machine learning in has better learning ability due to its ability to adjust the nonlinearity in risk factors and further, morphological plaque-based inclusions are representations of risk factors unlike the blood biomarkers alone. Based on these hypotheses, this study showed fulfillment of three objectives: ML-based calculators are more powerful than: (I) statistical calculators without integrated factors (~13% improvement) and (II) with integrated risk factors (6% improvement) such as AtheroEdge (AtheroPoint, Roseville, CA, USA) composite risk calculator, and (III) ML-based conventional risk calculators (26% improvement), respectively. Furthermore, the ML-based integrated calculator showed higher predictive power (42% improvement) compared to the mean predictive power of the 12 types of the conventional risk calculator. The overall risk-stratification accuracy for the ML-based integrated risk calculator was 92.52% which was higher than all other statistical calculators. Even though the system was designed on a particular ethnic group, it is generalized enough to be extended to other ethnicities, classification paradigms, and other event equivalent gold standards.

Supplementary

Power analysis

The primary aim of the power analysis is to determine a sample size which is essential to prove the hypothesis of the study. In this study, we hypothesized that (I) ML-based calculator can provide a better risk stratification compared to the statically derived conventional risk calculators and (II) inclusion of integrated risk factors into the risk prediction model can further improve the risk stratification ability, irrespective of the type of CVD risk calculator (statistical or ML-based). This study was designed for patients with Japanese ethnicity, thus, the population in power analysis refers to the samples derived from the Japanese patients (202 patients or i.e., 395 CUS scans) who need to be risk-stratified. The reason for considering the 395 CUS scans as samples have been discussed in the “A note on Sample Size” section. In order to perform the power analysis, we considered the 95% CI, 5% of the margin of error (MoE), and 0.5 as data proportion ($\widehat{p}$). Assuming a normally distributed sample set, z-score (Z*) of 1.96 was obtained using standard z-table for a confidence level of 95%. The MoE ensures that the study results should not exceed the tolerance band of ±5% of the true population. The desired sample size (n) can be computed using, $n=\left[{(z\ast )}^2\times \left(\frac{\widehat{p}(1-\widehat{p})}{Mo{E}^2}\right)\right]$ (133). The resultant sample size with a 95% confidence level and 5% of MoE was ~384. Thus recruited sample size (395) was ~3% higher compared to the desired sample size of 384.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The patients were approved by the institutional review board of Toho University, Japan and written consent was obtained from all the study participants. The study was approved by Toho University Japan (Ohashi Ethics Committee, Authorization No. 13-56).

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.