Multimodality carotid plaque tissue characterization and classification in the artificial intelligence paradigm: a narrative review for stroke application

Abstract

Cardiovascular disease (CVD) is one of the leading causes of morbidity and mortality in the United States of America and globally. Carotid arterial plaque, a cause and also a marker of such CVD, can be detected by various non-invasive imaging modalities such as magnetic resonance imaging (MRI), computer tomography (CT), and ultrasound (US). Characterization and classification of carotid plaque-type in these imaging modalities, especially into symptomatic and asymptomatic plaque, helps in the planning of carotid endarterectomy or stenting. It can be challenging to characterize plaque components due to (I) partial volume effect in magnetic resonance imaging (MRI) or (II) varying Hausdorff values in plaque regions in CT, and (III) attenuation of echoes reflected by the plaque during US causing acoustic shadowing. Artificial intelligence (AI) methods have become an indispensable part of healthcare and their applications to the non-invasive imaging technologies such as MRI, CT, and the US. In this narrative review, three main types of AI models (machine learning, deep learning, and transfer learning) are analyzed when applied to MRI, CT, and the US. A link between carotid plaque characteristics and the risk of coronary artery disease is presented. With regard to characterization, we review tools and techniques that use AI models to distinguish carotid plaque types based on signal processing and feature strengths. We conclude that AI-based solutions offer an accurate and robust path for tissue characterization and classification for carotid artery plaque imaging in all three imaging modalities. Due to cost, user-friendliness, and clinical effectiveness, AI in the US has dominated the most.

Introduction

Cardiovascular disease (CVD) is the leading cause of death globally, with nearly 18 million deaths annually (1). In the US alone, approximately 700,000 people die each year—the equivalent of 1 in 4 people dying every 36 seconds (2,3). CVD’s leading cause is atherosclerosis, now thought to be contributed by arterial inflammation (4,5). There are several risk factors for CVD and atherosclerosis, including hypertension, hyperlipidemia, diabetes, obesity, smoking, and a sedentary lifestyle (6,7). The process of atherosclerosis is characterized by excessive plaque deposits in the arterial wall (8). In the case of an unstable plaque, the associated fibrous cap breaks, causing a form of thrombosis (9). The thrombosis leads to embolism, which, in turn, causes a blockage in blood flow resulting in a stroke or myocardial infarction (10).

The plaque, which forms over time, can be detected using imaging modalities like magnetic resonance (MR), computer tomography (CT), and ultrasound (US) (11). Ordinarily, it can be challenging to determine the nature of plaque. Its various components—formed at different stages during its maturation—are spatially distributed in the arterial walls. Thus, the contrast visible in standard imaging does not clearly differentiate between components (12,13). This can be due to (I) partial volume effect in magnetic resonance imaging (MRI) (14) or (II) varying Hausdorff values in plaque regions with different components in CT (15), and (III) attenuation of echoes reflected by the plaque during US causing acoustic shadowing from calcification (16). Despite these challenges, imaging modalities can still offer clues about the different components of the plaque, tapped by the software-based advanced characterization and classification techniques (17-19).

This study aims to better understand the physics of each modality and its interpretation capabilities (20,21). Further, we explore how the detection algorithms detect and characterize and classify plaque tissue components, the so-called tissue characterization and classification of the carotid artery (TCCCA). This narrative review’s secondary objective is to understand the clinical requirements for a reliable, accurate, fast, and user-friendly TCCCA based on Artificial Intelligence (AI).

Several attempts have been made to perform tissue characterization and classification on carotid artery plaque (17,22). However, no in-depth reports have summarized advanced artificial intelligence (AI) methods for this application. TCCCA algorithms have changed over time to adapt to new technological developments (21,23), such as machine learning (24,25) and deep learning (26-30) in MR (31,32), CT, and US (33-40). Plaque characterization utilizing AI-based methods can assist in clinical decisions such as when to consider revascularization (41,42).

Search strategy

For this review, we followed the search strategy as shown in Figure 1. We used three repositories to retrieve all relevant articles related to carotid classification and characterization. Among the repositories, Knimbus is a digital library that contains well-known repositories like JSTOR, Oxford, Science Direct, Springer, Ebsco, and IEEE. We searched for “ultrasound carotid classification”, “ultrasound carotid characterization”, “MRI carotid classification”, “MRI carotid characterization”, “CT carotid classification”, “CT carotid characterization”, “PET carotid classification”, “PET carotid characterization”, “IVUS carotid classification” and “IVUS carotid characterization”. After applying advanced filters such as publication year (as a range), article type, age, journal, and relevance, we found 4,829 articles in the PubMed repository, 211 in IEEE, and 56,868 in Knimbus. Using exclusion criteria—that publications must be current within ten years and specific to carotid tissue classification and characterization—and further filtering by title, abstract, and relevance, we arrived at 1,253, 125, and 1,523 articles, respectively. After duplicates were removed, our final tally was 229 articles. Although we did not employ systematic review and meta-analysis (SRMA) methods, our search strategy was very similar.

Figure 1.

Search strategy showing inclusion and exclusion criteria.

Search by imaging modality

We computed the distribution of articles by modality, and Figure 2A describes the articles’ classification accordingly. We found that 60% dealt with US imaging techniques. US imaging is widely available, convenient, radiation-free, and low cost; thus, this technology’s usage was more common and more likely to appear in the publication. Ultrasound imaging is the most commonly used imaging technique for applying AI methods (94.59%), followed by CT (30%) (43,44) (yellow color) and MRI (35%) (45) (brown color).

Figure 2.

Classification of articles by (A) imaging modality. (B) AI techniques. MA, manual; SA, statistical analysis; ML, machine learning; DL, deep learning; RL, reinforcement learning.

Search by AI-models

Figure 2B shows the distribution of the AI-models applied for TCCCA. Specifically, these constituted manual analysis (MA), statistical analysis (SA), machine learning (ML), deep learning (DL), and reinforcement learning (RL). We found that 70% of the articles used MA. Machine learning (54%) is a very common AI method in TCCCA (46). ML techniques have been used for various applications, including classifying tissue from the liver (47,48), thyroids (49), skin (50), plaque (19,51,52), and arrhythmia classification (53). ML is an older AI technique and requires manual interaction to extract features. Our search showed that the DL models had not been used often in TCCCA, as it is relatively new to medical imaging (54). Some of the key publications related to DL for medical imaging in stroke imaging are (26,55-58). RL represents 1% of all articles and one of the upcoming fields in AI-based diagnosis tools.

Atherosclerosis disease and plaque formation

Plaque development involves the engagement and transfer of smooth muscle cells (SMCs) from the media layer to the intima layer. These SMCs in the media layer move into the intima layer and form the bulk of the cellular auxiliary matrix known as the plaque creation process. These SMCs from the media layer may also enter the surface and create a layer called the fibrous cap (59), which has an elastic property that prevents it from cracking. However, the likelihood of developing a fracture also increases as this layer stiffens (60). Inflammatory cells break down SMCs to reinforce and stabilize the cap. The cap’s stiffness index determines the probability of plaque breakup. Thus, determining stiffness in a stroke risk assessment is critical. This is connected to differentiating hard tissues from soft tissues (19,25,61-63).

Effect of plaque formation in blood vessels

The pressure of developing plaque creates tension in the artery, which, over time, reduces the force-bearing capability of the fibrous cap. Figure 3 illustrates the plaque formation. The human carotid atherosclerotic plaque has properties such as fibrous pads, lipids, intra-plaque hemorrhage and thrombus, that shows non-linear conduct (64).

Figure 3.

Stages of plaque formation and rupture. (A) Healthy artery, (B) LDL penetration via intima layer, (C) dislodging of SMC, and (D) wall rupture (courtesy of AtheroPoint, Roseville, CA, USA). SMC, smooth muscle cell.

Arterial stiffness

The rupturing of this plaque then results in thrombosis or clot formation that leads to acute morbid events. Plaque formation initially begins with the deposition of small lipid and fatty materials in the intima, which can grow into complex plaque structures over time (65). The advanced plaque has a heterogeneous composition, with a lipid center, calcification, and fibrous deposits of the connective tissue (66).

Cause of arterial stiffness due to atherosclerosis

The top layer of atherosclerotic plaque, which includes SMCs, is cellular fibrosis. The elasticity of the SMCs determines the amount of force the arterial wall can withstand. The SMCs inside of the vascular walls contribute to vascular rigidity (67). During the replication process, removing the SMCs present in an artery prevents atherosclerotic plaque development. During the replication process, the reduction or removal of the SMCs present in an artery prevents atherosclerotic plaque development. During the replication process, the small number of SMCs present in an artery prevents atherosclerotic plaque development. The small number of SMCs present in an advanced atheroma splits into deoxyribonucleic acid (DNA), which results in programmed cell death, or apoptosis. As the artery grows, inflammatory cytokines induce apoptosis, helping the SMCs to adapt. The accumulation of SMCs may be a product of cell death and replication, causing arterial stiffness (68). Previous research has suggested that increased stiffness may also be caused by changes in the collagen ratio to elastin in the arterial media’s extracellular matrix (69). An example of different pathological components is shown in Figure 4.

Figure 4.

Pathological images were depicting different components of plaque. (A) Healthy wall, (B) neovessels, (C) calcified plaque, and (D) intraplaque hemorrhage (courtesy of Dr. Luca Saba, U of Cagliari, Italy).

Physics of multimodality imaging for carotid plaque acquisition

This section presents the different imaging modalities used for carotid plaque imaging (20).

Magnetic resonance imaging for carotid plaque

The MRI contains four components: (I) a magnetic coil, (II) a gradient coil, (III) a radio frequency (RF) coil, and (IV) a shim coil (70). A set of magnet coils made of superconducting metal-alloy produce powerful magnetic fields when a current passes through it. It also generates negligible heat, which is subsequently cooled by cryogenic helium. There are three concentric gradient coils located at the primary magnet coil. These gradient coils change the magnetic field’s direction along the x, y, and z-axes. RF coils are mounted concentric to the main magnetic field and serve as antennae to send RF energy to the tissue and receive the induced RF signals generated by the tissue (71). The MRI requires the aid of a homogenous magnetic field to localize the region of interest. Shim coils were used to homogenously adjust the magnetic field by controlling it via the computer. The human body contains 80% water. Thus, when a patient lies in the bore of the MRI machine, the hydrogen atoms in the tissue align themselves with the magnetic field of the MRI machine. The resultant MR signals are localized and construct an image using gradient field magnets. Inhomogeneity correction in MRI for carotids can be adapted (72). Figure 5A shows the MRI machine representation.

Figure 5.

Imagining machines of (A) MRI, (B) CT, and (C) carotid ultrasound scanning using a linear probe (MRI/CT images-courtesy of Dr. Luca Saba, Italy, and US image, courtesy of AtheroPoint, CA, USA). MRI, magnetic resonance imaging; CT, computed tomography.

Computed tomography imaging for carotid plaque

CT imaging gives the ROI of tissue without the need to superimpose on the adjacent structure (73). It uses an X-ray radiation attenuation coefficient (AC) when imaging organs. AC was calculated as the density of the X-ray beam attenuated by the tissue. The CT machine mainly consists of an X-ray tube with slip-ring technology and a detector. The detector continuously rotates around the patient during the diagnosis. This technology is called Helical CT, which has replaced the conventional CT (step and shoot) acquisition technique. The images acquired via CT are three-dimensional and allow multiplanar reconstruction. In a CT machine, the “pitch” parameter indicates the quality of the image acquired. If the pitch is less than one, then the acquired image is considered high-quality and oversampled, but increases the patient’s radiation exposure. Typically, the pitch value should be between 1 and 1.5 (74).

The thickness of the image slice depends on the X-ray collimator. The longer the collimator is exposed, the thicker the slice. However, the amount of radiation exposure increases and fails to planner reconstruction. Multiple detector array CT scanners overcome this problem. These scanners contain multiple detector arrays, and the collimator spacing is quite wide. Thus, it improves the imaging quality and slice thickness. Dual-source CT scanners have recently been developed, containing dual X-ray sources and detectors; these detectors are perpendicular to each other and rotate around the patient body. This setup cuts the acquisition time in half while still producing high-quality images.

A study was conducted recently that compares the image quality and radiation dose delivered to patients during computed tomography (CT) angiography (CTA) of the supra-aortic arteries using two single-source (SS) and two dual-energy (DE) CT scanners (75). Figure 5B illustrates a typical CT machine. Several types of energy levels are sometimes considered for the optimization of carotid plaque characterization (76).

Several applications for carotid plaque characterization and microbleeds were shown in CT (77,78). Semi-automated wall measurements were computed in CT. Baradaran et al. (79) recently showed the calcium volume measurement using 74 CT. Saba et al. (80) showed that the color scale instead of the conventional grayscale improves the diagnostic confidence, accuracy, and inter-observer agreement of the readers, particularly junior ones, in the diagnosis of internal carotid artery dissection on non-contrast CT.

Ultrasound imaging for carotid plaque

Ultrasound machines have a duplex scanner and a transducer with a linear broadband width of 4–7 MHz (multifrequency) and a resolution of 20 pixels/mm. Ultrasounds operate on the principle of the piezoelectric effect (81). The machine contains a transducer, a CPU, and a monitor. The transducer contains piezoelectric crystals, which act as the senders and receivers of ultrasound signals. The piezoelectric crystals generate sound waves when a current flows through them, and they generate a voltage when they receive a vibration (ultrasonic wave) (5). When the ultrasound waves fall on the ROI tissue, the tissue reflects the ultrasonic waves (echo), the transducer catches the reflected signals. It then constructs and displays an image in the monitor based on these signals.

Some prerequisites are necessary for successful image normalization. The following prerequisites were carried out in this study: (I) dynamic range was adopted. (II) Frames were averaged. (III) The time gain compensation curve that was sloping through the tissues was positioned vertically through the vessel’s lumen because the ultrasound beam was not attenuated when it passed through the blood. This was done to ensure that the adventitia of the anterior and posterior walls had similar brightness. (IV) The gain was adjusted. (V) Post-processing was done using a linear transfer curve. (VI) The ultrasound beam was maintained at an angle of 90 degrees to the arterial wall. (VII) The minimum depth was used to ensure that the plaque occupied a large part of the image. (VIII) The probe was adjusted so that the adventitia adjacent to the plaque was visible as a hyperechoic band used for normalization. The ultrasound machine at Atheropoint^TM is shown in Figure 5C.

The MRI cross-sectional scans of the carotid plaque are shown in Figure 6A (82). The CT cross-sectional scans of the carotid plaque are shown in Figure 6B (83). Figure 6C presents the plaque acquisition demonstrate in the US (40). Figure 7 provides a comparison of intraplaque hemorrhage (IPH) in a male patient’s CTA and MRI and histopathology (74).

Figure 6.

Carotid plaque scans from (A) MRI scans of ICAP; the yellow arrow represents intraplaque hemorrhage (A1) hyperintense on T1-wt MRI, (A2) hypointense on TOF, (A3) hyperintense on T1-wt MRI after contrast medium injection in sagittal, and (A4) axial view, (A5, A6); the red area represents the intraplaque hemorrhage in the axial section of T1-wt MRI. (B) [courtesy of (82)], (B) CT scans of the ICAP; the white arrow represents the plaque developed (B1) axial, and (B2) sagittal scan [courtesy of (83)]. (B) US scans of ICAP, (C1, C2) represent the symptomatic plaque images, (C3, C4) corresponding color Doppler images, and (C5, C6) delineated plaques (40) (courtesy of AtheroPoint, Roseville, CA, USA) [Permitted to reproduce Figure (A) and (B)]. MRI, magnetic resonance imaging; ICAP, internal carotid artery plaque.

Figure 7.

Comparison of the IPH visualization in (A) CTA, (B) MRA, and (C) the histopathology section; the white arrow represents IPH [source (84), Permitted to reproduce]. IPH, intraplaque hemorrhage; CTA, computed tomography angiography; MRA, magnetic resonance angiography.

Other modalities, like molecular imaging positron emission tomography (PET), can also study plaque biology (85) and find the sequential stages of inflammation in atherosclerosis. Intra-vascular ultrasounds yield promising results for understanding vulnerable carotid features. Also, IVUS was perceived as one of the best imaging technique for studying carotid (86).

Challenges in plaque classification/characterization and making a case for AI

The biggest challenge in plaque characterization is the fuzziness in the walls’ intimal-media region, immaterial of the imaging modality. This causes validations in contrast; for example, in ultrasound, symptomatic plaques are hyper-echoic (bright), having a high-lipid core, low calcium, and are vulnerable, unlike asymptomatic plaque, which is hypo-echoic, having poor lipid core, stable, and appears dark in ultrasound scans. Thus, characterizing this plaque with the naked eye is unreliable and subject to error and has inter-observer variability (87,88).

With the help of computer vision techniques and artificial intelligence, however, we can identify the plaque (89,90), in spite of variations and fuzziness in pixel distribution to be non-linear (91). Such images require machines with greater learning capabilities. Recent advancements in machine learning techniques offer precisely this; now, plaques can be identified, classified, and characterized using the “handcrafted” feature (19,38,40,52,61,62,92-99). These methods achieve the objective but lack reliability and stability due to variation in the features and performance of different AI models. Deep learning technology, meanwhile, can fill in the gaps using automatic feature extraction (55,100) through a convolutional neural network (CNN) that allows us to characterize plaques (55).

Artificial intelligence architectures

The nature of AI architecture design has gradually evolved over time, starting from manual methods and signal processing methods and eventually leading to learning methods (so-called AI models). Thus, we have divided this section into AI architectures by modality for TCCCA.

Manual, semi-automated, and statistical models using MR, CT, and the US

Several authors have used manual and statistical methods for TCCCA (101-108). In van Ingersleben et al. (109), the radiologist characterized the carotid plaque at the bifurcation by identifying calcium, lipid deposits, and fibrous plaques, based on ex-vivo imaging. Using MRI, Yuan et al. (32) showed the importance of carotid MR imaging and evaluated the plaque morphology and its composition rather than lumen stenosis. Chu et al. (110) showed that carotid lesion characterization requires the quantitative measurement of the carotid artery wall thickness. The authors followed the AHA guideline for characterization of the carotid lesions, namely type I-II, considered as near-normal wall thickness without calcification; type III, considered as diffuse wall thickening or small eccentric plaque without calcification; type IV-V, considered as a plaque with a lipid-rich necrotic core surrounded by fibrous tissue with possible calcification; type VI, considered as a complex plaque with a possible surface defect, hemorrhage, or thrombus; type VII, considered as calcified plaque; and type VIII, fibrotic plaque without a lipid core and with possible small calcifications. In MRI, Zhao et al. (111) showed the role of high-risk plaque (HRP), defined as plaques having luminal surface disruption or a lipid-rich necrotic core occupying >40% of the wall, or intraplaque hemorrhage compared to stenosis in 1,047 recruited subjects. The authors demonstrated that HRP using carotid MRI was 1.5x greater than stenosis of ≥50%. These surface characteristics that originated from the composition had a greater stroke risk, unlike the lumen constriction.

In CT, Wintermark et al. (112) showed a CT-based degree of stenosis (DoS) by comparing a semi-automated method against a neuroradiologist’s visual assessment using Kappa statistics (so-called gold standard), yielding an agreement of 0.918 (P<0.001). Further, Chien et al. (113) conducted a cross-sectional study in which the authors characterized the CTA plaque by correlating the symptoms (or risk factors). These symptoms included hypertension, age, and carotid bruit. Univariate and multivariate analyses were performed, demonstrating the P value close to 0.0. Wintermark et al. (114) conducted a cross-sectional study to explore the CT-based features of carotid plaque (namely lumen area, wall volume, number of calcium clusters, fibrous cap thickness, number of lipid clusters, and the location of the largest lipid clusters) against the two types of strokes (acute carotid stroke patients and non-acute carotid stroke patients). The Causative Classification System identified the gold standard for Ischemic Stroke with the help of a neuroradiologist. After undergoing TCCCA of the CT carotid wall, acute carotid stroke patients demonstrated a significantly greater difference between the CA wall ipsilateral’s appearance to the side of their infarct (OR >1.0 with P=0.0) when compared against the non-acute carotid stroke patients. Li et al. (115) characterized several CT carotid wall-based features, namely, (I) maximum plaque height (MPH), (II) soft plaque, and (III) carotid arterial stenosis (DoS) and associated with 10-year atherosclerotic cardiovascular disease (ASCVD) (into two groups having a cut-off of 7.5%), thereby showing concordance results but not perfect overlap.

In the US, Griffin et al. (116) identified high-risk patients by characterizing the carotid plaque in the US using the juxtraluminal black area (JBA) as a risk predictor in the univariate logistic regression paradigm. The ground truth had three classes based on mild, moderate, and severe carotid stenosis. SPSS 10.0 was used to compute the regression coefficient, followed by the odds ratio (OR). The authors demonstrated by changing the cut of JBA >8 mm² and GSM <15, they were able to detect identify a high-risk group of 188 plaques that contained 142 (77%) of the 185 symptomatic plaques [odds ratio (OR), 6.7; 95% confidence interval (CI), 4.08–10.91, P<0.001], (sensitivity: 77%; specificity 66%; positive predictive value 75%; negative predictive value 68%). In another work, Nicolaides et al. (17) demonstrated that a number of baseline clinical characteristics and ultrasonic plaque features are independent predictors of subsequent ipsilateral cerebral or retinal ischemic (CORI) events. The author uses “Plaque Texture Analysis Software v3.4” to calculate the risk of the individual patient based on the Cox model. In (117), the same authors discussed the image acquisition technique for 3-D ultrasound image reconstruction from a series of 2-D ultrasound scans using “Philip QLAB and Plaque Texture Analysis Software v3.4”. The authors extracted the GSM, JBA, plaque area, and SGLDM-difference entropy (SGLDM-DE) from the 2-D and 3-D images. The authors found no linear relationship between 2-D GSM and its corresponding 3-D GSM (P=0.24), and there is a moderate relation between 2-D JBA and its corresponding juxtraluminal black volume (JBV). The authors found that JBA and JBV are higher in symptomatic than asymptomatic (P=0.004 and P=0.002). The authors observed a good linear relation between 2-D plaque area and 3-D plaque volume. It is observed that plaque area is higher in symptomatic plaque (P=0.18), but in the plaque, the volume is not showing that characteristic (P=1.0). SGLDM-DE is showing a poor relationship between 2-D and 3-D. Loizou et al. (118) extracted the same texture features from plaque components and identified the stroke risk by measuring the statistical difference of the symptomatic and asymptomatic cases. The authors showed 71 texture features to be different between symptomatic and asymptomatic. Jamthikar et al. (119) from the AtheroPoint team studied a narrative review on artificial intelligence techniques in ultrasound imaging modality for tissue classification, characterization, and CVD risk assessment.

ML has started to become more prevalent in the field of AI, as it provides a way to extract knowledge from different covariates of a cohort. This knowledge can be utilized to assess the non-linearity between predictors and distinguish cardiovascular predictors from cerebrovascular predictors. Figure 8 depicts the general architecture of the ML system that consists of offline and online systems. The risk stratification was predicted online, as shown in Figure 8.

Figure 8.

Generalized global ML architecture for tissue classification using MRI, CT, and US (courtesy of AtheroPoint, Roseville, CA, USA). GS, gold standard; PE, performance evaluation; AUC, area under the curve; SE, sensitivity; SP, specificity; DOR, diagnostic odds ratio; MRI, magnetic resonance imaging; CT, computed tomography.

Machine learning for plaque characterization in MRI

Winn et al. (31) characterized 20 carotid fibrous caps in T2-weighted MRI using four observers (radiologists). The ROI was generated by comparing T2-weighted MRI against histological images taken from carotid endarterectomy. The AUC came out to be 0.75. The authors further characterized fibrous cap or rupture, having detection and characterization evaluated as 90% and 98%, respectively. Validation of 3-D ultrasound imaging for plaque vulnerability is crucial for clinical viability. Chiu et al. (120) did exactly that by registering 3-D segmented carotid ultrasound against 3-D MRI using a surface-based image registration algorithm, as shown in previously developed methods (121,122). The authors demonstrated an average error rate of less than 1 mm in the in-vivo study and 0.3 mm in the phantom study. The carotid artery geometry requires imaging more than 3 cm in length from the collar bone to the jaw. Not always, the plaque can be captured in the wall of these arteries. This is because CCA bifurcates into ICA and ECA. As a result, some plaque is identified, some partially identified, and some are entirely missed. Murata et al. (123) designed a motion-sensitized driven equilibrium prepared rapid gradient echo (3-D MERGE) acquisition protocol to characterize carotid plaque using black blood MRI. The authors analyzed 97 subjects consisting of 194 carotid arteries (70 men and 27 women, mean age 60 years). The authors identified 136 plaques; 68 (50%) were within, 46 (33.8%) were partially outside, and 22 (16.2%) were entirely outside of 2-D MRI coverage. Guan et al. (124) developed two kinds of segmentation algorithms, namely structural SVM (SSVM) and Bayes segmentation, to detect the internal wall. The authors compared the two segmentation algorithms’ results and demonstrated SSVM to be more robust than Bayes. The authors achieved a misclassification rate of 16.9% in Bayes and 9.6% in SSVM.

Zhang et al. (125) assessed the importance of using simultaneous non-contrast angiography and intraplaque hemorrhage (SNAP) to detect the lipid-rich/necrotic core (LRNC) from among 1,436 scans obtained by 3T MRI and 3-D SNAP sequence. The authors compared the classification accuracies of manually segmented plaques with plaques segmented by SNAP. The accuracies of NB, SVM, RF, gradient boosting decision tree (GBDT), and ANN were higher than 88% when the manually segmented plaque was used. However, when SNAP was used, the same classifiers achieved higher than 78% accuracy (P<0.0001). Studies (126) have been conducted for the registration of MRI volumes using a different protocol for plaque characterization.

Machine learning for plaque characterization in CT

Zhu et al. (127) investigated CT Angiogram (CTA) for quantitative features to predict 10 years’ worth of atherosclerotic cardiovascular disease (ASCVD) risk using 117 CTA scans. The authors built two semi-automated linear regression models with continuous and dichotomous features. The models yielded ASCVD risk scores of 18.87%±13.26% and 18.39%±11.6% (P<0.0001), and the mean biases between observed ASCVD and predicted ASCVDs were −1.954%±10.88% and −1.466%±12.04%, respectively. The most accurate prediction for the ischemic stroke depends on the optimal spatial coverage of the carotid plaque. Arora et al. (128) studied the optimal coverage area required to distinguish between stroke and non-stroke patients. The authors found that 20 mm coverage on each side of the carotid bifurcation offers optimal results. Further, the authors used a CT-automated classification algorithm, multivariate, and univariate analysis on different components of plaque obtained from 136 patients.

He et al. (129) used optical coherence tomography (OCT) to quantitatively assess the plaque vulnerability of 31 patients based on ex-vivo carotid plaques. The authors accurately classified the fibrous, calcified, and lipid core components of plaque by extracting texture features and pixel-wise features using an RF classifier. Accuracies of 80%, 62%, and 83.1% were achieved with sensitivity and specificity (%) combinations of (80.5, 91.2), (64.7, 90.7), and (87.5, 80.2). Acharya et al. (92) classified CT carotid artery images from 20 patients as symptomatic and asymptomatic by extracting LBP features and wavelet features with SVM. Their classifications were 88% accurate. Further, they characterized the plaque using the Atheromatic index (AtheroPoint, CA, USA), as shown in Figure 9A.

Figure 9.

Local assessment of plaque echo-morphology, in terms of (A,B,C) hypoechogenicity and (D) heterogeneity [Source (33), Permitted for reproduction].

Machine learning for plaque characterization in the US

Most studies carried out within the ML framework have used ultrasounds for TCCCA. Such studies are typically divided into two parts: one based on grayscale information and another based on non-grayscale information (so-called point-based information). Several ML methods have been tried on various organs such as thyroid gland (130-132), liver (47,48,133), ovarian (134-137), carotid plaque characterization (98,138), coronary (139,140), stroke risk (19,52,141), lung tissue characterization (142).

3-D carotid plaque characterization was tried by Seabra et al. (143) mainly to overcome the weakness due to subjectivity with data acquisition and operator-dependent selection. In their characterization model, the authors applied a labeling procedure using graph cuts, which allow them to identify, locate, and quantify vulnerable plaque. The authors tested their method on five patients who had undergone 2-D ultrasound scans and validated their synthetic data method. The same authors in (33) proposed a Bayesian technique for estimating the volume inside plaque by removing speckle noise in the ROI. The authors compared the proposed method with a gold standard, and the quantified volumes P40 (40% hypoechogenic voxels) were 56.76 and 50.62 mm^3, respectively, while GSM <32 were 49.31 and 44.29 mm³, respectively. The author’s assessment of ecomorphology is shown in Figure 9.

Seabra et al. (144) proposed a classification model using an ML classification technique (Adaboost) and tested it on 146 US scans using leave one patient out cross-validation protocol (LOPO CV). The authors extracted the texture features fused with clinical information, achieving an accuracy rate of 99.2% and 100% sensitivity. Afonso et al. (145) developed a computer-aided diagnosis tool for measuring plaque rupture risk using an activity index and an enhanced activity index (EAI). The authors were able to characterize the plaque echogenicity using the EAI.

Christodoulou et al. (97) classified 330 carotid ultrasound scans (CUS) as symptomatic and asymptomatic using a self-organizing map (SOM) and k-nearest neighbor through which they extracted texture features spatial gray level dependence matrix (SGLDM), gray level difference statistics (GLDS). With the help of first-order statistics, the researchers achieved a 70% diagnostic yield (DY). The author (25) extracted ten different Law texture energy features and fractal dimension features from CUS. The author then fed these features into SOM and k-NN for classification, achieving 73.1% and 68.8% accuracy rates, respectively. Loizou et al. (146) proposed a classification model to investigate the texture feature variability obtained from 120 longitudinal ultrasound videos in M-mode during the cardiac cycle. The authors automatically segmented the plaque using a snake algorithm. Further, the authors classified it as either systolic or diastolic using SVM with texture features, achieving 100% accuracy and a ROC of 1.0. Doonan et al. (147) studied the correlation between the texture and echo density features of plaque based on 104 bilateral US scans. The authors extracted the features using commercial “plaque texture analysis software” and PCA. The correlation between these two types of features was between 0.211 and 0.641 (P<0.0001).

Our group had implemented several carotid characterization techniques in the machine learning framework. Acharya et al. (40) proposed a classification technique based on 346 US scans in which texture, LBP, and Law’s Texture Energy (LTE) features were extracted using SVM. The authors achieved an accuracy of 83%, a sensitivity of 84.4%, and a specificity of 79.7%. In this study (93), the authors extracted texture-based features from 99 US scans and fed them into SVM using RBF kernel. This method achieved an accuracy of 91.7%, a 97% sensitivity, and a specificity of 80%. The authors then characterized the plaque using an asymptomatic carotid index. In this study (94), the authors extracted discrete wavelet transform features and texture features. They then fed the features into SVM with a polynomial kernel of order 2, achieving 83.7% accuracy, a sensitivity of 80%, and a specificity of 86.4%. Molinari et al. (95) proposed a data mining framework for classifying symptomatic and asymptomatic plaque using bi-dimensional empirical mode decomposition and entropy features. They achieved an accuracy of 91.43%. The researchers also studied the relationship between coronary plaque and carotid IMT (139,148). Table 1 lists the reviewed literature regarding ML techniques for TCCCA.

Table 1. The literature on TCCCA using ML techniques.

SN#	ML model	Modality	References
1	SVM	US	(19,34-40,61,94-96,146,149-151)
2	KNN	US	(13,40,95,97,98,152)
3	Adaboost	US	(61)
4	UAI	US	(145,153)
5	PNN	US	(34,37,40,95,149)
6	Adaboost	US	(144,154)
7	SOM	US	(97,152,155)
8	ANN	US	(35,156-158)
9	NB	US	(35)
10	LR	US	(36)
11	LDA	US	(95)
12	QDA	US	(95)
13	DT	US	(95)
14	RF	MRI	(159)
15		CT	(129)
16	DT, NB, RF, LR, NN	CT	(160)
17	SVM	CT	(92)

TCCCA, tissue characterization and classification of the carotid artery; ML, machine learning; US, ultrasound; MRI, magnetic resonance imaging; CT, computed tomography.

Deep learning strategies using MRI, CT, and the US

Deep learning (DL) methods have changed the paradigm of AI. They have accelerated the application of AI techniques in medical imaging owing to the flexible and reliable strategies associated with them. A significant drawback of the ML model was solved by using automatic feature extraction via convolution neural networks (CNN). In the DL model, all acquired grayscale images are fed into the convolution layers. This process is followed by max-pooling/average pooling layers for automatic feature extraction. These extracted features are subsequently fed into a fully connected network for classification and risk assessment. The global architecture of a DL system is shown in Figure 10A. DL models learn complex patterns from input training images and then use these patterns to predict target labels. There are several models besides CNN that are popular in medical imaging. Some of these are the U-Net, Autoencoders, recurrent neural networks (RNNs), long short-term memory (LSTM), Region-based Convolutional Neural Networks (R-CNN), and Mask R-CNN. U-Net (a combination of encoders and decoders) plays an important role in the segmentation and classification processes (55,161-163). Figure 10B shows the architecture of U-Net. The encoders compress the image, and the decoders do the opposite. RNN is typically used when time series or historical clinical data are essential (164). Meanwhile, LSTM is useful when the input data is present within the time sequence (165). Finally, RCNN and Mask R-CNN are used for segmentation (166).

Figure 10.

The DL System architectures of (A) The global architecture, (B) U-Net, (C) CNN, and (D) LSTM memory cell (courtesy of AtheroPoint, Roseville, CA, USA). DL, deep learning; CNN, convolutional neural network; LSTM, long short-term memory.

Figure 10C represents a conventional convolution neural network with a convolution layer followed by a max-pooling layer. Figure 10D depicts an LSTM recurrent neural network. It consists of three gates. The forget gate removes information that is no longer useful in the cell state, the input gate provides additional information that is required for the cell state. The output gate extracts all useful information from the current cell state.

Demirer et al. (167) developed a GUI-based toolbox using a deep neural network (DNN) for 2-D/3-D segmentation and classification and integration with third-party scripting languages. The researchers used this tool to annotate 1,843 arteries and 294 coronary CTA atherosclerotic plaques in 23 days. This GUI used NoSQL as its database, while TensoFlows was used for DNN. In other work, Lee et al. (168) developed a DL model and used it to classify 6,556-lumen images of OCT scans using CNN. The authors achieved sensitivity and specificity rates of 84.8% and 91.4% (fibrolipidic) and 97.8% and 95.7% (fibrocalcific).

Lekadir et al. (169) proposed a CNN model that classifies plaque components into lipid core, fibrous cap, and calcified tissue. The authors did this by extracting 90,000 patches from 50 in vivo CUS. The authors achieved a correlation coefficient of 0.90 between automatic measurement and expert measurement (see Figure 11 for a visual representation of CNN classification). Skandha et al. (55) categorized 346 CUS scans as either symptomatic or asymptomatic using optimized deep CNN. They also characterized plaque higher features using mean feature strength (MFS) and bispectrum. The authors achieved an accuracy of 95.66% and an AUC of 0.956 (P<0.0001). Research on the use of DL in TCCCA is still emerging (170); other researchers have used DL to investigate the chest (29,171), coronary (172), liver (28), IMT wall (173,174) of patients, as well as lumen characterization (26) and carotid risk measurement in diabetic patients (30,56) and Rheumatoid arthritis (175) in arthritic patients in particular.

Figure 11.

A visual example of DL-based CNN classification results shown in color representing different components of plaque [source: (169), Permitted for reproduction]. CNN, convolutional neural network; DL, deep learning.

Transfer learning model in MR, CT, and the US

DL is a rapidly growing technology, but it has a minor drawback (i.e., the high-performance hardware requirements for and availability. This issue is resolved by the transfer learning technique, i.e., by using a DL model that has been pre-trained for natural images. The weights are transferred to the current target model by the researcher. Then, the researcher can train the FCN for target label prediction (27), thus avoiding the problem associated with the availability of high-quality computational devices. The global architecture of the TL system is shown in Figure 12.

Figure 12.

The global architecture of the TL models (courtesy of AtheroPoint, Roseville, CA, USA). TL, transfer learning.

Reinforcement learning (RL)

RL is a unique AI technique that enables a model to learn complex patterns using trial-and-error methods based on its own experience (176). Figure 13 presents an architectural view of reinforcement learning.

Figure 13.

Reinforcement learning architecture (courtesy of AtheroPoint, Roseville, CA, USA).

AI similarities and differences between MR, CT, and US

While we have seen that AI models independently operate for TCCCA using MR, CT, and US, but it is necessary to study the differences and similarities. Table 2 shows the difference between MR, CT, and US for TCCCA using AI. We distinguish the three modalities using the three prominent AI models, namely, ML, DL, and TL. These AI models are divided into different blocks, such as segmentation, feature extraction, classification, and performance metrics. The corresponding references are also attached for ready reference. Table 3 shows the similarities between the modalities when it comes to AI models. The differences and similarities are self-described and explanatory. We, therefore, will not elaborate in detail here.

Table 2. Difference between MR, CT, and US for TCCCA using AI.

	MRI	CT	US 2-D/3-D
ML	(123-125,159,177-179)	(92,129,160)	(25,35-40,52,61,62,93-98,139,144,145,149,150), (153,156-158,180-185)
	Segmentation: Bayes Clustering, Structural Support Vector Machines, Manual, SNAP, 3-D Hybrid Segmentation, 3-D MARGE	Segmentation: automatic	Segmentation: Manual, Simple Linear Iterative clustering, RealAdaboost, Bayesian
	Extracted Features: Surface disruption features, SIFT, Morphological, Intensity Features of IR, Intensity features of reference acquisition (REF), TOF-MRA and BBMRI, histological	Extracted Features: Texture features and relative position of pixels, LBP and wavelet transform, Texture features	Extracted Features: Multiresolution Features, Bi-dimensional empirical mode decomposition, and entropy features, texture features, automatic, Rayleigh Mixture Model, Histogram, Texture, Morphological, Monogenic, Wavelet energies, co-occurrence matrix, 1st order statistics, Multilevel binary morphological, second-order statistics spatial gray level dependence matrices, ACRS Clinical, discrete wavelet and higher-order spectra, 2-D DWT, Degree of stenosis, DWT with Averaging. Envelop Radio-Frequency, statistical features, fractal dimension, laws texture energy, Fourier power spectrum, Spatial Based Plaque Feature, 3-D Plaque Feature Extraction, Neighbourhood Gray Tone Difference Matrix, Quadratic Programming Feature Selection, Minimal Redundancy Maximal Relevance, Mutual Information Quotient, Spectral Conditional Mutual Information, Cramer’s V test, neighborhood gray-tone difference matrix, 3-D Fractal Dimension
	Classification: RF, t-test and logistic regression, SSVM, NB, SVM, RF, GBDT, ANN, KNN, SVM, DT,	Classification: RF, SVM RBF, DT, NB, LR, NN	Classification: SVM and Probabilistic Neural Network, SVM RBF, SVM Polynomial, linear, LibSVM, DT, AtheroRisk, Adaboost, self-organizing map, KNN, ANN, 3-D Blanket, SVM with 3-DUS
	Performance metrics: AUC Ranges: 0.95; ACC ranges (%): 87, 88, 76, 87.5, 90 Misclassification Rate (%): 9.6	Performance Metrics: ACC Ranges (%): 83.1, 88, 69	Performance Metrics: AUC Ranges: 0.649, 0.732, 0.905; ACC Ranges (%): 85, 91.43, 82.4, 83.5, 73.7, 77.18, 76, 91.7, 83.7, 90.66, 83.7, 99.2, 73.1, 73.72, 68.8, 69.3, 81.82, 80.38, 81
DL	(186,187)	(29,188)	(55,161,162)
	Segmentation: U-NET, DeepMAD morphological active contours with the iterative framework, 3-D Level set, Automatic	Segmentation: LVO	Segmentation: U-Net, Dilated U-Net, 3-D U-Net
	Extracted Features: Morphological features	Extracted Features: Automatic	Extracted Features: Automatic
	Classification: DeepMAD	Classification: DeepSymNet, Faster R-CNN	Classification: Optimized CNN, Dynamic CNN
	Performance Metrics: ACC Ranges (%): 99.1, 92.6, 89.16	Performance Metrics: AUC Ranges: 0.88, ACC Ranges (%): 83	Performance Metrics: ACC Ranges (%): 95.66, Dice Coefficient Ranges: 96.6, 84
TL	(189)		(55)
	Segmentation: Automatic	<NF>	Segmentation: Manual
	Extracted Features: 23 different morphological	<NF>	Extracted Features: Automatic
	Classification: Linear Discriminate Classification	<NF>	Classification: VGG16
	Performance Metrics: ACC Ranges (%):90	<NF>	Performance Metrics: ACC Ranges (%):83.33

MR, magnetic resonance; CT, computed tomography; US, ultrasound; TCCCA, tissue characterization and classification of the carotid artery; AI, artificial intelligence; IR, inversion recovery; BBMRI, black blood MRI; LBP, local binary patterns.

Table 3. The similarity between MR, CT, and US for TCCCA using AI.

● All the segmentation and classification was attempted on 2-D slices without considering the 3-D spatial information

● All the modalities have implemented segmentation of the wall as its first step

● The centreline algorithm was adapted to extract the orthogonal slices to the blood flow for all three modalities

● All the modalities have attempted tissue characterization

● ML has been attempted on all the three modalities for TCCCA

● SVM, RF, and DT are the common ML classifiers adapted by three modalities

● DL has been attempted on all the three modalities for TCCCA

● U-Net is the most common architecture used in the DL framework for all three modalities.

● CNN is the most popular architecture tried for all modalities

● TL is the least adopted among all the architectures

● Accuracy and AUC are the common performance metrics for all the three modalities

MR, magnetic resonance; CT, computed tomography; US, ultrasound; TCCCA, tissue characterization and classification of the carotid artery; AI, artificial intelligence.

Carotid artery disease and its link to coronary artery disease

Since CVD covers both stroke and myocardial infarction, it is therefore important to look at the AI trends for CVD prediction using carotid ultrasound. Accurate cardiovascular event predictions are necessary since it saves the lives of patients. This section shows how we can use the plaque deposited in the carotid artery to predict cardiovascular events (so-called myocardial infarction). This not only reduces the costs of invasive coronary artery disease screening but also offers a non-invasive imaging option for predicting cardiovascular events early. This subsection is divided into three parts. Part A presents the genetic makeup of atherosclerosis disease in the carotid artery vs. the coronary artery. This is accompanied by the visualization of plaque using non-invasive carotid imaging. Part B presents scientific evidence gathered from several studies that have linked plaque in the carotid artery with coronary stenosis. Finally, part C discusses the use of novel multiclass AI techniques for predicting coronary artery risk based on carotid plaque levels detected via ultrasound.

A: atherosclerotic components and genetic makeup in carotid and coronary arteries

Atherosclerosis is a systemic condition. It has been well-established that plaque seen in the coronary artery has the same set of components as plaque in the carotid artery, especially in the bulb or bifurcation region (190). Several studies have cited the presence of plaque components such as lipid, fibrosis, fibrin, and calcium in both the arteries (I) via IVUS or CT (in the coronary artery) (191) and (II) via MR/CT/BUS (in the carotid artery) (5,106,125,192). An example of the link between the plaque burden in the carotid and coronary arteries can be seen in Figure 14. This similarity of plaque composition has been validated by several pathology studies (193-195).

Figure 14.

Carotid and coronary arteries representation of (A) and (B) visuals, (C) and (D) ultrasound scans of carotid and coronary arteries with calcium components, and (E) a cross-section of the plaque representing blood flow disruption due to calcium deposition along artery walls [courtesy of AtheroPoint™, CA, USA, source: (140), permitted for reproduction].

B: the relationship between carotid and coronary artery disease

Several studies have verified that carotid artery plaque measurements are directly related to coronary artery disease and cardiovascular disease risk (5,196-198). Clinical trials (199) have revealed that carotid IMT is a useful surrogate marker for coronary vascular disease. Further, in (196), the authors used cIMT for carotid and coronary arteries while using an ultrasound framework. The authors in (200) showed maximum plaque height (MPH) as a measure to evaluate the coronary artery disease risk. Further, authors in (201-203) showed the carotid bulb’s use to assess coronary artery disease risk. Table 4 depicts the correlation between carotid-to-coronary and clearly establishes the validity of estimating the risk of CVD, MI, or CVE based on carotid plaque levels.

Table 4. Studies are showing the relationship between carotid plaque and coronary arteries.

SN	Ref	Relationship between carotid vs. coronary arteries	Technique	Performance metrics
1	(204)	Irregularities in IMT and its effect on the CoVD	Root mean square error, multiple logistic regression	Odds ratio: 5.43 (P=0.003)
2	(205)	An increase in the thickness of the IMT affects the CoVD	Discriminant analysis and mean IMT	Sensitivity: 65%
				Specificity: 80%
3	(206)	Effect of carotid plaque vulnerability on coronary artery disease using IPN	The Kaplan-Meier analysis	IPN score >1.25, P=0.004
4	(207)	Atherosclerosis risk factor and calcium score of middle-age men in the femoral and carotid arteries improves the CVD risk prediction in coronary	Femoral odds ratio, carotid odds ration	AUC: 0.665 to 0.719
5	(208)	AI and IVUS based framework for measuring the coronary risk assessment from the cIMT and validated relation between two arteries	SVM with RBF, poly order 1, 2, 3, and linear	ACC (%): 94.95 AUC: 0.95
6	(209)	Correlation between cIMT (without and with bulb) and coronary SYNTAX score	SYNTAX score, polyline distance method	cIMT error: 0.0099±0.00988 mm; AUC: 0.69, 0.67
7	(201)	Automated 3-D ultrasound-based carotid plaque quantification is a useful screening tool for CAD	Stacked-contour method & 16-segment model	Sensitivity (%) 98.0, 93.9
8	(200)	An increase of cIMT thickness and plaque measurements are indicative of the presence of epicardial coronary stenosis	Mean far distal carotid intima-media thickness, maximum plaque height, total plaque area	An optimal threshold value of cIMT thickness of 0.82 mm
				Plaque height of 1.54 mm
				Plaque area: 25.6 mm2
9	(210)	Pre-operative coronary angiography before CEA	Kaplan-Meier analysis	Survival analysis at six years by Kaplan-Meier estimates was 95.6
10	(211)	Effect of calcium score on coronary artery over carotid wall plaque	Cox proportional hazards models, C-statistics	Coronary vascular disease Hazard ratio: 1.78, P<0.001 Carotid vascular disease Hazard ratio 2.09, P<0.001
11	(203)	Left main coronary artery atherosclerosis is related to maximum common cIMT by measuring with carotid ultrasonography	Plaque area	P<0.05
12	(202)	Contralateral carotid artery stenosis and high-intensity carotid plaque on T1 weighted MRI predicts the CoVD	Clinical, multivariate logistic analysis	Odds ratio: 5.7 (P<0.01)

CoVD, coronary vascular disease; IPN, intraplaque neovascularization; CAD, coronary artery disease; CEA, carotid endarterectomy.

C: AI framework for prediction of coronary artery disease based on the carotid artery

Several studies have been carried out to predict carotid and coronary artery disease risk in the ML framework (151,212,213). Recently, an ML-based strategy was developed for determining the risk of coronary artery disease using the carotid artery as a gold standard (139,214,215). ML was also applied to identify coronary artery disease patients using the greyscale features of left ventricle ultrasound scans (216).

In recent research, intravascular neovascularization (IPN) (206) was used to detect plaque in carotid arteries. The observations were then used to predict coronary artery disease based on univariate logistic regression. The authors in (217) extended the role of IPN, along with carotid B-mode ultrasound (CBUS) and office-based biomarkers (OBBM), in the ML framework to predict coronary artery disease. The ML method provided superior coronary artery disease risk assessments, as it was able to differentiate between the non-linearity between risk factors (predictors) and the gold standard (CVE). These strategies can also be applied to carotid plaque data while using coronary angiography score (CAS) as the gold standard. Such an automated system is likely to become a prominent CAD detection system in the future. Further, the team (55) published the DL strategy for stroke risk prediction using the gold standard as carotid risk labels, which can be extended to coronary risk prediction if the gold standard was taken from the coronary artery.

Critical discussion

Image acquisition, normalization, and automated segmentation/manual delineation

In CT, the Hounsfield Unit (HU) value plays an important role in carotid plaque characterization (218). Saba et al. (219) recently studied the correlation between the carotid computed tomography (CT) Hounsfield unit (HU)-based plaque attenuation values measured using dual-energy CT (DECT) scanner and brain leukoaraiosis (LA). The study showed an association between HU attenuation of the carotid artery plaques and the LA volume at low keV energy levels.

In the US, the transducer angle (angle of insonation) plays an important role in ultrasound imaging. The angle of insonation is equal to the angle of the ultrasound beam relative to the tissue of interest (220). Maximizing the amplitude of the target’s echo through the transducer provides optimal imaging at specular reflector (a perpendicular incident beam causes perpendicular reflection) per, and amplitude depends on the orientation, size, and its surface characteristics. Thus, obtaining the specular reflector is essential for imaging in an ultrasound.

Classification accuracy depends on the normalization methods used as well as the intensity levels of the images. Normalization is the first step after image acquisition; it is a statistical process that compares inappropriate scaling. In general, MRI, CT, and US imaging undergo normalization to generate more interpretable final images.

Plaque delineation in the existing works was done manually (52,61,62,93,221,222) using a mouse and video card. Medical practitioners selected the plaque region in the full scan and created a mask over it to cut the plaque of the ROI. This method is error-prone and requires practitioners skilled in delineation. Using advanced AI, like RCNN and Mask RCNN, may help in the automatic segmentation of plaque (223).

3-D imaging and carotid geometry

Three-dimensional medical imaging is required in order to understand an organ’s spatial information. In 3-D MRI, all the slices of 2-D MRI are joined from top to bottom using advanced computational techniques (224,225). Similarly, in CT scans, a series of 2-D scans are mapped to create a 3-D visualization. In the US as well, 2-D US scans were mapped to a 3-D visualization; however, this visualization carries an in-depth analysis of the tissue that actually reduces operator dependence (184). 3-D imaging offers a more accurate diagnosis for advanced atherosclerosis (226). See Figure 15 for 3-D MRI. Saba et al. (228) showed that the index of asymmetry in the carotid artery wall thickness (CAWT) was used as a further parameter to stratify the risk of symptoms related to the carotid artery.

Figure 15.

Visual 3-D MRI cross-sectional representations of severe stenosis in the right internal artery: (A) axial, (B) sagittal, and (D) coronal. (C) represents the 3-D segmentation of lumen, vessel wall, and plaque components, Green: calcification, Yellow: plaque components, and Blue: background matrix [courtesy of AtheroPoint™, CA, USA, source: (227), Permitted for reproduction]. MRI, magnetic resonance imaging.

3-D Optimization

AI techniques must be optimized to achieve more accurate predictions and risk assessments. 3-D optimization is a novel technique that varies with the hyperparameters of the model; variation may exist in the number of features, the layers, the nodes, the data size, or the optimal point at which the AI model will give the best results (55). Figure 16 depicts the 3-D optimization of CNN (courtesy of Atheropoint^TM).

Figure 16.

Left: 3-D optimization of DCNN showing CNN layers vs. augmentation vs. accuracy. Right: 3-D bar chart of the optimized DCNN (11 CNN layers and 5-fold augmentation). [courtesy of AtheroPoint™, CA, USA, source: (55), Permitted for reproduction]. DCNN, deep convolutional neural network; CNN, convolutional neural network.

Multi-class classification

AI studies on plaque classification have focused on binary classification (40,52,55,61,62,93,94) i.e., symptomatic (unstable) vs. asymptomatic (stable). However, in reality, this classification is more of a spectrum, including symptomatic, moderate symptomatic, weakly symptomatic, and asymptomatic. It is thus a multiclass classification problem. Dealing with a problem of this nature demands a great deal of data analysis to determine the misclassification ratio (100). Our team considered the problem of carotid plaque to be one of multiclass classification. Tandel et al. (229) performed a multiclass classification (from three classes to six classes) when exploring the different stages of brain tumors in T2 MRI brain scans using seven AI techniques. Figure 17 represents the ROC curves and performance metrics of the multiclass classification using DL and ML (229).

Figure 17.

Example showing (A) performance of ML vs. DL training models (classifiers) in multiclass framework utilizing T2-weighted MRI brain scans, and (B) corresponding ROC plots of ML vs. DL in the multiclass framework [courtesy of AtheroPoint™, CA, USA, source: (229), Permitted for reproduction]. ML, machine learning; DL, deep learning; MRI, magnetic resonance imaging.

Hardware analysis

Advanced AI applications require sophisticated hardware (GPUs). We worked on two supercomputers designed for deep learning applications—one at Banaras Hindu University (BHU) (Param Shivay), in Varanasi, and the other at Bennett University (BU), in New Delhi. BHU’s supercomputer is a grid computer, while BU’s is a standalone machine developed by Nvidia. Training a DL model at BHU took 9 seconds per epoch, whereas, at BU, that time increased to 18 seconds; the grid computation structure at BHU required less training time. BHU and BU have the same Nvidia GPUs (i.e., Nvidia DGX V100). However, every job at BU is assigned to a single GPU because it is a standalone system. Meanwhile, the system at BHU has 233 nodes; each job is assigned to 40 CPU nodes and 2 GPU nodes, meaning jobs can run concurrently.

Interface, validation, and extensions

Interfacing AI techniques with real-time data through a graphical user interface has the potential to lighten the radiologist’s burden of image data curation, annotations, segmentation, and risk prediction (167). AI tools such as qQuant, ImageJ, and RADSpa can help radiologists access and characterize malignancy in tissues.

The probability scores generated by AI models can be validated using both ground truth values and neovascularization (206). In the latter technique, a score is assigned to the plaque depending on the microbubbles present; based on this, an inter-plaque neovascularization score is calculated and used to predict cardiovascular events.

Similar work can be done on the coronary, aorta, or peripheral arteries and neural organs; numerous AI applications exist in the area of tissue classification and characterization. Further comparison of carotid plaques generated in different geological locations would also yield more sophisticated results.

Conclusion

This review presented the AI-based models for multimodality TCCCA. A search strategy was adopted using a standardized protocol that adapted inclusion and exclusion criteria. The order of AI models in multimodality TCCCA was US>CT>MRI> IVUS>PET. Over 95% of studies adopted 2-D imaging during the AI model design. Typically, the system consists of segmentation, followed by classification. Several studies are leaning towards automated segmentation of carotid plaque, unlike semi-automated. This study classified AI models into five categories: manual, statistical, machine learning, deep learning, and transfer learning. The order of the number of AI models implemented in TCCCA was ML>DL>TL. The ML was dominated in the order of SVM>k-NN>ANN, where the most prominent feature extracted was the texture. In DL, CNN architecture was most adopted for classification and U-Net for segmentation. VGG16 was most popular in TL. Our observation showed a low number of validations were conducted when using AI models. One reason could be the lack of publicly available databases. It is expected that AI models will dominate the TCCCA industry for the next decade.

Supplementary

The article’s supplementary files as

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.