Acoustic Radiation Force Beam Sequence Performance for Detection and Material Characterization of Atherosclerotic Plaques: Preclinical, Ex Vivo Results

Abstract

This work presents preclinical data demonstrating performance of acoustic radiation force (ARF) based elasticity imaging with five different beam sequences for atherosclerotic plaque detection and material characterization. Twelve trained, blinded readers evaluated parametric images taken ex vivo under simulated in vivo conditions of 22 porcine femoral arterial segments. Receiver operating characteristic (ROC) curve analysis was carried out to quantify reader performance using spatially-matched immunohistochemistry for validation. The beam sequences employed had high sensitivity and specificity for detecting Type III+ plaques (Sens: 85%, Spec: 79%), lipid pools (Sens: 80%, Spec: 86%), fibrous caps (Sens: 86%, spec: 82%), calcium (Sens: 96%, Spec: 85%), collagen (Sens: 78%, Spec: 77%), and disrupted internal elastic lamina (Sens: 92%, Spec: 75%). 1:1 single-receive tracking yielded the highest median areas under the ROC curve (AUC), but was not statistically significantly higher than 4:1 parallel-receive tracking. Excitation focal configuration did not result in statistically different AUCs. Overall, these results suggest ARF-based imaging is relevant to detecting and characterizing plaques and support its use for diagnosing and monitoring atherosclerosis.

I. Introduction

Atherosclerosis ranks among the leading causes of cardiovascular related mortality in the United States [1]. The disease progresses slowly and can remain completely asymptomatic until very late stages, making it difficult to identify and treat. Furthermore, atherosclerotic lesions may or may not develop into what are considered “vulnerable plaques”, a term associated with those plaques at greatest risk for rupture and a subsequent ischemic event. Vulnerable plaques are most often not the largest or the most obstructive but an elusive range in the middle, with their relative risk governed by their composition and structure [2–5]. A vulnerable plaque is distinguished by a number of elements including: a thin fibrous cap, large lipid-rich necrotic core, increased plaque inflammation, positive vascular remodeling, increased vasa-vasorum neovascularization, and intra-plaque hemorrhage. Diagnosing atherosclerosis can therefore be broken into two main challenges; 1) identifying the existence of plaques in the vasculature, and, if found, 2) determining plaque composition and structure to stratify the relative risk.

Traditionally, minimally invasive techniques such as X-ray angiography, as well as noninvasive techniques such as ultrasound carotid intima-media thickness (CIMT) and Doppler flow measurements, have been used to detect atherosclerosis [6–10]. These techniques, however, can give false negatives in arteries that have undergone vascular remodeling [11] and are incapable of characterizing the material composition of the plaque. Acoustic radiation force (ARF) based elasticity imaging modalities have been proposed as a possible method for arterial imaging and plaque characterization [12–21]. Briefly, ARF elasticity imaging techniques use ultrasonic energy to impart a stress on the tissue that creates a measurable deformation, which can be used to infer the underlying composition of the tissue. Generally, these techniques fall into two categories; qualitative techniques that measure deformation within the region of excitation (ROE) caused by the transfer of momentum from the longitudinal wave, and quantitative techniques that measure the deformation from the shear wave that propagates transversely from the ROE. A subset of both of these categories is evaluated in this study [22, 23].

A number of preliminary acoustic radiation force impulse (ARFI)-based atherosclerosis imaging studies have been published with matched histology [12, 13, 20], suggesting that plaque characterization is feasible with this technique due to the large contrast in mechanical properties of plaque features. Summarizing the results of the three studies briefly, collagen deposits, calcium, and fibrous caps were correlated with areas of decreased peak displacement (PD), consistent with the expected response of stiff materials, while lipid pools were correlated with areas of increased PD, consistent with the expected response of soft materials. In [13], it was also reported that degradation of the internal elastic lamina (IEL) was correlated with an increase in tissue recovery time (RT). The IEL is a thin layer of elastin (on the order of 1–2 µm) that separates the intima and media and, when disrupted, is associated with increased plaque rupture risk [24].

One aspect of ARFI-based plaque characterization that has not yet been extensively investigated is the difference in performance between various beam sequence configurations. An ARFI-based beam sequence consists of two types of ultrasonic pulses: long-duration ARF excitation pulses, which deform the tissue, and standard B-mode pulses, which track the deformation. Previous studies have shown that the beamforming of these pulses can significantly impact a number of factors, including displacement tracking accuracy, contrast-to-noise ratio (CNR), tissue heating, and frame rate [25–29]. For example, it has been modeled that the focal configuration, or F/# (which is given by the ratio of focal depth, zto aperture width, d), of the excitation pulse can mitigate displacement underestimation due to shearing artifact [25, 26]. Shearing artifact is a phenomenon that occurs when the distribution of scatterer motion within the point spread function (PSF) of the tracking beam is non-uniform and leads to averaging, and consequently underestimation, of ultrasonic displacement tracking. A large F/# excitation pulse (relative to the tracking pulse) deposits energy over a wide lateral region of tissue within the tracking resolution cell to create a relatively uniform displacement field and less shearing. A large F/# excitation also changes the ARF distribution in the axial dimension; the axial range over which displacement is constant is increased while the force magnitude in a given position decreases [29].

In addition to F/#, receive beamforming methods such as parallel receive tracking, where multiple A-lines are beamformed from a single transmit event, can be used to reduce the number of ARF excitations necessary to interrogate a given lateral range, thereby decreasing tissue heating and increasing frame rate [28]. Depending on the parallel beam spacing and excitation pulse beamwidth, the displacements observed in the A-lines furthest from the region of excitation (ROE) could be due entirely to shear wave propagation, which may not be uniform in a heterogeneous medium such as an atheroma.

The purpose of this study was to quantitatively assess the performance of ARF-based techniques for atherosclerosis imaging with a reader validation study. Atherosclerotic arteries were harvested from a dietary hypercholesterolemic porcine model and imaged ex vivo under simulated in vivo conditions. Twelve blinded readers were recruited and trained to read images created with ARF-based techniques. Plaque detection and characterization performance for each reader was assessed using receiver operating characteristic (ROC) curve analysis and a pathologist’s reading of spatially matched immunohistochemistry. ARF-based imaging performance results assessed under simulated in vivo conditions are compared to published in vivo MRI and CT results to demonstrate the potential relevance of ARF-based ultrasound methods to detecting and characterizing atherosclerotic plaques.

II. Methods

A. Animal Model

Diet-induced hypercholesterolemic (DH) pigs were used as the model for atherosclerosis. DH pigs are genetically normocholesterolemic (NC) swine that are fed a high-fat diet (20% beef tallow, 1% cholesterol or ~10 g cholesterol daily, 0.75% cholate) to induce hypercholesterolemia and cause the animals to spontaneously develop atherosclerotic lesions that closely mimic histopathology seen in humans. Detailed descriptions of the lipid and hemostatic profiles of DH swine, as well as a summary of the atherogenic diet, are available in [30]. For this study, nineteen phenotypically normal pigs were selected from the closed colony at the Francis Owen Blood Research Laboratory at the University of North Carolina, Chapel Hill, and thirteen of those pigs were put on an atherogenic diet for 12 months prior to imaging. The remaining six were continued to be fed a regular low-fat diet (3.58% crude fat including 60 to 65 mg cholesterol total daily), the same diet that the DH pigs were on prior to this study. All animal procedures were approved by the University of North Carolina Institutional Animal Care and Use Committee (IACUC).

B. Ultrasonic Imaging, Data Collection, and Data Processing

Left and right iliac arteries were harvested from all pigs (13 DH and 6 NC), yielding 38 arteries, each ~20 to 25 cm in length. The mean (± standard deviation) age of DH animals at necropsy was 3.8 (± 0.8) years, while the mean age of NC animals was 4.6 (± 2.2) years. These arteries were stripped of excessive adventitial connective tissue, mounted, and pressurized statically to 80 mmHg in a saline water bath kept at room temperature (22 °C), as illustrated in Figure 1. To assist in mounting, a scaffold was erected across the top of the water tank with two clamps that extended into the water. The arteries were fitted with barbed plastic tubing connectors inserted into either end of the vessel, and each end was placed in a clamp to hold the artery in place. Using the clamps, arteries were stretched to approximately 1.5 times their recoil length to estimate in vivo lengths [31]. One end of the vessel was attached, via the barbed connector, to a hose coming from a Masterflex LS 7550-30 computerized pump drive (Cole-Parmer Instrument Co. Chicago, IL, USA), while the other end was connected to a hose that lead to a GE Druck DPI 104 digital pressure gauge (Druck Ltd. Leicester, UK). To convert the output of the flow pump from pulsatile to laminar flow, a Windkessel-type pulse damper (EW-07596-20, Cole-Parmer Instrument Co. Chicago, IL, USA) was placed in series with the water pump and the artery. In some cases, branch points in the arteries that were cut during tissue harvesting were sutured closed to maintain pressure in the artery. Branch points that had sutures in the imaging field of view (FOV) were not included as candidate imaging locations.

Figure 1.

Pressurization apparatus for ex vivo imaging. Excised arteries were mounted in a saline water bath between ~20 kPa tissue mimicking gelatin phantoms and statically pressurized to 80 mmHg to simulate in vivo imaging conditions during diastole. The saline bath was kept at room temperature (22 °C).

To simulate in vivo imaging conditions, two homogeneous soft tissue mimicking phantoms with a Young's modulus of ~20 kPa were created and placed above (~15–18 mm thickness) and below (~30–40 mm thickness) the vessel during imaging. The phantom recipe consisted of the following ingredients (with percent composition by mass in parentheses): type-A gelatin (13%; Acros Organics, Geel, Belgium), Photo-Flo (2%; Eastman Kodak Co., Rochester, NY, USA), n-propanol (1%), powdered graphite (1%), and de-ionized water (83%). The acoustic attenuation of the phantom was determined to be 0.2 (± 0.02) dB/cm/MHz, based on 5 repeated measurements. A modulus of ~20 kPa was selected to mimic the modulus of soft tissue (e.g. muscle) surrounding peripheral vessels in vivo [32]. Phantom modulus values were determined by measuring shear wave velocities in the phantom materials (in separate, homogeneous, 64 cm³ blocks) using a time-to-peak algorithm [33]. The arterial mimicking phantoms were poured from the same phantom mixtures as those measured. The SWV and Young’s modulus measurement methods were validated in a model 049A Computerized Imaging Reference Systems phantom (CIRS, Norfolk, VA, USA) with 25 kPa background material; the measured SWV was 2.81 ± 0.08 m/s, which matches the expected SWV of 2.89 m/s for a 25 kPa material. The arteries were not completely embedded in the tissue mimicking material to preserve the tissue for immunohistochemical processing after imaging. The phantom above the artery was held in place by the transducer, while the phantom below the artery was held in place by weights. Pressure was applied, via the transducer face, to the upper pad such that it would not float away or move during acquisition.

Imaging was performed with a Siemens SONOLINE Antares™ imaging system (Siemens Medical Solutions USA, Inc., Ultrasound Division), equipped for modifiable beam sequencing for research purposes and a VF7-3 transducer. The transducer was secured in a holder connected to a digital three-axis motion controller mounted on an optical air table (Newport Corp., Irvine, CA, USA). The transducer was positioned to image the arteries longitudinally and was translated down the length of each artery to generate up to four non-overlapping FOVs.

ARF ensembles consisted of two reference lines, one ARF excitation impulse, and 60 tracking lines with an 11.5 kHz pulse repetition frequency (PRF). ARF excitations were 300 cycles at 4.21 MHz, and both tracking and reference lines were two cycles at 6.15 MHz. Two types of ARF excitations were combined with three types of tracking to generate five total beam sequences. The two types of excitations included: 1) single excitation with an F/1.5 focal configuration (SP1.5), and 2) single excitation with an F/3 focal configuration (SP3). The three types of tracking included: 1) single-receive (SRx) in the ROE, 2) 4:1 parallel-receive (ParRx) in the ROE [28], and 3) 4:1 parallel-receive lateral to the ROE (LatRx), the last being a method comparable to shear wave elasticity imaging (SWEI) [23]. On transmit, SRx tracking pulses had either an F/1.5 or F/3 focal configuration, matching the corresponding excitation F/#. On receive, all sequences used dynamic focusing and aperture growth (F/0.75). Figure 2 depicts the different beam sequences and associated labeling scheme used in this study. For SRx tracking, 40 ROEs and tracking locations laterally spaced 0.53 mm apart were used for an overall lateral FOV of 2.1 cm. ParRx tracking used 15 ROEs laterally spaced 1.4 mm apart and 60 tracking locations spaced 0.35 mm apart for a 2.1 cm lateral FOV. LatRx tracking was performed using a single ROE centered laterally, with 60 tracking locations laterally spaced 0.35 mm apart spanning a 2.1 cm lateral FOV. ARF ensembles were acquired using wiperblading, a scanning mode that acquired lines in a non-serial order across the lateral FOV to minimize heating and reduce interference between consecutive ARF excitations. A single ensemble was captured from the far left of the FOV, then the center, then one position to the right of the far left, then one position right of center, etc., such that no two ensembles were captured in two consecutive lateral locations.

Figure 2.

Diagrams of the five different ARFI beam sequences that were investigated in this study. Downward arrows represent excitation beams, while upward arrows represent tracking beams. The width of the downward arrows signifies the focal configuration (F/#), with the narrow arrows representing F/1.5 and the wide arrows representing F/3. Figures are not drawn to scale.

One-dimensional normalized cross correlation using an interpolation factor of 4, a search window length of 80 µm, and a kernel length of 376 µm (i.e. 1.5λ, where λ is the wavelength of the tracking pulse assuming a speed of sound of 1540 m/s) was applied to the raw, radio frequency (RF) data ensembles to measure axial motion induced by ARF [34]. ARF displacement profiles were then processed to reject luminal and reverberation signals using the algorithm previously described in [35]. The algorithm was also implemented to automatically segment arterial wall tissue. Finally, parametric images were adaptively rendered by automatically adjusting color scale limits according to the median parameter value measured in the segmented arterial wall (which ranged from 0 to median + [3 × parameter standard deviation]). Rendered parametric images included peak displacement (PD) and recovery time (RT) for non-LatRx sequences and shear wave velocity (SWV) for all LatRx sequences. SWV was calculated using the lateral time-to-peak (TTP) methods described in [33] using least squares linear regression for regions measuring 0.5 mm axially and 3 mm laterally. Due to this windowing approach and the positioning of the ROE, the effective FOV in the LatRx images was reduced by ~6 mm (3 mm lost on the edges from windowing, and 3 mm lost due to the ROE). Matched B-modes were acquired and used for histological registration but were not presented to the readers during the course of the reader evaluation study.

A total of 22 arterial locations were imaged, each using the five different beam sequences with four matched acquisitions per sequence to ensure successful data acquisition. Twelve imaging locations focused on distal and ten on proximal arterial walls (where “proximal” and “distal” are relative to the transducer). The number of imaging locations was determined by the number of areas where plaques developed in the arteries. Due to the spontaneous nature of plaque generation in these animals, it was not guaranteed that disease would be present in every DH pig; only 6 of the 26 DH arteries yielded plaques that could be imaged, and from these 6 arteries, 10 non-overlapping imaging FOVs, each containing plaques, were acquired. Similarly, 12 non-overlapping “control” data sets were taken from 5 NC arteries. Therefore, data from 11 harvested arteries were included in this study.

Following ex vivo imaging, the upper tissue-mimicking phantom was removed, and a biologically inert carbon particle solution (undiluted Carbon Black Dispersate No. 8; Eberhard Faber, Bedminster, NJ, USA) was deposited from a 28-gauge syringe onto the exterior of the proximal arterial wall following the entire length of the transducer. While the carbon was being deposited, the needle was visualized with B-mode to ensure that the particles were co-located with the imaging plane. Approximately 10 deposits were made along the imaging FOV. The nature of the carbon particles was such that they diffused into the outermost adventitial layers of the vessel, making a semi-permanent dark stain that remained even after formalin fixation. This stain served as the marker for the imaging plane, and all sectioning was performed using it as a reference. The carbon particles were both macroscopically visible during sectioning and microscopically visible after histological processing.

C. Histology

After imaging, arteries were removed from the pressurization apparatus and fixed in 10% buffered formalin. Using the macroscopically visible carbon markings, portions of the arteries corresponding to the imaging plane were cut out with a scalpel and embedded in paraffin for spatially-matched immunohistochemical analysis. The arteries were sectioned and stained with hematoxylin and eosin (H&E), Von Kossa (VK) for calcium, Lillie’s modified Masson's Trichrome (LMT) for collagen and fibrin, and Verhoeff-Van Gieson (VVG) for elastin. Slides containing the stained sections were digitized using an Aperio Scanscope (Aperio Technologies Inc., Vista, CA, USA) at 20x magnification.

The microscopy images were then divided into four laterally adjacent, equally sized subsections spatially corresponding to four laterally adjacent, equally sized quarters in the ARFI images. A pathologist with experience in atherosclerosis then graded each subsection using the criteria established by the American Heart Association (AHA) Committee on Vascular Lesions [2, 36, 37] (Table 1). First, the pathologist graded plaque absence or presence on a 7-point scale, indicating the grade of the plaque when applicable (Table 2). If a plaque was present in a given subsection, the pathologist also graded the compositional and structural features of the plaque (including: collagen deposition, calcium deposition, degradation of the IEL, the presence of lipid pools, and fibrous caps) on 3- or 5-level scales (Table 2). The pathologist’s analysis was facilitated using a custom graphical user interface (GUI) developed in Matlab (Mathworks Inc., Natick, MA, USA). Table 3 lists the pathologist’s analysis for the entire data set, which shows the sample sizes in the gold standard used to assess performance.

Table 1.

Classification system for atherosclerotic plaques established by the AHA Committee on Vascular Lesions [2, 36, 37].

Type	Plaque Composition
I	Isolated macrophage foam cells
II	Multiple foam cell layers formed, fatty streak
III	Pre-atheroma with isolated extracellular lipid pools
IV	Atheroma with confluent extracellular lipid core
Va	Fibroatheroma surrounded by fibromuscular tissue layers with lipid or necrotic core (sometimes classified as simply type V)
Vb	Calcification predominates (sometimes classified as Type VII)
Vc	Fibrous tissue changes predominate, absent/minimal lipid core (sometimes classified as Type VIII)
VI	Fissured, ulcerated, hemorrhagic, thrombotic lesion

Table 2.

Rating system used by the pathologist. In general, the ratings were done in a qualitative fashion, i.e. the ratings of “High” and “Low”, or “Large” and “Small”, were subjectively determined by the pathologist rather than being determined by a quantitative measurement. Ratings for collagen were made with regards to the “average” atherosclerotic plaque, e.g. a collagen rating of 2 indicates a mild decrease in the collagen levels relative to an average plaque as judged by the experience of the pathologist. Black vertical bars indicate where each rating system was dichotomized to perform ROC analysis (see ‘Statistical Analysis’). Ordinal values that are used for statistical analysis are given at the bottom of the table.

Rating Category	Pathologist’s Rating Levels
Plaque Detection	Plaque Not Present	Type I	Type II	Type III	Type IV	Type V	Type VI
Collagen	Severe Decrease (>50%)	Mild Decrease (<50%)	No Change	Mild Increase (<50%)	Severe Increase (>50%)	-	-
IEL	Intact	Somewhat Disrupted	Heavily Disrupted	-	-	-	-
Calcium	None	Low	High	-	-	-	-
Lipid Pool	None	Small	Large	-	-	-	-
Fibrous Cap
Ordinal Value	1	2	3	4	5	6	7

Table 3.

Number of plaque types and features as determined by the pathologist collapsed according to the dichotomization thresholds (see ‘Statistical Analysis’). Each entry represents the number of image subsections under each rating, with each row totaling 88 (22 arterial FOVs × 4 subsections/FOV), the total number of subsections rated by the pathologist.

Rating Category	True Negatives	True Positives
Plaque Detection	None, Type I - II	Type III – V(a/b/c)*
	71	17
Collagen	No Plaque Present or Decreased Compared to Average Plaque	Comparable or Increased Compared to Average Plaque
	65	23
IEL	Intact	Degraded
	64	24
Calcium	Not Present	Present
	82	6
Lipid Pool	Not Present	Present
	73	15
Fibrous Cap	Not Present	Present
	73	15

^*The pathologist could rate up to Type VI plaques, but no arteries in this study developed plaques of this type

D. Reader Training

All 110 parametric image sets (5 beam sequences × 22 non-overlapping arterial FOVs) were evaluated by 12 trained, blinded readers with varying backgrounds in biomedical imaging and levels of prior experience analyzing ARFI/SWEI images (ranging from less than one year (n = 7) to greater than ten (n = 1) years). Prior to beginning the study each reader underwent a training regimen. The readers were given a handout that included a number of items: instructions on the reader graphical user interface (GUI), a brief tutorial on the manifestation of atherosclerosis, examples of images from previous studies with matched histology, and a table of expected responses of ARFI/SWEI parameters for each class of a plaque feature (Table 4). As part of the atherosclerosis tutorial, the readers were told that 1) plaques are generally fibrotic, 2) calcium depositions tend to be focal, and 3) fibrous caps may develop on top of large lipid pools or necrotic regions. Furthermore, the readers were instructed to 1) make no assumptions of the number of plaques in an image, 2) assess each subsection independently of other subsections in an image set, and 3) make no assumptions about mutual exclusion of plaque features (e.g. if calcium is observed, this does not exclude the possibility of a lipid pool or any other feature). Table 4 was modeled after a cardiovascular MRI study [38] and modified to reflect ARF-based imaging characteristics. Response predictions for each category were assigned based on published vascular data with matched histology when possible [12, 13, 20]. In the cases where no published literature existed, grades were predicted from unpublished preliminary data collected by the authors. Readers were allowed to reference all training materials during the image evaluation without restriction.

Table 4.

Expected ARF-induced responses of different plaque features in parametric images, presented to the reader as part of the training regimen. A ‘−’ indicates that the metric should be low relative to adjacent arterial tissue (i.e. peak displacement of calcium should be relatively low). A ‘+’ indicates that the metric should be high relative to adjacent arterial tissue (i.e. recovery time of a disrupted IEL should be relatively high). A ‘0’ indicates that the metric should remain unchanged. A ‘/’ is used to indicate a situation where two results may occur (i.e. recovery time with collagen deposition may be decreased, or may be unchanged). Multiple symbols, e.g. ‘+++’, indicate that the metric is expected to change substantially.

Tissue Classification	PD	RT	SWV
Collagen Deposition	−	−/0	+
Disrupted IEL	0	+	−/0
Calcium Deposition	−	0	+
Lipid Pool/Necrosis	+++	0	−
Fibrous Cap	−	−/0	+

E. Reader Evaluation Study

A custom GUI was developed to show the image sets in a random order to the reader and to save responses to the reader’s hard drive. Upon starting, the GUI loaded the first of 110 image sets displaying the full 2.1 cm FOV captured in each parametric image. The reader was then prompted to assign an overall image quality rating of 1 (“very bad”) to 5 (“very good”) to the set. This perceptual image quality value was later used as an exclusion criterion for reader responses (described further in the section titled “ROC Analysis”). Next, a black box outline was overlaid on the parametric images representing the first of four equally spaced subregions (−1.0 cm to −0.5 cm in the lateral dimension). The reader was then asked to assign a rating for plaque presence for the boxed subregion using a five-point scale: 1 (plaque definitely absent), 2 (plaque probably absent), 3 (unsure if plaque absent or present), 4 (plaque probably present), 5 (plaque definitely present). If the reader believed that a plaque was present in a given subregion, she/he was also asked to judge the compositional and structural elements of the plaque (including: collagen deposition, calcium deposition, degradation of the IEL, the presence of lipid pools, and fibrous caps). After the reader finished rating the first subsection, the GUI moved the black box to the second of the four subregions (−0.5 cm to 0 cm in the lateral dimension), and the reader was again asked for a rating. This process continued until all 4 subregions of the image set were rated. Table 5 lists the rating system used by the readers and the ordinal numbers associated with each response. As mentioned previously, multiple matched acquisitions were taken. Two of the four acquisitions were selected for each image set and randomized across reader such that six readers read acquisition 1 and six readers read acquisition 2.

Table 5.

Rating system used by the readers. Ordinal values that are used for statistical analysis are given at the bottom of the table.

Rating Category	Reader’s Rating Levels
Plaque Detection	Plaque Definitely Not Present	Plaque Probably Not Present	Unclear	Plaque Probably Present	Plaque Definitely Present	-
Collagen	Definitely Not Present	Probably Not Present	Unclear	Probably Present	Definitely Present	-
Calcium
Lipid Pool
IEL	Definitely Intact	Probably Intact	Unclear	Probably Disrupted	Definitely Present	-
Fibrous Cap	Not Present	Definitely Intact	Probably Intact	Unclear	Probably Disrupted	Definitely Disrupted
Ordinal Value	1	2	3	4	5	6

F. ROC Analysis

The performance of different beam sequences to detect and materially characterize atherosclerotic plaques was evaluated using receiver operating characteristic (ROC) curve analysis, with area under the curve (AUC) used as a metric of performance. A detailed description of ROC calculation from ordinal reader responses is given in Appendix A along with an example from the data set. Briefly, the ROC analysis was performed by first dichotomizing the pathologist’s response into “case” (feature is present) and “control” (feature is absent) groups for each plaque feature. Next, frequencies of ordinal reader responses were organized along pathologist ratings of “case” or “control”. The “case” and “control” rows in this frequency table were then assumed to be discrete random variables latently distributed as normal distributions, and the parameters of these distributions were estimated with maximum likelihood estimation (MLE) and AUC values were calculated. Following the multi-reader methods of [39], summary ROC curves were created for sequences with the highest AUC values, and “optimal” operating points were chosen from the median summary curves to report sensitivity and specificity. The operating point was selected by maximizing the sum of sensitivity and specificity. Image sets that were rated by readers as having an image quality of “Very Bad” or “Bad” were excluded from the ROC analysis. ROC computation was performed using the R programming language and development environment (R Foundation for Statistical Computing, Vienna, Austria).

To test for performance differences between beam sequences, pairwise Wilcoxon rank sum tests were performed on the resulting distributions of AUCs. To assess inter-reader variability for a given beam sequence and feature, pairwise Spearman’s rank correlation coefficients (SCC) were computed for every reader combination. And finally, to highlight the impact of reader experience, the AUC of Reader 1 (the most experienced reader in the study) was compared to the median AUC for a given beam sequence by subtracting the median from Reader 1’s AUC.

III. Results

The results for plaque detection are presented first, followed by the results for the five plaque components investigated for plaque characterization (calcium, lipid pools, fibrous caps, degraded IEL, and collagen). The responses of one reader were excluded from the results because she/he did not fully complete the evaluation. Finally, due to the large number of beam sequences and plaque features, ROC curves are not shown for brevity.

A. Plaque Detection

Figure 3(a) shows boxplots of AUC and SCC for plaque detection across all five beam sequences. SP3-SRx yielded the highest median AUC (0.899) for plaque detection, followed by SP1.5-ParRx (0.854). Summary ROC curve analysis for this sequence showed an optimal operating point corresponding to a sensitivity of 0.847 and a specificity of 0.791. The two LatRx sequences (SP3-LatRx and SP1.5-LatRx) had the two lowest median AUCs (0.771 and 0.758, respectively), but had large IQRs and were not statistically different from SP3-SRx, the highest median AUC sequence. Mean shear wave group velocity was 6.6 ± 1.1 m/s (range: 4.4 – 8.1 m/s) in control arteries, and 6.7 ± 2.6 m/s (range: 4.0 – 13.2 m/s) in disease arteries. SP3-SRx also had the highest SCC between readers with a median value of 0.583.

Figure 3.

Boxplots of AUC and SCC vs. beam sequence for plaque detection (a) and individual plaque features: calcium (b), lipid pool (c), fibrous cap (d), degraded IEL (e), and collagen deposition (f). Whiskers represent the highest and lowest data points still within 1.5 IQR, and outliers are depicted with circles. AUC value of the most experienced reader is indicated with an ‘x’. Letters ‘a’ through ‘g’ are used to indicate statistical differences (α = 0.05), where each letter identifies the sequence for comparison (a = SP3-SRx, b = SP1.5-ParRx, c = SP3-ParRx, d = SP1.5-LatRx, e = SP3-LatRx). Sequences marked with multiple letters were statistically different from multiple sequences.

It is important to note that the dichotomization threshold used for the plaque detection ROC analysis was Type III plaques (i.e. plaques rated as Type I or Type II were counted as true negatives). Therefore the AUCs reported in Figure 3(a) are only valid for plaques of Type III (pre-atheroma) and above. Performance was compromised when the threshold was set to include Type I (isolated macrophage foam cells) and Type II (fatty streaks) plaques, with the highest median AUC dropping to 0.612. This suggests that beam sequences used in this study do not perform well at detecting the earliest stages of plaque formation.

B. Plaque Characterization

The median beam sequence AUCs for calcium detection ranged from 0.914 to 0.779 (Figure 3(b)). SP1.5-LatRx had the highest median AUC of 0.914, but was also seen to have a high IQR, and was not statistically different from any of the other sequences. SP3-SRx had the second highest median AUC of 0.892, but had a much smaller IQR and was statistically different from SP3-ParRx (p < 0.009). The summary ROC curve for SP3-SRx showed generally high performance for calcium detection, with an optimal operating point for the “average” reader at 0.964 sensitivity and 0.852 specificity. The median (25^th and 75^th percentiles) SCC across all sequences was 0.56 (0.36 – 0.66). The AUCs for Reader 1 were, on average, substantially lower than the medians for each beam sequence (by 19.6 percentage points). Reader 1’s AUC with SP3-SRx (52.0 percentage points lower than the median), was a considerable outlier despite the sequence having the highest median AUC for calcium detection across all the readers. Examples of plaques with calcium are shown in Figure 4(a) and (b).

Figure 4.

Three examples of ARFI images with matched histology. Stains include: hematoxylin and eosin (H&E) (shown in (a.1), (b.1), and (c.1)); Lillie’s modified Masson's trichrome (LMT) staining collagen blue (shown in (a.2), (b.2), and (c.2)); Von Kossa (VK) staining calcium black (shown in (a.3) and (b.3)); and Verhoeff-Van Gieson (VVG) staining elastin black (shown in (c.3)). Black arrows are used to indicate plaque features (C = calcium, L = lipid, F = fibrous cap or fibrosis). In (a), an advanced Type Vb atheroma, spanning a lateral range of approximately −7.5 to −2.5 mm, is seen with a large calcium deposit located to the left part of a lipid pool. ARFI peak displacement image (a.4), taken with SP3-ParRx, shows an area of low displacement (blue) indicating a stiff material spatially matching the area of calcium, and an area of high peak displacement (red) indicating a soft material spatially matching the area of lipid pool. An area of slightly lower displacement (compared to the lipid pool) spans the top of the plaque in the position of the fibrous cap. In (b), an advanced type Vb atheroma, spanning a lateral range of approximately −5 to 0 mm, is seen with a large calcium deposit underneath a small lipid pool and fibrous cap. ARFI peak displacement image (b.4), taken with SP3-SRx, shows an area of low displacement (green/blue pixels) indicating a stiff material spatially matching the extent of the calcium deposition. An area of slightly elevated displacement is seen just above the area of low displacement, which spatially correlates with the small lipid pool and fibrous cap above the calcium. In (c), a type IV plaque spans the lateral range of −5 to 1 mm, with significant collagen deposition. The ARFI peak displacement image, taken with SP1.5-ParRx, shows a substantial decrease in peak displacement (blue) spatially aligned with the area of fibrosis. The ARFI recovery time image shows a gradient of increased recovery time across the plaque spatially located in areas where the IEL has started to degrade and duplicate. Zoomed in VVG images ((c.3.1), (c.3.2), and (c.3.3)) show the degraded IEL at three points along the artery, in areas where the recovery time is fast (~0.2 msec), slow (~0.8 msec), and moderate (~0.6 msec), respectively. In (c.3.1) the IEL is duplicated but generally intact, and RT is fast; in (c.3.2) the IEL is substantially disrupted and chaotic, and RT is slow; and in (c.3.3) the IEL is somewhat degraded, and RT is slightly faster than in (c.3.2).

The median beam sequence AUCs for lipid pool detection ranged from 0.888 to 0.726 (Figure 3(c)). SP3-SRx had the highest median AUC of 0.888 and was statistically different from SP1.5-LatRx (p < 0.04). The summary ROC curve for SP3-SRx lipid pool/necrotic core detection showed an optimal operating point for the “average” reader at 0.797 sensitivity and 0.857 specificity. The median (25^th and 75^th percentiles) SCC across all sequences was 0.56 (0.39 – 0.67). The AUCs for Reader 1 were, on average, slightly lower than the medians for each beam sequence (8.8 percentage points). Examples of plaques with lipid pools are shown in Figure 4(a) and (b).

The median beam sequence AUCs for fibrous cap detection ranged from 0.916 to 0.721 (Figure 3(d)). SP3-SRx had the highest median AUC of 0.916, followed by SP1.5-ParRx with a median AUC of 0.825. The SP3-SRx sequence was statistically different from every other sequence (p < 0.03) except SP3-LatRx. The summary ROC curve for SP3-SRx showed an optimal operating point for the “average” reader at 0.855 sensitivity and 0.819 specificity. The median (25^th and 75^th percentiles) SCC across all sequences was 0.50 (0.32 – 0.64). The AUCs for Reader 1, as with lipid pools, were, on average, slightly lower than the medians for each beam sequence (by 5.9 percentage points).

The median beam sequence AUCs for degraded IEL detection ranged from 0.889 to 0.692, with SP3-SRx having the highest median AUC (Figure 3(e)) and being statistically different from SP1.5-LatRx. The corresponding summary ROC curve for the “average” reader using a SP3-SRx sequence revealed a sensitivity of 0.916 and specificity of 0.754 for detecting degraded IEL. The median (25^th and 75^th percentiles) SCC across all sequences was 0.49 (0.34 – 0.63). The AUCs for Reader 1 varied considerably around the corresponding median AUC value for each beam sequence. For SP1.5-LatRx, Reader 1’s AUC was very high, 0.986 (29.3 percentage points higher than the median), but low for SP3-SRx, 0.603 (20.2 percentage points lower than the median). An example of a plaque with a degraded IEL is shown in Figure 4(c).

The median beam sequence AUCs for collagen deposition detection ranged from 0.832 to 0.726 (Figure 3(f)). SP1.5-ParRx had the highest median AUC but was followed closely by SP3-SRx, and SP3-ParRx, all of which were statistically different from the lowest median AUC sequence (SP1.5-LatRx). The summary ROC curve for SP1.5-ParRx showed an optimal operating point for the “average” reader at 0.784 sensitivity and 0.765 specificity. The median (25^th and 75^th percentiles) SCC across all sequences was 0.54 (0.38 – 0.65). The AUCs for Reader 1 were, on average, slightly lower than the medians for each beam sequence (6.1 percentage points). Reader 1’s AUCs for SP1.5-LatRx and SP3-LatRx were the furthest from the median, at 10.1 and 16.9 percentage points lower, respectively.

To give the results of this study context to the field of atherosclerosis imaging, ARFI performance numbers (ex vivo in porcine femorals) are loosely compared in Table 6 to published numbers for two other non-invasive atherosclerosis imaging modalities where the data was taken in vivo in human carotids: MRI, and X-ray CT [40, 41].

Table 6.

Comparison of ARF beam sequence sensitivity and specificity for plaque detection and characterization (measured ex vivo in porcine femoral arteries under simulated in vivo conditions) versus published sensitivity and specificity for MRI and CT (measured in vivo in human carotids) [40, 41].

		ARF-Based Ultrasound		CT		MRI
		(Ex vivo, procine femoral)		(In vivo, human carotid)
Detection		Sensitivity	Specificity	Sensitivity	Specificity	Sensitivity	Specificity
Type III-Va,b,c		85%	79%	-	-	80–84%	90–98%
Type unreported		-	-	88–95%	90–96%	-	-
Characterization
Calcium		96%*	85%*	100%	100%	76–80%	86–94%
Lipid/soft		80%*	86%*	76%	74%	91–98%	65–100%
Fibrous cap		86%*	82%*	-	-	81%	90%
Degraded IEL		92%	75%*	-	-	-	-
Collagen		78%†	77%†	86%	94%	-	-

Values marked with ‘*’are reported from SP3-SRx sequence results

Values marked with ‘†’ are reported from SP1.5-ParRx sequence results.

C. Image Exclusions

Out of 1210 image set readings (22 arteries × 5 beam sequences × 11 readers), 432 were excluded by reader assessment of perceptual image quality, resulting in an overall exclusion fraction of 0.357. Figure 5(a) shows the image exclusion fraction by artery. The median exclusion fractions for control and diseased arteries were similar, at 0.309 and 0.400 respectively, and these groups were not statistically significantly different (p > 0.11). Figure 5(b) shows the image exclusion rate by reader. The median exclusion fractions for readers with greater than one year of experience and less than one year of experience were substantially different, 0.109 and 0.455 respectively, and these groups were statistically significantly different (p < 0.02). Figure 5(c) shows the image exclusion rate by beam sequence. SP3-LatRx had the highest exclusion fraction (0.628), while the SP1.5-ParRx had the lowest exclusion fraction (0.128).

Figure 5.

(a) Fractional image exclusion rate versus artery. The exclusion rate for control arteries was not statistically different (p > 0.11) from the exclusion rate for disease arteries. (b) Fractional image exclusion rate versus readers. The exclusion rate for readers with less than 2 years of experience was statistically higher from the exclusion rate for readers with greater than 2 years of experience (p < 0.02). Note that Reader 1 was the most experienced reader and had the lowest fractional exclusion rate, 0.007. (c) Fractional image exclusion rate versus beam sequence.

IV. Discussion

Comparing the distributions of AUC values achieved by the readers (Figure 3), a number of observations can be made about the performance of different beam sequences for detecting and materially characterizing atherosclerotic plaques. Overall, the F/# of the excitation pulse did not have much of an impact on readers’ AUCs for plaque detection or characterization. A large F/# excitation when tracking within the ROE could be beneficial for two reasons: first, the shearing in the tracking resolution cell would be reduced, improving displacement underestimation [26]; and second, the axial range over which the displacement is constant would be increased [29]. In this study though, there was no statistically significant differences in AUC performance for plaque detection or component characterization between the F/1.5 and F/3 parallel-receive tracking sequences (SP1.5-ParRx and SP3-ParRx). In combination with displacement tracking lateral to the region of excitation to observe shear wave propagation, F/1.5 and F/3 also did not have statistically different AUCs. Similar to the within-ROE tracking case, a larger F/# with lateral tracking could be advantageous because it would increase the axial range over which the shear wave is travelling parallel to the arterial wall. The lateral time-to-peak algorithm used to measure shear wave group velocity in this study assumed that the shear waves propagate parallel to the lateral dimension, an assumption more quickly violated with a more focused excitation [33]. In this study though, axial range likely was not a factor because none of the plaques exceeded ~2 mm in thickness, which could explain the lack of statistical differences in AUCs between F/1.5 and F/3.

The investigation of different tracking methods centered on two main questions: 1) what is the impact on performance of single versus parallel-receive tracking and 2) what is the performance of shear wave tracking lateral to the region of excitation. Parallel-receive tracking (ParRx), in which four tracking lines are simultaneously beamformed in positions centered around the region of excitation (ROE), results in slightly poorer lateral resolution and CNR relative to single-receive tracking (SRx) [28, 42]. This loss of resolution is attributable to both the widening of the tracking point spread function (PSF) and the increased beam spacing with fewer ARF excitations which, ultimately, could negatively impact the detection of small, focal features, such as small calcifications. Despite these drawbacks, the readers achieved comparable AUCs with parallel-receive sequences as compared to the single-receive sequences; for lipid pools, degraded IEL, and collagen neither SP1.5-ParRx nor SP3-ParRx was statistically different from SP3-SRx, the highest median AUC sequence, and for calcium and overall plaque detection, only one of the two parallel-receive sequences was statistically lower. Only in the case of fibrous caps were both parallel-receive sequences statistically significantly lower than SP3-SRx. Given that single-receive did not appear to have a large impact on reader AUC, parallel-receive tracking may be favored due to the factor of four reduction in incident acoustic energy and up to a factor of four increase in frame rate.

For lateral receive tracking, the results of this study showed that in the case of calcium, a stiff material, AUCs were not statistically significantly different from the sequence with the highest sensitivity and specificity. However, in the case of lipid pools, a soft material, the LatRx sequences exhibited relatively low AUC performance. Lipid pools are generally comprised of water, phospholipids, cholesterol monohydrate (present as plate-like crystals), and cholesterol esters (present as oils) [43, 44], and exist in either liquid or liquid-crystalline form, which may not support shear wave propagation. Therefore, poor LatRx performance in lipid regions may be related to inhibited shear wave propagation through the lipid region. Further studies are needed to understand the shear wave dynamics in such media.

While the AUC results presented in this study may indicate that SWV images yielded lower sensitivity and specificity, the sequence implementation must be taken into careful consideration. In both LatRx sequences, the FOV was significantly reduced compared to the within-ROE sequences due to two design choices. First, to measure SWV with more accuracy, a sliding window of ~3 mm (laterally) was used, meaning that 1.5 mm was lost from the left and right edges of the image from windowing. Second, the ROE, which was located in the center of the FOV in this case, was not moved during the sequence and, consequently, no SWVs were measured within it, reducing the FOV by an additional 2–3 mm. On top of the FOV reduction, the LatRx image sets were seen to have smaller magnitude displacements achieved away from the region of ARF excitation (due to high levels of shear wave attenuation), yielding noisy velocity measures near the edges of the image. As derived from the Cramér-Rao Lower Bound [45], the magnitude of jitter error for the imaging parameters used in this study was estimated to be ~0.75 µm (assuming 50% fractional bandwidth, 0.999 correlation coefficient, and 30 dB SNR). In the LatRx data, displacements at the edges of the image (8 mm laterally from the ROE) were measured to be ~1 µm, suggesting that the SWV values plotted on the edge of the FOV were calculated from displacement just above the absolute lower bound of the displacement tracking algorithm. Finally, it should be noted that shear wave reflections and the irregular boundary conditions created by the lesions may have impacted the time-to-peak estimates (and consequently the SWV measurements) drastically [46]. More sophisticated shear wave excitation techniques such as supersonic shear imaging (SSI) or signal processing techniques such as directional filtering could have improved the LatRx images and subsequently the AUCs achieved by the readers with these sequences [16, 47, 48].

It is interesting that, while the mean shear wave group velocity between control and diseased arteries was not statistically different, the standard deviation and range of group velocities was larger for diseased arteries. Direct artery-to-artery comparisons of SWVs can be confounded by variables such as pre-tension [49] and wall thickness [16], therefore an in-depth analysis of SWV as it pertains to plaque detection and material and structural characterization was outside the scope of this manuscript.

Table 6 shows a comparison of the sensitivities and specificities of the highest median AUC sequence to two other non-invasive atherosclerosis imaging modalities, MRI and CT. While the ARF-based approaches described here interrogate the mechanical properties of arteries to delineate plaque, MRI evaluates the nuclear magnetic resonance properties, and CT exploits x-ray attenuation properties [50]. Another important difference between the ARFI, MRI and CT data presented in Table 6 is that the CT and MRI performance numbers are reported for in vivo human carotids, while the ARFI numbers are reported for ex vivo porcine femorals under simulated in vivo imaging conditions. Therefore, the comparison in Table 6 is shown simply to give the results in this study context to the field of non-invasive atherosclerosis imaging and highlight the current state of the art. CT performs well for hard features (calcium, collagen), but does not do as well for soft features (lipid pools), while MRI has the reverse trend, leading the authors in [40] to conclude that MRI may be more promising to stratify the vulnerability of a plaque to rupture. Comparing the performance numbers for ARFI, it appears that ARFI is most sensitive to calcium deposits, but is also generally sensitive to soft components such as lipid pools. ARFI also had high sensitivity for detecting degraded IEL, which was particularly interesting because this feature is not currently detectable by other imaging modalities. As seen in [13], degradation of the IEL appears to change the elastic properties of the arterial wall substantially. So, although the actual feature is far below the resolution of the imaging system, the IEL’s impact on mechanical properties is detectable and could prove to be an important advantage of using ARF-based elastography techniques.

On average, the inter-reader agreement across all beam sequences and plaque features in this study was moderate. The median SCCs ranged from 0.641 (SP3-SRx, lipid pools) down to 0.390 (SP3- LatRx, fibrous caps), with generally large IQRs across all sequences. As expected from the high AUC values, SP3-SRx had the best inter-reader agreement, with SCCs being statistically higher than most other sequences for any given feature (in the case of lipid pools and fibrous caps, SCCs for SP3-SRx were statistically higher than all other sequences). The variability of the SCCs was most likely a consequence of the range of experience levels between the readers, with seven out of the eleven readers having never seen an ARFI image before the training regimen. For these readers, the training regimen was largely responsible for their performance, and readers had to rely heavily on the guidance table (Table 4) when interpreting images. Since ARF-based atherosclerosis imaging is a relatively new field, the training set was limited to one or two examples of each compositional element with one or two beam sequences. Providing more comprehensive training for the readers would likely improve the readers' ability to accurately characterize atherosclerotic plaques and reduce inter-reader variability.

Another interesting observation regarding reader experience was that the most experienced reader, Reader 1, had an equal or lower AUC than the median for most beam sequence and plaque feature combinations (Figure 3). One explanation for this phenomenon is that Reader 1 rated more of the “poor quality” images than the inexperienced readers, and therefore had a more challenging task. As seen in Figure 5(b), the experienced readers had a significantly lower exclusion rate compared to the inexperienced readers; in fact, out of the experienced readers, Reader 1 had the lowest exclusion rate, excluding only 1 out of 110 images. This suggests that as a reader gains experience, she/he is able to make relatively accurate assessments on images that an inexperienced reader would not be able to read.

ARFI imaging for atherosclerosis has already started to be translated to human subjects on a limited basis and has shown promise at identifying regions of mechanical contrast [15, 18]. The next step, as suggested in [15], includes measuring the sensitivity and specificity of the imaging technique in vivo, with matched immunohistochemistry studies. It would be expected that in vivo sensitivity numbers may initially be poorer than those reported in this study due to a number of physiological challenges (motion artifact, intraluminal pressure variation, oblique imaging perspectives, increased imaging depths in obese patients, etc.), but with the development of real time ARFI imaging solutions [51], clever beam sequencing [17, 52], and improvements to motion filtering [53, 54] equal or better sensitivities may be achievable.

Finally, due to the extreme complexity and heterogeneity of the atherosclerotic disease process, a larger study than this would provide the possibility of performing a more robust statistical analysis of detection of all compositional elements. The arteries included in this analysis provided a broad spectrum of atherosclerotic manifestations, but a wide range of possibilities was not represented. For example, this analysis did not include arteries with intra-plaque hemorrhage or ruptured plaques, preventing any assessment of beam sequence performance for detecting thrombus formation. A recent review concluded that thrombi forming at the sites of plaque rupture have an important role in promoting the rapid progression of the underlying lesion which can lead to future ischemic events [55]. Therefore, performance for detecting thrombus and plaque hemorrhage should be assessed. This study also did not quantify the effect of lesion size on reader performance. It has previously been shown that the resolution limit of transcutaneous ARFI ultrasound is generally between 0.5 to 1.0 mm, depending on the imaging parameters used and the underlying mechanical properties of tissue [56]. These numbers though, were derived from experiments done in phantoms and porcine liver, so their pertinence to heterogeneous tissue such as atherosclerotic plaque is unclear. Quantifying the resolution limits of ARFI on features and structures in plaques is an area for future research.

V. Conclusions

Based on the sensitivity and specificity numbers calculated in this ex vivo study, ARF-based ultrasound has been demonstrated to be relevant for detecting type III and higher plaques in peripheral vasculature, and for characterizing plaques features such as calcium and collagen deposits as well as lipid pools and fibrous caps (the latter two features being linked to plaque vulnerability). This study also showed that ARFI is relevant for detecting degraded IEL, while no other imaging method has been demonstrated in this regard. As such, degraded IEL detection may prove to be a significant advantage of ARF-based imaging. Furthermore, this study demonstrated that 1:1 single-receive tracking yielded the highest median AUC values for plaque detection and for delineating all of the compositional elements valuated. In many cases though, AUCs from the 4:1 parallel-receive sequences were comparably high and not statistically significantly different from the single-receive tracking sequence, suggesting these sequences may be preferred for in vivo application due to their improved frame rate and lower energy requirements. ARF-based imaging modalities are relatively facile, as even relatively unskilled readers achieved high performance marks, but more extensive reader training may improve AUC values further. Overall, these results add to the growing body of evidence that suggests ARF-based imaging is relevant to detecting plaques and describing their composition and structure with potential application to diagnosis, monitoring, and assessment of lipid lowering treatment efficacy.

Appendix A

A. ROC Calculation

The general setting and notation of the ROC calculation can be described as follows. There are I blinded readers involved, indexed by i = 1, 2, … , I; J images involved, indexed by j = 1, 2, … , J; and K subsections involved, indexed by k = 1, 2, … , K, that represent the number of sections into which each image is broken. The value X_i,j,k will represent the evaluation of the i^th reader on the k^th subsection of the j^th image. On the histological side, a similar labeling scheme is used; Y_j,k denotes the histological evaluation on the k^th section of j^th image, which represents the gold standard. For this study, one pathologist was recruited to read the gold standard histology slides so the ith indexing is omitted for simplicity for Y.

Recall from the section titled “Reader Evaluation Study”, each reader rated features on a graded scale that has a naturally increasing order (e.g. “Definitely Not Present” “Probably Not Present” “Unclear” “Probably Present” “Definitely Present”). To represent these rating levels numerically, each is assigned a monotonically increasing integer; the possible values of X_i,j,k are 1, 2, …, nwhere a value of 1 is equivalent to a “Definitely Not Present” response, a value of 2 is equivalent to a “Probably Not Present” response, etc. Similarly, the pathologist’s ratings, Y_j,k, are represented numerically as 1, 2, …, m. It is important to note that values of m and n (i.e. the maximum possible value for Y_j,k and X_i,j,k, respectively) are not necessarily the same, and depend on the feature that is being rated (e.g. for calcium, m = 3 and n = 5).

To begin the ROC calculation, all pairs of (X_i,j,k Y_j,k) are organized into an m-by-n frequency table, which represents the histogram of reader responses given the corresponding histological responses. Next a threshold, tis chosen on Y_j,k to dichotomize the m levels of the gold standard into “control” and “case” classes. In other words, a Y_j,k value less than or equal to t means the k^th section of the j^th image does not have the feature in question; whereas a Y_j,k value greater than t means the k^th section of the j^th image does have the feature present. After dichotomization, the m-by-n table becomes a 2-by-n table.

Following Ch. 4.2.2 of [57] the two rows of the frequency table are considered as two discrete random variables latently distributed as normal distributions (referred to as Z₁ and Z₂ for “control” and “case” respectively), whose parameters can be determined with maximum likelihood estimation (MLE). The correspondence between the discrete X_i,j,k distribution and the continuous normal distributions is described by n-1 parameters, or cuts, (referred to as c₁, c₂, …, c_n-1) which can take on any value between - ∞ to ∞. Without loss of generality, Z₁ can be assumed to have zero mean (µ₁ = 0) and standard deviation of 1 (σ₁ = 1) by rescaling the n-1 cuts appropriately. This assumption does not change the discrete probability mass distribution of X_i,j,k, and consequently, Z₂ can be described as a function of µ₂, σ₂, c₁, c₂, …, c_n-1, and solved for by maximizing the log-likelihood function:

$$\text{log}(L({\mu }_2,{\sigma }_2,c_1,\dots ,c_{n-1}))=E11*\text{log}(\mathrm{\Phi }(c_1;\mu =0,\sigma =1))+E12*\text{log}(\mathrm{\Phi }(c_2;\mu =0,\sigma =1)-\mathrm{\Phi }(c_1;\mu =0,\sigma =1))+\cdots +E1n*\text{log}(1-\mathrm{\Phi }(c_{n-1};\mu =0,\sigma =1))+E21*\text{log}(\mathrm{\Phi }(c_1;\mu ={\mu }_2,\sigma ={\sigma }_2))+E22*\text{log}(\mathrm{\Phi }(c_2;\mu ={\mu }_2,\sigma ={\sigma }_2)-\mathrm{\Phi }(c_1;\mu ={\mu }_2,\sigma ={\sigma }_2))+\cdots +E2n*\text{log}(1-\mathrm{\Phi }(c_{n-1};\mu ={\mu }_2,\sigma ={\sigma }_2))$$ (A1)

where Φ (x; µ ,σ) is the cumulative distribution function (CDF) of the normal distribution, and the E_m,n variables represent the frequency counts of reader responses. The resultant parameter estimates are denoted as (µ̂₂, σ̂₂, ĉ₁ …, ĉ_n-1) and from these parameters a continuous ROC curve can be drawn based on µ̂₂, and σ̂₂.

Note that this statistical analysis assumes all subsections as independent, but, due to the study design, not all subsections are truly independent from each other. Although readers were asked to rate each subsection independently of any other subsection, readers viewed each arterial segment in its entirety while rating each of the four individual subsections. This means that there is an inherent correlation structure in the reader response data, as readers most likely rated plaques based on information from adjacent subsection. Therefore, the rational for assuming independence needs to be justified as this assumption was made in this study.

Mathematically, the correlation structure between subsections in an MLE operation would be represented as follows (in a simple example where MLE is being used to estimate the mean, µ, from some number of observations,${\{Y_i\}}_{i=1}^n$):

$$\frac{d}{d\mu }\text{log}(L(\mu ))=-\frac{1}{{\sigma }^2}\{\frac{d}{d\mu }[Y_1-\mu \cdots Y_n-\mu ]\}{\left[\begin{array}{cccc}1& 0& \cdots & 0\\ & 1& \cdots & 0\\ & & \ddots & ⋮\\ & & & 1\end{array}\right]}^{-1}\left[\begin{array}{c}{Y}_1-\mu \\ ⋮\\ Y_n-\mu \end{array}\right]=0$$ (A2)

The center matrix represents the “correlation structure” between 4 subsections viewed simultaneously. Unfortunately, quantifying the true correlation structure in a study where human observers are looking at images is difficult because each reader combines information in a unique and subjective way. In the ROC analysis presented in this manuscript the correlation structure matrix is set to the identity matrix to indicate observations are independent from each other. Statistical theory indicates that using the identity matrix (i.e. assuming independence) can still achieve a consistent estimate of µ₂ and σ₂ (if the true correlation structure was known, the estimate of µ₂ and σ₂ would be consistent and efficient), as the correlation structure mostly affects more subtle statistics like the variance of µ̂₂, and σ̂₂ [58].

B. ROC Example

To demonstrate these methods, the AUC will be calculated for Reader 5 using beam sequence SP3-ParRx to identify lipid pools. The m-by-n frequency table depicting reader response vs. pathologist response is as follows (Note: Reader 5 rated one image set as having “Bad” image quality, so responses for only 84 subsections are given, out of a maximum of 88):

		Reader 1 Response
		Def. Not Pres. (X=1)	Prob. Not Pres. (X=2)	Unclear X=3	Prob. Pres. (X=4)	Def. Pres. (X=5)
Pathologist Response	None (Y=1)	42	15	0	12	0
	Small (Y=2)	1	0	0	8	1
	Large (Y=3)	0	0	0	5	0

The next step is to dichotomize the pathologist’s responses. In the case of lipid pools, the pathologist used a 3-point scale (1 – no lipid pool, 2 – small lipid pool, and 3 – large lipid pool), and the threshold was chosen such that t > 1, indicating that both small and large lipid pools would be counted as “positives” as follows:

		Reader 1 Response
		Def. Not Pres. (X=1)	Prob. Not Pres. (X=2)	Unclear X=3	Prob. Pres. (X=4)	Def. Pres. (X=5)
Pathologist Response	None (Y=1)	42	15	0	12	0
	Small Or Large (Y=2,3)	1	0	0	13	1

Finally the distributions Z₁ and Z₂ are estimated; µ₁ and σ₁ are assumed to be 0 and 1 respectively, and µ̂₂, and σ̂₂ are calculated to be 4.52 and 2.57, respectively, from MLE. These distributions are depicted graphically in Figure A1(a). Note that the area under the ROC curve (AUC) can be computed at this point without actually plotting the ROC curve itself as the AUC is defined as the P(Z₂ > Z₁). Once the distributions are known, a continuous ROC curve can be plotted by sliding the decision threshold (indicated by a vertical line in Figure A1(a)), and computing the false positive rate (FPR, the x-axis of the ROC curve) and the true positive rate (TPR, the y-axis of the ROC curve). The FPR is given as the probability that Z₁ is greater than the decision threshold, while the TPR is given as the probability that Z₂ is greater than the decision threshold. The continuous ROC curve is depicted in Figure A1(b), with an AUC value of 0.95.

Figure A1.

(a) Estimated normal distributions representing the variables “control” (Z₁) and “case” (Z₂) which depict Reader 5’s responses for lipid pools using SP3-ParRx. For the Z₁ distribution, the mean (µ₁) and standard deviation (σ₁) were assumed to be 0 and 1 respectively. For the Z₂ distribution the mean and standard deviation were estimated by maximum likelihood estimation (MLE) as follows; µ̂₂ = 4.52 and σ̂₂ = 2.57. The vertical line represents one possible decision threshold and the corresponding regions of the two distributions are labeled accordingly (TN – true negative, FN – false negative, FP – false positive, TP – true positive). An ROC curve is computed by sliding the decision threshold along the X-axis to compute both false positive rate (FPR) and true positive rate (TPR). (b) Corresponding ROC curve depicting Reader 5’s performance using beam sequence SP3-ParRx to identify lipid pools. The result for the reader is depicted as a solid line, while the result for a random classifier is depicted as a dashed line. The discrete operating points (estimated with MLE) are depicted as square boxes plotted on top of the reader curve. In this case, the area under the curve is 0.95.