Fully automated and robust analysis technique for popliteal artery vessel wall evaluation (FRAPPE) using neural network models from standardized knee MRI

Abstract

Purpose:

To develop a fully automated vessel wall (VW) analysis workflow (fully automated and robust analysis technique for popliteal artery evaluation, FRAPPE) on the popliteal artery in standardized knee MR images.

Methods:

Popliteal artery locations were detected from each MR slice by a deep neural network model and connected into a 3D artery centerline. Vessel wall regions around the centerline were then segmented using another neural network model for segmentation in polar coordinate system. Contours from vessel wall segmentations were used for vascular feature calculation, such as mean wall thickness and wall area. A transfer learning and active learning framework was applied in training the localization and segmentation neural network models to maintain accuracy while reducing manual annotations. This new popliteal artery analysis technique (FRAPPE) was validated against manual segmentation qualitatively and quantitatively in a series of 225 cases from the Osteoarthritis Initiative (OAI) dataset.

Results:

FRAPPE demonstrated high accuracy and robustness in locating popliteal arteries, segmenting artery walls, and quantifying arterial features. Qualitative evaluations showed 1.2% of slices had noticeable major errors, including segmenting the wrong target and irregular vessel wall contours. The mean Dice similarity coefficient with manual segmentation was 0.79, which is comparable to inter-rater variations. Repeatability evaluations show most of the vascular features have good to excellent repeatability from repeated scans of same subjects, with intra-class coefficient ranging from 0.80 to 0.98.

Conclusion:

This technique can be used in large population-based studies, such as OAI, to efficiently assess the burden of atherosclerosis from routine MR knee scans.

1 |. INTRODUCTION

Atherosclerosis, a leading cause of death worldwide,¹ is a systemic disease that leads to plaque formation or luminal narrowing in multiple vascular beds and can cause clinical events through blood flow obstruction to the heart (coronary heart disease), brain (ischemic stroke), or lower extremities (peripheral vascular disease).²

Vessel wall (VW) MRI, through the use of black blood imaging, has been effective at visualizing normal and diseased arteries and characterizing atherosclerotic lesions.^3,4 VW MRI has previously been used in clinical trials and in natural history studies to identify populations with high cardiovascular risks, or to monitor vascular disease progression,^5,6 particularly in the carotid and coronary arteries. Due to the expenses in the MRI scan, most previous works involving vessel wall analysis were limited in subjects and times of scans. In these studies, careful and comprehensive manual review was needed. Unfortunately, manual review is not scalable to large population-based studies, because it is impractically time-consuming.

The Osteoarthritis Initiative (OAI)⁷ (URL: https://www.clinicaltrials.gov. Unique identifier: NCT00080171) is a multicenter, prospective cohort study of knee osteoarthritis, sponsored by the National Institutes of Health. This cohort would be of interest for cardiovascular research because osteoarthritis and cardiovascular disease share a number of risk factors including age and obesity, and both are associated with physical inactivity. This massive dataset (bilateral knee MRI in 4796 subjects up to eight timepoints over a period of 96 mo, over three million images in total) provides high-quality 3D VW MRI images with the popliteal artery wall (in axial view) clearly visible on the 3D DESS sequence. Therefore, it is ideal for research on vessel wall features as MR biomarkers⁸ and the relationship of these biomarkers with cardiovascular risk.⁹

We hypothesize that quantitative VW analysis of the popliteal arteries will provide new insights into atherosclerotic cardiovascular disease, beyond the original focus of the OAI study on knee osteoarthritis. Artery wall quantification requires drawing the lumen and outer wall contours, then deriving vessel wall feature biomarkers, such as wall thickness. Most quantitative studies relied on manual vessel wall segmentation,¹⁰ which is challenging and subject to reader variability and fatigue.¹¹ Some lumen and outer wall boundaries are not clearly demarcated. Artery and vein may be adjacent and easy to confuse. Bifurcations lead to irregular vessel wall shapes. All these factors increase the review difficulty and time. In our experience, the manual review for a single knee at a single timepoint (more than 60 images per knee) takes up to 3 h. An accurate and fully automated VW evaluation technique is, therefore, essential for large population studies. Some computer assisted tools for vessel wall segmentation exist,^12–17 but manual operations such as initial seed-point placement and region of interest selection are still needed. In addition, none of these methods are adapted to popliteal vessel wall segmentations.

Therefore, we aim to develop an automated vessel wall analysis technique with minimal human intervention. This would be applicable not only to OAI data, but also to other studies using similar acquisition protocols. Fully automated vessel wall analysis faces three main technical challenges: artery localization, vessel wall segmentation, and adequate annotation. For artery localization, the technique must identify the correct artery as the analysis target. Veins and other small artery branches could be mistaken for the popliteal artery, and the technique must be robust to normal and pathologic variants including bifurcation and stenosis. For vessel wall segmentation, valid vessel wall contours must be defined even from images with flow artifacts or signal loss as subtle deviation of contours might lead to large errors in vessel wall measurements. Finally, the training, validating, and testing of machine learning models, especially deep neural network models, require a large number of high-quality manual annotations.

In this work, we developed a fully automated and robust analysis technique for popliteal artery evaluation (FRAPPE) using innovative machine learning techniques, including object tracking and deep neural networks to solve the technical challenges. We validated the performance of FRAPPE in the OAI dataset by comparing its measurements with manual measurements and estimating scan-rescan repeatability. We also performed a preliminary assessment of FRAPPE’s ability to discriminate between diseased and non-diseased arteries by comparing FRAPPE-based measurements between subjects with high and low cardiovascular risk.

2 |. METHODS

2.1 |. FRAPPE techniques

Five steps are included in FRAPPE: artery detection, tracklet refinement, vessel wall segmentation, feature calculation, and export & display. Workflow is shown in Figure 1.

FIGURE 1.

A, Workflow of FRAPPE with one axial image slice as an example. B, Bounding box is detected shown in green rectangles. C, The x and y coordinates of boxes from all slices are refined using a tracklet approach. D, Lumen (red) and outer wall (blue) contours are generated using the segmentation model. E, Vascular features are calculated using contours. F, Segmentation results can be exported and visualized, with the red arrow indicating the slices in A,B,D

2.1.1 |. Artery detection

The bounding boxes (minimum encompassing rectangles around the target) of popliteal arteries in each image from the 3D MRI are predicted from a Yolo V2¹⁸ model.

2.1.2 |. Tracklet refinement

Artery detection might initially include false objects or missed popliteal arteries in certain images. A tracklet refinement step¹⁹ is thus used to combine the neighboring detection information to generate the centerline of the popliteal artery of interest. Patches along the centerline are extracted for vessel wall segmentation.

2.1.3 |. Vessel wall segmentation

A vessel wall segmentation technique based on another neural network model is applied. The segmentation neural network structure with regression and segmentation branches as two outputs is shown in Figure 2. The segmentation is in polar coordinate system so that it reduces errors near artery bifurcations and ensures contour continuity. Each image along with its neighbors are converted to the polar coordinate system, then concatenated in 3D polar stacks (height of 180, width of 256, depth of 3, channel of 1). A sliding window with the height of 80 and step of 1 is used to extract polar patches (height of 80, width of 256, depth of 3, channel of 1) along the height dimension from the polar stacks for the neural network to segment vessel wall. The lumen and outer wall boundaries can be predicted from the regression branch of the network, then converted back to the Cartesian coordinate system as the final vessel wall contours. From the segmentation branch of the network, a probability map can be acquired to calculate the segmentation confidence score for estimation of segmentation quality²⁰. Dice loss²¹ and mean absolute error loss are used for the segmentation and regression branches. Adam optimizer²² is used to control the learning rate. Please refer to Chen et al¹⁹ for detailed descriptions for techniques of artery localization and vessel wall segmentation.

FIGURE 2.

Convolutional neural network structure used for vessel wall segmentation

2.1.4 |. Feature calculation

Vessel features for each image are calculated²³ when lumen and outer wall contours are identified from the vessel wall segmentation. The available features for each image generated from FRAPPE are listed in Table 1. Image-based features can be further combined into artery-based features, such as maximum of normalized wall index.

TABLE 1.

Available features generated from FRAPPE for each image and the method to group image-based features into artery-based features

Feature category	Image-based features	Feature explanation	Method to group into artery-based feature
Vessel wall thickness	Max wall thickness	Maximum in-plane distance between lumen and outer wall contours along a vector originating from the artery centerline	Maximum among all images
	Mean wall thickness	Mean distance between lumen and outer wall contours	Mean of all images
	Eccentricity ratio	Max wall thickness/Min wall thickness	Mean of all images
Vessel wall area features	Area of vessel wall	Area between the lumen and outer wall contours	Mean of all images
	Area of lumen	Area within the lumen contour	Mean of all images
	Total vessel area	Area inside the outer wall contour	Mean of all images
	Normalized wall index	Area of vessel wall / total vessel area × 100%	Mean and maximum of all images

Abbreviation: FRAPPE, fully automated and robust analysis technique for popliteal artery evaluation.

2.1.5 |. Export and display

The artery features and confidence scores for each image are exported to a MySQL database in the local network for storage and statistical analysis. During the automated process, images failed in any of the steps are flagged for further manual confirmation or correction. A visualization tool²⁴ can be used for 3D visualization of artery lumen and wall structures if manual inspections are needed. The contour editing tool²³ can be used if manual corrections are needed.

2.2 |. FRAPPE training

Machine learning techniques used in FRAPPE require training procedures. Unlike traditional training methods requiring large number of human annotations, FRAPPE training includes two phases: model development by transfer learning, and fine tuning using active learning.

Transfer learning techniques were applied to train the localization and segmentation models, aided by limited human annotations. Initiated from previous models for carotid artery applications¹⁹ where human annotations of 32,591 image slices from 1157 subjects are available, models could be trained using a relatively small dataset of popliteal artery images to reduce the need for tedious human labeling.

To improve the robustness of FRAPPE in quantifying the images that are particularly challenging, such as abnormal artery structures, poor image quality or with substantial artifacts, a tuning phase is needed. With feedback from an expert human reviewer (G.C., with over 10 y of experience in vascular review), the hyper-parameters in FRAPPE were adjusted based on the reviewer’s qualitative comments on the initial FRAPPE results, in order to achieve better agreements with manual review. In addition, active learning techniques were applied. Batches of challenging samples, such as images in artery bifurcations or with calcified plaques, with low confidence scores, were chosen for manual corrections, then the corrected images were used for further training, so that the performance of FRAPPE on difficult samples was iteratively improved.

Before FRAPPE was deemed acceptable for its final performance evaluation, an assessment was made of 1) whether FRAPPE achieved performance comparable to human inter-rater variation, so that the iterative training procedure could stop; 2) whether FRAPPE was reliable for all images, or whether it was necessary to discard FRAPPE segmentations of images with failed artery localization or vessel wall segmentation and replaced by manual segmentation; 3) whether results should be discarded from the most proximal and distal images in the 3D image where image quality is systematically poorer.

2.3 |. Dataset selection

All data were from the OAI,⁷ for which the study procedures were approved by local Institutional Review Boards and all participants provided informed consent. Briefly, subjects underwent MRI of the knee using a rigorously standardized acquisition protocol on four identical Siemens 3T MRI scanners. By initial inspection of a limited amount of data, the 3D DESS sequence in the OAI provided valuable vessel wall information for analysis. The imaging parameters for 3D DESS are: repetition time/echo time:16.32/4.71 ms, in-plane resolution: 0.36 × 0.36 mm, slice thickness: 1.5 mm, field of view: 140 × 140 mm.

As our research focuses on cardiovascular risk assessments, we identified two groups of subjects who, on the basis of baseline clinical and demographic information, had, respectively, low or high risk for cardiovascular disease. This allowed us to compare results between high- and low-risk groups and to enrich the datasets used for training, validation, and testing with individuals at higher risk of atherosclerosis and plaque in their popliteal arteries. The high-risk group included subjects ≥65 y old with a history of smoking, history of hypertension, body mass index (BMI) ≥ 25 kg/m², and at least one of seven additional prior events or risk factors: 1) operation to unclog or bypass arteries in legs; 2) stroke, transient ischemic attack, blood clot or bleeding in brain; 3) heart attack; 4) diabetes; 5) current smoker; 6) BMI ≥ 30 kg/m²; 7) age ≥ 75 y old. The low-risk group included subjects <55 y old who never smoked, were not hypertensive, had BMI < 25 kg/m², and had none of the seven additional risk factors specified for the high-risk group.

Altogether, two training sets, two validation sets, and four testing sets from the OAI database were employed in FRAPPE development, tuning, and in final performance evaluation. These datasets are summarized in Table 2. Training Set 1 and Validation Set 1 were randomly selected for model training during the technical development phase. The Training Set 2, Validation Set 2, Testing Set 1, and Testing Set 2 were randomly selected, but with the high-risk group enriched to comprise approximately one-third of the sample, about three-times the prevalence in the entire OAI dataset. Testing Set 1 included 25 subjects with a full quantitative review by human reviewers for comparison with FRAPPE. Testing Set 2 included 225 subjects (including the 25 Testing Set 1 subjects) which underwent a qualitative review of FRAPPE’s performance, further described in the Image Review section. Testing Set 3 was designed to assess repeatability of vascular features measured by FRAPPE and was randomly selected from the subjects who had 24-mo and 30-mo scans available. Testing Set 4 was selected from the baseline scans of the high- and low-risk groups.

TABLE 2.

Datasets used in this study

Phase	Dataset name	Number of subject/image	Side	Manual review	Selection method	Purpose
Technical development	Training Set 1	23/1326	Index	Reviewer 1	Simple random sample	Train the neural network model for artery localization and segmentation
	Validation Set 1	2/117	Index	Reviewer 1		Monitor the training procedure and tune parameters
Fine tuning and Validation	Training Set 2	100/7372	Index	N/A	Stratified random sample enriched with high-risk subjects	Further model tuning in a larger dataset, with reviewer’s help to identify mistakes and confirm improvements
	Validation Set 2	10/743	Index	Reviewer 1 (N = 10), Reviewer 2 (N = 10)		Contours drawn by both reviewers to compare quantitatively to decide when to stop tuning. Also assess inter-rater variability
	Testing Set 1	25/1843	Index	Reviewer 1 (N = 12), Reviewer 2 (N = 13)		Used for performance evaluation in the quantitative assessment
	Testing Set 2	225/16633	Index	N/A		Used for performance evaluation in the qualitative assessment
Performance evaluation	Testing Set 3	50/3711(24 mo), 50/3738 (30 mo)	Index	N/A	Simple random sample	Used for performance evaluation in the repeatability assessment
	Testing Set 4	50/3562 (high risk), 50/3536 (low risk)	Both	N/A	Stratified random samples of the high-risk and low-risk groups	Used for evaluating feature differences between high and low risk subjects

Images from both sides were used for analysis in Testing Set 4, and a random side was chosen for the other sets. Training/Validation Sets 2 and Testing Sets 1–4 were selected to achieve equal representation from each of the four participating sites. Training/Validation Sets 2 and Testing Sets 1–2 included a single scan per subject, with the timepoint (baseline, 12-mo, etc.) chosen at random.

2.4 |. Image review

Human reviewer 1 (G.C.) drew vessel wall contours on Training Set 1, 2, Validation Set 2, and half of Testing Set 1 (N = 12). She also did the qualitative review for Validation Set 2, and Testing Set 2. Reviewer 2 (W.L., a newly trained qualified reviewer) drew vessel wall contours on Validation Set 2, and the other half of Testing Set 1 (N = 13) independent of reviewer 1. Inter-reader agreement was evaluated using Validation Set 2. Both reviewers analyzed 61 images centered at the tibial medial spine in all cases and all reviews were performed independent of FRAPPE. To further evaluate the segmentation performance on normal vessel wall versus vessel wall with plaques, reviewer 2 subdivided each slice in Testing Set 1 into “no plaque,” “plaque without calcification,” and “plaque with calcification” groups based on a modified American Heart Association (AHA) lesion classification for MRI.²⁵

The qualitative assessment was designed to evaluate image and segmental quality, and to take less than 5 min per case, so a much larger sample size could feasibly be evaluated compared to the quantitative image reviews of Validation Set 2 and Testing Set 1. Overall image quality of each scan was rated on a 4-point scale (1 = poor, arterial wall and lumen boundaries were unidentifiable; 2 = adequate, wall and luminal boundaries were identifiable, but wall components were not; 3 = good, wall and luminal boundaries were identifiable, but there was some uncertainty in the detection of wall components; 4 = excellent, wall and lumen boundaries and wall components were identifiable with no uncertainty). The contours generated by FRAPPE on each image were examined and rated as having major segmental errors or not. Major errors were defined as cases where 1) the contours drawn by FRAPPE were clearly inconsistent with the underlying images in such a way that, in reviewer 1’s experience, a trained human reviewer would not draw them; and 2) the error potentially could have a noticeable effect on vascular features (e.g., wall thickness or lumen area) derived from the contours, either due to a substantial error on a single image, or a more moderate error present on several images. Major errors were also classified by type of error, including: unable to segment artery, wrong structure segmented (e.g., wrong artery or vein), overly thin wall, overly thick wall, highly irregular contours, and others.

2.5 |. Statistical analysis

Agreement between FRAPPE and manual review measurements (Validation Set 2 and Testing Set 1) was evaluated using the Dice similarity coefficient (DSC)²⁶ as the metric for pixel-wise segmentation and using the mean difference (bias), the coefficient of variation (CV), the intraclass correlation coefficient (ICC), and Bland-Altman plots as metrics of the image- and artery-based vessel features in Table 1. Similar methods were used to summarize inter-reader agreement (Validation Set 2) and the repeatability of the vessel features (Testing Set 3). Differences in artery-based features between high- and low-risk groups (Testing Set 4) were summarized using means and the area under the receiver operating characteristic (ROC) curve (AUC). The qualitative review (Testing Set 2) was summarized using counts and percentages. Throughout, 95% confidence intervals (CIs) were calculated for important parameters. For image-based analyses and the analysis of Testing Set 4, which included left and right arteries, CIs were calculated using the non-parametric bootstrap with resampling by subject to account for any dependence among the multiple observations per subject.

To evaluate the robustness of the segmentation confidence, the ranked correlation coefficient between the confidence score and DSC was calculated using generalized estimating equations, accounting for non-independence between slices from the same subject.

To compare the segmentation performance of FRAPPE with state-of-the-art neural network models, 3D Cartesian based U-net ²⁷(previously adopted in vessel wall segmentation²⁸), and Mask-RCNN²⁹ (ResNet-101³⁰ as backbone, pretrained on the ImageNet³¹ dataset) were trained using the same transfer learning and active learning procedure for popliteal vessel wall segmentation with the same dataset. The segmentation performances were evaluated using DSC. Paired T-test was used to evaluate the significance of the segmentation differences.

All statistical calculations were conducted with the statistical computing language R (version 3.4.2 and 3.5.1; R Foundation for Statistical Computing, Vienna, Austria).

3 |. RESULTS

3.1 |. FRAPPE training and validation

Baseline characteristics of OAI subjects included in the training, validation, and testing sets are summarized in Supporting Information Table S1. After transfer learning using Training set 1, the FRAPPE model was further tuned in Training set 2 with feedback provided by reviewer 1. Images with a confidence score lower than 0.8 were further reviewed by reviewer 1 and corrected if necessary, to provide additional training to the algorithm. In addition, most artifacts and low-quality images were at the most distal or proximal slices of the 3D image, so the first and last five images were excluded from further analysis.

After three iterations of active learning, FRAPPE achieved a DSC of 0.81 (95% CI: 0.79 to 0.83) in Validate Set 2, comparable to the DSC between two human reviewers of 0.85 (95% CI: 0.83 to 0.87). Examples showing the improvement for segmenting challenging images with calcified plaque or severe flow artifacts through active learning is shown in Figure 3. Agreement in measurements of vascular features between FRAPPE and human reviewers as measured by the CV and ICC were also similar to the level of agreement between the two human reviewers (Supporting Information Table S2). For example, the FRAPPE-reviewer CV for mean wall thickness was 6.5% compared to the inter-reader CV of 6.8%.

FIGURE 3.

Missed segmentation of calcified plaques (pointed by the red arrows) and irregular contours caused by severe flow artifacts (pointed by the green arrows) before active learning (middle). Contours improved after additional training using active learning technique (right). Contours are not affected by the neighboring vein (pointed by the yellow arrow), a benefit for using the polar segmentation model

From Validation Set 2, only one image was flagged as failures in artery localization due to partial coverage of the artery. Considering the low occurrence, there was no need to add the missing contours manually, so no human corrections were applied in other assessments.

3.2 |. Accuracy of FRAPPE in the testing sets

In Testing Set 1, the DSC of 0.79 (95% CI: 0.77–0.81) was similar to that estimated in Validation Set 2, and significantly higher than 3D Cartesian based U-net (mean DSC of 0.77, P value < 1e-5) and Mask-RCNN (mean DSC of 0.67, P value < 1e-5). The quantitative assessment of image-based and artery-based vascular features in Testing Set 1 are shown in Table 3. Relative to human reviewers, the CV and ICC estimates for FRAPPE were 8.7% and 0.73 for mean wall thickness, 10.0% and 0.90 for mean wall area, and 2.9% and 0.99 for mean lumen area. Bland-Altman plots of selected features are shown in Supporting Information Figure S1. Some of the segmentation results on challenging images were shown in Figure 4, demonstrating good performance of FRAPPE on bifurcation images, vessel wall with low contrasts, vessel wall with plaque, and when the artery is close to the vein. After slice-based plaque labeling, 1276 slices were grouped as “no plaque,” 59 slices as “plaque without calcification,” 56 slices as “plaque with calcification.” The DSCs (0.79, 0.82, 0.82) and vessel wall features are comparable between three groups, indicating similar performance of FRAPPE in segmentation of normal vessel wall and vessel wall with plaques. Detailed comparison results between three groups are shown in Supporting Information Table S3.

TABLE 3.

Agreement between FRAPPE and human reviewers in the Testing Set 1

Variable	Reader		Difference between FRAPPE and human reviewers
	FRAPPE	Human	Mean	(95% CI)	CV	(95% CI)	ICC	(95% CI)
Image-based measurements (n = 1391)
Mean wall thickness, mm	0.81 ± 0.16	0.76 ± 0.21	0.05	(0.02, 0.08)	12.3%	(9.4, 15.3)	0.74	(0.65, 0.79)
Max wall thickness, mm	0.99 ± 0.38	0.96 ± 0.49	0.03	(−0.03, 0.07)	17.8%	(13.2, 22.2)	0.84	(0.77, 0.90)
Eccentricity ratio	1.43 ± 0.51	1.59 ± 0.74	−0.16	(−0.24, −0.07)	22.6%	(17.9, 26.8)	0.72	(0.55, 0.80)
Wall area, mm2	18 ± 6	17 ± 7	1.2	(0.2, 1.9)	14.5%	(10.3, 18.2)	0.87	(0.83, 0.89)
Lumen area, mm2	30 ± 12	30 ± 13	−0.1	(−0.6, 0.4)	5.0%	(3.5, 6.8)	0.99	(0.98, 0.99)
Total vessel area, mm2	48 ± 17	47 ± 18	1.1	(0.4, 1.7)	4.7%	(3.3, 6.6)	0.98	(0.97, 0.99)
Normalized wall index, %	38 ± 5	36 ± 6	1.9	(0.6, 2.8)	8.3%	(7.1, 9.6)	0.71	(0.60, 0.78)
Artery-based measurements (n = 25)
Mean wall thickness, mm	0.81 ± 0.10	0.75 ± 0.15	0.05	(0.02, 0.09)	8.7%	(6.4, 10.7)	0.73	(0.35, 0.89)
Max wall thickness, mm	1.82 ± 0.73	1.83 ± 1.00	−0.01	(−0.25, 0.24)	22.4%	(13.6, 32.3)	0.78	(0.56, 0.90)
Mean eccentricity ratio	1.43 ± 0.21	1.59 ± 0.36	−0.16	(−0.24, −0.07)	11.8%	(8.4, 14.5)	0.68	(0.22, 0.87)
Mean wall area, mm2	18 ± 5	16 ± 6	1.2	(0.4, 2.1)	10.0%	(7.1, 12.4)	0.90	(0.73, 0.96)
Mean lumen area, mm2	30 ± 11	30 ± 11	−0.1	(−0.7, 0.4)	2.9%	(1.8, 4.1)	0.99	(0.99, 1.00)
Mean total vessel area, mm2	48 ± 16	46 ± 17	1.1	(0.4, 1.7)	2.8%	(2.0, 3.9)	0.99	(0.97, 1.00)
Mean normalized wall index, %	38 ± 3	36 ± 5	1.9	(0.8, 3.0)	6.2%	(4.8, 7.5)	0.70	(0.30, 0.88)
Max normalized wall index, %	46 ± 8	46 ± 10	0.1	(−1.7, 1.9)	6.5%	(3.9, 9.1)	0.88	(0.75, 0.95)

Abbreviations: CV, coefficient of variation; FRAPPE, fully automated and robust analysis technique for popliteal artery evaluation; ICC, intraclass correlation coefficient.

FIGURE 4.

Example of FRAPPE generated contours (right column images: red contour, lumen boundary; blue contour, outer wall boundary) in challenging images (original images shown in the left column, zoomed-in images shown in the middle column) with low image contrast around vessel wall boundary, bifurcation (top row), plaque (middle row), and when artery is close to the vein (bottom row)

The ranked correlation coefficient between confidence score and the DSC is 0.234 with P-value < .001, indicating lower confidence is likely to lead to lower segmentation performance.

Testing Set 2 contained 225 subjects. Of these scans, 211 (94%) were rated as good or excellent image quality (rating = 3–4), 14 (6%) had adequate quality (rating = 2), and no scans had poor quality (rating = 1). Flow artifacts were found in 81 (36%) of cases. Other types of artifacts were less common: motion-related (n = 1), inadequate fat suppression (n = 2), substantial partial voluming (n = 9), and other types (n = 6).

From the 225 scans in Testing Set 2, the FRAPPE segmentations on 14,055 images were reviewed. Of these images, 165 were rated as having a major segmentation error (rate: 1.2%, 95% CI: 0.6–1.9%). The specific types of errors included FRAPPE outlining the wrong structure (wrong artery or vein; n = 66 [0.5%]), highly irregular contours (n = 38 [0.3%]), overly thin wall (n = 30 [0.2%]), overly thick wall (n = 24 [0.2%]), unable to segment (n = 7 [0.05%]), and other types (n = 7 [0.05%]).

3.3 |. Repeatability of vascular features

Repeatability of artery-based vascular features in Testing Set 1 is summarized in Table 4. Repeatability was high for most features with CV < 5% and ICC ≥ 0.94 except for mean eccentricity ratio (CV = 12.2%, ICC = 0.80) and max normalized wall index (CV = 6.2%, ICC = 0.86).

TABLE 4.

Repeatability statistics of FRAPPE artery-based vessel wall measurements from Testing Set 3

Variable	Scan*		Difference		Repeatability
	First	Second	Mean	(95% CI)	CV	(95% CI)	ICC	(95% CI)
Mean wall thickness, mm	0.86 ± 0.11	0.87 ± 0.11	0.00	(−0.01, 0.01)	2.9%	(2.2, 3.7)	0.95	(0.92, 0.97)
Max wall thickness, mm	2.00 ± 0.55	2.04 ± 0.54	−0.04	(−0.14, 0.06)	12.2%	(9.4, 15.0)	0.80	(0.66, 0.88)
Mean eccentricity ratio	1.52 ± 0.25	1.53 ± 0.25	−0.01	(−0.03, 0.01)	3.6%	(2.5, 4.7)	0.95	(0.91, 0.97)
Mean wall area, mm2	19 ± 5	19 ± 5	0.1	(−0.1, 0.4)	3.4%	(2.6, 4.2)	0.98	(0.97, 0.99)
Mean lumen area, mm2	30 ± 11	29 ± 11	0.6	(0.1, 1.1)	4.4%	(2.8, 6.3)	0.98	(0.97, 0.99)
Mean total vessel area, mm2	49 ± 15	48 ± 15	0.7	(0.1, 1.3)	3.3%	(2.0, 4.9)	0.99	(0.98, 0.99)
Mean normalized wall index, %	40 ± 4	40 ± 5	−0.4	(−0.8, 0.0)	2.6%	(2.0, 3.3)	0.94	(0.90, 0.97)
Max normalized wall index, %	50 ± 8	51 ± 8	−0.3	(−1.5, 1.0)	6.2%	(4.7, 7.5)	0.86	(0.76, 0.92)

Abbreviations: CV, coefficient of variation; FRAPPE, fully automated and robust analysis technique for popliteal artery evaluation; ICC, intraclass correlation coefficient.

^*Values are mean ± SD.

3.4 |. Comparison of high- and low-risk groups

Artery-based vascular features in subjects with high and low risk for atherosclerosis (Testing Set 4) are summarized in Table 5. Mean eccentricity ratio (AUC: 0.79, 95% CI: 0.68–0.89, P = .005), max wall thickness (AUC: 0.73, 95% CI: 0.60–0.85, P = .002), and mean wall thickness (AUC: 0.71, 95% CI: 0.57–0.84, P = .004) were most able to discriminate between high- and low-risk groups based on the AUC.

TABLE 5.

Comparison of FRAPPE vessel wall measurements between high and low atherosclerosis risk groups

Variable	Risk Group		Difference
	High (N = 50)	Low (N = 50)	Mean	(95% CI)	P-value	AUC	(95% CI)
Mean wall thickness, mm	0.82 ± 0.08	0.76 ± 0.06	0.06	(0.02, 0.10)	0.004	0.71	(0.57, 0.84)
Max wall thickness, mm	1.97 ± 0.56	1.57 ± 0.43	0.40	(0.16, 0.63)	0.002	0.73	(0.60, 0.85)
Mean eccentricity ratio	1.47 ± 0.20	1.34 ± 0.15	0.13	(0.04, 0.22)	0.005	0.79	(0.68, 0.89)
Mean wall area, mm2	17 ± 4	16 ± 4	1.2	(−0.9, 3.3)	0.25	0.60	(0.44, 0.75)
Mean lumen area, mm2	27 ± 8	28 ± 11	−1.0	(−6.3, 4.2)	0.72	0.50	(0.34, 0.67)
Mean total vessel area, mm2	45 ± 12	44 ± 15	0.2	(−7.1, 7.3)	0.95	0.53	(0.37, 0.69)
Mean normalized wall index, %	39 ± 3	38 ± 4	1.7	(−0.2, 3.5)	0.073	0.65	(0.49, 0.79)
Max normalized wall index, %	49 ± 6	46 ± 7	2.9	(−0.5, 6.2)	0.091	0.64	(0.49, 0.77)

Abbreviations: AUC, area under the receiver operating characteristic curve; CI, confidence interval; FRAPPE, fully automated and robust analysis technique for popliteal artery evaluation.

3.5 |. Computation time

Based on our workstation with Intel Xeon CPU E5–1650 v4 @3.6GHz 6 cores, 64 GB memory, NVIDIA GeForce GTX TITAN V on Windows 10, the processing time for Testing Set 2 for artery localization, vessel wall segmentation, and feature calculation is 65.2 ± 19.8 s, 279.1 ± 48.4 s, and 114.3 ± 14.7 s, respectively. The whole process took less than 8 min per artery on average.

4 |. DISCUSSION

4.1 |. Feasibility

On the basis of comprehensive assessments using a wide range of images from the OAI dataset, vessel wall is clearly visible in MR knee images and vessel wall features can be robustly and automatically extracted.

Robustness of the algorithm was demonstrated in three ways.

1. From the qualitative review (Testing Set 2), MR images originally designed for osteoarthritis were shown to have sufficient quality for quantitative vessel wall analysis, and FRAPPE worked well overall across images of varying quality, vascular anatomy and artifacts with an overall error rate of 1.2% of images. From the quantitative review (Testing Set 1), FRAPPE had a DSC of 0.79 (over 0.7 considered as excellent agreement,³² and significantly better than two other neural network methods using the same learning procedure and dataset) and produced vascular features with relatively good agreement (ICC > 0.71 for image-based measurements, ICC > 0.68 for artery-based measurements) with those produced by human review.

2. Six-month repeatability of generated MR vessel wall biomarkers by FRAPPE compared favorably with previous work in the carotid artery.³³

3. For challenging situations, such as images with calcified plaques, due to the active learning technique, FRAPPE showed good performance after selecting only a few similar cases with human annotations as additional training set. The same approach could be applied if other unseen challenging problems are identified to further improve the performance.

Errors in FRAPPE are usually on images from the most distal and proximal segments of the artery in the scan where the signal loss becomes severe in MR; therefore, the first and last five images were excluded. Outer wall segmentation has more problems than the lumen due to weaker contrast with neighboring tissues. In subjects with higher cardiovascular risk, it is much more likely that severe plaques might reduce the flow and lead to more challenging segmentation. Arterial bifurcations or trifurcations did not have noticeable impact on the completeness of vessel wall segmentation, which is one of the advantages of the polar coordinate-based segmentation applied in FRAPPE.

The process time for FRAPPE was reasonable, so that analysis of popliteal arteries in large-scale datasets is realistic. The popliteal artery appears in over three million images in the OAI dataset so manual segmentation is completely impractical, and it could take up to 67 y (8 h per workday) if using manual review. The analysis is still challenging if using some semi-automated review methods that require human interactions such as region of interest selection or manual modifications on predicted contours. The realistic processing time comes from three main factors.

1. The fully automated workflow in FRAPPE demonstrates no human interventions are needed to generate high-quality vessel wall features, even if a few images might have errors or low segmentation confidence.

2. GPU acceleration allows fast predictions of artery locations and segmentations.

3. With MySQL database handling results from multiple workstations in the local network, paralleling processing is available. Based on the current processing time, with 10 instances of FRAPPE, all OAI subjects (each has bilateral knee scans) at baseline can be processed within a week.

4.2 |. Significance

Because of the accuracy and efficiency of FRAPPE, the OAI dataset with multiple timepoints can be analyzed within a reasonable time. Based on previous MR work in other vessels such as the carotid,⁵ the vessel wall features generated from FRAPPE are likely to offer imaging biomarkers of cardiovascular risk useful in both research and clinical environments.

As the pilot data, Testing Set 4 including subjects with both high and low cardiovascular risks validated our assumption that some FRAPPE extracted vessel wall features would be reflective of these differences in risks. The relationships between popliteal vessel wall and atherosclerotic cardiovascular disease should be further studied and characterized with a larger sample. Considering the large number of subjects, multiple timepoints of scans, the large coverage of knee (at least 90 mm), and the variety of atherosclerotic conditions of subjects in this information rich dataset, the OAI is promising for further research on popliteal vessel wall and atherosclerotic cardiovascular disease in a large population study.

The OAI 3T MRI protocol was developed through consensus³⁴ and has become a de facto standard for knee MRI research, so we believe FRAPPE could have wide applicability wherever knee MRI data are acquired. Importantly, FRAPPE could provide an add-on popliteal artery analysis of standard knee MR scans without adding any additional work for radiologists. Although vessel wall features for popliteal arteries have not usually been reported from routine knee scans, these features could provide additional assessments of cardiovascular risks in asymptomatic patients. Subjects identified with thickened vessel wall can then be referred for more detailed examinations for atherosclerosis.

4.3 |. Limitations

The original OAI protocol has no repeated scans at the same timepoint. The closest approximation to these repeated scans were the subset of subjects who underwent MRI at both 24 and 30 mo. It is possible that some subjects had disease progression within 6 mo, in which case repeatability may be underestimated. Furthermore, image registration was not performed. We expect the repeatability of FRAPPE to be further improved with good vessel-targeted registrations.

FRAPPE was developed using the 3D DESS sequence. Applications on other MR vessel wall sequences is available with additional training sets.

Only vessel wall thickness and area measurements are provided in the current FRAPPE version. More features and functions can be added in the future, such as vessel wall signal intensity measurements, automated high-risk plaque detection, and plaque components segmentation.

5 |. CONCLUSIONS

Automated analysis of popliteal vessel wall from standardized knee MRI is feasible, demonstrated from the OAI dataset. A fully validated FRAPPE has the potential for accurate and repeatable vessel wall measurement of popliteal arteries while minimizing human effort and analysis time, which may play an important role for future artery wall analysis in both atherosclerosis research and clinical care.

Supplementary Material

supp info

FIGURE S1 Bland-Altman plots of selected artery-based vascular features measured by FRAPPE and human reviewers in Testing Set 1. Comparisons of FRAPPE and human review measurements of all vascular features are summarized in Table 3

TABLE S1 Subject baseline characteristics

TABLE S2 Agreement between FRAPPE and human reviewers in Validation Set 2

TABLE S3 Agreement between FRAPPE and human reviewers in the Testing Set 1 divided by the plaque category