Artificial Intelligence to Assist in Exclusion of Coronary Atherosclerosis During CCTA Evaluation of Chest Pain in the Emergency Department: Preparing an Application for Real-world Use

Abstract

Coronary computed tomography angiography (CCTA) evaluation of chest pain patients in an emergency department (ED) is considered appropriate. While a “negative” CCTA interpretation supports direct patient discharge from an ED, labor-intensive analyses are required, with accuracy in jeopardy from distractions. We describe the development of an artificial intelligence (AI) algorithm and workflow for assisting qualified interpreting physicians in CCTA screening for total absence of coronary atherosclerosis. The two-phase approach consisted of (1) phase 1—development and preliminary testing of an algorithm for vessel-centerline extraction classification in a balanced study population (n = 500 with 50% disease prevalence) derived by retrospective random case selection, and (2) phase 2—simulated clinical Trialing of developed algorithm on a per-case (entire coronary artery tree) basis in a more “real-world” study population (n = 100 with 28% disease prevalence) from an ED chest pain series. This allowed pre-deployment evaluation of the AI-based CCTA screening application which provides vessel-by-vessel graphic display of algorithm inference results integrated into a clinically capable viewer. Algorithm performance evaluation used area under the receiver operating characteristic curve (AUC-ROC); confusion matrices reflected ground truth vs AI determinations. The vessel-based algorithm demonstrated strong performance with AUC-ROC = 0.96. In both phase 1 and phase 2, independent of disease prevalence differences, negative predictive values at the case level were very high at 95%. The rate of completion of the algorithm workflow process (96% with inference results in 55–80 s) in phase 2 depended on adequate image quality. There is potential for this AI application to assist in CCTA interpretation to help extricate atherosclerosis from chest pain presentations.

Background

New-onset chest pain remains a predominant clinical presentation to an emergency department (ED), and it can lead to significant concurrent costs from testing and/or anxiety, as well as to future financial and/or time expenses following initially abnormal findings [1, 2]; this burden persists despite acute coronary syndrome (ACS) being a relatively infrequent final diagnosis (i.e., in only 17% of ED chest pain cases [3]). However, due to the clinical or medical-legal consequences of failing to recognize ACS (i.e., in up to 4.4% of ACS cases [3]), ED chest pain patients continue to be frequently hospitalized even when they are at little risk (e.g., with low-to-intermediate pretest probability) of “obstructive” coronary artery disease (CAD) (i.e., ≥ 50% luminal diameter stenosis) [2, 4]. This has fostered opportunity for alternative approaches to chest pain evaluation, such as coronary computed tomography angiography (CCTA).

Validated by its high Sensitivity for atherosclerosis detection, as well as its high negative predictive value (NPV) and Specificity supporting disease exclusion, per coronary artery or major branch, CCTA has become a well-established option for CAD evaluation in symptomatic ED patients [1, 2, 4, 5]. Such use of CCTA in chest pain patients to effectively eliminate CAD as the cause of symptoms is now endorsed by professional appropriateness guidelines [6], and it is considered especially suitable in the clinical setting of low-to-intermediate CAD probability as is found in most (i.e., over 80% [7]) ED presentations for suspected coronary-related chest pain [6]. Moreover, direct patient discharge from the ED to home based on a “negative” CCTA examination (often defined by absence of obstructive disease [5]) is supported by its (1) noninvasiveness and patient acceptability, (2) safety and positive outcome projection (e.g., absent cardiac deaths, stable-to-decreased major adverse cardiac event (MACE) occurrence, and stable-to-decreased repeat visits or admissions following ED discharge), and (3) generic cost-savings (funds and time) to healthcare systems (e.g., shortened and less costly initial visits; reduced post-visit expenses) [1, 2, 4, 8–11].

However, for the full clinical benefit of this application to be realized, the complete exclusion of any coronary atherosclerosis, not only obstructive CAD, within the entire coronary artery tree per chest pain patient is essential because the presence of even mild degrees of atherosclerotic plaque on CCTA is known to predispose to ACS [12, 13]. In addition, if coronary atherosclerosis is detected on CCTA, and quantitation or display of atherosclerotic plaque burden is desired for treatment planning or prognostication [14], accurate delineation of the coincident disease-free arteries, branches, or segments becomes a prerequisite.

The widespread utilization of this CCTA option in the ED setting is being increasingly challenged by mounting pressure from referring clinicians for rapid turnaround of definitive imaging results from CCTA interpreters, in order to expedite ED-patient throughput [15]. With rising demands on medical professionals related to more-and-more stressful healthcare delivery environments and communication complexities, frequent disturbances of physician workflow and concentration during an imaging interpretation are growing prospects for professional mistakes [16, 17]. This could potentially result in a falsely negative analysis of CCTA examinations and ultimately to an unwanted early and dangerous discharge of a chest pain patient from an ED due to undetected coronary atherosclerosis [3]. The application of artificial intelligence (AI) in medical imaging has the potential to both enhance yield and reduce human error in diagnostic assessments through the assistance it provides to qualified interpreting physicians [18, 19], such as the experienced and skilled ones that oversee CCTA examinations.

In recent years, AI-based approaches have been used for the direct detection and characterization of known coronary atherosclerosis on CCTA [20–23]. On the other hand, to our knowledge, there has been no prior application of AI to the direct exclusion of atherosclerosis on CCTA for the aforementioned reasons, especially in the ED setting.

In this study, a two-phase approach was used to develop, assess, and prepare an AI algorithm for a “real-world” deployment aimed at prospectively assisting a qualified interpreting physician in accurate and prompt exclusion of coronary atherosclerosis on CCTA in ED chest pain patients. This preparation includes capabilities for concurrent return of vessel-by-vessel inference results to the interpreter regarding the entire coronary artery tree per chest pain patient. This is accomplished within a graphical user interface (GUI) viewer, which also incorporates standard clinical image display capabilities, allowing inference acceptance or rejection along with regular CCTA image re-examination as needed for corroboration.

Materials and Methods

Following a brief description of the (1) basics of CCTA imaging methodology used, (2) coronary artery image-processing performed, and (3) AI resources employed, this section describes the two phases leading to the creation, assessment, and pre-deployment of an AI algorithm to assist the qualified interpreting physician in the exclusion of coronary atherosclerosis on CCTA. Phase 1 was focused on the development and preliminary evaluation of the vessel-based algorithm grounded on a balanced (diseased vs non-diseased) study population derived by retrospective random selection of CCTA cases and supported by data augmentation (DA). Phase 2 was concerned with the simulated clinical Trialing of the developed vessel extraction-based algorithm in a more “real-world” study population composed of a recent series of consecutive ED cases with chest pain evaluated using CCTA. It allowed a pre-deployment evaluation of the proposed case-based application, which concurrently provides to the interpreter the algorithm inference results by vessel-by-vessel graphic display integrated into a clinically capable viewer of the same GUI.

CCTA Imaging Methodology

All CCTA examinations were performed using American College of Radiology (ACR)-accredited single-source (Definition AS Plus, Edge, go.Top) or dual-source (Definition Flash, Force) multi-detector systems (Somatom: Siemens Healthineers, Forchheim, Germany), with ECG-based referencing of image-data acquisition [24]. Following standard processes [25], heart-rate control (target ≤ 60 beats/minute using intravenous Metoprolol ≤ 30 mg) and vasodilatation (using sublingual Nitroglycerin 0.4 mg) preceded injector-controlled (Stellant: Medrad, Indianola, PA) administration of a non-ionic/low-osmolar contrast agent (intravenous Iohexol-350 80 ml, followed by saline 50 ml, at 5–6 ml/s) (General Electric Healthcare Medical Diagnostics, Chicago, IL). Emphasizing the “as low as reasonably achievable” principle [26], limitation of radiation exposures was achieved using (in decreasing order of preference): (1) single-beat helical, (2) prospectively triggered sequential, or (3) pulse-modulated retrospectively gated helical acquisitions [24, 25].

Coronary Artery Image Processing

The stepwise CCTA image-processing methodology used per case included (1) segmentation of the coronary arteries in the entire cardiac CT volume and (2) use of a previously described GUI [27, 28] for both manual coronary artery analysis and the evaluation of automatic vessel-centerline extractions (CT Cardio-Vascular Engine: Siemens Healthineers, Forchheim, Germany), combined with proprietary technology [29], performed on coronary arteries or branches either with or without atherosclerosis [30] (Appendix 1).

AI Resources

AI computational support was provided by multiple workstations and servers containing integrated graphics processing units (GPUs) (Quadro GV100, TITAN V (Volta), GTX 1080 Ti, and GTX TITAN X: Nvidia, Santa Clara, CA) [31]. These systems enable technologies (e.g., Clara Medical Imaging [32]) with distributed functionality as a node cluster for processing optimization (e.g., Linux-based Kubernetes v1.8 for managing variable-generation GPUs [33]). For this project, dedicated servers sited in the local data center, while functioning within the medical center network, supported primarily either algorithm development (DGX systems: Nvidia, Santa Clara, CA) (aka “Training Servers”) in phase 1 or algorithm inference processing (DGX1 system: Nvidia) (aka “Inference Server”) in phase 2.

Phase 1: Algorithm Development and Preliminary Evaluation

Study Population

With local IRB approval, a retrospective electronic-record search led to the identification of a standard-of-care experience (May/2013–October/2018) with CCTA for new-onset chest pain in non-hospitalized patients. This experience was reflected in clinical CCTA examination reports which were generated by either of the following: (1) 1 of 8 interpreting physicians with verification of American College of Cardiology Foundation (ACCF)/American Heart Association (AHA) Cardiovascular Computed Tomography Experience (CCTE) at level-2 [34], but more often by (2) 1 of 2 interpreters with ACCF/AHA CCTE at level-3 [34], as well as ACR Certification of Advanced Proficiency in Cardiac Computed Tomography (COAP) [35] and validation by the Certification Board of Cardiovascular Computed Tomography (CBCCT) [36].

From this derived collection of de-identified patient data, non-consecutive CCTA reports were randomly selected until the predetermined 500-case (including 200 cases from a prior technology validation report [30]) and 50% disease prevalence targets were reached; the disease prevalence in the phase 1 study population was deliberately inflated in order to minimize issues related to sample size and model Training. Due to their potential to confound AI algorithm production, this selection process purposely excluded cases with (1) retained metallic materials related to prior CAD intervention (e.g., stents, bypass graft markers, sternal sutures) or (2) other internal treatment-related materials (i.e., surgical or implanted devices, such as prosthetic valves or pacemakers). However, the image characteristics of the remaining cases were otherwise relatively “real-world” with the inclusion of all image qualities (e.g., equivocal from motion-related artifact) and complicating coronary anatomies (e.g., anomalous origin/course or myocardial bridges).

The final categorization of each selected case (i.e., coronary artery tree per chest pain patient) as: (1) “Diseased,” with non-obstructive (i.e., at most insignificant stenosis < 50% everywhere) or obstructive (stenosis ≥ 50% anywhere) atherosclerotic plaque formation, or as (2) “Normal” (totally free of coronary atherosclerosis), required CCTA examination review/re-review by a level-3 ACCF/AHA CCTE, ACR COAP, and CBCCT investigator [34–36].

Ultimately, the desired study population (age: range 20–91/average 50 years old; gender 52% male), consisting of 250 confirmed Diseased cases (125 non-obstructive and 125 obstructive) and 250 confirmed Normal cases, was established (Table 1). All CCTA examinations had been performed in ED-based (91%) or ambulatory clinic-based (9%) settings; cases of ACS (very uncommon at our institution) were unintentionally unrepresented.

Table 1.

Phase 1: algorithm development and preliminary evaluation

		Vessel-based algorithm data subsets
Phase 1 study population (n = 500)		Training	Validation	Testing
Expert “ground truth” case class	Diseased (n = 250 including 125 non-obstructive / 125 obstructive)	150	50	50	-
	Normal (n = 250)	150	50	50	-
		Algorithm-development operations
Level of operation		Training	Validation	Testing	Restored-case testing
Vessel extraction (MPV display)	Plaque-annotated (n = 2539)	2364*	50	125	-
	Atherosclerosis-free (n = 3420)	2304	50	1066	-
Case	Diseased				50 # &
	Normal				50
Applied to MPVs from vessel-centerline extractions	Resulting from sixfold DA of 394 Plaque-annotated MPV per extraction	*	-	-	-
	Use of single MPV per extraction	-	X	X	-
	Inclusion of extraction MPVs of atherosclerosis-free branches with “upstream” plaque annotations	-	-	-	# (n = 538 added to complete Diseased-case image datasets)
	Inclusion of completely atherosclerosis-free extraction MPVs	-	-	-	& (n = 353 added to complete Diseased-case image datasets)

The vessel-based algorithm development (including Training, Validation, and Testing) utilized mosaic projection views (MPVs) of vessel-centerline extractions following the courses of primary coronary arteries or branches which themselves did vs did not demonstrate atherosclerosis. If atherosclerosis was present along the vessel extraction, its extent was manually annotated (plaque-annotated) as nonobstructive or obstructive plaque [27]. During Training, sixfold data augmentation (DA) of plaque-annotated MPVs provided increased representations of atherosclerotic vessels and ultimately improved Training subset balance [30]; a single MPV without DA was used in both Validation and Testing of algorithm performance. Case-level Testing was conducted following the restoration of each Diseased case by the addition of previously excluded MPVs

Coronary Image-Data Curation

Image-data curation included ground truth annotation of the extent of any identified coronary artery atherosclerotic plaque (Appendix 2) [27, 30] by the most experienced investigator who has 35 years of practice in cardiovascular imaging, in addition to the aforementioned CCTA qualifications. As previously described in detail, this annotation process included the following: (1) automatic centerline delineation of all detectable coronary artery arteries and branches; (2) manual assessment of atherosclerotic-plaque distribution along the course of each vessel based on interactive multi-planar reformations, thin maximum-intensity projections, and vessel-short-axis tomographic stacks; and (3) color-coded segmental assignment of the associated basic degree of luminal stenosis (non-obstructive vs obstructive) when plaque was identified; the lack of any such labeling served as the local indicator of total absence of plaque [27, 30].

Algorithm Development

Establishing Classification

In phase 1, the development of the algorithm for coronary vessel classification on the Training Servers was initiated by the establishment of a 3:1:1 distribution [37] of randomly selected study population cases for Training (n = 300), Validation (n = 100), and Testing (n = 100); each of these subsets demonstrated a 1:1 Diseased-to-Normal case ratio (Table 1). Next, depending on the assigned subset and vessel centerline-extraction condition (i.e., “plaque-annotated” vs “atherosclerosis-free”), corresponding types of mosaic projection views (MPVs), produced as previously validated for DA [30], were selectively recruited from cases, as follows, in order to establish data partitioning for vessel-based algorithm Training, Validation, and Testing.

For the 250 Normal cases, the MPVs of all vessel extractions were included in all three phases of AI algorithm development. On the other hand, for the 250 Diseased cases, while MPVs of all plaque-annotated extractions were used, the following were excluded for algorithm development in order to avoid both unrecognized similarities with atherosclerotic vessels and further dataset imbalance favoring Normal [30]: (1) extraction MPVs of atherosclerosis-free branches with “upstream” plaque annotations and (2) completely atherosclerosis-free extraction MPVs (Table 1).

In order to promote further the relative balance of diseased vs non-diseased for algorithm development, a DA strategy was used to randomly permute and augment sixfold the 394 plaque-annotated MPVs in the Training subset [30], thereby creating 2364 (i.e., 394 × 6) unique representations of atherosclerotic vessels (Table 1). Traditional DA routines (e.g., random rotation) were also performed [38–41].

On the other hand, DA was not performed on (1) atherosclerosis-free MPVs of Normal cases in the Training subset (n = 2304); (2) the entire Validation subset of MPVs (both atherosclerotic and non-atherosclerotic); and (3) the entire Testing subset of MPVs (both atherosclerotic and non-atherosclerotic). In the Validation subset, only one MPV was randomly selected per Diseased case (n = 50 MPVs) and per Normal case (n = 50 MPVs), in order to maintain balance and avoid bias in the final algorithm selection [30]. In the vessel extraction-level Testing subset, a single MPV per plaque-annotated extraction (n = 125 MPVs), as well as for each atherosclerosis-free extraction (n = 1066 MPVs), was used (Table 1).

Testing on the case level in phase 1 (i.e., preliminarily evaluating the algorithm in differentiating between cases with vs without coronary atherosclerosis) was performed after full restoration of the Diseased cases. Restoration was accomplished by inclusion of the aforementioned initially excluded MPVs (n = 538 atherosclerosis-free with “upstream” plaque annotations, and n = 353 completely atherosclerosis-free), for completion of Diseased-case image-data sets (Table1).

Modeling

A transfer learning (TL) strategy [38–41], which as been previously validated [30], was used to help prepare the AI model for CCTA classification (Appendix 3).

Phase 2: Algorithm Trialing and Pre-deployment Assessment

Study Population

With local IRB approval, the aforementioned GUI [27, 28] was installed on a standard Picture Archiving and Communication System (PACS) workstation supporting commercially available advanced image-evaluation software (syngo.via: Siemens Healthineers, Forchheim, Germany). This allowed Trialing of the developed algorithm in a fashion simulating “real-world” use on a recent standard-of-care series (January/2019–April/2020) of 100 consecutive suitable patients with histories of ED-indicated CCTA evaluation of new-onset chest pain in the setting of low-to-intermediate pretest CAD probability. Out of the total of 106 case records accessed to achieve this goal, only 6 cases were excluded based on (1) inadequate image quality for AI inference generation although reported as completely free of atherosclerosis by the interpreting physician (n = 4, representing an algorithm workflow-completion level of 100/104 = 96%), (2) CT scanner malfunction resulting in incomplete CCTA examination (n = 1), and (3) presence of a potential AI confounder (n = 1 with implanted defibrillator).

In this phase 2 series, each clinical CCTA-examination underwent review/re-review by a level-3/COAP/CBCCT investigator [34–36] for final categorization as (1) Diseased, with non-obstructive or obstructive atherosclerotic plaque formation, or (2) Normal. Finally, this 100-consecutive-case study population from the ED (age: range 22–70/average 45 years old; gender 58% male) consisted of 28 confirmed—Diseased and 72 confirmed—Normal cases, reflecting a disease prevalence of 28% atherosclerotic cases (22 non-obstructive and 6 obstructive); as in phase 1, there were no confirmed ACS cases in this series.

Algorithm Trialing

During algorithm Trialing in phase 2, the CCTA image-data of each of the 100 study cases was initially reviewed by the interpreting investigator, as normally done in standard-of-care interpretations, using the aforementioned advanced clinical viewer (i.e., syngo.via) for the identification of the cardiac phase with perceived optimal diagnostic quality (often end-diastolic at 65–75%). Once this phase was identified, the following basic steps occurred: (1) the optimal image-data volume was manually routed by the interpreter (by single mouse click) to the Inference Server for the previously described automatic vessel-centerline processing prior to algorithm assessment; (2) an algorithm inference was promptly returned (55–80 s later, depending on processing and/or network demands) to the GUI for a vessel-by-vessel graphic overlay indicating any algorithm-detected atherosclerosis on the displayed coronary artery tree (i.e., composite 3D display of centerline-extracted coronary vessels); and (3) interpreter acceptance or rejection of inference results, with opportunity for concurrent CCTA image re-examination and corroboration using the standard multi-planar reformation and thin-maximum-intensity projection displays of the GUI, as needed based on indicated likelihood of coronary atherosclerosis [42] (Appendix 4).

Statistical Analysis

In phase 1, Testing of the performance of the AI algorithm for vessel extraction classification (plaque-annotated vs atherosclerosis-free) in the 500-random case study population was evaluated using the area under the receiver operating characteristic curve (AUC-ROC) methodology [43]. For standard analysis of Testing results (especially NPV), confusion matrices were used to reflect expert “ground truth” determinations vs AI algorithm predictive determinations of atherosclerosis presence vs absence at both the vessel extraction level and case level [38–41]. As previously mentioned, case-level Testing was performed after Diseased case restoration by addition of previously excluded: (1) extraction MPVs of atherosclerosis-free branches with “upstream” plaque annotations and (2) completely atherosclerosis-free extraction MPVs.

In phase 2, algorithm Trialing was performed on the 100-consecutive-case study population from the ED, using confusion matrices to reflect expert determinations vs algorithm determinations of atherosclerosis presence vs absence on a per-case basis.

Results

Phase 1: Algorithm Development and Preliminary Testing

In the phase 1 500-case study population, expert ground truth was established for 3420 atherosclerosis-free vessel-centerline extractions from the 250 Normal cases, as well as for 569 (i.e., 394 + 50 + 125) plaque-annotated extractions from the 250 Diseased cases (Table 1). While all 3989 (i.e., 3420 + 569) vessel extractions were converted to MPVs for Training, Validation, and Testing during algorithm development, only the 394 plaque-annotated MPVs in the Training subset underwent DA (creating 2364 different representations of atherosclerotic arteries/branches) [30].

The optimized algorithm developed in phase 1 revealed potentially very high performance at the vessel extraction level, with an AUC-ROC of 0.96 measured (Fig. 1). When the Testing results were reviewed further (Table 2), the confusion matrix statistics confirmed the potential strength of the algorithm for total exclusion of atherosclerosis per vessel extraction, with calculated high NPV of 98.21% (95% CI 97.28–98.83%) and Specificity of 92.68% (95% CI 90.95–94.17%).

Fig. 1.

Area under the receiver operating characteristic curve (AUC-ROC) curve from Testing of vessel-based algorithm performance

Table 2.

Statistical results from Testing at vessel extraction level

Phase 1 algorithm performance testing: vessel extraction level (decision threshold 0.5)
	Formula	Confusion Matrix Values	Calculated Value	95% CI
Sensitivity	$\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$	107/(107 + 18)	85.60%	78.20 to 91.24%
Specificity	$\frac{\mathrm{TN}}{\mathrm{FP}+\mathrm{TN}}$	988/(78 + 988)	92.68%	90.95 to 94.17%
PPV	$\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$	107/(107 + 78)	57.84%	52.27 to 63.22%
NPV	$\frac{\mathrm{TN}}{\mathrm{FN}+\mathrm{TN}}$	988/(18 + 988)	98.21%	97.28 to 98.83%
Accuracy	$\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}+\mathrm{TN}}$	(107 + 988)/(107 + 78 + 18 + 988)	91.94%	90.25 to 93.42%

FN false negative, FP false positive, NPV negative predictive value, PPV positive predictive value, TN true negative, TP true positive

Nevertheless, with the ultimate goal being to assess AI in screening for the absence of atherosclerosis anywhere within a chest pain patient’s coronary artery system on CCTA, case-level evaluation was the major focus. To that end, the developed algorithm was applied in phase 1 to per-case evaluation in the Testing subset (including 50 restored Diseased cases and 50 Normal cases) (Table 3). The results revealed a case-level NPV of 95.24% (95% CI 73.61 to 99.31%) and Specificity of 40.00% (95% CI 26.41 to 54.82%). There was false-negative (FN) determination in only 1 case which demonstrated a single atherosclerotic narrowing of mild severity (25–49% stenosis) immediately after the origin of the vessel (Fig. 2).

Table 3.

Statistical results on preliminary testing of randomly selected cases

Phase 1 algorithm performance Testing: Randomly Selected-Case Level
	Formula	Confusion matrix values	Calculated value	95% CI
Sensitivity	$\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$	49/(49 + 1)	98.00%	89.35 to 99.95%
Specificity	$\frac{\mathrm{TN}}{\mathrm{FP}+\mathrm{TN}}$	20/(30 + 20)	40.00%	26.41 to 54.82%
Disease prevalence	$\frac{\mathrm{TP}+\mathrm{FN}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}+\mathrm{TN}}$	(49 + 1)/(49 + 30 + 1 + 20)	50.00%	-
PPV	$\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$	49/(49 + 30)	62.03%	56.48 to 67.27%
NPV	$\frac{\mathrm{TN}}{\mathrm{FN}+\mathrm{TN}}$	20/(1 + 20)	95.24%	73.61 to 99.31%
Accuracy	$\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}+\mathrm{TN}}$	(49 + 20)/(49 + 30 + 1 + 20)	69.00%	58.97 to 77.87%

FN false negative, FP false positive, NPV negative predictive value, PPV positive predictive value, TN true negative, TP true positive

Fig. 2.

Single false-negative case-level determination during preliminary algorithm Testing A focal mild (24–49%) atherosclerotic stenosis (arrows) immediately after the origin of the left circumflex coronary artery, with bifurcation of the left main coronary artery, is shown on the volume-rendered (upper left) image and multi-planar reformations (upper right and lower left). A VGG16-based saliency map (lower right) superimposed on the MPVs (unique 18 × 1 matrices of arranged 2D projections—newer version) of this case demonstrates a focal “hot spot” region indicating a discriminative area by the algorithm (corresponding to manually demarcated segment of mild atherosclerosis [yellow]) along projection images from the vessel extraction. However, this “hot spot” went under-represented due to the predicted probability for plaque of only 0.02, below the diagnostic threshold used (i.e., 0.5)

Phase 2: Algorithm Trialing and Pre-deployment Assessment

When the developed algorithm was applied to per-case Trialing in the phase 2 study population (including 28 Diseased cases and 72 Normal cases), the workflow experience was operationally successful (Fig. 3) and the results resembled phase 1 preliminary case-level results (Table 4). NPV of 94.74% (95% CI 82.27 to 98.59%) and Specificity of 50.00% (95% CI 37.98 to 62.02%) were calculated at the case level in phase 2. The results revealed FN determination in only two cases; both demonstrated single-vessel (both left anterior descending coronary artery) non-obstructive atherosclerosis (< 25% and 25–49% stenosis) immediately beyond branch origins (Fig. 4). Although not leading to case-level FN determinations, there were 5 additional vessel extraction-level FN determinations (3 right coronary artery, including 2 main portion and 1 posterior descending branch; and 2 left circumflex coronary artery, including 1 main portion and 1 obtuse marginal).

Fig. 3.

Algorithm inference feedback on detected atherosclerosis. Non-obstructive atherosclerosis (yellow brackets) spanning the levels of the left main coronary artery (LMCA) to the proximal-mid left anterior descending (LAD) coronary artery is demonstrated on multi-planar-reformation and thin-maximum-intensity projection images from a commercial clinical viewer (Above). The GUI display (Below) supports comparable interactive-viewing capabilities, as well as a graphic overlay of inference results from plaque detection (red) on the vessel-centerline extracted coronary tree (gray). Algorithm-based detection of atherosclerosis in the LMCA, proximal LAD (from LMCA bifurcation to origin of first diagonal branch [D1]) and mid LAD (from D1 original to origin of second diagonal branch [D2]) conveys distally to the “downstream” non-diseased distal LAD and D1-D3. Without disease detected in the left circumflex coronary artery (LCx) or right coronary artery (RCA), no overlays are applied to their vessel distributions

Table 4.

Statistical results from Trialing on consecutive cases

Phase 2 algorithm performance trialing: consecutive-case level
	Formula	Confusion matrix values	Calculated value	95% CI
Sensitivity	$\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$	26/(26 + 2)	92.86%	76.50 to 99.12%
Specificity	$\frac{\mathrm{TN}}{\mathrm{FP}+\mathrm{TN}}$	36/(36 + 36)	50.00%	37.98 to 62.02%
Disease Prevalence	$\frac{\mathrm{TP}+\mathrm{FN}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}+\mathrm{TN}}$	(26 + 2)/(26 + 36 + 2 + 36)	28.00%	-
PPV	$\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$	26/(26 + 36)	41.94%	35.93 to 48.19%
NPV	$\frac{\mathrm{TN}}{\mathrm{FN}+\mathrm{TN}}$	36/(2 + 36)	94.74%	82.27 to 98.59%
Accuracy	$\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}+\mathrm{TN}}$	(26 + 36)/(26 + 36 + 2 + 36)	62.00%	51.75 to 71.52%

FN false negative, FP false positive, NPV negative predictive value, PPV positive predictive value, TN true negative, TP true positive

Fig. 4.

False-negative case-level determinations during algorithm Trialing false-negative determinations were observed in only 2 cases; both demonstrated single-vessel non-obstructive atherosclerosis of the left anterior descending (LAD) coronary artery immediately beyond a branching point. In case A (above), curved and straight multi-planar reformations show minimal (< 25%) atherosclerotic stenosis (red arrows) in the LAD after its origin from the left main coronary artery; an adjacent very small ramus intermedius branch is noted. In case B (below), straight and curved multi-planar, as well as volume rendered (right), reconstructions show focal atherosclerosis causing mild (25–49%) stenosis (yellow arrows) of the mid LAD just beyond the origin of the first diagonal branch

Discussion

Building on a shared view that non-obstructive coronary artery atherosclerosis detected during CCTA evaluation of chest pain should not be ignored [44, 45], this work was focused on the added diagnostic value of AI in assisting the qualified interpreting physician. The importance of totally excluding coronary atherosclerosis aided by AI, particularly in typical ED chest pain patients of low-to-intermediate pretest CAD probability, has implications beyond facilitating discharging to home, operational efficiencies, and financial benefits; failure to detect even mild coronary plaque formation can leave the patient in jeopardy of a future MACE [12, 13]. Therefore, the confident confirmation of a completely normal-appearing CCTA to help determine minimal risk in an ED chest pain patient should be considered to be an important objective along with the detection of CAD.

While the current project depended heavily on previously described work related to the GUI development for image display and annotation [27] and to the validation of methods for image-data processing and algorithm preparation [30], it represents significant stepwise progress toward clinical implementation including future support of the continuous learning inherent in AI [38–41]. Relative to the prior reports [27, 30], the contributions made by the current project include the following: (1) compared to the number of coronary artery representations (813 diseased and 953 normal) previously used to demonstrate the value of TL and DA towards algorithm preparation [30], this project incorporated 3.1 times more (2539 diseased) atherosclerotic vessel representations and 3.6 times more (3420 normal) atherosclerosis-free vessel representations in order to support the development of a more robust/less-brittle algorithm for vessel evaluation; (2) unlike the previous validation project [30], this project extended its scope to evaluating algorithm performance at the case level (i.e., the entire coronary artery tree per chest pain patient) in 2.5 times more study cases (500 cases, compared to 200); and (3) this project included a simulated case-based clinical implementation of the developed algorithm in another study population (additional 100 cases) with demonstration of a functional workflow, including graphic display of inference results to the interpreter within a viewer resembling a clinical viewer, all incorporated into same GUI used for image annotation [27].

Based on the results of this study, AI could soon play a significant role in assisting the qualified interpreting physician in achieving this objective of coronary atherosclerosis exclusion. First, they indicate a very high performance of the developed AI algorithm at the vessel level (i.e., AUC-ROC = 0.96); the small number of FN determinations resulted in very high NPV (i.e., 98%) and high Specificity (i.e., 93%) per vessel extraction. Multiple prior validation studies have also demonstrated a strong ability of CCTA to exclude atherosclerosis per vessel [4, 5, 46, 47], but this has necessitated labor-intensive manual analysis which may be difficult to support, for the aforementioned reasons, within an often stressful and distracting ED environment. Regardless of the interpretation environment, independent sources of human error in segmental CCTA interpretation, resulting in under-appreciation of atherosclerosis (i.e., FN determinations), are recognized and are most prevalent in the left circumflex coronary artery system; they include the following (in decreasing order): tortuosity, small diameter, limited mean luminal opacification, absence of plaque calcification, and vein crossing [48]. In this project, FN determinations by the AI algorithm were most often noted immediately beyond branch points.

More pertinent to the practical goal of totally excluding coronary atherosclerosis per chest pain patient for their clinical disposition and treatment planning in the ED, our AI algorithm demonstrated very high NPV (95%) in both phase 1 (disease prevalence of 50%) and phase 2 (disease prevalence of 28%) study populations, thereby indicating that the post-test probability of an ED chest pain patient having no coronary atherosclerosis, given a completely atherosclerosis-free AI-evaluated CCTA result, approaches absolute, with low likelihood of a FN determination [49]. On the other hand, our previously described high Specificity per-vessel extraction was not observed in either case population (40% and 50%, respectively), reflecting the equivocal ability of our algorithm to alone correctly designate an ED chest pain patient without coronary atherosclerosis as being completely atherosclerosis-free, with significant potential for false-positive determinations warranting thorough manual evaluations for suspected plaque formations [49]. Even when CCTA examinations are manually interpreted, decline in Specificity per-patient, compared to per-segment or per-vessel basis, has been reported [4, 5, 46, 47].

Hence, neither the views of the authors nor these results endorse the use of the developed algorithm for autonomous AI-based CCTA interpretation; this was not anticipated, especially when it is recognized that expert review is still essential due to potential for other symptom-producing abnormalities, such as congenital coronary artery anomalies. However, the results do help validate its use as a pre-view or post-view assistant to the qualified interpreting physician in preventing mistakes of diagnostic omission that might otherwise lead to unrecognized risks of MACE by failing to detect coronary atherosclerosis prior to ED discharge without plans for further evaluation or treatment. In addition, to that end, the results indicate that our algorithm workflow is robust with (1) reliable performance when image quality is not poor (100 of 104 = 96% of real-world cases without obvious confounders demonstrated algorithm workflow completion with inference feedback), (2) acceptable turnaround time (55–80 s) of graphic inference results, and (3) post-inference opportunity for the interpreting physician to consider and corroborate inference results by easy interactive re-review of CCTA images within the same clinically capable GUI.

Limitations

Limitations in Study Population

Unlike prior reports on the use of AI for anatomic/physiologic detection or characterization of known coronary artery atherosclerotic plaque [20–23], this project focused on the AI-based exclusion any atherosclerosis. Apparent deficiencies in independent (not a goal) detection of atherosclerosis (per-case positive predictive values of 42–62%) are likely related to both phase 1 and phase 2 study populations having included (1) relatively “real-world” examinations spanning all image qualities (rendering workflow incomplete in 4 phase 2 cases with very poor imaging) and (2) a high prevalence of vessel extractions which were atherosclerosis-free or had non-obstructive (often very mild) atherosclerosis, partially ameliorated in phase 1 but left in “real world” status in phase 2. In order to correct for this latter issue for algorithm development in phase 1, disease prevalence was purposefully inflated to 50% during case selection in order to minimize the sample-size concern; consequently, the calculated sample-size need (i.e., 146 cases) was reduced to a level surpassed by the combined Training-Validation subsets (i.e., 400 cases) and approximated by the Testing subset (i.e., separate 100 cases). Data balancing was also employed to ensure disease representation throughout the vessel-based algorithm development process [30].

Limitations in Technology

A variety of technique options are available for vessel-centerline extraction, and each has relative inherent strengths and deficiencies [50], beyond practical restrictions related to poor image quality, reduced intravascular enhancement, and severe vessel tortuosity. Technical limitations of the fully automatic centerline methodology applied in this study have been described [27, 30], and cumulatively, these limitations may have adversely affected vessel extraction and ultimately algorithm development.

In addition, known shortcomings with centerline recognition at vessel bifurcations may have contributed to the occurrence of missed AI algorithm detection of focal atherosclerotic plaques in this study [51]. Nevertheless, the three depicted instances leading to FN cases involved failed detection of non-obstructive degrees of atherosclerosis, regarded to be “negative” in many prior studies of CCTA in ED chest pain-patients [5]. Also, planned work to support the continuous learning process of our AI algorithm (in contrast to traditional computer aided diagnosis [38–41]) is expected to improve its future performance, thereby reducing the potential for such FN inference results.

Lastly, there are impending clinical implementation issues related to the current algorithm workflow, including its (1) requirement for manual selection of the cardiac phase with optimal image quality which may eventually be overcome with technical options (e.g., automatic phase selection [52], optimization via motion correction [53]); (2) inability to graphically display local regions of highest probability for plaque detection, or provide for probability threshold adjustment, which may be overcome with 3D and/or neural network-dependent approaches in the future; (3) ongoing dependencies on interfacing of commercially based (e.g., centerline technology) and locally developed approaches (e.g., MPV production), also potentially benefitting from the same approaches.

Limitations in Study Design

The current study successfully simulated acceptable workflow and yield associated within a real-world clinical implementation of the developed algorithm using the attributes of the GUI on a case-by-case basis. However, it did not objectively assess the case-by-case value added (e.g., detection of missed plaque, time saving) from use of the algorithm, compared to an AI-unassisted interpretation, prospectively within a clinical experience. Although this assessment had originally been planned, failure to secure local IRB approval owing to the impracticality of fulfillment of expectations for direct individual-patient consenting (e.g., interpreting physician unable to consistently interface with patient due to distance or time of day) prior to processing the CCTA data for algorithm assessment precluded the inclusion of this aspect of the investigation; however, because of its importance, it will be revisited in future phases of this research.

Conclusion

This report describes the methodical development of an AI algorithm, and supporting workflow, which appears to have reached a level of performance conducive to clinical deployment for assisting a fully qualified interpreting physician in confirming the total absence of any coronary artery atherosclerosis on CCTA during a chest pain patient evaluation in the ED setting. The AI application’s validated combination of (1) very high NPV at the case level, (2) efficient return of results from algorithm inference, and (3) graphic overlay of inference results on the coronary artery tree display, with allowance for subsequent interactive re-examination of CCTA images within the same “user friendly” GUI, promotes its readiness for clinical implementation. It also anticipates the expectation for support of continuous algorithm improvement, inherent in AI. Provision of such AI-based assistance to the qualified interpreting physician in confidently discounting coronary atherosclerosis as the cause of chest pain in an ED chest pain patient could help to avoid under-diagnosis or under-treatment due to human error.

Abbreviations

ACCF

American College of Cardiology Foundation

ACS

Acute coronary syndrome

ACR

American College of Radiology

AHA

American Heart Association

AI

Artificial intelligence

AUC-ROC

Area under the receiver operating characteristic curve

CAD

Coronary artery disease

CBCCT

Certification Board of Cardiovascular Computed Tomography

CCTA

Coronary computed tomography angiography

CCTE

Cardiovascular computed tomography experience

COAP

Certification of Advanced Proficiency in Cardiac Computed Tomography

DA

Data augmentation

ED

Emergency department

FN

False-negative

GPU

Graphics processing unit

GUI

Graphical user interface

IRB

Institutional review board

MACE

Major adverse cardiac event

MPV

Mosaic projection view

NPV

Negative predictive value

PACS

Picture Archiving and Communication System

TL

Transfer learning

Appendix 1

Because all vessel-centerline extractions originated at the transition from the aortic root to either the left main coronary artery or right coronary artery (standard in commercial coronary artery image-processing systems), the sharing of an “up-stream” arterial segment between “downstream” segments of the same primary artery or its branches is common. In this study, while the identification of each vessel extraction was based on the vessel of termination, its classification as “plaque-annotated” or “atherosclerosis-free” was established by the presence or absence, respectively, of atherosclerosis anywhere along the vessel extraction, even when atherosclerosis was located only before the true origin of a branch (Fig. 5).

Fig. 5.

Classification of vessel-centerline extractions. Significant luminal narrowing (arrows) of the left anterior descending (LAD) coronary artery is demonstrated on both a volume-rendered image (above, with automatically extracted centerlines indicated on right) and curved multi-planar images along the vessel centerlines (below). The “upstream” atherosclerotic proximal LAD dictates the classification of both the “downstream” atherosclerosis-free mid-distal LAD and first diagonal branch (D1) as “plaque-annotated.” Hence, D1 is considered a diseased vessel course due to the proximal LAD atherosclerosis; for it to be considered atherosclerosis-free requires all segments proximal to the branch to also be free of atherosclerosis. On the other hand, the left circumflex coronary artery (LCx) and its first obtuse marginal branch (OM1) are entirely atherosclerosis-free along their vessel extractions (D2 = second diagonal branch; RI = ramus intermedius branch)

Appendix 2

The GUI-based manual annotation process involved regional color-coding of basic stenosis grade (i.e., non-obstructive vs obstructive) along the vessel courses as shown on rotatable 3D branching coronary artery trees, representing the combined display of the multiple vessel-centerline extractions per case [27] (Fig. 6).

Fig. 6.

GUI for ground truth annotation of coronary arteries and branches. The same case as represented in Fig. 5 helps to demonstrate the subsequent GUI-based image-data curation with ground truth annotation of coronary artery atherosclerotic plaque extent. This process involves regional color-coding of basic stenosis grade of the previously described significant stenosis (arrow) of the proximal left anterior descending (LAD) coronary artery; both non-obstructive (< 50% luminal narrowing [light pink]) and obstructive (≥ 50% luminal narrowing [dark pink]) portions of the plaque are shown as segmental coatings on the 3D branching arterial tree (D1 = first diagonal branch; LCx = left circumflex coronary artery; RI = ramus intermedius branch)

The annotated vessel extractions (i.e., plaque-annotated) were then converted to 2D mosaic projection view (MPV) displays in order to facilitate data augmentation (DA) of the size of the atherosclerotic-vessel component for Training in algorithm development [30]. DA is accomplished by re-ordering projections, thereby creating new MPVs per vessel used to augment the Training subset (Fig. 7). Through its improvements to both modeling and diseased vs non-diseased balance within the Training subset [37], this DA methodology enhanced AI algorithm development, especially when combined with transfer learning (TL) using pre-trained weighting [30, 37–41].

Fig. 7.

Mosaic projection view (MPV) representations of disease-annotated vessel extractions Three MPVs (unique 9 × 2 matrices of arranged 2D projections—original version [30]) of coronary vessel atherosclerosis with variable stenosis are displayed as color maps (right); corresponding plaques are shown on curved CCTA reconstructions (left) (LAD = left anterior descending, LCx = left circumflex, RCA = right coronary artery)

Appendix 3

Inception-V3 (https://cloud.google.com/tpu/docs/inception-v3-advanced) served as the base convolutional neural network. Model weights pre-trained on ImageNet (http://www.image-net.org/) were used for TL during algorithm Training [30, 38–41]. To refine modeling for CCTA datasets, the final Inception-V3 layer was replaced by a fully connected 1024-node layer with a rectified linear unit (25% dropout to avoid overfitting), followed by sigmoid output function for binary classification [30] (Fig. 8).

Fig. 8.

Modified Inception-V3 convolutional neural network used for algorithm development

Training utilized the Keras library (https://keras.io/) with a TensorFlow-1.8 backend (https://www.tensorflow.org/). Initial learning rate was 0.001 on a stochastic gradient descent optimizer (decay factor 1e−6, with momentum 0.900 and mini-batch size 8); re-training was terminated at 120 epochs. A binary cross-entropy loss function was monitored during Training, and the resulting model was saved only if there was improvement in the Validation accuracy [30].

Appendix 4

For phase 2, Clara software stack (https://developer.nvidia.com/clara-medical-imaging) was used because of its support of (1) an interface to Digital Imaging and Communications in Medicine (DICOM) standards (www.dicomstandard.org), (2) a durable deployment of Kubernetes (https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/#clusters-containing-different-types-of-nvidia-gpus), and (3) seamless modular integration with existing image-visualization/analysis tools, including a PACS or a viewer.

The integrated workflow for pre-deployment of the algorithm developed in phase 1 included the following steps: (1) the interpreting physician opened the desired CCTA image-data in a commercial clinical viewer (https://www.siemens-healthineers.com/en-us/medical-imaging-it/advanced-visualization-solutions/syngovia) in order to select the most diagnostically optimal cardiac phase to undergo evaluation by the algorithm; (2) the selected volume was manually forwarded to the DICOM server from where it was sent to the server supporting the GUI (https://www.mevislab.de/) for coronary artery image-processing (including vessel-centerline extraction followed by production of straightened multi-planar reformations (MPRs); (3) following conversion, straightened-MPRs were sent to a Clara Deploy DICOM Adapter (https://docs.nvidia.com/clara/deploy/ngc/DicomAdapter.html) which monitors incoming DICOM images and initiates classification by the algorithm; (4) Clara Deploy platform, hosting a TensorRT Inference Server (https://docs.nvidia.com/deeplearning/sdk/triton-inference-server-guide/docs/index.html), is prompted to make predictions; (5) inference results are updated in the database (https://www.mongodb.com/), signaling readiness for feedback to the interpreting physician; and (6) inference probability values exceeding the threshold (0.5) are graphically displayed as overlays on the coronary artery tree displayed by the GUI.

Using a continuous model integration strategy, an additional model can be evaluated by updating the CLARA Deploy server, via the backend server without workflow disruption, for future selection by the interpreting physician. The application described in phase 2 and pre-deployed for simulated clinical use is illustrated below (Fig. 9).

Fig. 9.

Algorithm pre-deployment architecture; while blue components reflect a traditional radiology workflow, orange components are enhancements for AI algorithm deployment with graphic inference feedback to the interpreting physician via the GUI

Author Contribution

All listed authors attest to have made significant individual contributions to this research in one or more of the following components:

• Study conception and design

• IRB approval process

• Image-data acquisition, management, or annotation

• Imaging data or basic clinical data collection or compilation

• Technical advising (e.g., image processing, AI methodology)

• Results organization, review, or statistical evaluation

• Report preparation, review, or submission

The first draft of the manuscript was written by RDW, and all authors commented on versions of the manuscript, as well as read and approved the originally submitted version. This major revision of the manuscript was written by RDW, with selective input from authors as needed.

Funding

This research did not receive specific grant funding. It was partially supported by unrestricted philanthropic funds, as well as by in-kind technical support via industry collaborations under separate institutional Master Research Agreements (MRAs), as follows.

1. Donation from the RDW.

2. MRA with Siemens Healthineers (technical support).

3. MRA with NVIDIA Corporation (technical support).

Data Availability

Without current IRB approval to release data or code, both remain securely stored at RDW. With publication of this report, and if given prior IRB approval, open-source release of de-identified patient data and algorithm details will be strongly considered.

Declarations

Ethics Approval

All procedures performed related to human participants were approved and conducted in accordance with the ethical standards of the Biomedical Sciences Institutional Review Board of the Office of Responsible Research Practices of RDW supporting the Human Research Protection Program (HRPP). The mission of the HRPP is to protect the rights, dignity, welfare, and privacy of the human subjects in all research conducted on behalf of the university (regardless of funding) by adhering to the principles outlined in the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research report entitled, Ethical Principles & Guidelines for the Protection of Human Subjects of Research (“Belmont Report”), and the regulations of the Department of Health and Human Services, Food and Drug Administration, and other applicable agencies. Consistent with the HRPP, all procedures involving human participants in this research were in accordance with the ethical standards of the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Consent

With approval of waiver of informed consent by the local IRB RDW, this research was conducted retrospectively based solely on standard-of-care imaging data and basic clinical data.

Conflict of Interest

All authors declare that they have no conflicts of interest or competing interests, beyond the aforementioned industry employment of two technical collaborators (TPO by Siemens Healthineers and AHH by NVIDIA Corporation).