Abstract
Purpose
This study aimed to determine if an integrated structured reporting (SR) tool incorporating results from artificial intelligence (AI) enabled coronary stenosis quantification (AI-CSQ), AI-enabled coronary plaque analysis (AI-CPA), and fractional flow reserve CT (FFRCT) can increase reader efficiency for coronary computed tomography angiography (CCTA), while reducing cognitive load.
Methods
Eighty CCTAs from patients with stable chest pain or symptoms suggestive of coronary artery disease (CAD) were analyzed by five independent readers from two clinical sites. CCTA datasets were used to generate combined reports for AI-CSQ, AI-CPA, and FFRCT. Each reader randomly interpreted 40 cases with the SR tool (which provided a prepopulated fully editable template) and 40 cases without the SR tool using site-specific clinical workflows (400 total cases). Reporting times from initiation of case interpretation through report signing were measured with and without the SR tool and stratified by CAD-Reporting and Data System (CAD-RADS) 2.0 category. Reader confidence was recorded using a 5-point Likert Scale, and cognitive load (e.g., working memory load) was determined using a 9-point scale, with comparison by SR tool use. Inter-reader agreement associated with CCTA interpretation with versus without the SR tool also was compared.
Results
Reporting time per case was 40.2% shorter with versus without the SR tool (6.0 ± 2.0 vs. 10.0 ± 3.3 min; p < 0.001), independent of CAD-RADS 2.0 category. Readers were more likely to report very confident reads with the SR tool than without (48% vs. 29%; p < 0.001). Cognitive load also was significantly lower with the SR tool than without (3.8 ± 1.2 vs. 6.1 ± 1.5; p < 0.001). Report agreement was significantly higher with versus without the SR tool (94.6%±2.3% vs. 45.3%±9.2%; p < 0.001).
Conclusion
The SR tool significantly improved reader efficiency and decreased work effort when interpreting comprehensive CCTA exams.
Keywords: Coronary computed tomography angiography (CCTA), Coronary stenosis, Fractional flow reserve CT (FFRCT), AI-enabled coronary plaque analysis (AI-CPA)
Introduction
Coronary computed tomography angiography (CCTA) has become the imaging test of choice to evaluate patients with stable chest pain symptoms for the suspected diagnosis of coronary artery disease (CAD) [1]. The ability to directly visualize the coronary arteries with CCTA reduces false positives, and the addition of fractional flow reserve CT (FFRCT) has been shown to improve the likelihood for revascularization [2]. Accordingly, CCTA adoption in the United States has increased [3], and this trend is expected to continue.
Due to the complexity of CCTA image interpretation [4], the accurate assessment and reporting of CAD severity is time consuming. An artificial intelligence (AI) informed coronary stenosis quantification (AI-CSQ) tool recently showed promise to reduce reporting times for stenosis assessment [5]. However, the addition of newer technologies such as of AI-enabled coronary plaque analysis (AI-CPA) and FFRCT lengthen the report(s) generated for each CCTA exam [6, 7].
Due to rising CCTA report complexity with AI-CPA and FFRCT, tools are needed to reduce reading times and perceived workload. Possibly, structured reporting (SR) tools with prepopulated and fully editable templates offer these solutions. Therefore, the aim of this study was to determine if an integrated SR tool can improve reader efficiency for contemporary CCTA reporting that includes AI-CSQ, AI-CPA, and FFRCT. To our knowledge, this is the first study to combine these components into a single SR tool for efficiency assessment during CCTA interpretation.
Methods
A total of 80 patients with stable chest pain or symptoms suggestive of CAD had CCTAs performed utilizing a broad spectrum of CT scanners and vendors. All patients provided written informed consent for de-identified data to be used for research purposes. CCTAs of acceptable image quality with complete information for AI-CSQ, AI-CPA, and FFRCT were used. Details of these techniques, including accuracy, have been previously published [5–7]. In brief, AI-CSQ identifies vessels with mild (30–49%), moderate (50–69%), or severe (≥ 70%) stenoses. AI-CPA quantifies coronary plaque by total plaque volume (TPV), calcified plaque volume, non-calcified plaque volume, and low-attenuation plaque volume. FFRCT provides physiological data for stenoses ≥ 30%, as well as at the terminus of the three main epicardial coronary arteries.
Five readers from two sites interpreted 80 CCTA datasets to generate combined reports for coronary stenosis severity using AI-CSQ, AI-CPA, and FFRCT. Calcium scoring was not performed or reported. The average reporting time for a single case was estimated to be 12.9 min based on sample cases completed by a single reader during study planning. It was hypothesized that reading times would be 10% faster with the SR tool. Assuming a standard deviation of 3.0 min, a sample size of 200 (40 cases read with the SR tool by 5 readers) cases would provide > 90% power to detect a significant difference in average reporting time.
Each reader interpreted 40 cases with and 40 cases without the SR tool (400 studies in total). The SR tool provided a prepopulated and fully editable template of the findings for each CCTA dataset. For cases interpreted without the SR tool, reporting reflected site-specific clinical workflows using non-standardized institutional report templates with basic prepopulated fields, and free-text dictation. To mirror clinical practice, studies were interpreted in batches of 10 CCTAs per day using local site software for post-processing and reporting. CCTAs were interpreted by both cardiologists (n = 3) and radiologists (n = 2), including Society of Cardiovascular CT Level 2 (n = 2) and Level 3 (n = 3) readers [8]. Case order and use of the SR tool was determined by a random number assignment of each study to ensure all 80 CCTAs were read both with and without the SR tool by different readers.
Baseline demographics were collected for patients included in this study, including CT scanner characteristics. The primary endpoint was the difference in average per case CCTA reporting time with the SR tool versus without. Reporting time was manually timed by each reader and included the total time for stenosis quantification using AI-CSQ, and reporting for AI-CPA and FFRCT. Differences in the average per case CCTA report time also were stratified by CAD-Reporting and Data System (CAD-RADS) 2.0 category. Sensitivity analyses were performed stratified by specialty (cardiology vs. radiology) and reader level (Level 2 vs. Level 3).
Secondary endpoints of reader agreement, reader confidence, and cognitive load were examined by use of the SR tool. Reader confidence was recorded using a 5-point Likert Scale. Cognitive load (e.g., working memory load) was determined using a 9-point scale, with scores ≥ 6 indicating high mental effort [9]. For each patient, reports with or without the SR tool were analyzed for text similarity (percentage agreement) via cosine similarity using a Python NLP tool (scikit-learn).
Categorical variables were reported as frequency and percentage while continuous variables were recorded as mean ± standard deviation. Statistical significance for categorical variables was tested using the chi-square method and for continuous variables using the student’s t-test procedure. Statistical significance was defined as p < 0.05. SAS Version 9.4 (Cary, NC) was used for all analyses.
Results
Baseline characteristics for the patients included in this study and CCTA findings are shown in Table 1. A total of 49 patients (61%) had stenoses ≥ 50% on AI-CSQ, and 55 patients (69%) had a FFRCT ≤0.80. Average TPV was 471 ± 523 mm3.
Table 1.
Demographics, Risk Factors, and Findings of CCTA Exams (N = 80)
| Characteristic | |
|---|---|
| Age, mean ± SD (years) | 62 ± 11 |
| Male, n (%) | 54 (68) |
|
Race White, n (%) Black, n (%) Asian, n (%) Other, n (%) Unknown, n (%) |
17 (21) 2 (3) 13 (16) 20 (25) 28 (35) |
| BMI, mean ± SD (kg/m2) | 26 ± 3.4 |
| Diabetes, n (%) | 6 (8) |
| Hyperlipidemia, n (%) | 45 (56) |
| Hypertension, n (%) | 38 (48) |
| Smoker, n (%) | 29 (36) |
|
Coronary Stenosis <50%, n (%) ≥50%, n (%) |
31 (39) 49 (61) |
|
AI-CPA TPV, mean ± SD (mm3) Mild, n (%) Moderate, n (%) Severe, n (%) Extensive, n (%) |
471 ± 523 16 (20) 16 (20) 32 (40) 16 (20) |
| FFRCT ≤ 0.80, n (%) | 55 (69) |
|
CT Scanner Model GE Revolution CT, n (%) Phillips Electric Brilliance 64, n (%) Phillips iCT 256, n (%) Siemens Somatom Definition AS, n (%) Siemens Somatom Definition Flash, n (%) Siemens Somatom Force, n (%) Toshiba Aquilion One, n (%) |
11 (14) 1 (1) 21 (26) 12 (15) 16 (20) 9 (11) 10 (13) |
AI-CPA = AI-enabled coronary plaque analysis; BMI=body mass index; CCTA=coronary computed tomography angiography; FFRCT=fractional flow reserve CT; SD=standard deviation; TPV=total plaque volume
Reporting time per CCTA was 40.2% shorter with the SR tool than without (6.0 ± 2.0 vs. 10.0 ± 3.3 min; 5.7, 6.3 ± 1.8, 2.2 vs. 9.6, 10.5 ± 3.0, 3.7; p < 0.001). A reduction in reporting time was observed with the SR tool regardless of CAD-RADS 2.0 category (Table 2). Significant reductions in reporting times were observed with versus without the SR tool for cardiologists (6.3 ± 2.0 vs. 10.8 ± 3.4 min; p < 0.001) and radiologists (5.5 ± 1.7 vs. 8.9 ± 2.8 min; p < 0.001). A similar reduction in reporting time per CCTA was observed with compared without the SR tool for Level 2 (6.0 ± 2.2 vs. 10.3 ± 4.0 min; p < 0.001) and Level 3 readers (5.9 ± 1.8 vs. 9.8 ± 2.8 min; p < 0.001).
Table 2.
CCTA Reporting Times With versus Without SR Tool by CAD-RADS 2.0 Category
| CAD-RADS 2.0 Category | Reporting Time in Minutes (Mean ± SD) (95% CI) |
||
|---|---|---|---|
| With SR Tool | Without SR Tool | p-value | |
| 1 |
4.0 ± 1.2 (3.4, 4.7) ± (0.9, 1.8) |
6.2 ± 2.0 (5.0, 7.3) ± (1.5, 3.3) |
0.003 |
| 2 |
5.1 ± 1.4 (4.7, 5.6) ± (1.1, 1.8) |
9.2 ± 3.1 (8.1, 10.2) ± (2.6, 4.1) |
< 0.001 |
| 3 |
6.4 ± 2.0 (6.0, 6.9) ± (1.7, 2.4) |
10.4 ± 2.8 (9.7, 11.0) ± (2.4, 3.4) |
< 0.001 |
| 4 |
6.4 ± 2.0 (5.9, 6.9) ± (1.7, 2.5) |
10.9 ± 3.5 (10.1, 11.8) ± (3.0, 4.2) |
< 0.001 |
| 5 |
6.5 ± 0.9 (5.7, 7.3) ± (0.6, 1.9) |
9.8 ± 2.9 (7.4, 12.3) ± (1.9, 5.9) |
0.015 |
CAD-RADS= Coronary Artery Disease-Reporting and Data System; CCTA=coronary computed tomography angiography; CI=confidence interval; SD=standard deviation; SR=structured reporting
Reader confidence was more likely to be reported as very confident when the SR tool was used versus without (48% vs. 29%; p < 0.001; Table 3). Average cognitive load was significantly lower with the SR tool than without (3.8 ± 1.2 vs. 6.1 ± 1.5; p < 0.001). A reduction in cognitive load was observed with the SR tool regardless of CAD-RADS 2.0 category (Table 4). Report agreement also was significantly higher with versus without the SR tool (94.6%±2.3% vs. 45.3%±9.2%; p < 0.001).
Table 3.
CCTA Reader Confidence With versus Without SR Tool
| Category | With SR Tool | Without SR Tool | p-value |
|---|---|---|---|
| Not at all confident, n (%) | 0 (0) | 0 (0) | < 0.001 |
| Not very confident, n (%) | 1 (< 1) | 1 (< 1) | |
| Neither, n (%) | 1 (< 1) | 6 (3) | |
| Fairly confident, n (%) | 102 (51) | 135 (68) | |
| Very confident, n (%) | 96 (48) | 58 (29) |
CCTA=coronary computed tomography angiography; SR=structured reporting
Table 4.
CCTA Cognitive Load With versus Without SR Tool by CAD-RADS 2.0 Category
| CAD-RADS 2.0 Category | Cognitive Load (Mean ± SD) (95% CI) |
||
|---|---|---|---|
| With SR Tool | Without SR Tool | p-value | |
| 1 |
2.6 ± 0.9 (2.2, 3.1) ± (0.7, 1.4) |
4.3 ± 1.1 (3.6, 4.9) ± (0.8, 1.8) |
< 0.001 |
| 2 |
3.1 ± 0.9 (2.8, 3.3) ± (0.7, 1.1) |
5.5 ± 1.2 (5.1, 5.9) ± (1.0, 1.6) |
< 0.001 |
| 3 |
4.0 ± 1.2 (3.8, 4.3) ± (1.0, 1.4) |
6.4 ± 1.5 (6.0, 6.7) ± (1.2, 1.7) |
< 0.001 |
| 4 |
4.2 ± 1.2 (3.9, 4.5) ± (1.0, 1.5) |
6.6 ± 1.4 (6.3, 6.9) ± (1.2, 1.6) |
< 0.001 |
| 5 |
4.3 ± 1.1 (3.3, 5.3) ± (0.7, 2.5) |
6.3 ± 1.7 (4.9, 7.6) ± (1.1, 3.4) |
0.021 |
CAD-RADS= Coronary Artery Disease-Reporting and Data System; CCTA=coronary computed tomography angiography; CI=confidence interval; SD=standard deviation; SR=structured reporting
Discussion
In this study, the use of a SR tool for CCTA interpretation significantly reduced reporting times, while improving reader confidence and reducing perceived cognitive load. CCTA reports reflected contemporary exams and included stenosis quantification by AI-CSQ, AI-CPA, and FFRCT. Overall, the findings of this analysis suggest a unique role for SR tools to improve CCTA reader efficiency.
To our knowledge, this is the first study to explore the role of a single technology to improve the reporting times for all components of a contemporary CCTA exam. CCTA stenosis quantification times have significantly improved with AI-CSQ [5]. However, similar tools incorporating AI-CPA and FFRCT have not been explored. AI-CPA has shown great benefit in changing preventive strategies to reduce cardiovascular risk through increased prescriptions for lipid-lowering agents [10], and FFRCT has demonstrated great benefit by appropriately identifying patients who benefit from referral for invasive coronary angiography and revascularization [2]. The incorporation of AI-CPA and FFRCT lengthens physician reporting time, as more data must be reported. With the rising demand for CCTA, AI-CPA, and FFRCT [3], SR tools will be needed to improve clinical workflows for cardiac imaging specialists.
The reduction in reading times and perceived cognitive load likely is explained by reduced manual data entry and enhanced mental organization during reporting with the prepopulated and fully editable SR tool. Additionally, the improvement in inter-reader agreement with the SR tool is due to the standardized nature of the tool rather than AI accuracy of the individual components. High-volume CCTA centers and growing practices with readers of varying skill levels will benefit from the incorporation of SR tools to reduce reading times and perceived cognitive load during CCTA interpretation. Standardized imaging reports through SR tools also will increase report clarity to provide clear and direct answers to clinical questions for referring providers.
There are several limitations of this study. CCTA reporting times reflected the assessment and reporting for coronary stenosis severity by AI-CSQ, AI-CPA, and FFRCT. Longer reporting times would be expected if non-coronary structures were evaluated. Similarly, FFRCT was reported for all studies regardless of stenosis severity, and this did not reflect clinical practice. The accuracy of the tools used in this study (AI-CSQ, AI-CPA, and FFRCT) was beyond the scope of the current study. However, the accuracies of these technologies have been reported previously [5–7]. The amount of editing performed by readers with the SR tool was not captured to fully understand if time savings came from the SR template or the accuracy of the AI-enabled data included. Race was missing for several of the CCTA scans included in this analysis, and this limits the study’s generalizability to diverse populations. Lastly, this study was conducted with only five readers from two sites and reporting times possibly are not generalizable to larger reader groups.
In conclusion, this analysis demonstrated the ability of a SR tool to greatly improve the reporting times for complete CCTA exams found in contemporary practice. As CCTA becomes more widely available, SR tools are greatly needed to improve reader efficiency and reduce workload burden. Future studies also are needed to measure the impact of this SR tool on real-world clinical workflows and patient outcomes.
Author contributions
All authors contributed to the study conception and design. Material preparation and data collection were performed by WO, HG, MC, CS, MM. Study analyses were performed by NN. The first draft of the manuscript was written by WO and all authors commented on previous versions of the manuscript. All authors read and approved of the final manuscript.
Funding
This work was supported by research support from Heartflow.
Data availability
The datasets generated and/or analyzed during the current study are not publicly available due to proprietary restrictions.
Declarations
Ethics approval and informed consent
This is a secondary analysis of de-identified data. The institutional review board at Cone Health confirmed that no ethics approval is required. In accordance with the Declaration of Helsinki, all patients provided written informed consent for de-identified data to be used for research purposes.
Competing interests
Heartflow is the sponsor of the SMART-REPORT study. Wesley T. O’Neal has served as a speaker for Heartflow. Autumn Birch receives salary and equity in Heartflow and is a full-time employee. Sarah Mullen receives salary and equity in Heartflow and is a full-time employee. Nicholas Ng receives salary and equity in Heartflow and is a full-time employee. Michael Morris has served as a speaker for Heartflow.
Footnotes
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Gulati M, Levy PD, Mukherjee D, Amsterdam E, Bhatt DL, Birtcher KK, Blankstein R, Boyd J, Bullock-Palmer RP, Conejo T, Diercks DB, Gentile F, Greenwood JP, Hess EP, Hollenberg SM, Jaber WA, Jneid H, Joglar JA, Morrow DA, O’Connor RE, Ross MA, Shaw LJ. Evaluation and Diagnosis of Chest Pain: Executive Summary: A Report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines. J Am Coll Cardiol. 2021;78(22):2218–61. 2021 AHA/ACC/ASE/CHEST/SAEM/SCCT/SCMR Guideline for the. [DOI] [PubMed] [Google Scholar]
- 2.Douglas PS, Nanna MG, Kelsey MD, Yow E, Mark DB, Patel MR, Rogers C, Udelson JE, Fordyce CB, Curzen N, Pontone G, Maurovich-Horvat P, De Bruyne B, Greenwood JP, Marinescu V, Leipsic J, Stone GW, Ben-Yehuda O, Berry C, Hogan SE, Redfors B, Ali ZA, Byrne RA, Kramer CM, Yeh RW, Martinez B, Mullen S, Huey W, Anstrom KJ, Al-Khalidi HR, Vemulapalli S, Investigators P. Comparison of an Initial Risk-Based Testing Strategy vs Usual Testing in Stable Symptomatic Patients With Suspected Coronary Artery Disease: The PRECISE Randomized Clinical Trial. JAMA Cardiol. 2023;8(10):904–14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Reeves RA, Halpern EJ, Rao VM. Cardiac Imaging Trends from 2010 to 2019 in the Medicare Population. Radiol Cardiothorac Imaging. 2021;3(5):e210156. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Zimmerman ME, Batlle JC, Biga C, Blankstein R, Ghoshhajra BB, Rabbat MG, Wesbey GE 3rd, and, Rubin GD. The direct costs of coronary CT angiography relative to contrast-enhanced thoracic CT: Time-driven activity-based costing. J Cardiovasc Comput Tomogr. 2021;15(6):477–83. [DOI] [PubMed]
- 5.Morris MF, Chandrasekhar M, Gudi H, Schumann C, Benson B, Ng N, Mullen S, Huey W, O’Neal. A Study to Measure the Ability of AI-CSQ to suppoRt The busy CCTA reader: SMART-CT. J Cardiovasc Comput Tomogr. 2024;18(2):213–4. [DOI] [PubMed] [Google Scholar]
- 6.Narula J, Stuckey TD, Nakazawa G, Ahmadi A, Matsumura M, Petersen K, Mirza S, Ng N, Mullen S, Schaap M, Leipsic J, Rogers C, Taylor CA, Yacoub H, Gupta H, Matsuo H, Rinehart S, Maehara A. Prospective deep learning-based quantitative assessment of coronary plaque by computed tomography angiography compared with intravascular ultrasound: the REVEALPLAQUE study. Eur Heart J Cardiovasc Imaging. 2024;25(9):1287–95. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Koo BK, Erglis A, Doh JH, Daniels DV, Jegere S, Kim HS, Dunning A, DeFrance T, Lansky A, Leipsic J, Min JK. Diagnosis of ischemia-causing coronary stenoses by noninvasive fractional flow reserve computed from coronary computed tomographic angiograms. Results from the prospective multicenter DISCOVER-FLOW (Diagnosis of Ischemia-Causing Stenoses Obtained Via Noninvasive Fractional Flow Reserve) study. J Am Coll Cardiol. 2011;58(19):1989–97. [DOI] [PubMed] [Google Scholar]
- 8.Choi AD, Thomas DM, Lee J, Abbara S, Cury RC, Leipsic JA, Maroules C, Nagpal P, Steigner ML, Wang DD, Williams MC, Zeb I, Villines TC, Blankstein R. 2020 SCCT Guideline for Training Cardiology and Radiology Trainees as Independent Practitioners (Level II) and Advanced Practitioners (Level III) in Cardiovascular Computed Tomography: A Statement from the Society of Cardiovascular Computed Tomography. J Cardiovasc Comput Tomogr. 2021;15(1):2–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Pass F. Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. J Educ Psychol. 1992;84(4):429–34. [Google Scholar]
- 10.Rinehart S, Blankstein R, McCarthy CP, Januzzi JL, Scherer M, O’Neal WT, Green P, Puma J, Corrigan F, Datta J, Eichhorn J, Stuckey T, Khalique O, Ali Z, Ng N, Huey W, Mullen S, Rogers C, Shaw LJ, Investigators D. Guiding Preventive Care Strategies for Patients With Atherosclerotic Plaque on Coronary CTA: Primary Outcomes of the DECIDE Registry. JACC Cardiovasc Imaging. 2026;19(6):706–16. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The datasets generated and/or analyzed during the current study are not publicly available due to proprietary restrictions.