Artificial intelligence for pulmonary embolism detection: Is it comparable to radiology residents?

Abstract

Purpose

To assess the diagnostic performance of a commercial artificial intelligence (AI) software in detecting pulmonary embolism (PE) in emergency settings, compared with on-call radiology residents.

Methods

All consecutive emergency CT pulmonary angiographies (CTPA) performed over a 3-month period in the emergency department of a university hospital in patients with suspected PE, initially interpreted by the radiology residents during the emergency workflow and subsequently verified and approved by a board-certified radiologist, were concomitantly analyzed by an AI software for the presence of PE. The AI results were sent in a separate PACS partition and were not available for the preliminary and for the final reporting. Diagnostic performance of AI and residents for PE detection was assessed against the final report of the radiologist attending, which served as the reference standard.

Results

Among 594 CTPA examinations, PE was present in 82 patients (13.8% prevalence), including 41 (50%) proximally located emboli. Overall, AI achieved a sensitivity and a specificity of 89% and 99%, respectively, compared with 97.6% sensitivity and 99.2% specificity for residents (p > 0.05). For proximal PE, sensitivity was 97.6% for AI and 100% for residents (p = 1), whereas for peripheral PE, AI sensitivity was 80.5% versus 95.1% for residents (p = 0.08).

Conclusion

Radiology residents showed an overall better performance in detecting PE compared to the AI software. The AI software achieved good diagnostic performance for PE detection, particularly for proximal located PE.

Highlights

• •Radiology residents showed higher sensitivity than AI for PE detection on emergency CTPA.
• •The AI software demonstrated good overall diagnostic performance for PE detection.
• •The AI sensitivity was comparable to residents for proximal PE, but lower for peripheral emboli.

1. Introduction

Pulmonary embolism (PE) is a life-threatening condition, with a mortality approaching 30% if untreated [1], [2]. The demand for CT examinations in suspected PE has grown steadily, especially in emergency departments (ED) [3], as CT pulmonary angiography (CTPA) was established as the diagnostic gold standard [4].

Over the past years, various commercial and in-house artificial intelligence (AI) tools have been developed to support the detection of PE. Several studies have reported good diagnostic accuracy for AI [5], [6], [7], while others highlighted workflow benefits such as faster prioritization and faster CT reporting [8], [9], [10]. However, these investigations used various reference standards to assess AI diagnostic performance and evaluated different AI solutions either within their own hospitals or within teleradiology systems [7], [8], [9]. Only few studies included the performance of radiology residents, which was retrospectively evaluated in different clinical settings [7], [11] and none has compared AI with radiology residents issuing real-time reports in emergency within an academic hospital.

Thus, the aim of our study was to assess the diagnostic performance of a commercial AI tool for detecting PE on CTPA-scans, obtained in the emergency department, in patients with suspected PE, and to compare it with that of radiology residents providing immediate preliminary reports within the routine emergency workflow of our university hospital, using attending radiologists’ final reports as the reference standard.

2. Material and methods

2.1. Settings

Our single-center study received the IRB approval (IRB 2020–02812), which waived patients’ informed consent. The study was prospectively performed over a period of 3-month in the emergency radiology section of our university hospital; the analysis was performed retrospectively, based on radiology reports and AI results. A radiology resident is present 24/7, overseeing the CT activity and delivering the preliminary report for each CT examination. These reports are verified by the attending radiologists and once validated, they become the final interpretation for the CT examination. In our department, residents begin working in the emergency radiology section at the end of their 2nd year of postgraduate training, after completing their mandatory rotations in CT, sonography, and conventional radiology.

2.2. Study population

We included all consecutive CTPA examinations performed in emergency, during the study period, for suspected acute PE in adult patients (≥16 years old). We excluded CTPA performed for indications other than acute PE and CT examinations directly interpreted by the attending radiologist.

During the study period, we implemented a separate informatics workflow, independent of the standard CT interpretation system, to allow the direct transfer of CTPA examinations from the CT console to the commercially available software, trained for the automatic detection of PE. After analysis by the software, the positive AI-labeled images were directly sent to a separate PACS partition (research PACS) and were not accessible to residents or attending radiologists, during the interpretation of the CT examinations (Fig. 1).

Fig. 1.

Schematic representation of the emergency CT Pulmonary Angiograms (CTPA) reporting (A) and the AI workflow(B) during the study period. A: Standard workflow interpretation process for emergency CTPA, from the CT acquisition to the reporting workstation. B: Parallel workflow for the AI software, separated and independent from the reporting standard workflow. The AI outputs were sent directly to a research PACS and were not available to the radiologists for case interpretation.

2.3. AI software

The evaluation of the software for the detection of PE was conducted as part of a specific research protocol and not in the frame of its commercial use, which is a computer-assisted triage and notification tool, to assist radiologists with workflow triage by flagging suspected positive findings [12]. For this study, we used the commercially available software Aidoc (Aidoc Medical). Cases considered positive for PE by the software resulted in an output in the form of a heatmap, colour-labelling the location of the emboli on one CT image (Fig. 2); these images were directly sent and stored in the research PACS.

Fig. 2.

AI software heatmap, flagging pulmonary embolism. AI-generated heatmap highlights bilateral central filling defects in the main pulmonary arteries (arrows), consistent with acute central pulmonary embolism.

For all examinations, the AI results, presented in a binary mode (presence or absence of PE), were managed by an independent informatics analyst not involved in the study and released to the principal investigator six months after the study completion.

2.4. CT acquisition protocol

All CTPA examinations were performed using a standardized PE acquisition protocol on two multidetector CT systems located in the emergency radiology unit (256-multi-slices Somatom Force and 128-multi-slices Somatom Definition Edge, Siemens Healthineers). Intravenous contrast injection was done using a volume of 55 mL Accupaque (Iohexol, 350 mgI/mL, GE Healthcare AG), at a flow rate of 4 mL/, with an automated power injector and bolus-tracking of the contrast.

The same reconstructed series were sent from the CT acquisition console for routine clinical reading and for AI analysis.

2.5. Data collection

Clinical and radiological data were retrospectively collected from the Radiological Information System (RIS) of the Radiology Division of our hospital.

For all included examinations, we retrieved the following information from the CT reports: patient demographics (age and sex), PE status (positive, negative, or indeterminate) and PE location, the quality of the CTPA (documented in the final report as good or inadequate), and the presence of concomitant thoracic pathologies.

In our reporting system, radiologists classify CTPA as indeterminate for PE when PE could not be confirmed at the central, lobar, or segmental level [13]. The reported CT image quality is the attending radiologist’s global evaluation of the examination, including factors such as pulmonary artery opacification and artifacts.

For CTPAs reported positive for PE, we extracted the PE location, categorized as central (main pulmonary artery), lobar, segmental, or subsegmental. For the purpose of our study, we further grouped PE site into proximal PE (central and/or lobar level) and peripheral PE (segmental and/or subsegmental level).

All concomitant thoracic pathologies, unrelated to PE, were also extracted from the radiological reports and categorized into three groups: suspected pulmonary infections, pulmonary or mediastinal lesions (tumors), and pleural pathologies.

2.6. Reference standard

The final report of the attending radiologist was considered the reference standard for the presence or absence of PE. Both residents’ preliminary reports and the AI results were compared against this standard.

2.7. Statistical analysis

Statistical analysis was performed using R version 4.2.2. First, descriptive statistics were generated. Continuous variables were expressed as mean (± standard deviation), and categorical variables as counts (percentages). For the diagnostic performance, we calculated sensitivity, specificity, positive and negative predictive value for the AI algorithm and for the radiology residents, using the final radiological report as the reference standard. Examinations classified as indeterminate for acute PE were excluded from diagnostic performance analyses but were included in the descriptive statistics. The performance of AI and of residents were compared using the McNemar test for paired data. The 95% confidence intervals for proportions were computed using the Wilson method. We additionally analysed sensitivity and specificity for residents and AI in different subgroups of patients. The mean age of patients with and without PE were compared using the Welch’s t-test. A p-value of < 0.05 was considered statistically significant.

3. Results

A total of 626 CTPA examinations were performed during the study period. Nine examinations (1.4%) were excluded due to clinical indications unrelated to suspected PE (n = 2) or absence of the preliminary radiology resident report (n = 7), leaving 617 eligible studies. Of them, 23 examinations (3.7%) were excluded from the diagnostic performance analysis because the final attending radiologist’s report categorized them as indeterminate for PE.

The final study group consisted of 594 CTPA examinations, including 310 women (52.2%), with a mean patient age of 63 years (±17.5 SD). Patient characteristics are detailed in Table 1. Acute PE was diagnosed in 82 cases (13.8%), with comparable prevalence among men (13.7%) and women (13.9%). Patients with acute PE were significantly older than those without PE (mean age 66.6 vs. 62.4 years, p = 0.029). Among the positive cases, emboli were equally located at proximal (n = 41, 50%) and at peripheral levels (n = 41, 50%).

Table 1.

Characteristics of the patients who underwent CTPA for suspected pulmonary embolism in the emergency department.

	N (%)	PE (%)	No PE (%)	p-value
All	594	82 (13.8%)	512 (86.2%)
Gender				0.99
Women Men	310 (52.2%) 284 (47.8%)	43 (13.9%) 39 (13.7%)	267 (86.1%) 245 (86.3%)
Mean age (years)	63 (+/−17.5 SD)	66.6 (+/−15.8 SD)	62.4 (+/−17.7 SD)	0.029
Associated thoracic pathologies				0.72
Yes No	397 (67%) 197 (33%)	45 (11.3%) 37 (18.7%)	352 (88.7%) 160 (81.3%)
Reported CT quality				0.99
Good Inadequate	560 (94.3%) 34 (5.7%)	78(13.9%) 4 (11.8%)	482 (86.1%) 30 (88.2%)

PE: pulmonary embolism.

Coexisting thoracic pathologies were present in 397 patients (67%). Of these, 320 (80.6%) were consistent with suspected infectious pulmonary diseases, 19 (4.8%) with suspected malignant pathology, and 58 (14.6%) presented with isolated unilateral or bilateral pleural effusion.

3.1. Performance analysis

Compared with the attending radiologist’s final report, the AI software achieved an overall sensitivity of 89% (95%CI 80.4–94.1), specificity of 99% (95%CI 97.7–99.6), positive predictive value (PPV) of 93.6% (95%CI 85.9–97.2), and negative predictive value (NPV) of 98.3% (95%CI 96.7–99.1). Radiology residents demonstrated higher overall sensitivity: 97.6% (95%CI 91.5–99.3) for PE diagnosis compared with AI, while specificity: 99.2% (95%CI98–99.7), PPV: 95.2% (95%CI 88.4–98.1) and NPV: 99.6% (95%CI 98.6–99.9) were comparable. (Table 2).

Table 2.

Overall diagnostic performance of the AI software and of the radiology residents for the detection of pulmonary embolism.

	AI software % [95% CI]	Radiology residents % [95% CI]	p-value
Sensitivity (%)	89 (73/82) [80.4–94.1]	97.6 (80/82) [91.5–99.3]	0.07
Specificity (%)	99 (507/512) [97.7–99.6]	99.2 (508/512) [98.0–99.7]	0.99
PPV (%)	93.6 (73/78) [85.9–97.2]	95.2 (80/84) [88.4–98.1]
NPV (%)	98.3 (507/516) [96.7–99.1]	99.6 (508/510) [98.6–99.9]

Diagnostic performance of the AI software did not differ by patients’ age and sex. Sensitivity was 88.4% in women and 89.7% in men, with corresponding specificities of 99.3% and 98.8%, respectively. Across age groups, sensitivity and specificity for AI remained stable without significant variation.

AI sensitivity varied according to PE location. For proximal emboli (Fig. 3), sensitivity reached 97.6%, whereas it decreased to 80.5% for peripheral emboli. Among patients with coexisting thoracic pathologies, AI displayed a high diagnostic performance, with sensitivity of 91.1% and specificity of 98.6%.

Fig. 3.

True positive AI detection of pulmonary embolism. 3a: Axial CTPA image shows a filling defect in a segmental branch of the left upper pulmonary artery (arrow), consistent with pulmonary embolism. 3b: AI heatmap output correctly highlights the filling defect as positive for pulmonary embolism (arrow).

Residents’ performance for PE remained uniformly high across all subgroups, with sensitivities of 97.7% in women and 97.4% in men, 97.7% across all age groups, and 97.8% in patients with coexisting thoracic pathologies (Table 3).

Table 3.

Sensitivity and specificity of the AI software and of the radiology residents across subgroups of patients with suspected PE.

	Sensitivity			Specificity
	AI software % [95 CI]	Radiology residents % [95 CI]	p-value	AI software % [95 CI)	Radiology residents % [95 CI]	p-value
Gender
Women	88.4 (38/43) [75.5–94.9]	97.7 (42/43) [87.9–99.6]	0.22	99.3 (265/267) [97.3–99.8]	99.6 (266/267) [97.9–99.9]	0.99
Men	89.7 (35/39) [76.4–95.9]	97.4 (38/39) [86.8–99.5]	0.37	98.8 (242/245) [96.5–99.6]	98.8 (242/245) [96.5–99.6]	0.99
Age
< 65	89.7 (35/39) [76.4–95.9]	97.4 (38/39) [86.8–99.5]	0.37	99.2 (256/258) [97.2–99.8]	99.2 (256/258) [97.2–99.8]	0.99
≥ 65	88.4 (38/43) [75.5–94.9]	97.7 (42/43) [87.9–99.6]	0.22	98.8 (251/254) [96.6–99.6]	99.2 (252/254) [97.2–99.8]	0.99
CT quality
Good	89.7 (70/78) [81–94.7]	97.4 (76/78) [91.1–99.3]	0.11	99 (477/482) [97.6–99.6]	99.2 (478/482) [97.9–99.7]	0.99
Inadequate	75 (3/4) [30.1–95.4]	100 (30/30) [51−100]		100 (4/4) [88.6–100]	100 (30/30) [88.6–100]
Additional thoracic pathologies	91.1 (41/45) [79.3–96.5]	97.8 (44/45) [88.4–99.6]	0.37	98.6 (347/352) [96.7–99.4]	99.4 (350/352) [98–99.8]	0.45

When stratified by PE location, residents outperformed the AI in diagnosing peripheral PE 95.1% sensitivity for residents vs. 80.5% sensitivity for AI; AI software showed similar sensitivity for centrally located PE (97.6% sensitivity for AI and 100% for residents) (Table 4).

Table 4.

Sensitivity of the AI software and of the radiology residents according to pulmonary embolism location.

	Sensitivity		p-value
	AI software % [95% CI]	Radiology residents % [95% CI]
Overall	89 (73/82) [80.4–94.1]	97.6 (80/82) [91.5–99.3]	0.07
Proximal PE N = 41	97.6 (40/41) [87.4–99.6]	100 (41/41) [91.4–100]	1
Peripheral PE N = 41	80.5 (33/41) [66–89.8]	95.1 (39/41) [83.9–98.6]	0.08

PE: Pulmonary Embolism.

While residents consistently achieved higher sensitivity than the AI across all subgroup analyses, the differences did not reach statistical significance. Both AI software and radiology residents maintained high specificities and predictive values across the study population.

The AI software missed nine PE (false negatives) on CTPA, one at the lobar level and eight at segmental or subsegmental levels (Fig. 4). The AI mislabeled five negative examinations as positive (false positive) for PE, consisting of one case of lymph node incorrectly identified as an embolus (Fig. 5), and four cases where artifacts within the pulmonary arteries (one central and three peripheral) were misclassified as PE. Radiology residents missed two cases of PE (false negatives), at peripheral level; both were detected by the AI software. Residents incorrectly classified four cases without PE as positive (false positives); the AI software classified all four as negative. No cases labelled positive for PE by the AI software were missed by the attending radiologist.

Fig. 4.

False-negative AI detection of pulmonary embolism. Axial CTPA image shows a filling defect in a subsegmental branch of the right lower pulmonary artery (arrow), consistent with acute embolism. This case was considered negative by the AI software.

Fig. 5.

False-positive AI detection for pulmonary embolism. 5a: Axial CTPA image shows bilateral hilar lymph nodes (arrows). 5b: AI heatmap output falsely highlights the left hilar lymph node (arrow) as positive for pulmonary embolism.

4. Discussion

Our study compared the diagnostic performance of an AI software, to detect PE on CTPA performed in emergency, and that of radiology residents, obtained from their preliminary CT reports, issued during real life workflow, to the final report of attending radiologists. In our survey, the AI software achieved an overall sensitivity of 89% to detect PE, lower than the sensitivity of 97.6% for radiology residents, with similar specificity (99% for the AI software and 99.2% for the radiology residents). The AI software had a similar sensitivity and specificity in detecting PE across both sexes and all age groups, its diagnostic performance not being influenced by patients’ demographics.

The performance of AI in detecting PE in our study was consistent with previously published results from other centers, using the same software. The AI overall sensitivity of 89% lies within the reported range of 63–92.7% in previous analyses [5], [7], [8], [9], [14]. For instance, Batra et al. [8] reported a sensitivity of 83.3% using radiology reports as the reference standard for AI performance, closely matching our findings. Similarly, Schmuelling et al. [14] reported a sensitivity of 79.6% for AI in a retrospective evaluation of CTPAs performed in emergency settings. On the other hand, Cheikh AB et al. [7] documented an AI sensitivity of 92.6%, higher than that observed in our study; however, their reference standard for PE was based on a retrospective consensus assessment, incorporating imaging, radiology reports, and patient outcomes, rather than evaluating the algorithm in a routine emergency workflow. At the lower end, Rothenberg et al. [9] reported an AI sensitivity of 63%, a difference from our study that may be explained by their inclusion of cases from multiple centers, affiliated with a large hospital, as well as by the potential variability in patient selection, including outpatients or patients from different clinical settings, rather than exclusively ED patients with suspected PE.

Our study demonstrated that AI achieved superior diagnostic performance for proximally located emboli (sensitivity of 97.6%) compared with distally located emboli (sensitivity of 80.5%). This observation is consistent with the findings of Weikert et al. [5], who reported a comparable sensitivity of 95.7% for proximal emboli. A direct comparison for peripherally located emboli among studies, however, is not entirely possible, as classification of emboli according to location differ. In their analysis, Weikert et al. [5] reported sensitivities separately for segmental (93.3%) and subsegmental emboli (85.7%), whereas in our study, segmental and subsegmental emboli were analyzed jointly, yielding an overall sensitivity of 80.5%.

Our results further revealed that AI maintained a high diagnostic performance for PE detection in patients displaying associated thoracic pathologies (sensitivity of 91.1%, and specificity of 97.8%), suggesting that concomitant parenchymal abnormalities do not impair its diagnostic performance. This may be attributable to the training of the software on heterogenous datasets, including various associated thoracic pathologies.

In our study, AI demonstrated lower sensitivity in CTPA examinations of inadequate quality (75%) compared with those of good quality (89.7%). In contrast, Cheikh et al. reported a high AI sensitivity (94.4%) in a sub-cohort of patients with suboptimal CT quality, with AI performance comparable to that of radiologists [7]. A lower degree of arterial enhancement, important motion artifacts or a higher proportion of peripheral emboli in our subgroup of patients with inadequate CT quality, may explain the differences in AI performance between studies.

In our current analysis, the AI software demonstrated a small number of false positive and false negative cases for PE. The types of false positive and false negative cases are comparable with those reported in previous surveys [5].The similarity in the PE misclassifications by AI across centres suggests that they relate to the intrinsic properties of the software, rather than to site-specific factors.

To the best of our knowledge, only one previous study compared the same AI software as in our study with that of junior radiologists for PE detection [7], authors reporting a sensitivity of 91.5% for AI and of 93.5% for junior radiologists [7]. In our survey, the sensitivity was 89% for AI and 97.6% for radiology residents. Although the study designs and patient populations differ, these findings indicate very good performance of residents, with residents’ sensitivity exceeding those of AI across patient populations.

Our study showed that, in our center, radiology residents achieved a very high diagnostic performance for PE detection, regardless of PE location. The low discrepancy rate with attending radiologists was aligned with the 0.8% rate of major misinterpretations between residents and attendings, previously reported in a quality assessment evaluation of our activity [15] and slightly higher than those reported in other studies [7], [11], [16]. This may be explained by the residents’ stage of training, as only senior residents are responsible for reporting emergency cases [15].

Several limitations have to be acknowledged: first, the attending radiologist’s final report was used as the reference standard, and follow-up clinical records were not reviewed. This choice was intentional, as our aim was to reflect the routine emergency workflow, in which patient management is performed based on the attending radiologist’s interpretation. Second, we grouped segmental and subsegmental emboli together due to their common difficulty of detection and we did not further perform a subgroup analysis. Third, we did not evaluate inter-reader variability, because we wanted to assess our overall emergency radiology practice and not differences between radiology residents.

In conclusion, our study showed that radiology residents demonstrated higher overall diagnostic performance than the AI software when reporting for PE within an emergency workflow. The AI software achieved good diagnostic performance for PE detection, particularly for proximally located emboli.

CRediT authorship contribution statement

Pierre-Alexandre Poletti: Writing – review & editing, Resources, Methodology, Conceptualization. Thibaut Desmettre: Writing – review & editing, Resources. Alexandra Platon: Writing – review & editing, Writing – original draft, Project administration, Methodology, Investigation. Periktioni Crina Chrysostomou: Writing – review & editing, Writing – original draft, Investigation. Rodolphe Meyer: Writing – review & editing, Resources. Cyril Jaksic: Writing – review & editing, Formal analysis.

Declaration of Generative AI and AI-assisted technologies in the writing process

During the final check of the text, an AI tool (ChatGPT 5.2, Open AI, San Francisco, USA) was solely used to improve the language of the manuscript. After using this tool, the corresponding author reviewed and edited the content as needed and takes full responsibility for the content of the publication.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.