Diagnostic Accuracy of Deep Learning for Intracranial Hemorrhage Detection in Non-Contrast Brain CT Scans: A Systematic Review and Meta-Analysis

Abstract

Background: Intracranial hemorrhage (ICH) is a life-threatening medical condition that needs early detection and treatment. In this systematic review and meta-analysis, we aimed to update our knowledge of the performance of deep learning (DL) models in detecting ICH on non-contrast computed tomography (NCCT). Methods: The study protocol was registered with PROSPERO (CRD420250654071). PubMed/MEDLINE and Google Scholar databases and the reference section of included studies were searched for eligible studies. The risk of bias in the included studies was assessed using the QUADAS-2 tool. Required data was collected to calculate pooled sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) with the corresponding 95% CI using the random effects model. Results: Seventy-three studies were included in our qualitative synthesis, and fifty-eight studies were selected for our meta-analysis. A pooled sensitivity of 0.92 (95% CI 0.90–0.94) and a pooled specificity of 0.94 (95% CI 0.92–0.95) were achieved. Pooled PPV was 0.84 (95% CI 0.78–0.89) and pooled NPV was 0.97 (95% CI 0.96–0.98). A bivariate model showed a pooled AUC of 0.96 (95% CI 0.95–0.97). Conclusions: This meta-analysis demonstrates that DL performs well in detecting ICH from NCCTs, highlighting a promising potential for the use of AI tools in various practice settings. More prospective studies are needed to confirm the potential clinical benefit of implementing DL-based tools and reveal the limitations of such tools for automated ICH detection and their impact on clinical workflow and outcomes of patients.

1. Introduction

Intracranial hemorrhage (ICH) is a life-threatening condition and refers to any bleeding within the brain parenchyma and surrounding meningeal spaces [1]. It has been estimated that approximately 2 million individuals are affected by ICH worldwide annually [2]. It is an important cause of death and disability [3]. Early detection and initiation of appropriate treatment are critical to improve outcomes for ICH patients [4]. Non-contrast computed tomography (NCCT) is a standard imaging modality for ICH diagnosis and requires expertise to differentiate between blood and healthy brain parenchyma and assess its extent and location [5,6,7]. However, the diagnosis of ICH in emergency settings may be challenging due to a lack of time, the heavy workload of radiologists, and the shortage of expert radiologists in low-income settings [8,9,10]. While non-radiologist physicians or residents can compensate for these limitations, it may sometimes result in delays in diagnosis or misdiagnosis, potentially leading to life-threatening conditions [11,12,13]. This highlights the necessity for tools to help radiologists effectively and correctly identify ICH.

Artificial intelligence (AI) has, in recent years, had a very significant impact on most aspects of healthcare [14,15]. AI has the potential to improve radiologist efficiency, optimize workflow, and standardize ICH detection. Deep learning (DL), a subfield of machine learning (ML) and AI that utilizes artificial neural networks (ANN), possesses the capability to learn from data, making it a potent tool in computing with wide applications spanning numerous domains, including healthcare, visual recognition, and many others [16,17]. Unlike traditional machine learning, DL is able to learn the most critical features of a given task and perform tasks such as classification, detection, and segmentation on medical images simultaneously [18,19,20]. Convolutional Neural Network (CNN) is a specialized type of DL algorithm that has become popular in many computer-vision tasks and has recently gained attention in the health service area such as radiology. It uses convolutional layers that effectively extract local features from medical images [21,22,23].

Several previous studies have reported the efficacy and high accuracy of DL in the automated detection of ICH on non-contrast brain CT scans, which encourages the use of DL in clinical practice [20,24,25,26,27]. Regarding processing time, DL models have shown promising results, achieving a quicker first diagnosis compared to the time taken by radiologists to make their initial diagnoses [28]. Furthermore, it has been demonstrated that implementing DL for ICH detection from NCCT significantly reduced costs and shortened the length of emergency department stay [29]. Thus, it has the potential to help and improve the performance of radiologists in evaluating NCCT scans suspicious for ICH, especially in emergency scenarios, when time efficiency is required. By reducing the increasing workload, radiologists can devote their attention to more complex and challenging cases. Consequently, in this systematic review and meta-analysis, we aimed to update our knowledge of the per-scan diagnostic performance of DL-based models in detecting ICH.

2. Materials and Methods

This systematic review and meta-analysis was registered with PROSPERO (CRD420250654071) and written following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) and the Preferred Reporting Items for Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies (PRISMA-DTA) guidelines.

2.1. Search Strategy and Study Selection

The literature search was conducted in two public databases (PubMed/MEDLINE and Google Scholar) from inception to 23 February 2025, using combinations of the following keywords and MeSH terms: “Intracranial hemorrhage”, “Intracranial haemorrhage”, “ICH”, “Artificial Intelligence”, “AI”, “Machine Learning”, and “Deep Learning”. Two independent reviewers (A.K. and A.S.) screened the initial search to identify all eligible studies. Any disagreements between reviewers were resolved by reaching a consensus during the discussion. The reference lists of the included studies were also manually screened for additional pertinent articles.

We included studies that met the following criteria: (1) used a deep learning model for the detection of ICH of any type (intraparenchymal hemorrhage (IPH), subdural hemorrhage (SDH), epidural hemorrhage (EDH), intraventricular hemorrhage (IVH), and subarachnoid hemorrhage (SAH)) on NCCT scans, (2) published in peer-reviewed journals, (3) Full-text is available, (4) published in the English language, and (5) reported data on the diagnostic accuracy of the DL-based algorithms. Non-English manuscripts, case reports, case series, abstracts, preprints, conference papers, editorials, commentaries, book chapters, review articles, and studies that used non-deep learning methods were excluded. We also excluded studies that did not provide sufficient data on the diagnostic accuracy of DL-based models for ICH detection on NCCT scans.

2.2. Data Extraction

We extracted data from selected studies including first author, year of publication, study design, type of used databases for training and testing the DL model, size of the databases, type of DL model, gold standard, and performance results including accuracy, specificity, sensitivity, and Area Under the Curve (AUC). The true positive (TP), false positive (FP), true negative (TN), and false negative (FN) were also extracted, and if not reported, were calculated based on the provided values for sensitivity, specificity, and the total number of positive or negative NCCTs.

2.3. Risk of Bias Assessment

All included studies were evaluated for the potential risk of bias using a revised tool for quality assessment of diagnostic accuracy studies included in the systematic review (QUADAS-2) [30]. This tool comprises four domains: patient selection, index test, reference standard, and flow and timing.

2.4. Statistical Analysis

Statistical analysis in R v4.3.3 was conducted with the “meta” and “mada” packages. Values of TP, TN, FP, and FN were collected to compute pooled sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) with the corresponding 95% CI using random effects model. If in a study more than one DL model were compared, we used the performance results of the best-performing model. A summary receiver operating characteristic (SROC) curve was generated by the bivariate model using the “reitsma” function of the “mada” package. The AUC, which measures the entire two-dimensional area underneath the ROC curve, was also calculated to show the ability of DL models to identify ICH on NCCTs (the AUC of 1.0 indicates perfect discrimination, while an AUC of 0.5 indicates random chance).

Tau² and I² statistics were used to evaluate statistical heterogeneity testing. The heterogeneity was classified as low (0% to 25%), moderate (25% to 50%), or high (≥50%) based on the calculated I² [31]. Funnel plots were used to visualize publication bias and Egger’s regression test was performed to estimate publication bias in included studies. A p-value less than 0.05 was considered a significant publication bias.

3. Results

The primary search on databases resulted in 359 articles. Five eligible studies were also obtained by searching the reference section of the selected studies. In total, 121 full-text articles were reviewed for eligibility. At last, seventy-three studies were included in our qualitative synthesis [7,19,20,24,25,26,27,28,29,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95]. Given that in 15 studies the relevant data for calculating values of TP, TN, FP, and FN was unavailable, fifty-eight studies were selected for our meta-analysis (Figure 1). A summary of the characteristics of the included studies is provided in Table 1. Sixty-one studies were retrospective and nine were prospective studies. Three studies had a retrospective/prospective design.

Figure 1.

PRISMA flow diagram.

Table 1.

Characteristics of included studies.

Study	Design	Type of ICH	Datasets	DL Model	Ground Truth	Sensitivity (%)	Specificity (%)	Accuracy (%)	AUC	Comments on Performance
Abrigo 2023 [32]	Retrospective	ICH	Trained on RSNA public dataset/tested on a local dataset consisting of 1372 NCCTs	CNN	Radiologists or senior radiology trainees	73.8	79	78.6	0.842	Detection/per-scan
Alis 2022 [20]	Prospective	ICH	Local datasets/trained on 49,968 NCCTs/tested on 452 NCCTs	Joint CNN-RNN	Experienced neuroradiologists	96.41	95.79	96.02	0.961	Detection/per-scan
Altuve 2022 [24]	Retrospective	ICH	Publicly available Kaggle’s Head CT-Hemorrhage database including 200 NCCTs	ResNet-18 (deep residual CNN)	NA	95.65	96.2	95.93	NA	Detection/per-scan/the average performance is reported
Angkurawaranon 2023 [19]	Retrospective	ICH	Local dataset including 300 NCCTs	DL	Experienced neuroradiologists	82	90	89	NA	Detection of ICH/location-level performance is reported
Arbabshirani 2017 [33]	Retrospective	ICH	Local dataset/trained on 46,583 NCCTs/tested on 347 NCCTs	CNN with 3D-architecture	Experienced neuroradiologists	70	87	84	NA	Detection/per-scan
Arman 2023 [34]	Retrospective	ICH	Trained and tested on NCCTs from RSNA public dataset	DenseNet	Neuroradiologists	84.32	98.33	96.32	NA	Detection/per-slice
Babu 2024 [35]	Retrospective	ICH	Publicly available Kaggle dataset/1600 NCCTs were used for training, 600 images for testing, and 400 images for validation	Hybrid DenseNet 121 + LSTM models	Expert radiologists	98.99	NA	97.5	NA	Detection/per-scan
Bark 2024 [95]	Retrospective	ICH	Trained on a dataset of around 100,000 scans/tested on a local dataset of 2306 head CTs	HealthICH+ version 3.1.24 (3D CNN)	Neuroradiologist	NA	NA	NA	NA	Detection/per-scan/PPV of 82.3%
Buls 2020 [36]	Retrospective	ICH	Local dataset/tested on 388 NCCTs	Aidoc (CNN)	Board certified experienced neuroradiologists	84	94	93	NA	Detection/per-scan
Chang 2018 [37]	Prospective	ICH	Trained on a local dataset of 10,159 NCCTs/tested on a local dataset including 682 NCCTs	Hybrid 3D/2D mask ROI-based CNN	Board-certified radiologist	95.1	97.3	97	0.981	Detection/per-scan The accuracy for ICHs > 25, 5–25, 0.01–5.0, and <0.01 mL is 0.997, 0.977, 0.906, and 0.872, respectively
Chien 2022 [29]	Retrospective	ICH	Local dataset/tested on 238 NCCTs	Deep-CNN	NA	96.43	99.52	99.16	NA	Detection/per-scan
Chilamkurthy 2018 [38]	Retrospective	ICH	Trained on a local dataset (Qure25k dataset) including 21,095 NCCTs/tested on a local dataset (CQ500) including 491 NCCTs	Modified ResNet18 (CNN)	Experienced radiologists	94.7	70.1	NA	0.94	Detection/per-scan
Cho 2019 [25]	Retrospective	ICH	Trained on a local dataset of 5702 NCCTs	Cascade of CNNs and dual FCNs	A team of experienced neurologist, neurosurgeons, and emergency medicine doctors	97.91	98.76	NA	NA	Detection/per-scan
Choi 2024 [39]	Retrospective	ICH	Local dataset/tested on 2146 NCCTs	DL	Board-certified neuroradiologists	59.2	91.6	83.3	NA	Detection/per-scan
Colasurdo 2022 [40]	Retrospective	SDH	Local dataset/tested on 263 NCCTs	Deep-CNN	Experienced neuroradiologists	91.4	96.4	95.1	NA	Detection/per-scan
Coorens 2023 [41]	Retrospective	ICH	Local datasets/trained and validated on 16,348 NCCTs/tested on 4095 NCCTs	Masked loss U-Net architecture	NA	77	96.2	91.1	NA	Detection/per-slice
Cortés-Ferre 2023 [42]	Retrospective	ICH	Trained on the RSNA dataset/55 NCCTs were used for external validation	DL	Experienced radiologists, neurologists and neurocritical doctors	100	100	100	NA	Detection/per-scan
D’Angelo 2024 [28]	Retrospective	ICH	Local dataset/502 NCCTs	Dense-UNet architecture	Board-certified radiologists	90.37	94.85	91.24	NA	Detection/per-scan
Dawud 2019 [43]	Retrospective	ICH	Local dataset/Trained on 8855 and tested on 3790 NCCTs	AlexNet-SVM	NA	95	90	93	NA	Detection/per-slice
Del Gaizo 2024 [44]	Retrospective	ICH	Local dataset/58,321 NCCTs	CINA v1.0 device (DL)	Experienced emergency radiologists	75.6	92.1	91.7	NA	Detection/per-scan
Dyer 2021 [45]	Retrospective	ICH	Tested on a local dataset consisting of 390 NCCTs	EfficientNet neural network architecture	Consultant radiologists and neuroradiologists	98.8	92.5	NA	98.8	Detection/per-scan
Eldaya 2022 [46]	Retrospective	ICH	Tested on a local dataset consisting of 307 NCCTs	RAPID AI software (hybrid deep 2D–3D DCNN)	Board-certified or board-eligible neuroradiologists	91.9	84.4	85.3	NA	Detection/per-scan
Gibson 2022 [47]	Retrospective	ICH	Trained on 25,946 NCCTs/evaluated on 16,764 NCCTs from RSNA ICH dataset	Deep neural networks	Neuroradiologists	86	92	NA	0.95	Detection/per-scan
Ginat 2019 [48]	Prospective	ICH	Local dataset/tested on 2011 NCCTs	Aidoc (CNN)	Board-certified neuroradiologist	88.7	94.2	93.4	NA	Detection/per-scan
Ginat 2021 [49]	Prospective	ICH	Local dataset/tested on 8723 NCCTs	Aidoc (CNN)	Radiologist report	88.4	96.1	NA	NA	Detection/per-scan
Gruschwitz 2021 [50]	Retrospective/ prospective	ICH	Retrospective (Local dataset/tested on 872 NCCTs) Prospective (Local dataset/tested on 100 NCCTs)	Dense-UNet architecture	A resident radiologist with subsequent supervision of a board-certified radiologist				NA	Detection/per-scan Retrospective (sensitivity = 91.4/specificity = 90.4/accuracy = 90.8) Prospective (sensitivity = 81.8/specificity = 97.8/accuracy = 96)
Guo 2022 [51]	Prospective	ICH	Trained on a local dataset/tested on a local dataset consisting of 650 NCCTs	RoLo, a novel weakly supervised deep learning algorithm	Expert radiologists	97.9	91.5	NA	0.987	Detection/per-scan
He 2024 [52]	Retrospective	ICH	Trained and tested on the ASNR public dataset consisting of 750,000 head CTs	End-to-end deep multiscale convolutional feature fusion framework	Radiologists	NA	NA	NA	0.988	Detection/per-slice
Heit 2021 [53]	Retrospective	ICH	Trained on a local dataset including 805 NCCTs/validated on a local dataset consisting of 308 NCCTs	Hybrid deep 2D–3D CNN	Expert neuroradiologists	95.6	95.3	NA	NA	Detection/per-scan/performance on validation test is reported
Hofmeijer 2023 [54]	Retrospective	ICH	Trained and tested on a local dataset including 134 NCCTs	CNN	Neuroradiologists	NA	NA	NA	0.82	Detection/per-scan
Hopkins 2022 [55]	Prospective	ICH	Trained on RSNA dataset containing 21,784 NCCTs/tested on a local dataset including 726 NCCTs	CNN	Board-certified radiologists	94	94	NA	0.98	Detection/per-scan
Hu 2023 [7]	Ambispective	ICH	Three local datasets for external validation/trained on 931 NCCTs	Hybrid 2D/3D UNet deep-learning framework	Neurosurgeons and radiologists (not well defined)	81.2	99.9	99.7	NA	Detection/per-scan
Kang 2023 [56]	Retrospective	ICH	Trained on two publicly available datasets (RSNA and AI-Hub)/two local datasets including 986 NCCTs were used for external testing	2D U-net with the Inception module + weighted ensemble	Vascular neurologist and neuroradiologists	92.8	85.7	NA	95.3	Detection/per-scan
Kau 2021 [57]	Retrospective	ICH	Local dataset/tested on 2188 NCCTs	Aidoc (CNN)	Board-certified neuroradiologist	68.2	96.8	94	NA	Detection/per-scan
Kiefer 2024 [58]	Retrospective	ICH	Tested on a local dataset including 432 NCCTs	Dense-UNet architecture	Experienced radiologists	98.1	89.7	90.7	NA	Detection/per-scan
Kumaravel 2020 [26]	Retrospective	ICH	Trained and tested on a publicly available dataset (CQ500 with 451 NCCTs)	DCNN (AlexNet-SVM)	Experienced radiologists	99.86	99.86	99.86	0.999	Detection/per-slice
Kundisch 2021 [59]	Retrospective	ICH	Tested on a local dataset including 4946 NCCTs	3D DCNN (Aidoc)	Neuroradiologists	NA	NA	NA	NA	Detection/per-scan
Kuo 2019 [60]	Retrospective	ICH	Local datasets/Trained on 4396 head CTs/tested on 200 head CTs	Patch-based fully convolutional neural network (PatchFCN)	Board-certified neuroradiologist	100	90	NA	0.991 ± 0.006	Detection/per-scan
Lee 2018 [61]	Retrospective	ICH	Local datasets/Trained on 904 NCCTs/tested on 200 head CTs	DCNNs	Board-certified neuroradiologists	98	95	NA	99.3	Detection and classification/per-scan
Lee 2020 [62]	Retrospective	ICH	Local datasets/trained on 166 NCCTs/validation set included 84 cases	ANN	Board-certified neuroradiologists	78	80	NA	0.859	Detection/per-scan/performance on validation dataset is reported
López-Pérez 2022 [63]	Retrospective	ICH	Trained and tested on RSNA dataset	Multiple Instance Learning based on Deep Gaussian Processes (DGPMIL)	Neuroradiologists	95.7	NA	82.5	NA	Detection/per-scan
Majumdar 2018 [64]	Retrospective	ICH	Local dataset/trained on 60 NCCTs with ICH/tested on 69 NCCTs	CNN (modified U-Net)	Experienced radiologists	81	98	NA	NA	Sensitivity per lesion and specificity per normal case
McLouth 2021 [65]	Retrospective	ICH	Local dataset/tested on 814 NCCTs	CINA v1.0 device (DL)	Board-certified neuroradiologists	91.4	97.5	95.6	NA	Detection/per-scan True positive rate (%) for ICHs < 5, 5–25, and >25 mL is 71.8, 100, and 100, respectively
Monteiro 2020 [66]	Retrospective	ICH	Trained and tested on the CENTERTBI dataset (184 NCCTs in the training subset and 655 scans in the test subset)/external validation on a publicly available dataset (CQ500 dataset)	CNN	Experienced radiologists	90	51	NA	0.83	Detection/per-scan/performance on CQ500 dataset is reported
Nada 2024 [67]	Prospective	ICH	Local dataset/tested on 5600 NCCTs	Aidoc (CNN)	Experienced neuroradiologists	89	96	94	0.954	Detection/per-scan
Neves 2023 [27]	Retrospective	ICH	Local dataset/tested on 510 NCCTs	Caire ICH vR1 (Caire Health Inc.)	Experienced radiologists	97.52	100	98.05	0.9957	Detection/per-scan
Nishi 2021 [68]	Retrospective	Non-traumatic SAH	Local datasets/trained on 757 NCCTs/tested on 331 NCCTs	3D U-net (CNN)	Neurosurgery specialists	99	92	95	0.99	Detection/per-scan
Pettet 2024 [69]	Retrospective	ICH	Local dataset consisting of 1315 NCCTs	CNN (qER, developed by Qure.ai)	Experienced neuroradiologists and general radiologists	85.7	94.3	93.5	NA	Detection/per-scan
Phaphuangwittayakul 2022 [70]	Retrospective	ICH	Two public datasets (RSNA and PhysioNet) and one private dataset (CMU-TBI) including 321 NCCTs	CNN	NA	95.77	96.9	96.21	NA	Detection/per-slice
Rao 2022 [71]	Retrospective	ICH	Local dataset including 1164 NCCT images/Trained on 931 NCCT images/tested on 233 NCCT images	ResNet-based transfer learning model	Radiologists	99.4	99.7	99.6	100	Detection/per-slice
Roshan 2024 [72]	Retrospective	ICH	Local dataset including 4203 NCCTs	CNN (Viz.ai ICH)	Experienced neuroradiologists	85.3	98.3	NA	NA	Detection/per-scan Viz.ai ICH for intraparenchymal ICH >5 mL (99%) showed higher sensitivity than ICH <5 mL (99% and 84%, respectively). For SDH >10 mL it exhibited higher sensitivity than ICH <10 mL (90% and 77%, respectively)
Salehinejad 2021 [73]	Retrospective	ICH	Trained on 21,784 NCCTs from the RSNA dataset/external validated on a dataset including 5965 NCCTs	DCNN	A trained research assistant	91.3	94.1	93.8	0.954	Detection/per-scan/performance on external validation dataset is reported
Savage 2024 [74]	Prospective	ICH	Tested on a local dataset consisting of 3716 NCCTs	Aidoc (CNN)	Attending radiologists	87.8	94.3	93	NA	Detection/per-scan
Schmitt 2021 [75]	Retrospective	ICH	Local dataset/tested on 160 NCCTs	Brainomix algorithm (DCNN)	Board-certified neuroradiologist	91	89	NA	0.9	Detection/per-scan
Seyam 2022 [76]	Prospective	ICH	Local dataset/tested on 3017 NCCTs	Aidoc (CNN)	Board-certified neuroradiologist	87.2	93.9	93	NA	Detection/per-scan
Sindhura 2023 [77]	Retrospective	ICH	RSNA dataset (6122 NCCTs were used for training, 765 NCCTs were used for validation, and 765 NCCTs were used for testing)	Joint CNN-RNN	Neuroradiologists	93.16	97.1	95.5	NA	Detection/patient-level
Sreekrishnan 2023 [78]	Retrospective	ICH	Tested on a local dataset including 881 NCCTs	RAPID AI software (hybrid 2D–3D DCNN)	Neuroradiologists	97.84	99.52	NA	NA	Detection/per-scan
Teneggi 2024 [79]	Retrospective	ICH	Trained on 17,388 NCCTs/tested on two local datasets (CQ500 and CT-ICH with 436 and 75 NCCTs, respectively)	Attention-based CNN	Expert radiologists	NA	NA	NA	NA	Detection/per-scan/for CQ500 datset AUC = 0.92 and for CT-ICH dataset AUC = 0.95)
Thanellas 2023 [80]	Retrospective	SAH	Local datasets/trained on 1083 NCCTs/tested on two public external validation datasets (Zurich and CQ500 datasets) including 1379 NCCTs	CNN (U-Net)	NA for Zurich dataset and expert radiologists for CQ500 dataset	99	63	67	NA	Detection/per-scan
Tharek 2022 [81]	Retrospective	ICH	Trained and tested on a public dataset including 200 NCCTs (100 with ICH and 100 without)	CNN	NA	96.94	93.14	95	NA	Detection/per-scan
Trang 2024 [82]	Retrospective	ICH	Tested on a local datset consisting of 9736 NCCTs	Aidoc (CNN)	Board-certified or board-eligible neuroradiologists	85.2	94.3	93.1	NA	Detection/per-scan
Villringer 2024 [83]	Retrospective	ICH	Trained on 674,000 NCCT slices from RSNA/tested on 255 NCCTs from a local dataset	CNN + Efficient-Net-B3 architecture	Experienced radiologists	90	96	96	NA	Detection/per-scan/performance is the mean performance of AI vs. expert raters
Voter 2021 [84]	Retrospective	ICH	Tested on a local dataset including 3605 NCCTs	Aidoc (CNN)	CAQ-certified neuroradiologist	92.3	97.7	NA	NA	Detection/per-scan
Wang 2021 [85]	Retrospective	ICH	Trained on RSNA dataset/tested on two publicly available external datasets (PhysioNet-ICH with 75 and CQ500 with 491 NCCTs)	2D CNN	Experienced radiologists			NA		Detection/per-scan Performance on PhysioNet-ICH (sensitivity = 88.7/specificity = 94.4/AUC = 0.964) Performance on CQ500 dataset is reported (sensitivity = 91.4/specificity = 84.4/AUC = 0.949)
Wang 2023 [86]	Retrospective	ICH	Trained on 20,000 NCCTs from different centers/tested on a local dataset consisting of 527 NCCTs	VeriScout™ tool (CNN)	Radiology trainee with 2 years of specialty experience + a sub-specialty neuroradiologist	92	96	96	NA	Detection/per-scan
Wang 2024 [87]	Retrospective	ICH	Trained and tested on local datasets/validated on a local dataset including 778 NCCTs	2D CNN with modifications from U-Net and ResNet architectures	Two radiologists and one neurosurgeon	97.7	98.71	98.2	NA	Detection/per-scan/performance of model on validation dataset is reported
Wu 2024 [88]	Retrospective	ICH	Training + Cross-Validation on a local dataset including 10,699 NCCTs/CQ500 dataset including 491 NCCTs used for external testing	Transfer learning with 2D CNN + bidirectional LSTM + attention layer at study level	experienced radiologists	88.5	89.6	91.6	0.96	Detection/per-scan/performance of model on external dataset (CQ500) is reported
Ye 2019 [89]	Retrospective	ICH	Trained on a local dataset containing 2836 NCCTs/tested on 299 NCCTs	3D Joint CNN-RNN	senior radiologists	98	99	99	1	Detection/per-scan
Yedavalli 2023 [90]	Retrospective	ICH	Tested on a local dataset including 243 NCCTs	RAPID AI software (hybrid 2D–3D CNN)	board-certified general and neuroradiologists	96.2	99.5	NA	NA	Detection/per-scan
Yeo 2023 [91]	Retrospective	ICH	Trained on the public Kaggle dataset and tested on the CQ500 dataset with 491 NCCTs	Joint CNN-RNN	Experienced radiologists	95	81	88	0.966	Detection/per-scan
Yun 2023 [92]	Retrospective	ICH	Trained and validated on a local dataset including 104,666 NCCT slices/external validated on a local dataset consisting of 49,841 NCCTs	Joint CNN-RNN	Neuroradiologists	94.4	98.2	97.7	0.992	Detection/per-scan
Zhou 2022 [93]	Retrospective	ICH	Local dataset/351 NCCTs	DL (Res-Net18 and Dense-Net121)	Experienced radiologists			NA	NA	Detection/per-slice ResNet-18 (sensitivity of 100% and specificity of 87%) DenseNet-121 (sensitivity of 98% and specificity of 79%)
Zia 2022 [94]	Retrospective/ prospective	ICH	Tested on a local dataset including 1446 NCCTs	Aidoc (CNN)	Board-certified neuroradiologist	85.7	96.8	NA	NA	Detection/per-scan/performance of prospective dataset is reported

ICH; Intracranial hemorrhage, SDH; Subdural hematoma, SAH; Subarachnoid hemorrhage, DL; Deep learning, CNN; Convolutional neural networks, RNN; Recurrent neural networks, LSTM; Long Short-Term Memory, FCN; Fully convolutional networks, DCNN; Deep convolutional neural networks, AUC; Area under curve, PPV; Positive predictive value, NA; Not available.

The results of the risk of bias assessment are shown in Table S1. Domains related to patient selection and reference standard were associated with increased risk of bias.

In fifty-eight studies, the values for TP, TN, FP, and FN were reported or corresponding data was available to calculate these numbers for the calculation of pooled sensitivity, specificity, PPV, and NPV. A pooled sensitivity of 0.92 (95% CI 0.90–0.94, I² = 96%) and a pooled specificity of 0.94 (95% CI 0.92–0.95, I² = 99%) were achieved (Figure 2). Pooled PPV was 0.84 (95% CI 0.78–0.89, I² = 99%), and pooled NPV was 0.97 (95% CI 0.96–0.98, I² = 99%) (Figure 3). A bivariate model for the diagnostic accuracy of DL in the detection of ICH on NCCTs was performed (Figure 4). In this model, AUC was 0.96 (95% CI 0.95–0.97).

Figure 2.

Sensitivity and specificity of deep learning models in detecting intracranial hemorrhage on non-contrast computed tomography in included studies [7,19,20,24,25,27,28,29,32,33,35,36,37,38,39,40,42,45,46,47,48,49,50,51,53,55,56,57,58,59,60,61,62,65,66,67,68,69,72,73,74,75,76,78,80,81,82,84,85,86,87,88,89,90,91,92,94].

Figure 3.

Positive and negative predictive values of deep learning models in detecting intracranial hemorrhage on non-contrast computed tomography in included studies [7,19,20,24,25,27,28,29,32,33,35,36,37,38,39,40,42,45,46,47,48,49,50,51,53,55,56,57,58,59,60,61,62,65,66,67,68,69,72,73,74,75,76,78,80,81,82,84,85,86,87,88,89,90,91,92,94].

Figure 4.

Bivariate summary receiver operating characteristic (SROC) curve of deep learning models for detecting intracranial hemorrhage on non-contrast computed tomography in included studies.

3.1. Subgroup Analyses

We performed further analyses based on the study design. Forty-seven retrospective studies provided data for calculating the pooled diagnostic performance of DL models in ICH detection from NCCTs. The pooled sensitivity and specificity were 0.93 (95% CI 0.90–0.94, I² = 97%) and 0.93 (95% CI 0.91–0.95, I² = 99%), respectively (Figure 5). Pooled PPV was 0.85 (95% CI 0.78–0.90, I² = 99%), and pooled NPV was 0.97 (95% CI 0.96–0.98, I² = 99%) (Figure 6). The bivariate model based on the results from retrospective studies showed an AUC of 0.96 (95% CI 0.94–0.98) (Figure 7).

Figure 5.

Sensitivity and specificity of deep learning models in detecting intracranial hemorrhage on non-contrast computed tomography according to retrospective studies [7,19,24,25,27,28,29,32,33,35,36,38,39,40,42,45,46,47,53,56,57,58,59,60,61,62,65,66,68,69,72,73,75,78,80,81,82,84,85,86,87,88,89,90,91,92].

Figure 6.

Positive and negative predictive values of deep learning models in detecting intracranial hemorrhage on non-contrast computed tomography according to retrospective studies [7,19,24,25,27,28,29,32,33,35,36,38,39,40,42,45,46,47,53,56,57,58,59,60,61,62,65,66,68,69,72,73,75,78,80,81,82,84,85,86,87,88,89,90,91,92].

Figure 7.

Bivariate summary receiver operating characteristic (SROC) curve of deep learning models for detecting intracranial hemorrhage on non-contrast computed tomography according to retrospective studies.

According to the data from eleven prospective studies, a pooled sensitivity of 0.90 (95% CI 0.87–0.92, I² = 68%) and a pooled specificity of 0.95 (95% CI 0.94–0.96, I² = 85%) were achieved (Figure 8). Pooled PPV and pooled NPV were 0.79 (95% CI 0.71–0.85, I² = 93%) and 0.98 (95% CI 0.97–0.99, I² = 81%), respectively (Figure 9). The bivariate model based on the results from prospective studies showed an AUC of 0.97 (95% CI 0.95–0.98) (Figure 10).

Figure 8.

Sensitivity and specificity of deep learning models in detecting intracranial hemorrhage on non-contrast computed tomography according to prospective studies [20,37,48,49,50,51,55,67,74,76,94].

Figure 9.

Positive and negative predictive values of deep learning models in detecting intracranial hemorrhage on non-contrast computed tomography according to prospective studies [20,37,48,49,50,51,55,67,74,76,94].

Figure 10.

Bivariate summary receiver operating characteristic (SROC) curve of deep learning models for detecting intracranial hemorrhage on non-contrast computed tomography according to prospective studies.

3.2. Publication Bias

Funnel plots and Egger’s regression test were used to estimate publication bias in included studies (Figure S1a–d). Eggers’ test does not indicate the presence of funnel plot asymmetry in pooled specificity (p-value 0.9446), pooled PPV (p-value 0.0976), and pooled NPV (p-value 0.4335); however, it shows funnel plot asymmetry in pooled sensitivity (p-value 0.0135).

4. Discussion

ICH is a life-threatening condition, associated with a very high morbidity and mortality rate; thus, early diagnosis and treatment are critical to improve the prognosis of patients [96,97]. AI has the potential to help radiologists in ICH detection from NCCTs and enhance clinical workflow [46,59,76,98,99]. DL models can learn to identify subtle features associated with ICH that are difficult for radiologists to detect, thus saving time and directing radiologists’ attention to the most critical cases. In emergency settings, identifying bleeding as a binary classification is crucial in prompt intervention’s first step. In one study, AI-assisted re-evaluation of head NCCTs improved the mean accuracy of board-certified emergency physicians in identifying ICHs from 87.70% to 93.85% [98], which shows the potential usefulness of AI tools in reducing missed positive cases by radiologists in clinical practice. This systematic review and meta-analysis aimed to determine the diagnostic accuracy of DL models in detecting ICH (presence or absence) on NCCTs. The results showed that DL has a promising diagnostic performance in the automated detection of ICH from NCCTs with a high sensitivity and specificity of 0.92 and 0.94, respectively. The PPV, NPV, and AUC were also found to be 0.84, 0.97, and 0.96, respectively.

A DL model that has more sensitivity and fewer false positives can greatly reduce the workload of radiologists. Although the goal is to increase the sensitivity for the detection of true positive cases as much as possible, the feature-based balance, which is related to the detection of true negatives, should be considered the ultimate goal for an ideal DL model in emergency settings. Dyer et al. developed a DL model with EfficientNet neural network architecture for triaging patients with head CT abnormalities. The model was validated on a multi-center dataset and reached a sensitivity of 98.8% and a specificity of 92.5% for ICH detection from NCCTs [45]. Cho et al. proposed a DL model using a cascade of CNNs and dual fully convolutional networks (FCNs) for detecting the presence or absence of bleeding on NCCTs. In this model, the sensitivity and specificity were found to be 97.91% and 98.76%, respectively.

In recent years, several commercial AI tools have been developed to identify ICH in NCCTs. Aidoc is a commercially available, FDA- and CE-cleared CNN-based AI tool for detecting ICH from NCCTs. It has been trained and tested on CT scans from 9 medical centers and 17 different CT machines [57]. In a study by Buls et al., the Aidoc tool was used to detect ICH cases from a local dataset consisting of 388 NCCTs. The results showed a sensitivity of 84% and a specificity of 0.94% [36]. In another study, Ginat used Aidoc as a tool for prioritizing positive ICH cases on NCCTs. This study used a large local dataset of 8723 NCCTs from a single tertiary care center. The AI tool reached a sensitivity and specificity of 88.4% and 96.1%, respectively [49]. However, due to its single-center design, the findings may not apply to other practice settings. Conversely, in another study by Kau et al., the diagnostic accuracies of a second-year resident and the Aidoc differed highly significantly (p < 0.001) and both resident and neuroradiologist under time pressure outperformed the Aidoc in ICH detection from NCCTs [57].

RAPID ICH is another commercial AI tool that uses a hybrid deep 2D–3D deep CNN structure for detecting ICH on NCCTs [46,53]. Testing this AI tool on a single-center small dataset showed acceptable sensitivity, specificity, and NPV of 91.9%, 84.4%, and 98.7%, respectively, but a low PPV of 44.7%. In this study, the most common causes of high false-positive cases were motion, streak artifacts, calcifications, and dural thickening [46]. Another study by Heit et al. assessed the performance of the RAPID ICH tool for ICH detection from NCCTs. Testing the AI tool on a local independent dataset from the training dataset showed a sensitivity of 95.6% and a specificity of 95.3%. The tool also achieved a high PPV and NPV of 95.6% and 95.3%, respectively, for ICH detection [53].

Recently, another AI application called Viz.ai ICH has been developed to enhance and expedite the diagnosis of suspected ICH cases from NCCTs. Analyzing 4203 NCCTs, compared to the gold standard of the neuroradiologist’s impression, Viz.ai ICH demonstrated a sensitivity of 85% and a specificity of 98% enhancing radiologists’ workflow by correctly detecting NCCTs without bleeding as negative for ICH. For ICHs with larger volume/size, the sensitivity improved to 100% [72].

Another study by McLouth et al. validated a commercially available DL-based tool called CINA v1.0 device for detecting ICH [65]. The tool showed high sensitivity, specificity, and accuracy of 91.4%, 97.5%, and 95.6%, respectively, for ICH detection. The sensitivity reached 100% for medium (5–25 mL) and large (>25 mL) ICH. Del Gaizo tested the performance of the CINA v1.0 device on a large dataset for detecting ICH. The AI tool yielded reduced sensitivity and specificity compared with the published literature with a sensitivity of 75.6%, specificity of 92.1%, and accuracy of 91.7%. The authors concluded that the performance of this application for ICH detection was low in a low-prevalence environment (2.7%) because of the increased median time that the radiologists spent on reading false-positive and false-negative findings (1 min 14 s and 1 min 5 s, respectively), potentially reducing system efficiency [44]. This highlights the importance of testing AI tools in different clinical settings to achieve a better institutional decision-making system.

Since distinguishing ICH from bony adjacent regions on CT scans may be challenging for radiologists [78,100], applying image processing techniques by removing skull and facial bones will help retain only the intracranial structures and reduce the misdetection of ICH cases [60]. The present models have been specifically trained for ICH detection and not trained for the detection of other intracranial pathologies such as tumors, hydrocephalus, intracranial calcifications, and ischemic infarcts [32,33,39,95], resulting in false positive cases which may be seen as a drawback of the existing DL models. Another limitation in detecting ICH with AI tools is the difficulty of differentiating between normal brain tissue and some specific subtypes of ICH such as SDH in the subacute phase and small SAHs when mixed with cerebrospinal fluid (CSF) [56,76].

4.1. Limitations

This study has some limitations. Most of the included studies had a retrospective design, highlighting the gap in prospectively evaluating DL-based models in real-world clinical settings. The diversity of CT scanners, vendors, gold standards, and datasets utilized for training and validating DL methods makes it difficult to interpret the combined performance results.

Most studies did not provide sufficient separate data on the diagnostic performance of DL models for detecting different ICH subtypes (e.g., SAH, SDH, IVH, etc.). This could be a topic for future research to create datasets that include only specific ICH subtype cases to assess the diagnostic performance of DL models for each subtype. Additionally, there is limited information on the diagnostic performance of DL models concerning hemorrhage volume. This factor is critical, as smaller hemorrhages are more likely to be overlooked in emergency situations compared to larger, diffuse intracerebral hemorrhages. Because of that, in certain scenarios, subtle bleeding can be clinically significant (e.g., minor SDHs in anticoagulated patients or subtle cisternal aneurysmal SAH) and may benefit from assistance by DL algorithms. Moreover, there remains insufficient data on whether utilizing AI tools independently or as assistance for radiologists reduces the interpretation time and workload of radiologists. Furthermore, more studies are needed to assess the cost-effectiveness of using AI solutions as a diagnostic tool in clinical settings.

4.2. Ethical Implications and Responsibilities

Integrating DL models into radiological diagnostics offers many potential benefits, but also carries significant ethical implications and responsibilities. Strict adherence to data privacy regulations and robust cybersecurity measures must be implemented. There may be bias in training datasets; therefore, efforts should be made to make diverse and representative datasets and the benefits of using DL in radiology available to all populations, regardless of socioeconomic status or geographic location. DL models can be complex, making it difficult to understand how they arrive at their decisions, which raises concerns about their accountability and trust; thus, efforts to improve their transparency and explainability are needed. Radiologists must maintain their critical thinking skills and not blindly rely on reports generated by AI. In emergency scenarios when an AI model makes a mistake, determining liability can be problematic; therefore, clear guidelines and legal frameworks are needed. We need to validate and rigorously test DL models before applying them to clinical practice.

4.3. Directions for Future

AI tools can aid radiologists in screening CT scans, helping them focus on complex cases. In the future, they are anticipated to play an important role in the education of radiology residents. Where resources and expertise are limited, AI solutions can help diagnose abnormalities in CTs. Considering that less experienced radiologists might miss subtle hemorrhages, future studies should aim to compare the reports produced by the AI tools with those from radiologists to estimate false positives or false negatives accurately. Additionally, DL algorithms can standardize the interpretation of CT scans, reducing inter-observer variability in diagnosing ICH from NCCTs. In clinical settings, AI applications can act as a real-time second reader for images reported as normal, making sure that no positive cases are overlooked. Obtaining larger, more diverse, and accurately annotated datasets for training may increase the generalizability and reproducibility of DL models. Lastly, DL algorithm codes ought to be accessible publicly to researchers in the field so they can enhance and refine their models. Additional prospective multicenter studies with larger datasets are needed to confirm the efficacy of DL-based tools in identifying ICH in NCCTs.

5. Conclusions

This meta-analysis showed that the DL-based models have acceptable diagnostic performance for detecting ICH from NCCTs, underscoring a promising potential for the use of AI tools in various practice settings. More prospective studies are needed to confirm the potential clinical benefit of implementing DL-based tools for automated ICH detection and their impact on clinical workflow and patient outcomes, especially mortality.

Abbreviations

• The following abbreviations are used in this manuscript:

ICH	Intracranial hemorrhage
AI	Artificial intelligence
DL	Deep learning
NCCT	Non-contrast computed tomography
ANN	Artificial neural networks
CNN	Convolutional neural network
IPH	Intraparenchymal hemorrhage
SDH	Subdural hemorrhage
EDH	Epidural hemorrhage
IVH	Intraventricular hemorrhage
SAH	Subarachnoid hemorrhage
AUC	Area under curve
TP	True positive
TN	True negative
FP	False positive
FN	False negative
PPV	Positive predictive value
NPV	Negative predictive value
RNN	Recurrent neural networks
LSTM	Long Short-Term Memory
FCN	Fully convolutional networks
DCNN	Deep convolutional neural networks

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/jcm14072377/s1, Table S1: Quality Assessment of Diagnostic Accuracy Studies-2 (QUADS-2) risk of bias assessment. Figure S1: Funnel plots of sensitivity (a), specificity (b), PPV (c), and NPV (d).

Author Contributions

Conceptualization, A.K. and A.S.; methodology, software, validation, formal analysis, A.K.; investigation, A.K. and A.S.; data curation, A.K.; writing—original draft preparation, A.K.; writing—review and editing, A.K. and A.S.; visualization, A.K.; supervision, A.S. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

No new data were created in this study. All required data are available in the included studies.

Conflicts of Interest

The authors declare no conflicts of interest.

Funding Statement

This research received no external funding.