Application of convolutional network models in detection of intracranial aneurysms: A systematic review and meta-analysis

Abstract

Introduction

Intracranial aneurysms have a high prevalence in human population. It also has a heavy burden of disease and high mortality rate in the case of rupture. Convolutional neural network(CNN) is a type of deep learning architecture which has been proven powerful to detect intracranial aneurysms.

Methods

Four databases were searched using artificial intelligence, intracranial aneurysms, and synonyms to find eligible studies. Articles which had applied CNN for detection of intracranial aneurisms were included in this review. Sensitivity and specificity of the models and human readers regarding modality, size, and location of aneurysms were sought to be extracted. Random model was the preferred model for analyses using CMA 2 to determine pooled sensitivity and specificity.

Results

Overall, 20 studies were used in this review. Deep learning models could detect intracranial aneurysms with a sensitivity of 90/6% (CI: 87/2–93/2%) and specificity of 94/6% (CI: 0/914–0/966). CTA was the most sensitive modality (92.0%(CI:85/2–95/8%)). Overall sensitivity of the models for aneurysms more than 3 mm was above 98% (98%-100%) and 74.6 for aneurysms less than 3 mm. With the aid of AI, the clinicians’ sensitivity increased to 12/8% and interrater agreement to 0/193.

Conclusion

CNN models had an acceptable sensitivity for detection of intracranial aneurysms, surpassing human readers in some fields. The logical approach for application of deep learning models would be its use as a highly capable assistant. In essence, deep learning models are a groundbreaking technology that can assist clinicians and allow them to diagnose intracranial aneurysms more accurately.

Introduction

Intracranial aneurysms are a fairly prevalent vascular abnormality with a prevalence of 3.2%. ¹ Rupture of intracranial aneurysms may also lead to subarachnoid hemorrhage and be fatal. Aneurysmal subarachnoid hemorrhage is responsible for 85% of all subarachnoid hemorrhage cases. ² The mortality rate of subarachnoid hemorrhage due to aneurysms is reported to be 23 to 51 percent,^2–4 and it leads to 10 to 20 percent disability. ⁵ However, unruptured intracranial aneurysms are the most prevalent incidental finding in normal individuals. ⁶ This emphasizes the importance of detection of such lesions since there is an overall 1.4% risk per year for an intracranial aneurysm to rupture.^7–10 Increased use of various neuroimaging methods such as CT angiography and MRA has led to an increased detection rate of intracranial aneurysms. ¹¹

Convolutional neural network (CNN), a specific type of deep learning network architecture, has been implemented in recent years and can be applied for classification and detection in medical imaging.^12–14 Use of CNN may be particularly easy since it learns the outcome classifications automatically rather than needing any explicit categorization. ¹⁵ CNN has been used successfully in detecting various lesions in medical imaging such as pulmonary nodules, glaucoma in retinal fundus images, and even diagnosis of covid-19 in X-rays and CT scans.^16–18 These attempts at overcoming difficulties regarding automatic detection and segmentation of medical imaging have proven to be efficient and successful. ¹⁹

There have been numerous studies on the use of CNN in detection of intracranial aneurysms that have had controversial results. These studies were done in various methods and different choices of modalities for detection of intracranial aneurysms.^20–22 Addressing this vastness in features of studies previously carried out, we not only reviewed the outcome of articles discussing intracranial aneurysms, but also took a step farther and surveyed this issue in the eyes of clinicians.

Methods

Eligibility criteria

Studies which used CNN methods to detect intracranial aneurysms and reported the sensitivity of deep learning (DL) for detection of intracranial aneurysms were included in this study. The exclusion criteria were non-English articles, ongoing trials, modeled articles, abstracts, methodologies, conference papers, case reports and case series, editorials, letter to editors, and review articles.

Information sources

PubMed, Scopus, Cochrane library and Embase were searched by two separate authors simultaneously. References of articles were explored to cover any missed articles. The search process was performed in November 2021.

Search strategy

The related words and synonym used for artificial inelegance were: “Artificial Intelligence, Computational Intelligence, Machine Intelligence, Computer Reasoning, AI, Computer Vision System, Machine Learning, Deep Learning, Supervised Machine Learning, Unsupervised Machine Learning, Intelligence Retrieval and Neural Network. “These words were combined using “OR” and then combined with the words “Intracranial” and “Stenosis” to make the final search strategy. The filters that were used were Title-abstract filter for PubMed and Embase and Title-abstract-keywords for Scopus. No limits were used in the search process. The search strategy was kept broad to not only catch all related articles, but also study other aspects of using DL for vascular lesions of intracranial aneurysms; then, the articles related to the detection of aneurysms were selected. The details of search in each database are provided in supplementary material 1.

Selection and data collection process

The selection and data collection process were all performed manually by two separate author without using any automation tool in two steps; in case any conflicts were noticed between the researchers, a third author was consulted. All exports from each database was exported in an endnote library(Version 8; ^{23 )} then, the duplicates were removed. In the first step of selection process, titles and abstracts were read, and the selected articles entered the second step which was reading the full text of articles and checking to see if they were matched to our eligibility criteria. The included articles were discussed in the group, and we then decided which outcomes were needed to be extracted. Figure 1 reflects these processes in a PRISMA flowchart.

Figure 1.

PRISMA flow chart of the included studies determining sensitivity and specificity of clinicians and AI models for detection of intracranial aneurysms .383 articles were identified and after removing duplicates 198 articles were screened. 4 articles were added manually and finally 20 articles were included in this review.

Data items

We sought to extract the demographic data and outcomes from the studies. The year of publication, architecture of machine learning(ML) system, imaging modality, and number of training and test sets were also gathered as complementary data. Overall, location-based and size-based sensitivity and specificity regarding each modality were our main outcomes.

Study risk of bias assessment

Quality and risk of bias were assessed using national institute of health tool for observational studies. Two independent authors did it for each article. This tool consists of 14 questions that assess different parts of studies, and for each question there are three options of “Yes” for being according to that index, “No” for not consider it and cannot reported, or not applicable or not reported for other conditions.

Effect measures and synthesis methods

Effect size was determined using comprehensive meta-analysis version 2. ²⁴ Four different analyses were performed regarding each modality: 1) the overall sensitivity and specificity of ML for detection of aneurysm without considering the place and size of lesions, 2)sensitivity of ML for detection of aneurysm based on the location and size of aneurysm, 3)comparison of sensitivity of ML compared to human readers and to the combination of these two together, and 4) the changes in interrater agreement with and without use of ML model. Final data were reported as event rate (sensitivity, specificity and interrater agreement) in a forest plot. Heterogeneity of data was explored using I². Duval and Tweedie trim and fill method was used to determine the risk of publication bias. Sensitivity of the results to an article was assessed by dropping out the articles one-by-one and re-analyze the rest of articles.

Results

Study selection

Our search result included 383 articles. After omitting the duplicates, 198 articles remained for screening. Eventually, 16 articles were included for quantitative discussion. 26 articles were excluded because they had assessed other aspects of using ML for intracranial arteries rather than detection of aneurysms, and some were excluded because they had not reported sensitivity and/or specificity. 4 articles that were found from references of articles were added parallelly. PRISMA flowchart comprehensively displays this process (Figure 1).

Study characteristics

7 studies used computed tomography angiography (CTA),^25–31 9 used time of flight magnetic resonance angiography (TOF-MRA),^22,32–39 and 4 used Digital Subtraction Angiography (DSA).^40–43 18 out of 20 articles had retrospective study design, and the two others used randomized crossover ²⁵ and crossover design. ²⁷ 6 articles compared the sensitivity and/or specificity of ML model to the human readers.^{25,27,28,31,32,36} Sensitivity of ML and human reader based on location and size of aneurysms was reported in7^{22,30,33–36,38} and5^{22,26,33,34,36} articles, respectively.

Risk of bias in studies

Risk of bias and quality assessment was found to be desirable.

Results of synthesis and biases

Deep learning models could detect intracranial aneurysms with a lesion-wise sensitivity of 90.6% (CI: 87/2–93/2%) and a patient-wise sensitivity of 96.4%(CI: 95/2–97/3%) and specificity of 94/6%(CI: 0/914–0/966). CTA was the most sensitive and least specific modality for detection of intracranial aneurysms regardless of their size and location with a sensitivity and specificity of 92/0%(CI:85/2–95/8%) and 68/8%(CI: 34/8–90/1%), respectively. The highest specificity was reported in TOF-MRA (96/0%, CI: 93/3–97/6%). Overall sensitivity of the models for aneurysms more than 3 mm was above 98%(98%-100%) and 74/6%(CI: 70/8–78%) for intracranial aneurysms less than 3 mm, and CTA was more sensitive than modalities for intracranial aneurysms less than 3 mm (84/1%(63/1–94/2%). Sensitivity of DL models for posterior circulation aneurysms was the least, while the models detected ACA aneurysms with the most sensitivity(80.6%(CI:73/7–86.0%) versus 94/5(CI:89/9–97.1%). Clinicians diagnosed aneurysms with a lesion-wise and patient-wise sensitivity of 80/9% and 79.8 respectively. With the aid of artificial intelligence (AI), clinicians could mark aneurysms with a sensitivity of 93/7%(CI:0/686–0/990). It is worth mentioning that interrater agreement increased 0/193 unit after applying AI assistance (0/668 vs. 0/862). The details of our analyses are shown in Tables 1 and 2 and Figure 2. More detailed tables are included in supplementary material. Duval and Tweedie trim and fill method revealed no publication bias.

Table 1.

Characteristics and primary outcomes of the included studies determining sensitivity and specificity of clinicians and AI models for detection of intracranial aneurysms.

Study	Year	Architecture	Modality	Study design	Number of training set/validation/test set	Model	Overall sensitivity of model	Overall specificity of model (Rate)
Park et al. 25	2019	3D-CNN	CTA	Randomized Crossover	611/92/115	HeadXNet	Patient-wise: 94/9	Patient-wise: 66/1)
Faron et al. 32	2020	3D-CNN	TOF-MRA	Retrospective	58/10/17 Five-fold cross validation	DeepMedic	90
Joo et al. 22	2021	3D-CNN	TOF-MRA	Retrospective	421/47/332	3D-ResNet + 3D UNet	Patient-wise: 91.11 Lesion-wise: 92/26	Patient-wise: 93/91
Joo et al. 33	2020	3D-CNN	TOF-MRA	Retrospective	468/-/106/574	3D-ResNet	85/7%	98%
Shahzad et al. 26	2020	3D-CNN	CTA	Retrospective	79/215 Five-fold cross validation		82%
Bo et al. 27	2020	3D-CNN	CTA	Crossover	1186/152	GLIA-Net	Patient-wise:96/2 lesion-wise:85/8	Patient-wise:38/9 lesion-wise:19/3
Sichtermann et al. 34	2018	CNN	TOF-MRA	Retrospective	58/10/17 Five-fold cross validation	DeepMedic	90
Liu et al. 40	2021	3D-CNN	DSA	Retrospective	347/41/63	3D-Dense-UNet	88.4
Pennig et al. 28	2021	3D-CNN	CTA	Retrospective	79/126 Five-fold cross validation		85/7
Nakao et al. 37	2017	2D-CNN	TOF-MRA	Retrospective	300/50/100		70
Hou et al. 35	2020	1D-CNN	TOF-MRA	Retrospective	245/35/70		93/16	96/87
Hainc et al. 41	2020	CNN	DSA	Retrospective	565/141 Cross validation		79	79
Sohn et al. 36	2021	3D-CNN	TOF-MRA	Retrospective	600/110/332	3D-ResNet	Patient-lesion- wise :74/8	93/9
Shi et al. 29	2020	3D-CNN	CTA	Retrospective	927/100/150	DAResUNet	Patient-wise: 97/3(150) Lesion-wise:95/6	89/7
Zeng et al. 43	2019	3D-CNN	DSA		300		99/38	98/19
Dai et al. 30	2020	2D-RCNN	CTA	Retrospective	208/-/103	ResNet	94/3
Ueda et al. 38	2018		TOF-MRA	Retrospective	683/-/67Five fold cross validation	ResNet	93
Stember et al. 39	2018	2D-CNN	MRA + TOF MRA	Retrospective	250/250/86	U-Net	98/8
Yang et al. 31	2021	3D-CNN	CTA	Retrospective	534/534/400 Five fold cross validation		97/5
Jin et al. 42	2020	2D-CNN	DSA	Retrospective	249/98/146		Lesion wise:89/3 Patient wise: 97/7

CNN: Convolutional neural network, DLM: Deep learning model, CTA: Computed tomography angiography, TOF-MRA: Time-of-flight magnetic resonance angiography, DSA: Digital subtraction angiography, MRA: magnetic resonance angiography.

Table 2.

Results of meta-analysis of the included studies determining sensitivity and specificity of clinicians and AI models for detection of intracranial aneurysms.

Analyses	Subgroup	Outcome (% (CI 95%))
Overall Sensitivity of Model(lesion-wise)	Overall	90/6, CI: 87/2- 93/2
	CTA	92/0, CI: 85/2- 95/8
	DSA	90/2, CI: 82/6- 94/7
	MR	90/0, CI: 84/1- 93/9
Overall Sensitivity of Model(Patients-wise)		96/4, CI: 952- 973
Overall Sensitivity of Model	Overall	94/6, CI:91/4–96/6
	CTA	68/8, CI:34/8- 90/1
	DSA	93/2, CI: 50/3–99/5
	MR	96/0, CI: 93/3–97/6
Sensitivity of AI to detection of aneurisms base on their size	<3mm	74/6, CI: 70/8- 78/0
	3–5 mm	98/1, CI: 97/3- 98/6
	3–7 mm	94/3, CI: 89/7- 96/9
	>5mm	98/0, CI: 96/6- 98/8
	>10mm	100, CI: 99/8–100
Sensitivity of AI to detection of aneurisms base on their size	ICA	92/8, CI: 92/8- 88/8
	MCA	93/3, CI: 89/7- 95/7
	ACA	94/5, CI: 89/9- 97/1
	Posterior circulation	80/6, CI: 73/7- 86/0
Overall senstivity of clinicians to detect intracranial aneurysms (lesion-wise)		80/9, CI: 69/5- 88/7
Overall senstivity of clinicians to detect intracranial aneurysms (patient-wise)		79/8, CI: 78/6- 81/0
Sensitivity of clinicians to detection of aneurisms base on their size	<3mm	82/8, CI: 48/8- 96/0
	>3mm	90/8, CI: 80/5- 95/9
Sensitivity of clinicians to detection of aneurisms base on their location	ICA	84/3, CI: 50/3- 96/6
	MCA	94/6, CI: 93/2- 95/8
	ACA	67/7, CI: 12/3- 96/9
	Posterior circulation	85/6, CI: 58/2- 96/2
Overall senstivity of clinicians to detect intracranial aneurysms with the help of AI		92/2, CI: 87/2- 95/3
Interrater agreement before AI augmentation		66/8, CI: 34/4- 88/5
Interrater agreement after AI augmentation		86/2, CI: 83/9- 88/2

Figure 2.

Overall and based on modality senstivity of AI models to detect intracranial aneurysms (Lesion-wise)(Over:Event rate: 0/906, CI: 0/872–0/932, I²:96/42, Duval and Tweedie's trim and fill model not significant, senstivity analysis not significant, CTA: event rate: 0/920, CI: 0/852–0/958, I²:97/08, DSA: event rate: 0/902, CI: 0/826–0/947, I²:94/11, TOF-MRA: event rate: 0/900, CI: 0/841–0/939, I²:97/19).

Discussion

To the best of our knowledge, this is the first systematic review and meta-analysis on the determination of the performance of AI models and how they could aid clinicians for better diagnosis of intracranial aneurysms. In general, deep learning models were able to detect aneurysms regardless of their size and location with an acceptable sensitivity compared to human readers. Additionally, DL models assisted the clinicians to detect more lesions and also increased inter-observer agreement. Our discussion will focus on the interpretation and analysis of our results from a clinical standpoint, rather than details of model development.

Reviewed studies can be categorized into 2 groups: first, studies that investigated the performance of DL models detection of intracanal aneurysms and second the studies which not only performed the previous test, but also compared these models to the performance of human readers. Before discussing the interpretation of these studies in more detail, it should be noted that determining true positive aneurysms (ground truth) was performed by experienced clinicians who were not error-free. An example of this can be found in the Ueda et al.'s study, where the model detected aneurysms that were not detected by the reference model. ³⁸ This would be a double-edged sword. On the one hand, it demonstrates that these models can identify aneurysms that even a highly experienced clinician would not find. On the other hand, it casts doubt on the accuracy of all results. Nonetheless, we cannot make any other choice than accepting the reference clinician's opinion as the ground truth.

The search for additional aneurysms may be stopped because the human reader has already found one lesion, a phenomenon called satisfaction of search. ⁴⁴ The importance of this will be most apparent in emergency settings when missing a lesion can lead to a catastrophe. Even though this phenomenon may concern human readers, DL models could simply prevent it via searching for an infinite number of lesions. A study only reported patient-wise sensitivity; that is, when the model found a lesion, it stopped searching for additional ones, an observation we would like to call the satisfaction of search for AI model. ²⁵ This is an undesirable feature if we ever want to rely on DL models solely without the supervision of a clinician due to staff shortages. Accordingly, we analyzed the sensitivity of detection of aneurysms both patient-wise and lesion-wise. Our results indicate that machine learning is more sensitive than human readers alone, but when combined altogether, the results will be more promising than either alone.

Size

The diameter of intracranial aneurism is an important determinant of whether an intervention is needed or not. According to the stroke council of the American Heart Association and Greving et al., in patients without a history of SAH with aneurysms below 7 mm, observation is preferred to intervention, but for those above 7 mm and patients who develop symptoms from intradural aneurysms, treatment becomes necessary.^45,46 The risk of rupture is estimated to be 0/36% per year for aneurysms with a diameter of 3–4 mm, while treatment is suggested for patients with a high risk of rupture.^47–49 While asymptomatic patients may not require urgent intervention for <3 mm aneurysms, analyzing the sensitivity of DL model and comparing it to human readers shed light on the limitations of these models in detecting such aneurysms besides the fact that the incidence of these aneurysms are increasing.^49,50 The sensitivity of human readers to detect these aneurysms was pretty more compared to DL models. Clinicians’ detection rate was solely based on MR modality due to limitations of the analyzed studies, where machine learning could detect these tiny aneurysms more accurately if we only consider CTA. The hazard ratio for the risk of rupture for aneurysms 7–9 mm was reported 3/3, 9/1 for 10–24 mm and 76/3 for aneurysm sizes more than 25 mm compared to tiny aneurysms. ⁵¹ Our study showed an average sensitivity of 94% for aneurysms above 3 mm of diameter, and it reached a sensitivity of 100% for those greater than 10 mm. These sensitivities were higher than those of the clinicians and demonstrates the power of machine learning.

Location

The international study of unruptured intracranial aneurysms (ISUIA) has stratified the risk of the rapture (RoR) of aneurysms based on the location into low, intermediate, and high. Cavernous carotid artery aneurysms RoR is considered low, anterior circulation and internal carotid arteries are considered intermediate, and posterior circulation high. ⁵² The neurological approach to aneurysms of the posterior territory is complicated due to their location, size of these aneurysms, and connection of vessels. Aneurysms of the posterior territory account for 15% of intracranial aneurysms.^53–55 Pooled results showed less sensitivity of DL models in detection of posterior circulation aneurysms compared to human readers. Sensitivity of DL models in CTA for detection of posterior circulation aneurysms was more than of human readers in the same manner that DL models based on CTA were more successful than human readers in detecting small sized aneurysms. Anterior circulation makes up 85% of total intracranial aneurysms. ⁵⁶ DL models could detect anterior circulation aneurysms with a higher detection rate compared to those of clinicians and the same result was observed for aneurysm of the ICA either, but human readers showed a more sensitivity rate for detection of MCA aneurysms. These differences can be justified due to various reasons. It can be argued that posterior circulation contains more bony structures, which could decrease the accuracy of CNN models. An aneurysm may be missed or mis-detected in the case of overlap of vessels or curvatures of an artery.

Modality

For unruptured intracranial aneurysms, DSA is considered the diagnostic gold standard since it provides precise information about the size, shape, and the orifice size of the aneurysm and can be used to determine future treatment options. ⁵⁷ However, the use of CTA or MRA may be preferred due to being less invasive, having lower cost, and being more agile and accessible in versatile facilities. ⁵⁸ Regardless of their advantages, CTA and MRA come with some limitations that make them an unfitting substitute for DSA, for example, limitation of CTA on detection of small aneurysms or aneurysms in MCA and some parts of ICA^59,60 and artifacts of TOF-MRA when there is tortuosity in the vessels or a turbulent flow. ⁶¹ These limitations increase the amount of false positive and negative cases which decrease the overall sensitivity and specificity of these modalities regardless of a human's interpretation or that of an AI-based model. Also, contrast material should be injected for CTA and DSA that may be contraindicated for a number of patients where those undergoing MRA should be MRI compatible.^57,62,63 Removal of bony parts were used in several studies in the preprocessing step. It should be noted that without a sufficient model or human supervision, some calcified lesions may also be omitted, which leads to misdiagnosis. ⁵⁷ By and large, the modality with most sensitivity in the detection of aneurysms is desirable. The analysis pointed out that models using CTA had the highest sensitivity and those based on DSA and MRA had approximately the same lower rates. CTA-based model also showed better performance for detection of lesions under 3 mm and in the posterior circulation. It was a question for authors if machine learning can detect previously treated aneurysms that recur. Metal elements used for treated aneurysms produce artifacts that decrease the image quality and make it difficult for the model to detect the regrown aneurysms. ⁶⁴ Some studies found that the CNN model could successfully find new aneurysms that were previously coiled in CTA.^30,33

Aneurysmal subarachnoid hemorrhage

Aneurysmal subarachnoid hemorrhage(aSAH) is a serious complication of intracranial aneurysms rapture. ⁶⁵ SAH has a high mortality rate both in the short and long term and accounts for a 30-day mortality rate of 45%.^66–69 Common signs and symptoms of SAH are headache, nausea, vomiting, mesangial signs and decreased level of consciousness. ⁷⁰ CT scan without contrast is the best primary workup for diagnosis of SAH and has a sensitivity of 98.7% for detection of SAH in the first 6 h.^71,72 It should be noticed that most aneurysms do not rupture, have no symptoms, and sometimes may even be found incidentally; however, in case they rupture and are missed, the results would be fatal. ⁶⁴ Two studies designed a DL model to detect aneurysms in the context of aneurysmal SAH based on CTA. Both models could detect aneurysms with an acceptable sensitivity and proved that machine learning could be helpful in the emergency setting for detection of such aneurysms.^26,28

Morphology of aneurysms

There is another classification for brain aneurysms based on their shape. Aneurysms could be divided into groups, saccular(90%), and fusiform(3–13%).^73,74 Fusiform aneurysms have a lower risk of rupture and are mostly located on the posterior circulation. ⁷⁵ Due to their shape, fusiform aneurysms are difficult to detect by deep learning models. Some studies have even omitted them in their training sets and trained the model just to diagnose saccular ones. Yang et al. examined this issue more deeply and asked whether the sensitivity of the model was affected by saccular or fusiform aneurysms. While the model showed high sensitivity across both classes, it was higher for saccular aneurysms. ³¹

Limitations

In almost all analyses, there was a high level of heterogeneity. Some explanations may interpret this issue. The first and most important issue is different models used by articles. The second reason may be different inclusion and exclusion criteria and number of data sets. Another reason could be different pre-processing algorithms used before training and testing data such as data augmentation, removal of bony parts, and intensification of relevant parts. We have performed many subgroup analyses to reduce heterogeneity, but it was not successful. Another limitation of our study was the scarcity of modalities in different subgroups. Specificity was not reported by studies based on the location and size of aneurysms to be analyzed.

Conclusion

To address whether or not machine learning is reliable to detect aneurysms in routine clinical practice and emergencies, for a variety of sizes and locations, we found that CNN models had an acceptable sensitivity for detection of intracranial aneurysms, surpassing human readers in some fields. Nevertheless, some studies reported low specificities in CTA models which may limit its usage. Thus, the logical approach will be using machine learning as an assistant, not as a sole decision maker. Moreover, CTA had the best performance among modalities regarding sensitivity and could swiftly detect aneurysms in pitfalls rather than other modalities. In essence, deep learning models are a groundbreaking technology that can assist clinicians and allow them to diagnose intracranial aneurysms more accurately.