Artificial intelligence for detecting traumatic intracranial haemorrhage with CT: A workflow-oriented implementation

Abstract

The objective of this study was to assess the performance of an artificial intelligence (AI) algorithm in detecting intracranial haemorrhages (ICHs) on non-contrast CT scans (NCCT). Another objective was to gauge the department’s acceptance of said algorithm. Surveys conducted at three and nine months post-implementation revealed an increase in radiologists’ acceptance of the AI tool with an increasing performance. However, a significant portion still preferred an additional physician given comparable cost. Our findings emphasize the importance of careful software implementation into a robust IT architecture.

Introduction

Non-contrast cranial computed tomography (NCCT) is widely available and remains the gold standard imaging modality for diagnosing intracranial haemorrhage (ICH), offering an excellent sensitivity and specificity. ¹ The overall incidence of spontaneous ICH is 24.6 per 100,000. ² Early detection of ICH enables physicians to start treatment early. Very early intervention may help arrest active bleeding and limit haematoma expansion. ³ Mortality rates following ICH vary depending on the duration of follow-up, with short-term mortality typically lower than long-term mortality ranging from 34% to up to 53% in a one-year period.^4–8

Research and clinical application of artificial intelligence (AI) has increased in the context of healthcare and become more relevant. AI has found use in the setting of the emergency room (ER), ⁹ in the context of triaging patients with a potential diagnosis of COVID-19 ¹⁰ and even in order to diagnose acute conditions such as strokes. ¹¹

Multiple studies have evaluated the use of AI in the acute setting for the detection of ICH on non-contrast cranial CT (NCCT), demonstrating promising diagnostic accuracy. Among the commercially available products, the AIDOC-software has been investigated in multiple peer-reviewed publications, with high sensitivity and specificity consistently reported across a variety of clinical settings. AIDOC is able to demonstrate a sensitivity between 84 and 99% and specificity between 93 and 99% in detecting intracranial haemorrhages.^12–19 These papers highlight not only AIDOC’s diagnostic performance but also its potential to assist with triage and prioritization in emergency radiology.

We hypothesized that the introduction of the AIDOC AI software would lead to moderate acceptance within the department and serve as a useful adjunct tool.

Materials and methods

We obtained institutional review board (IRB) approval for this prospective analysis.

Study design

We implemented a commercially available, AI-based detection tool for algorithm-aided detection and prioritization of ICH in August 2021. In October 2021, we performed an analysis of all NCCTs within a period of three weeks and compared radiologist reports with the notifications of the AI algorithm. Both 4 mm and 1 mm thick slices acquired on a Siemens Somatom CT scanner were sent to AIDOC for analysis. Finally, we have included a case study.

A neuroradiologist with 20 years of experience reviewed all CT scans and served as the reference standard, determining whether the original interpreting radiologist or the AI algorithm provided the correct assessment. After this three-week analysis, we conducted a survey in our university department (general hospital of Salzburg) to gauge all radiologists’ acceptance of the AI software. The survey included a question evaluating whether the algorithm was perceived to provide more value than the addition of a resident position (Figure 1).

Figure 1.

How the algorithm marks an intracranial haemorrhage (ICH) using computer vision.

In May 2022, that is, after an additional six months of constant usage of the AI algorithm in clinical practice by the radiologists, another three-week analysis and a second user survey were performed as described above.

We then compared our calculated sensitivity and specificity to other studies analyzing the AI algorithm.

False-negatives were defined as cases in which either the original interpreting radiologist or the AI algorithm misinterpreted the scan as not containing an ICH, when there was an ICH. False-positives were defined as cases, where the original interpreting radiologist or the AI algorithm detected an ICH, although no ICH was detectable on the scan. Discrepancies were defined as reports that didn’t contain an ICH according to the original radiologist’s interpretation, while they were flagged positive by the AI algorithm.

AI algorithm

The cloud-based AI algorithm (AI-Doc, Tel Aviv, Israel) is a medical grade deep learning algorithm, which is meant to be used as an interpretative aid to detect ICH. The algorithm is FDA approved and CE marked. The algorithm automatically analyzes every scan in the background at all hours and flags scans suspicious of containing ICH, prompting a widget popping up. The algorithms accuracy has been proved by multiple studies.^17,20,21 The scans are sent pseudonymized to the provider. When the analysis by the artificial intelligence algorithm is completed, the scans with the results are sent back and then the pseudonymization is reversed by the hospital’s IT department. During the duration of the study, the head of the radiology and the head of the neuroradiology department held biweekly meetings to give AI-Doc feedback.

Statistical analysis

For the statistical analysis, we used sensitivity, specificity, positive predictive value, and negative predictive value. We used SPSS to perform the statistical analyses.

Results

First analysis

From 22.09.2021 until 14.10.2021, a total of n = 372 NCCTs were read and interpreted by radiologists and the AI algorithm. There were no NCCTs, which fit in our time frame that were excluded. Out of these 372, n = 43 depicted an ICH. Out of the total 372, n = 34 scans were not interpreted by the AI algorithm equalling 9%. Out of those 34 scans, 11 included ICHs equalling 32% of scans not interpreted by the AI algorithm depicting an ICH (Table 1).

Table 1.

Table depicting 372 NCCTs interpreted by both an algorithm and a radiologist during the first analysis.

n = 372	True positive	True negative	False positive	False negative	Sensitivity	Specificity	Positive Predictive Value	Negative Predictive Value
AI algorithm	39	317	13	3	92.9%	96%	75%	99%
Radiologist	38	329	0	5	88.4%	100%	100%	99%

Second analysis

From 04.04.2022 until 24.04.2022, a total of n = 351 NCCTs were read and interpreted by radiologists and the AI algorithm. Out of these 351, n = 51 depicted an ICH. Out of the total 351 there were no NCCTs, which fit in our time frame that were excluded (Table 2).

Table 2.

Table depicting 351 NCCTs interpreted by both an algorithm and a radiologist during the second analysis.

n = 351	True positive	True negative	False positive	False negative	Sensitivity	Specificity	Positive Predictive Value	Negative Predictive Value
AI algorithm	40	289	11	11	78.4%	96.3%	78.4%	96.3%
Radiologist	44	299	1	7	86.3%	99.7%	97.8%	97.7%

First survey

Results of the first survey conducted in November 2021 with 27 out of 40 radiologists working at the university department in Salzburg, three months after having started using the algorithm, have been presented in Table 3.

Table 3.

Table visualizing the results of the first survey rated based on agreement.

n = 27	Strongly agree	Agree	Somewhat agree	Disagree	Strongly disagree
The algorithm makes me feel more secure in my reporting	2 (7.4%)	7 (25.9%)	11 (40.7%)	4 (14.8%)	3 (11.1%)
The algorithm is practicable for use with the PACS	2 (7.4%)	13 (48.2%)	9 (33.3%)	3 (11.1%)	0 (0%)
The algorithm increasing the speed of my reporting	1 (3.70%)	0 (0%)	4 (14.8%)	12 (44.4%)	10 (37.0%)
The algorithm makes me feel pressed for time	0 (0%)	1 (3.70%)	1 (3.70%)	8 (29.6%)	17 (62.9%)
The algorithm detected ICH that I overlooked	2 (7.4%)	0 (0%)	4 (14.8%)	12 (44.4%)	9 (33.3%)
Should the algorithm remain as a tool in the department?	5 (18.5%)	7 (25.9%)	7 (25.9%)	5 (18.5%)	3 (11.1%)
Considering the added value of the algorithm, would an additional resident position be more useful?	11 (40.7%)	8 (29.6%)	2 (7.4%)	3 (11.1%)	3 (11.1%)

Second survey

Results of the second survey conducted in May 2022, six months after the first survey and a total of nine months after having started using the algorithm, have been presented in Table 4. The survey was again conducted with radiologists in the university department in Salzburg (9 out of 40 radiologists) as follows:

Table 4.

Table visualizing the results of the second survey rated based on agreement.

n = 9	Strongly agree	Agree	Somewhat agree	Disagree	Strongly disagree
The algorithm makes me feel more secure in my reporting	2 (22.2%)	5 (55.6%)	1 (11.1%)	0 (0%)	1 (11.1%)
The algorithm is practicable for use with the IMPAX	3 (33.3%)	4 (44.4%)	1 (11.1%)	1 (11.1%)	0 (0%)
The algorithm increasing the speed of my reporting	1 (11.1%)	2 (22.2%)	1 (11.1%)	2 (22.2%)	3 (33.3%)
The algorithm makes me feel pressed for time	0 (0%)	0 (0%)	1 (11.1%)	3 (33.3%)	5 (55.6%)
The algorithm detected ICH that I overlooked	1 (11.1%)	1 (11.1%)	3 (33.3%)	1 (11.1%)	3 (33.3%)
Should the algorithm remain as a tool in the department?	3 (33.3%)	2 (22.2%)	2 (22.2%)	1 (11.1%)	1 (11.1%)
Considering the added value of the algorithm, would an additional resident position be more useful?	5 (55.6%)	1 (11.1%)	1 (11.1%)	0 (0%)	2 (22.2%)

Case study

Small and clearly visible post-traumatic intra-axial haemorrhage in the right frontal lobe on CCT was detected by the radiologist and AIDOC. In the follow-up CCT performed the next day, the AI software AIDOC no longer detected this small haemorrhage, although it had not changed (Figure 2).

Figure 2.

A Small and clearly visible post-traumatic intra-axial haemorrhage in the right frontal lobe on NCCT, which was recognized by the radiologist and AIDOC. In the follow-up NCCT the next day, AIDOC did not recognize or flag the haemorrhage, although it had not changed.

Discussion

How did the algorithm preform?

The diagnostic detection rate of the AI software for intracranial haemorrhages was like that of the interpreting radiologists, but the software occasionally misclassified benign findings as haemorrhages that radiologists could readily attribute to other changes. As mentioned in the Introduction section, previous studies found 84–99% and specificity between 93 and 99% in detecting intracranial haemorrhages,^12–19 which is consistent with our results. The only notable discrepancy in our study was a lower sensitivity for the algorithm in the second analysis (78.4%) compared to previous studies. The algorithm’s sensitivity in the first analysis and the specificity in both analyses were in range (Figure 3).

Figure 3.

A known parenchyma defect labelled as an intracranial haemorrhage (ICH) by the artificial intelligence algorithm (false positive).

It is important to remark that not all the NCCTs were labelled or analyzed by the algorithm and there was no form of feedback as to which scans were left out. In the first analysis out of 372 NCCTs, n = 34 scans were not interpreted by the AI algorithm equalling 9%. Out of those 34 scans, 11 included an ICH equalling 32% of scans not interpreted by the AI algorithm including an ICH.

Our study shows that the integration of a specific automated diagnostic head CT scan into the more complex, general CT scan schemes of a full-service hospital revealed unexpected workflow problems of the local components of the AI system. We found that approximately one-third of intracranial haemorrhages would have gone undetected if automated CT diagnosis alone had been relied upon, due to non-obvious errors in software integration. Therefore, successful implementation of such software within a clinical IT architecture is a critical factor. The smooth clinical performance of the AI software is highly dependent on the full functionality of the underlying technical infrastructure.

Although our study suggests that the AI solution is able to identify small haemorrhages, some of these findings may be inconsequential. For example, a small but clearly visible right frontal haemorrhage in NCCT was initially detected by AIDOC, but not the next day. This suggests an inconsistency in the detection performance of the software, and further research into the reliability of such automated detection systems seems necessary.

The majority of the analyzed NCCTs were in the context of trauma. Not a single case of intraventricular haemorrhage was detected by the algorithm. The algorithm demonstrated strong performance in identifying discrete subarachnoid haemorrhages. The algorithm did not provide any classification of the haemorrhage subtype detected. This is relevant information since different types of ICH have different clinical consequences. The algorithm also did not analyze the size of ICH. An ICH volume below 30cm³ had the lowest mortality rates, compared to a volume of 30–60m3 and a volume of above 60cm³.^7,23,24

Opinion of department

The study shows a trend towards higher acceptance of the algorithm over time, as reflected in the survey responses. This increased acceptance correlates with an improved, observed performance of the algorithm during the second analysis, as several parameters reflecting its diagnostic contribution improved within the six-month interval. Importantly, this refers to enhanced performance in clinical use and not to any technical modification of the algorithm itself. In particular, the two survey items stating that the algorithm makes the radiologist feel more secure in reporting and that the algorithm should remain as a tool in the department increased markedly. The proportion of respondents who agreed or strongly agreed that the algorithm made them feel more secure rose from 33.3% to 77.8%. Whether the algorithm should remain as a tool in the department increased from 44.5% to 55.6%. These findings suggest that acceptance of the algorithm within the department improved significantly over time.

Approximately two-thirds of respondents strongly agreed or agreed in both surveys that an additional resident position would be more useful. Unlike the AI tool, a resident can identify not only ICH but also other urgent findings such as a hydrocephalus or a midline shift. This was elected as a question, because the algorithm costs approximately the same as the yearly salary of a resident. Lastly, in the first survey, 7.41% strongly agreed or agreed that the algorithm detected an ICH that they would have overlooked, while in the second survey 22.2% strongly agreed or agreed.

Possible benefits of an algorithm

We identify several potential clinical applications for the algorithm. The algorithm alerts physicians to possible ICH, which aids in rapidly evaluating NCCTs in the emergency setting. In some centres, suspected ICH cases can be marked as a priority study allowing for interpretation within an hour from scan acquisition. ²⁵ This is particularly important, as acute deterioration due to haemorrhage expansion commonly occurs within initial 3–4.5 hours of symptom onset.^26–28 Algorithms can also support physicians in triaging patients. ²⁹

Artificial intelligence has even been found to be beneficial for reducing the throughput read time and can assist in reprioritizing the reading work list of radiologists. ²² In our radiology department, we implemented a deep learning algorithm, which detects ICH on NCCTs and alerts radiologists via a pop-up notification widget highlighting the suspected finding. This enables radiologists to reprioritize their exam list accordingly and focus their attention on a potentially time-sensitive emergency.

Over the past decades, the amount of ordered exams within the hospital setting has increased considerably. ³⁰ The rising workload may contribute to an increase in the error rate of radiologists. ³¹ AI algorithms have been shown to improve diagnostic accuracy of interpreting physicians, particularly among junior physicians with less experience in the reading of NCCT scans. ³² Even board-certified radiologists, when compared to sub-specialist neuroradiologists, can demonstrate discrepancies of up to 12%.^33,34 Double reading by a sub-specialist, in general, led to high rates of changed reports. ³⁵ Given that a second look by a radiologist improves the rate of correct reporting, the same must be true when algorithms are used for second look.^36–39

Strengths and weaknesses of the study

Several limitations should be addressed when interpreting our results. First, the examinations in this study were conducted in a single hospital. Therefore, while we have demonstrated that our results generalize well to independent datasets obtained at our hospital centre, further work is necessary to evaluate performance on a variety of vendors and scanning protocols at other institutions. Whereas most centres typically use a 4 mm slice thickness, we used 1 mm slices for analysis. It remains unclear whether section thickness contributed to the under detection of some of the urgent findings. And further evaluation of the algorithm with thin-section images would be needed to assess for improved sensitivity. The overall sample size in our study was relatively small and more research is required.

Another limitation of the study is the relatively low number of respondents in the initial survey. The change in the number of respondents in the second survey also may limit generalizability of the findings and increase the risk of bias.

Conclusion

The experience with the AIDOC software at our institute demonstrates that this AI software serve as a useful supporting tool in the narrow diagnostic segment of ICH detection but requires expert supervision. When given the choice, radiologists expressed preference in an additional physician compared to the AI algorithm. Highest level of careful software implementation into the complex IT architecture is vital.

Appendix.

List of abbreviations

ICH

intracranial haemorrhage

NCCT

non-contrast cranial computed tomography

ER

emergency room

IRB

institutional review board

AI

artificial intelligence

IT

information technology

NLP

natural language processing

Ethical statement

Ethical approval

All human studies have been approved by the appropriate ethics committee and have therefore been performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki.

ORCID iDs

Selim Abed https://orcid.org/0009-0003-0989-0249

Johannes Pfaff https://orcid.org/0000-0003-0672-5718