Towards Reproducible Results: Validating CT Hemorrhage-Detection Algorithms on Standard Datasets

We read with great interest the work by Chang et al¹ entitled, “Hybrid 3D/2D Convolutional Neural Network for Hemorrhage Evaluation on Head CT.” Using a custom hybrid 3D/2D variant of the feature pyramid network, they have developed algorithms with excellent accuracy for detecting and classifying bleeds and quantifying bleed volumes. This study has the essential role of showing the importance of using a Convolutional Neural Network (CNN) for automated reporting of hemorrhages in CT of the brain. This would be useful in detecting traumatic brain injury–related bleeds and various spontaneous intracranial hemorrhages. The accuracy in distinguishing different types of bleeds is also impressive. The bleed identification time for the trained algorithm is 0.121 seconds, which is incredibly fast, given that a radiologist would take 3–5 minutes on average.

There is, however, a substantial drop in sensitivity of small intracerebral hemorrhages (ICHs) and extradural hematomas (EDHs) in the test datasets. This drop could be due to overfitting on the training dataset because the number of small (0.01–5.0 mL) ICHs and EDHs is relatively lower on the training set.

As the authors have pointed out, they have trained their algorithm limited to their institution imaging system, and performance of the tool may drop, given the heterogeneity in data acquisition in various machines. This overfitting on the training dataset would delay adoption of the tool in other clinical settings.

Presentation of the results in standard publicly available test datasets (http://headctstudy.qure.ai/dataset) such as CQ500² would make the results more accurate and comparable. It would also be helpful if the authors could host the testing dataset used in the validation of their tool as a comparison for future studies in automated hemorrhage detection and quantification.