Deep Learning Enabled Brain Shunt Valve Identification Using Mobile Phones

Abstract

Background and Objective:

Accurate information concerning implanted medical devices prior to a Magnetic resonance imaging (MRI) examination is crucial to assure safety of the patient and to address MRI induced unintended changes in device settings. The identification of these devices still remains a very challenging task. In this paper, with the aim of providing a faster device detection, we propose the adoption of deep learning for medical device detection from X-rays.

Method:

In particular, we propose a pipeline for the identification of implanted programmable cerebrospinal fluid shunt valves using X-ray images of the radiologist workstation screens captured with mobile phone integrated cameras at different angles and illuminations. We compare the proposed convolutional neural network with published methods.

Results:

Experimental results show that this approach outperforms methods trained on images digitally transferred directly from the scanners and then applied on mobile phones images (mean accuracy 95% vs 77%, Avg. Precision 0.96 vs 0.77, Avg. Recall 0.95 vs 0.77, Avg. F1-score 0.95 vs 0.77) and existing published methods based on transfer learning fine-tuned directly on the mobile phone images (mean accuracy 94% vs 75%, Avg. Precision 0.94 vs 0.75, Avg. Recall 0.94 vs 0.75, Avg. F1-score 0.94 vs 0.75).

Conclusion:

An automated shunt valve identification system is a promising safety tool for radiologists to efficiently coordinate the care of patients with implanted devices. An image-based safety system able to be deployed on a mobile phone would have significant advantages over methods requiring direct input from X-ray scanners or clinical picture archiving and communication system (PACS) in terms of ease of integration in the hospital or clinical ecosystems.

1. Introduction

Magnetic Resonance Imaging is the most commonly used imaging modality for diagnosis and surveillance of various neurosurgical conditions, including hydrocephalus, symptomatic intracranial cysts, pseudotumor cerebri, and hygromas, which often require placement of implantable devices. (Capitanio et al., 2016). MRI utilizes strong magnetic fields that may interact with implanted medical devices with ferromagnetic properties, causing modification of their settings or heating (Lavinio et al., 2008). Patients undergo a rigorous screening process prior to MRI examination to ensure the implants they have are MR safe, and assure patient safety. The screening process includes the completion of a pre-MRI procedure screening form that addresses questions regarding implanted device details such as name of the manufacturer, device model/serial number, and MRI compatibility (Sawyer-Glover and Shellock, 2000). In a survey conducted at Memorial Hermann Hospital – Texas Medical Center with 50 responders involved in the MRI clearance process (providers, schedulers, nurses, technologists), 5-10% of patients referred for MRI do not have information about implanted devices in their body (Giancardo et al., 2018). As a consequence, radiologists need to infer the implanted device by visually inspecting the X-rays, accessing clinical picture archiving and communication system (PACS) or querying image databases.

This process often delays and many times precludes the MRI exam (Kim et al., 2016; Paré and Trudel, 2007; Treichel et al., 2012). Accordingly, there is an urgent need for a streamlined method for automated identification of implanted medical devices.

Medical smartphone-based applications have huge potential to assist clinical decision-making across the globe (Watson et al., 2019). Their use is popular, with 85% of medical professionals using smartphones and 30-50% using medical applications in their clinical practice (Watson et al., 2019). Modern smartphones combine both computing and communication features in a single device and are small enough to hold or store in a pocket, making access effortless at the point of care. New mobile device models have high-quality cameras, powerful processors and operating systems, large memories, and high-resolution screens. Mobile devices have essentially become handheld computers and are becoming essential tools in clinical practice (Lee Ventola, 2014). The main advantage of a mobile phone application for shunt valve identification is the ease of integration as there is no need to install any new software on the workstations or interfere with the PACS.

Advances in deep learning (DL) are enhancing medical imaging. In particular, DL approaches based on deep convolutional neural networks (DCNNs) has shown excellent performance in various applications such as segmentation, classification and registration (de Vos et al., 2019; Gabr et al., 2020; Sujit et al., 2019). The quality of images captured using mobile phones can be easily deteriorated due to real-world settings such as illumination, shading, background, and phone orientation. Over the past few years, deep learning has become the dominant approach in machine learning to solve real-world problems such as object recognition, face recognition, video classification and scene understanding.

Recent research work by Giancardo et al., (Giancardo et al., 2018) demonstrated the feasibility of machine learning and DL models for automated identification of 5 types of cerebrospinal fluid shunt valves (CSF -SVs) that are used for hydrocephalus treatment. However, the authors did not test their approach on images acquired with a mobile phone nor with images not containing valves.

In this work, we aim to develop a custom DCNN model based on VGG16 (Simonyan et al., 2014) for automated identification of brain shunt valve images captured from clinical X-rays using smartphone camera suitable to be interfaced with smartphone – based mobile applications. We focus on 3 types of CSF -SV (Codman-Hakim, Strata II-NSC and Sophysa Polaris). Fig.1 shows the pipeline of the envisioned shunt valve identification framework. If the implanted device is not known for a patient, the operator/radiologist takes a picture of the clinical X-ray image from the PACS using a handheld smartphone device. The DL enabled smartphone identifies the type of shunt valve and a database is queried to show the appropriate safety information. Based on the information retrieved by the smartphone, the radiologist will decide whether to proceed or not with the MRI.

Fig. 1.

Pipeline of the envisioned shunt valve identification framework. Mobile phone application is integrated as part of the PACS.

Our main contributions are:

• We propose a new pipeline to identify implantable device MRI compatibility using mobile phone-quality images.

• We estimate the domain-shift-induced generalization error of two deep learning approaches with medical X-rays captured from PACS and with mobile phone images

• We evaluate the performance of the models with and without images of X-rays containing valves, thereby simulating operator errors.

In Section 2 of this paper, we describe the dataset, network architecture and evaluation metrics in detail. The results are presented in Section 3 and discussed in Section 4. We conclude our work and discuss the scope of future work in Section 5.

2. Material and methods

2.1. Data acquisition and expert evaluation

The dataset was acquired as a part of a quality improvement project approved by our Institution (Quality Improvement Project Registry, no. 2017-017) (Giancardo et al., 2018). The UTHealth IRB Office reviewed and approved the study protocol. A total of 1978 skull X-rays from 659 patients that included the CSF-SV were captured from different angles using a handheld smartphone (iPhone 11 pro) from the clinical X-rays selected from PACS. For regular capture this phone uses the 12MP regular camera ∣ 26mm f/1.8 ∣ Optical image stabilization mode. Camera was focused to keep the shunt valves positioned at the center of the image. Three images for each subject were captured from different perspectives, scales and brightness and along with background objects such as bone structures, craniotomy hardware or catheters as shown in Fig. 2. The images were labeled by an expert neuroradiologist and a trained research associate (EB and AA). Region of Interest (ROI) was selected by cropping images keeping the valve in the center and eliminating the background. This was done to simulate a region of interest selection by the operator. Also, a ‘No Valve’ class type was created by cropping the background region without a valve. The region of interest that did not include a valve, simulates areas erroneously chosen by the operator. The specific 4 class grouping is as follows: No valve (n = 298), Codman – Hakim (n = 705), Medtronic Strata II-NSC (n = 741) and Sophysa Polaris SPV (n = 234). All images were anonymized. Fig.2 shows sample images taken from the database.

Fig. 2.

Examples of CSF-SV X-ray valve images in the dataset. Each row shows 5 random samples of valve types used in the study. From the top, row 1: No-valve (n = 298), row 2: Codman – Hakim (n=705), row 3: Sophysa Polaris (n = 234), row 4: Strata II- NSC (n = 741). ‘n’ represents the number of samples in the database. All the shunt valve brands in the database are currently used in US.

2.2. Preprocessing

All cropped images have different sizes, and the mobile phone used in our experiments output color images, even if X-rays images are evaluated by radiologists in greyscale. To facilitate their use as input to the deep learning model and maintaining a low number of parameters to be learned, all images were preprocessed as follows: 1) converting RGB image to the grayscale intensity image; 2) using a Gaussian filter to remove screen noise; 3) the cropped images with variable matrix size are reshaped to a standard matrix size of 300 x 300 pixels; 4) image intensity was normalized in the range [0,1] to facilitate the training.

2.3. DCNN Architecture

The architecture of the DCNN used for classification of valve types is shown in Fig. 3. The architecture was inspired by the building blocks of the VGG16 model which achieves 92.7% top-5 test accuracy in ImageNet, which is a dataset of over 14 million images belonging to 1000 classes. These building blocks have a relatively low number of parameters (for a CNN) to be learned, which is important for a dataset as small as ours. The network has a total of 11,087 trainable parameters. The DCNN architecture comprises of four consecutive convolutional-pooling layers, a convolutional – global average pooling (GAP) layer, a fully connected layer and an output layer. The four convolutional pooling layers use the same fixed 3 × 3 convolutional kernel and 2 × 2 pooling kernel, but have 8, 16, 16 and 16 neurons, respectively.

Fig. 3.

Architecture of the DCNN model. conv: convolutional layer; pool: max pooling layer; FC: fully connected layer.

The fully connected layer has 15 neurons, which are all connected to the final output SoftMax layer with 4 neurons for no valve/ Codman – Hakim/ Strata II–NSC/ Sophysa Polaris SPV shunt valve type classification. The network used rectified linear unit (ReLu) as the activation function.

2.4. Enhanced - Xception Network

This network has been described in Giancardo et al (2018) and it is currently the only published method tested on a similar dataset as ours. The authors used the Xception network architecture (Chollet, 2017), a DCNN inspired by Inception V3 pre-trained on the ImageNet dataset to classify five types of shunt valves. The last fully-connected layer of the network was removed and a max pooling layer was added to generate the feature vector. The threshold for foreground objects was estimated using the non-parametric Otsu thresholding approach (Otsu, 1979). The network is named as Enhanced-Xception Network. The generated feature vector was classified using a linear logistic regression classifier with L2 regularization

2.5. Design of Experiments

One major goal of our work is to estimate the domain-shift-induced generalization error of two deep learning approaches with medical X-rays captured from PACS and with mobile phone images. For the purpose of comparing different components, we propose five scenarios of training and testing for the two deep learning approaches explained in sections 2.3 and 2.4.

The five scenarios of experiments done using the two deep learning models are described below:

1. Training and testing the model using X-ray images taken from PACS.

2. Training and testing the model using images acquired from mobile phone camera not including ‘no valve’ class.

3. Training and testing the model using images acquired from mobile phone camera including ‘no valve’ class.

4. Training the model using images acquired from mobile phone camera not including ‘no valve’ class and testing the model on X-ray images taken from PACS.

5. Training the model using X-ray images taken from PACS and testing the model on images acquired using mobile phone camera.

2.5.1. Training the DCNN

In the experiments using a single source, i.e., PACS images or mobile phone images, the dataset was split into 60% for training, 20% for validation and 20% for testing. The splits were randomized on subject level, such that images originating from the same subject were not placed into different data splits. Data augmentation was used to artificially create new training data from existing training data. All images were augmented using rotations (range=20°), horizontal and vertical shifting (range = 0.2), and horizontal and vertical flipping.

The network coefficients were trained by optimizing the categorical cross-entropy loss function with a learning rate equal to 0.001 and a batch size of 64 using the Adam optimizer (Kingma and Ba, 2014) with parameters β1 = 0.9 and β2 = 0.999. The learning rate and the batch size were optimized using a grid search. To account for the imbalance in the training dataset, class weights were used. These weights were inversely proportional to the number of images in each category. Early stopping is used to counteract overfitting, whereby training is stopped when overfitting begins to take place.

The model was built in Python using the Keras Library (Chollet, 2015) and TensorFlow (Abadi et al., 2016) was used as the computing backend. Training was performed using NVIDIA® GeForce® RTX 2080 Ti graphics processing units (GPUs).

2.5.2. Transfer -Learning with Xception Networks

In experiments using a single source, i.e., PACS images or mobile phone images, the dataset was split into 10 folds, each fold maintained the same class distribution of the complete dataset and splits were performed at subject level as described in the section above. One-fold was left out as testing set and the classifier was trained on the remaining 9 folds. This operation was iteratively performed for all folds making sure that the classifier is reset at each iteration (Giancardo et al., 2018).

2.6. Model Evaluation

The performance of the deep learning models was evaluated using the receiver operating characteristic (ROC) curve. The area under the ROC curve (AUC) was computed and precision (also known as positive predictive value), recall (also known as sensitivity) and F1-score, i.e., the harmonic mean of precision and recall were calculated. All confidence intervals were computed using the non-parametric bootstrap procedure, using 1000 repetitions and reporting the 5th and 95th percentiles. P-values were computed using the two-tailed Mann–Whitney U test to reject the null hypothesis that each valve is indistinguishable from the others in a 1 vs all strategy.

3. Results

Class-level performance metrics using the deep CNN model trained and tested on 3 shunt valve images and “no valve class” acquired using mobile phone is shown in Table 1. No valve and Sophysa Polaris valve classes were classified with a F1 score of 0.88 and 0.84, whereas F1 score of valve classes were higher which ranged from 0.97 on Codman-Hakim class to 0.96 on Strata II-NSC class. The ROC curve in Fig. 4 shows excellent classification performance from the proposed DCNN model using four class shunt valve datasets acquired by mobile phone. The micro-average and the macro-average ROC curves in addition to the ROC curve for each of the target classes were evaluated. The micro-average and a macro-average AUC were both equal to 0.99. Moreover, the AUC for class 0 (No-valve), Codman – Hakim (class 1), Sophysa Polaris SPV (class 2), and Strata IINSC (class 3) were 0.98, 1.00, 0.99 and 0.97 respectively. The performance metrics of the two deep learning models for each experimental scenario, (i.e.) training-testing within the same datasets and across datasets are compared in Table 2. Overall comparison shows that the proposed deep convolutional networks clearly outperform the published methods tested by achieving an accuracy of 92–96% (confidence intervals (CI) [88–97]/ [94–96]) vs 77–82% (CI [75-79]/ [79-85]). Irrespective of the training dataset, the Xception model showed similar performance when tested on images acquired using mobile phone images (Accuracy = [0.75 – 0.77], Avg. Precision = [0.75 – 0.77], Avg. Recall = [0.75 – 0.77], Avg. F1-score = [0.75 – 0.77]). However, we found that the performance of the proposed DCNN model for classification of shunt valves significantly decreases (Accuracy = 0.60, CI [0.57-.62], Avg. Precision = 0.69, CI [0.67-.71], Avg. Recall = 0.60, CI [0.57-.62] and Avg. F1-score = 0.59, CI [0.57-.61]) when the model trained on X-rays from PACS was tested on images acquired using mobile phone. Whereas, when the same model was tested on images from PACS, it showed significant improvement in performance (Accuracy = 0.92, CI [0.88-.97], Avg. Precision = 0.94, CI [0.91-.98], Avg. Recall = 0.92, CI [0.88-.97], Avg. F1- Score = 0.93, CI [0.88-.97]). Similarly, irrespective of the training dataset, the Xception model showed no significant difference in performance when tested on images acquired from PACS (Accuracy = 0.82, Avg. Precision = [0.82 – 0.83], Avg. Recall = 0.82 and Avg. F1- score = 0.82). While the proposed model performs best when trained on mobile images and tested on PACS images (Accuracy = 0.96, [0.94 – 0.98], Avg, Precision = 0.96, CI [0.94 – 0.98], Avg. Recall = 0.96, CI [0.94 – 0.98], Avg. F1-score = 0,96, CI [0.94 – 0.98]). For real time implementation, the proposed DCNN model was trained and tested on images acquired using mobile phone with or without the ‘no valve’ class showed good performance (Accuracy = [0.94 – 0.95], Avg. Precision = [0.94 – 0.96], Avg. Recall = [0.94 – 0.95], Avg. F1. Score = [0.94 – 0.95]). The performance of the experiment done by training and testing the proposed DCNN model with mobile phone images including ‘no valve’ class is shown in Table 1 and Fig. 4.

Table 1.

Class-level performance metrics using the deep CNN model trained and tested on 3 shunt valve image classes and “no valve class” acquired using mobile phone. N = number of samples per class.

Valve type	N	Precision	Recall	F1 score	p-value
No Valve	78	0.88	0.87	0.88	***p<0.001
Codman-Hakim	117	0.97	0.97	0.97	***p<0.001
Strata II -NSC	171	0.98	0.95	0.96	***p<0.001
Sophysa Polaris SPV	48	0.80	0.90	0.84	***p<0.001

Fig. 4.

ROC curves of the DL model trained with No valve (class 0), Codman – Hakim (class 1), Sophysa Polaris SPV (class 2), and Strata II-NSC (class 3) types of shunt valves

Table 2.

Classification performance of the proposed method with the enhanced- xception network

Machine Learning Methods	Inclusion of no valve class	Training		Testing		Accuracy [Confidence Interval]	Avg. Precision	Avg. Recall	Avg. F1- score
		PACS	Mobile phone images	PACS	Mobile phone images
Transfer learning with		×		×		0.82, [0.79-.85]	0.83, [0.79-.86]	0.82, [0.79-.85]	0.82, [0.79-.85]
Enhanced Xception net			×		×	0.77, [0.75-.79]	0.77, [0.75-.79]	0.77, [0.75-.79]	0.77, [0.75-.79]
(Giancardo et al., 2018)	×		×		×	0.75, [0.73-.77]	0.75, [0.74-.77]	0.75, [0.73-.77]	0.75, [0.73-.77]
			×	×		0.82, [0.79-.85]	0.82, [0.79-.86]	0.82, [0.79-.85]	0.82, [0.79-.85]
		×			×	0.77, [0.75-.79]	0.77, [0.75-.79]	0.77, [0.75-.79]	0.77, [0.75-.79]
Proposed DCNN		×		×		0.92, [0.88-.97]	0.94, [0.91-.98]	0.92, [0.88-.97]	0.93, [0.88-.97]
			×		×	0.95, [0.93–.97]	0.96, [0.94-.97]	0.95, [0.93-.97]	0.95, [0.93-.97]
	×		×		×	0.94, [0.91-.96]	0.94, [0.92-.96]	0.94, [0.91-.96]	0.94, [0.91-.96]
			×	×		0.96, [0.94-.98]	0.96, [0.94-.98]	0.96, [0.94-.98]	0.96, [0.94-.98]
		×			×	0.60, [0.57-.62]	0.69, [0.67-.71]	0.60, [0.57-.62]	0.59, [0.57-.61]

4. Discussion

In this work, we described a DL enabled automatic implanted shunt valve identification system using mobile phone images for MRI safety. Our results indicate that a deep learning-based algorithm can achieve high F1-score by classifying the valves correctly. We visually inspected the valves that were misclassified (see Fig. 10). In all cases, we noticed large foreign objects, low contrast or acquisition angles not well represented in the dataset. These issues are likely to be solved by increasing the dataset size and using better augmentation techniques. In general, the algorithm was able to classify very challenging samples of valves imaged at skewed angles, scales and locations in the skull as shown in Fig. (6-9).

Fig. 10.

Row 1 shows random valve images selected from the database that were not correctly predicted by the DCNN model. Row 2 shows the corresponding saliency map showing the pixels that had the greatest influence on prediction in yellow. No valve (a) was misclassified as Sophysa Polaris (d). Class Strata II-NSC (b) was misclassified as Sophysa Polaris (e). Sophysa Polaris SPV (c) was misclassified as Strata II-NSC (f).

Fig. 6.

Row 1 shows random no-valve images selected from the database that were correctly predicted by the DCNN model. Row 2 shows the corresponding saliency map showing the pixels that had the greatest influence on predicted class (No-valve) in yellow

Fig. 9.

Row 1 shows random Sophysa Polaris SPV images selected from the database that were correctly predicted by the DCNN model. Row 2 shows the corresponding saliency map showing the pixels that had the greatest influence on predicted class (Sophysa Polaris SPV) in yellow

The domain-shift-induced error induced by training with images coming from PACS and testing on mobile phone images is significant for both models, with the proposed DCNN model showing the largest drop in performance. This is likely due to the fact that handheld mobile phone images have varied acquisition angles, brightness and size of valves which are not widely represented in images acquired from PACS. This observation is confirmed by the performance obtained by training the models on mobile phone images and testing them on PACS images. In this case, both models did perform better, but while the Xception-based model had marginally better F1 scores (0.82 vs 0.77), the proposed DCNN had a major boost in performance (0.96 vs. 0.56). When we compare the performance of the proposed DCNN trained on mobile phone images and tested on both domains, there was no measurable domain-shift-induced error. In addition, this model handled the inclusion of the “no valve class” significantly better than the Xception-based model.

As opposed to the methods described in (Giancardo et al., 2018), the proposed DCNN had much fewer trainable parameters, allowing it to be fully trained with valve images without any transfer learning. In addition to the improvement in its generalization ability, it makes the model more suitable to run on mobile phones with limited computational resources.

The pipeline proposed can be easily integrated into the hospital workflow as it does not require any software installation in the hospital workstations or PACS. Additionally, the images captured with the mobile phones do not have any metadata containing protected health information (PHI), thereby drastically reducing security concerns in the translation of the project to clinical practice. Our pipeline assumes that clear criteria of safety have been established with the support of an implantable device registry. Such registry should be constantly updated as new implantable devices enter the market.

This study presents some limitations. Firstly, while a smartphone-like device considerably lowers the integration barrier in a clinical environment, drastically increasing the ease of IT deployment, it also significantly reduces the quality of the images acquired and adds more uncertainty of scale and image contrast.

Secondly, being a retrospective study based on clinical data, we had a small number of valve types. More valve types need to be included in training the DCNN models in future works, so that the models can be clinically useful. Thirdly, the automatic image recognition component, while achieving excellent classification performance, did not provide spatial localization of the valves and require the operator to manually select the region of interest. However, the effort required in manually selecting the valve is minimal. Therefore, perfect localization of shunt valve in the X-ray image would not be strictly required. Additional studies applying our valve and other implantable device identification software in clinical practice are underway. In fact, the algorithm proposed is not specific to a particular type of implanted device and could be readily adapted by retraining the algorithm on other datasets. The possibility of applying this model to other types of implantable devices in multiple body parts has the potential to significantly simplify the workflow of busy radiology departments and facilitate the clearance of patients for MRI.

5. Conclusion

The present study has demonstrated the feasibility of DCNN for automated identification of brain shunt valves using mobile phone images for MRI safety. As the features of mobile phones and their usage in clinical applications expand, they are expected to become even more widely incorporated into every aspect of clinical practice. The developed methodology can be deployed on a mobile phone and easily integrated in the hospital or clinical ecosystems. While initial tests with the proposed DCNN models for shunt valve identification are promising, further evaluation, validation and development of best practice standards are greatly needed before it can be fully integrated into clinical practice. With the implementation of such measures, the automated brain shunt valve identification system may ultimately be used to provide meaningful, accurate, and timely information and guidance to the radiologists for the purpose of improving patient outcomes.

Fig. 5.

Confusion matrices of (a) transfer learning with Enhanced Xception – net (b) proposed DCNN models trained and tested using images acquired from mobile phone camera including ‘no valve’ (nv) class. Codman – Hakim (ch), Strata II-NSC (nsc) and Sophysa Polaris SPV (sp) are the types of shunt valves.

Fig. 7.

Row 1 shows random Codman-Hakim images selected from the database that were correctly predicted by the DCNN model. Row 2 shows the corresponding saliency map showing the pixels that had the greatest influence on predicted class (Codman-Hakim) in yellow

Fig. 8.

Row 1 shows random Strata II-NSC images selected from the database that were correctly predicted by the DCNN model. Row 2 shows the corresponding saliency map showing the pixels that had the greatest influence on predicted class (Strata II-NSC) in yellow

Paper Highlights.

We propose a deep learning-based system for the identification of implanted programmable cerebrospinal fluid shunt valves using X-ray images of the radiologist workstation screens captured with mobile phone integrated cameras. This is significant because an image-based safety system able to be deployed on a mobile phone is a promising safety tool for radiologists and would enhance ease of integration in the hospital or clinical ecosystems.

To our knowledge, using mobile phones images of the radiologist workstations for object detection has received little to no attention from the medical image computing community. However, it could greatly facilitate the deployment of deep learning-based imaging systems in hospitals.