Automated Segmentation of the Median Nerve in the Carpal Tunnel using U-Net

Abstract

Nerve area and motion in carpal tunnel syndrome (CTS) are currently under investigation in terms of prognostic potential. Therefore, there is increasing interest in non-invasive measurement of the nerve using ultrasound. Manual segmentation is time consuming and subject to inter-rater variation, providing an opportunity for automation. Dynamic ultrasound images (n=5560) of the carpal tunnel from 99 clinically diagnosed CTS patients were used to train a U-Net shaped neural network. The best results from the U-Net were achieved with a location primer as initial region of interest for the segmentations during finger flexion (Dice Coefficient of 0.88). This is comparable to the manual Dice measure of 0.92 and higher than the resulting automated Dice measure of the wrist flexion (0.81). Although there is a dependency on image quality, a trained U-Net can reliably be used in the assessment of ultrasound-acquired median nerve size and mobility, considerably decreasing manual effort.

Introduction

Carpal tunnel syndrome (CTS) is characterized by tingling and numbness in the distribution area of the median nerve due to compression at the carpal tunnel. No widely accepted gold standard for CTS diagnosis exists. Commonly, CTS diagnoses are made based on the combination of clinical symptoms, physical examination and neuroconductive studies (NCSs) with or without electromyography (Surgeons. 2016). Based on the severity, local guidelines and patient preference, treatment can include splinting, corticosteroid injections and/or surgical division of the transverse carpal ligament.

Ultrasonography has emerged as a relevant imaging modality for patients with CTS. Currently, it is mostly used as an addendum to the classic NCS during the diagnostic work up, with sensitivity and specificity reaching similar levels (Fowler, et al. 2011) and high (ICC>0.9) intra-rater reliability (El Miedany, et al. 2015, Fowler, et al. 2015, Ghasemi-Esfe, et al. 2011, Gonzalez-Suarez, et al. 2018). Studies show however that inter-rater reliability is more prone to variation with reported ICC values between 0.57-0.94 (Gonzalez-Suarez, et al. 2018, Schrier, et al. 2019, Van Doesburg, et al. 2010). Although NCSs serve as a direct measure for alterations in the function of the nerve, it is a test with limited ability to predict clinical severity or intervention outcome (Finsen and Russwurm 2001, Naranjo, et al. 2009, Surgeons. 2016). Ultrasound allows direct visualization of the nerve, its position in the carpal tunnel, and its relation to the surrounding structures, making it a helpful guidance tool e.g. during injection (Evers, et al. 2017). Research has also suggested that median nerve cross sectional area (CSA) at the level of the carpal tunnel inlet is a reliable diagnostic feature (Buchberger, et al. 1991, Wong, et al. 2004, Yesildag, et al. 2004, Ziswiler, et al. 2005). In addition to size, there is a decrease in nerve mobility in CTS patients (Ellis, et al. 2017, Erel, et al. 2003, Kuo, et al. 2016, Nakamichi and Tachibana 1995, Nanno, et al. 2015, Schrier, et al. 2019, Van Doesburg, et al. 2012, Wang, et al. 2014) which can be quantified using dynamic ultrasound.

Segmentation of the nerve not only helps to acquire nerve size, but also allows for tracking during finger- and wrist motion. Patients with CTS often have edematous, fibrotic connective tissue surrounding the median nerve, causing inhibition of nerve motion. Nanno et al. showed significant changes in median nerve motion after CTR (Nanno, et al. 2017). Nerve dynamics may thus hold useful information pertaining to CTS pathophysiology, but so far there has been limited research associating this with clinical outcome. Segmentation allows for frame to frame tracking of the nerve, visualizing nerve pathways and allowing for quantification of nerve motion.

With the increase in interest in using US in different facets of CTS patient assessment, an automated nerve segmentation method could aid in the reliability of the measure, decrease time and cognitive efforts and decrease inter-rater variability.

Materials and Methods

General

Data used in this study is derived from a registered prospective cohort study (Clinicaltrials.gov: NCT02219555) which was reviewed and approved by the Institutional Review Board (IRB# 14-003444). This study was aimed at assessing the prognostic value of dynamic ultrasound by measuring the median nerve mobility and connective tissue response in CTS patients before treatment. Written informed consent was obtained from all participants in accordance with the Declaration of Helsinki (Association 2013).

Dataset and ultrasound protocol

Images were collected at baseline and one or three months after corticosteroid injection or surgical intervention respectively. Idiopathic CTS patients were evaluated by specialized physicians at a single tertiary medical center with a dedicated hand clinic. Patients were recruited if they met the following inclusion criteria: clinical diagnosis of CTS (based on classic symptoms including numbness or tingling in the distribution area of the median nerve), age between 21-80 years, symptom duration for at least four weeks in at least two digits on one hand, and full understanding of English language. Exclusion criteria were a previous surgical release, tumor, mass or deformity in hand/wrist, pregnancy related CTS, and any of these diagnoses: cervical radiculopathy, peripheral nerve disease, thyroid disease, rheumatoid arthritis or other inflammatory arthritis, osteoarthritis in wrist, diabetes, renal failure, sarcoidosis, amyloidosis or major trauma to ipsilateral hand or wrist. If patients experienced bilateral CTS, only the hand with the most severe (self-reported) symptoms was included.

Images were taken just proximal to the level of the carpal tunnel inlet defined as the proximal margin of the flexor retinaculum (Figure 1A). The nerve was identified by its shape (oval) with a hyperechoic delineation (epineurium) and a “honeycomb”-like appearance due to the perineurial septations between the fascicles. Patients were asked to flex their fingers and their wrists in two separate motions. Generally, the median nerve will move in dorso-ulnar direction (Figure 1B). All images were made using a Philips iE33 (Royal Philips Electronics, Amsterdam, the Netherlands) ultrasound machine with a L15-7io linear transducer. Three sonographers were involved in the acquisition of the images. Two sonographers were residents with 1 year of ultrasound experience. They were trained for a period of multiple weeks by a dedicated MSK sonographer (10+ years of experience), all blinded to any clinical outcome measures at the time of acquisition. Measurements were performed according to a strict protocol designed specifically for this study by the MSK sonographer. Manual analysis of the DICOM images was done using in-house software (Analyze v12.0, Biomedical Imaging Resource, Mayo Clinic, Rochester MN, USA) by two experienced researchers who demarcated the median nerve using a polygon. A gross representation of nerve mobility was needed, so segmentation was done eleven times per cineloop using 33 frame intervals. From 99 patients with an average of 5.1 videos a library was created containing 505 videos with 5560 annotated frames. For the training of U-Net images were sampled from 3336 (60% of the total) frames. 1112 (20%) images were used for validation, and 1112 (20%) for tests.

Figure 1:

(A) The probe was positioned just proximal to the carpal tunnel inlet with the patient’s hand strapped to a board while supported with a gelpad and kept in neutral position using towels. (B) Example of ultrasound showing median nerve before (upper) and after wrist motion (lower). Notice the typical dorso-ulnar translation.

Neural Network design and software

The ultrasound videos were converted from DICOM to PNG format using MatLab 9.5 (MathWorks, Natick, MA, USA). Subsequent parts of the system were programmed in Python 3 (Open source, Python Software Foundation, Beaverton, OR, USA) using a.o. Numpy and the TensorFlow 1.5 (Open source, Google, Mountain View, CA, USA) libraries.

An Artificial Neural Network, a U-Net model, was adapted from Ronneberger, Fisher & Brox (Ronneberger, et al. 2015). Two implementations of the U-Net model were trained: The first model is based on single frame segmentation. In this model every single frame of a video is assessed individually without prospective information or feedback from earlier segmented frames. A second version of the model was made using focus windows. In this model, spatial information from the previous segmented frame is used to redirect the focus of the search area for the next frame, hypothetically increasing the odds of a correct segmentation. The first step in this model was to crop the frame to the relevant section –focus window– of the image.

Tests were performed with the segmentation of the first frame done manually and with the first frame segmented by the single frame network (referred to as the 3^rd model, “fully automated”). Figure 2 displays the overall layout of the U-Net model starting with a single image entering at the top left. A single frame (.png file) was downsized from 1024x768 pixels to 512x384 (in the single frame model) or, like in this diagram, cropped to 320x240 pixels around the nerve when using focus windows. On the downslope - left side - of the U shaped model the information of the image gets transformed with an added dimension (z-axis) containing abstract information and the condensed shape in x and y direction; getting smaller in x and y directions and deeper in the z direction. On the upslope - right side- of the U shape the information is scaled back up to the original size of the image with a single depth where the segmentation emerges with help of the location info from the same sized shape from the left side as depicted by the horizontal arrows across the U. Finally, the image with the segmentation of 320x240 is positioned back in a 1024x768 image where it was cropped from. The arrows depicting convolutional filters and transpose convolutional filters are convolution operations with parameters that are trained using the Neural Network methodology, making U-Net a Convolutional Neural Network.

Figure 2:

Diagram of U-Net model.

The U-net model used for the initial frame of the video uses images and masks of 512x384, the U-net model used for the consecutive frames uses images and masks of 320x240 (no compression) unless the size of a nerve was bigger than the 320x240 * 0.25 mm/pixel= 8mm x 6mm.

The model with focus window uses the mask of frame n-1 to determine the focus area for frame n. For the training set however, the 11 annotated frames from the total of 334 frames in the video were not consecutive frames but 33 frames apart. In order to train the model to be adaptable to a moving nerve, as is seen during finger and wrist flexion, the focus window was subjected to a random geometric translation around the known mask n (Figure 3).

Figure 3:

From left to right: 1) Original frame n; 2) Truth mask n with a focus window to be translated randomly; 3) The translated focus window cut from frame n; 4) The cut from mask n.

Primary Outcome

During the training, as well as the final comparison with the manual segmentation, the mean Dice Coefficient (95% confidence interval) was used as primary outcome. This measure represents the relative overlap of two segmentations with “0” indicating no overlap at all and value “1” representing perfect overlap. This measure therefore represents both morphological as well as spatial accuracy.

Results

The resulting Dice Coefficients for both models are shown in Table 1. The first model was marginally outperformed by the focused frames model, and only if the first frame was defined manually by a user (0.82 versus 0.85). Additionally, better results were acquired using images gained during finger flexion irrespective of U-Net model, but most prominently seen in the focused frames – manual approach (0.88 versus 0.81). During the design it was noted that in particular cases, especially where the nerve showed larger excursions and the quality of the images was diminished, the resulting masks were less accurate, leading to a cascade of failed segmentations. After automated segmentation the system produced an assortment of data for each frame of the video, providing potential research metrics (Figure 4).

Table 1:

Dice Coefficients (95% Confidence interval) for separate U-Net strategies

	1) Single frames	Focus windows
		2) Manual first frame	3) First frame defined by "Single frames" U-Net
Finger flexion dataset (n=594)	0.86 (0.85 – 0.88)	0.88 (0.87 – 0.89)	0.83 (0.81 –0.86)
Wrist flexion dataset (n=518)	0.77 (0.75 – 0.80)	0.81 (0.79 – 0.83)	0.80 (0.78 – 0.82)
Both flexion datasets (n=1112)	0.82 (0.81 – 0.84)	0.85 (0.84 – 0.86)	0.82 (0.81 – 0.83)

Figure 4:

(A) Example of an ultrasound image manually segmented (red dotted outline). A cineclip of 331 frames was recorded and manually segmented every 32nd frame to acquire 11 datapoints to quantify the nerve ‘s motion. (B) Example of an ultrasound frame with an automated overlay in orange. At the top there is a real-time display of several nerve parameters including area, perimeter and location coordinates.

Using the center point of the manual/ automated overlay of the nerve, nerve size, perimeter and absolute coordinates were abstracted. The coordinates were used to track the nerve and calculate the overall motion (see Suppl. A for an example video).

Discussion

This study shows that a U-Net model can be used to accurately assess two-dimensional median nerve characteristics and mobility in ultrasound-based images using a medium-sized dataset. Performance can be enhanced by using a single manual segmentation to add prospective spatial information to the model. In clinical practice the system can aid in getting to a consistent method of diagnosing CTS using ultrasound. Another purpose of the model is to aid in predicting outcomes of CTS remedies. The computer easily outputs various data concerning the nerve: next to the standard measures area and perimeter, the movement of the center of the nerve compared to the previous frame, or the change in shape compared to previous frame. With the model, this can be done for all the images in a videoclip creating a source of information on the dynamics of the nerve that, together with other structured data like patient specifics or possibly existing NCS data, can be used to dial in on a predictive model informing both physician and patient on chances of success facilitating their joint decision-making process.

With decreased nerve mobility as a potential marker for CTS diagnosis (Ellis, et al. 2017) as well as prognosis in those undergoing injection (Schrier, et al. 2019), the results of our study help in the analysis and assessment of the true added value of nerve mobility in clinical evaluation. Studies show that intra-rater reliability generally is high (ICC>0.9) when measuring the nerve cross sectional area, but inter-rater reliability is lower with reported ICC values between 0.57-0.94 (Gonzalez-Suarez, et al. 2018, Schrier, et al. 2019, Van Doesburg, et al. 2010). Since the intra-rater performance of a U-Net model is 1.00, its added value is only limited by its accuracy. Comparing the accuracy to the before mentioned inter-rater accuracies starting at 0.57, the system performed better with scores up to 0.88. Still the main advantages are the increased number of measurements, consistency of measurement as the algorithm will always provide the same segmentation, and speed. With an optimized hardware setup the analysis should be finished within minutes and be well applicable in the context of patient examination.

A challenge in dynamic measurements with US is the change in image features during motion created by the normal anisotropic characteristics of the nerve. Moreover, the shape of the nerve resembles the flexor tendons present in the carpal tunnel. Where a human interpreter would use anatomical knowledge and positive feedback tests (patient moving a finger), our model does not have this information. Consequently, the system can err from the truth. For practical usability it is important to know when this happens so the outlier can be eliminated from further analysis or perhaps a new clip of higher quality can be recorded. A way to cope with this is to add a visual check of the resulting video clip of seven seconds. Improvements to the algorithm can be made for instance by employing Active Contour Model as post processing step. If the visibility of the nerve can be maintained with wrist flexions, the model will perform better for those as well. Preprocessing images for sharpening, denoising, etc. is done by convolutional filters. As U-Net is a convolutional network, ideally its contents already covers those functionalities. However established filters from classic - non AI - computer image analysis as preprocessing step before the U-Net could be tested for improving the results.

Limitations of this study are the single ultrasound system that was used: other probes will yield different images and will need additional training for the network to perform. Chances are there that the model misses the targeted segmentation. A visual check by the physician in clinical context and an outlier detector in research context are recommended.

For future research we can look at the model itself with added preprocessing and postprocessing to increase accuracy. To extend the use of the model as a practical tool for a physician further development can be made to create a 3D image of the median nerve. Finally, to use the information from the model for predicting chances of treatments being successful further research is needed to link the attained data with for instance NCS to historic outcomes of those treatments.

Supplementary Material

1

Suppl. A: Example video showing the nerve including the generated overlay during finger flexion.