Clinical validation of automated hippocampal segmentation in temporal lobe epilepsy

Abstract

Objective

To provide a multi-atlas framework for automated hippocampus segmentation in temporal lobe epilepsy (TLE) and clinically validate the results with respect to surgical lateralization and post-surgical outcome.

Methods

We retrospectively identified 47 TLE patients who underwent surgical resection and 12 healthy controls. T1-weighted 3 T MRI scans were acquired for all subjects, and patients were identified by a neuroradiologist with regards to lateralization and degree of hippocampal sclerosis (HS). Automated segmentation was implemented through the Joint Label Fusion/Corrective Learning (JLF/CL) method. Gold standard lateralization was determined from the surgically resected side in Engel I (seizure-free) patients at the two-year timepoint. ROC curves were used to identify appropriate thresholds for hippocampal asymmetry ratios, which were then used to analyze JLF/CL lateralization.

Results

The optimal template atlas based on subject images with varying appearances, from normal-appearing to severe HS, was demonstrated to be composed entirely of normal-appearing subjects, with good agreement between automated and manual segmentations. In applying this atlas to 26 surgically resected seizure-free patients at a two-year timepoint, JLF/CL lateralized seizure onset 92% of the time. In comparison, neuroradiology reads lateralized 65% of patients, but correctly lateralized seizure onset in these patients 100% of the time. When compared to lateralized neuroradiology reads, JLF/CL was in agreement and correctly lateralized all 17 patients. When compared to nonlateralized radiology reads, JLF/CL correctly lateralized 78% of the nine patients.

Significance

While a neuroradiologist's interpretation of MR imaging is a key, albeit imperfect, diagnostic tool for seizure localization in medically-refractory TLE patients, automated hippocampal segmentation may provide more efficient and accurate epileptic foci localization. These promising findings demonstrate the clinical utility of automated segmentation in the TLE MR imaging pipeline prior to surgical resection, and suggest that further investigation into JLF/CL-assisted MRI reading could improve clinical outcomes. Our JLF/CL software is publicly available at https://www.nitrc.org/projects/ashs/.

Highlights

• •20-subject normal atlas is sufficient for automated segmentation of any subject.
• •Automated segmentation correctly lateralizes the epileptogenic focus in 92%.
• •Automated segmentation is predictive of seizure freedom at a two-year timepoint.
• •Automated segmentation matches and assists neuroradiology lateralization.
• •Automated segmentation correctly lateralizes 78% of nonlesional patients.

1. Introduction

Temporal lobe epilepsy (TLE) is the most common medically-refractory form of epilepsy, and approximately 20% of TLE patients are nonlesional (MRI-negative), with no apparent hippocampal sclerosis (HS) on imaging (Carne et al., 2004; Cascino et al., 1991). Due to the severe nature of the disease, surgical intervention is often required to achieve seizure freedom, and MRI is featured prominently in pre-surgical evaluation. Since 87% of all surgically resected patients have some degree of histopathological change, accurate identification of the epileptogenic focus, aided by radiology reads, is essential and has been associated with improved surgical outcomes, such that up to 83% of patients with a well-identified seizure focus can achieve good surgical outcome (Cascino et al., 1996; Siegel et al., 2001; Cohen-Gadol et al., 2006). Additionally, further progress in identification of MRI lesions could improve outcomes for lesional (MRI-positive) patients, due to the extent of lesion resection being associated with surgical outcome (Awad et al., 1991). Furthermore, it is thought that the inability to adequately visualize the epileptogenic focus prior to potential surgical resection contributes to the worse outcomes nonlesional patients experience following surgery, with lesional TLE patients being 2.7 times more likely to achieve seizure freedom following surgery (Téllez-Zenteno et al., 2010).

Currently, MRI scans of TLE patients are evaluated qualitatively by a neuroradiologist to identify areas of HS and atrophy for presurgical evaluation, findings which are typically associated with better outcomes following successful surgical resection (Paglioli et al., 2004). Additionally, ipsilateral volume loss in HS is correlated with seizure frequency, with medically refractory TLE being associated with progressive hippocampal atrophy (Fuerst et al., 2003). To quantitatively evaluate abnormalities in hippocampal volume requires segmentation, which without automated tools is time-consuming and prone to human error. Despite the availability of automated hippocampal volumetry tools, they are rarely used in the clinical pre-surgical evaluation pipeline, in part due to limited validation of their efficacy vis-à-vis the clinical standard of qualitative evaluation (Pardoe et al., 2009). However, studies have shown that volumetric changes alone in the hippocampus are associated with outcome, particularly in patients with hippocampal sclerosis (Jack et al., 1994).

Recent advances in multi-atlas, patch-based label fusion techniques have led to robust automatic hippocampus segmentation methods that are competitive with expert human segmenters in terms of reliability (Coupé et al., 2011; Wu et al., 2014; Zhang et al., n.d.; Iglesias and Sabuncu, 2015; Wang et al., n.d.). Such methods have been extensively evaluated in the context of Alzheimer's disease and could allow for earlier detection and treatment of the disease (Leung et al., 2010). To automatically segment a novel “target” image, these methods utilize a set of similar expert-labeled example images, known as atlases. Each atlas image is registered nonlinearly onto the target image, and the warped atlas segmentations are fused to form a consensus segmentation of the hippocampus in the target image. A particular implementation of this strategy, joint label fusion with corrective learning (JLF/CL), has achieved leading performance in a range of applications, including in international competitions (Wang and Yushkevich, 2013). This JLF/CL algorithm is implemented through the repurposed open-source software tool “Automated Segmentation of Hippocampal Subfields” (ASHS) for T1-weighted images, available at https://www.nitrc.org/projects/ashs/ under the Penn Temporal Lobe Epilepsy T1-MRI Whole Hippocampus ASHS Atlas: ASHS 1.0 Compatible release entry with the filename ashs_atlas_penntle_hippo_20170915.tar. Despite its name, ASHS can be easily trained to segment structures other than hippocampal subfields, including the whole hippocampus—of note, we are using JLF/CL through ASHS with T1-weighted images passed in for both inputs (no T2-weighted imaging used) to segment out the whole hippocampi, and not the subfields.

Our paper seeks to evaluate the performance of JLF/CL in the context of whole hippocampus segmentation in clinical MRI scans obtained during presurgical evaluation of TLE patients. From the methodological perspective, we seek to determine whether the composition of the atlas set in JLF/CL (in terms of the proportion of patients with hippocampal abnormality) significantly affects segmentation accuracy. From the clinical perspective, we seek to evaluate the efficacy of using quantitative measures of hippocampal volume derived by JLF/CL as an alternative or complement to qualitative neuroradiological assessment for lateralization of the TLE seizure focus. Lateralization accuracy by JLF/CL and neuroradiologists is evaluated against the gold standard of seizure-free outcome after surgical resection.

2. Materials and methods

2.1. Study population

The 59 subjects (47 TLE and 12 controls) all underwent 3 T T1-weighted brain MRI scans performed for clinical purposes (Supplementary Table 1). TLE patients consisted of consecutive TLE surgical patients from the Penn Epilepsy Center who were scanned between April 2005 and September 2015, and for whom surgical outcome information was available. This cohort of 47 TLE patients were all evaluated prior to epilepsy surgery in the Penn Epilepsy Center multidisciplinary epilepsy surgery case conference comprised of board-certified neuroradiologists, neurologists, and neurosurgeons. Clinical determination of seizure lateralization leading to resection was determined for patients with a combination of video-EEG, MRI, and PET, as well as confirmed with pathology (Supplementary Table 1). The average age of all patients was 36 (range 18–66), composed of 33 females and 14 males. In the patient group, 23 underwent left temporal resection and 24 right temporal resection. Control subjects had routine clinical brain MRI acquired prior to diagnosis of psychogenic non-epileptic events on video EEG monitoring, as opposed to seizures (Supplementary Table 1). The average age of all controls was 37 (range 24–49), composed of eight females and four males.

We retrospectively retrieved the subjects' MRI scans from the Hospital of the University of Pennsylvania PACS, under an approved Institutional Review Board protocol of the University of Pennsylvania.

2.2. Image acquisition

A 3 Tesla Siemens Trio Scanner at the Hospital of the University of Pennsylvania was used to acquire T1-weighted (MPRAGE) images of all patients following a clinical epilepsy scanning protocol. All subjects had sufficient imaging without significant noise distortion, enabling both manual and automated segmentations. All 47 surgically-resected TLE patients were characterized based on radiology reports into left-sided, right-sided, and nonlateralized (bilateral or none) hippocampal volume loss; this enabled the comparison of the clinical radiologic information used during clinical case conferences for epilepsy surgery to JLF/CL lateralizations, mimicking the potential future utility of implementing our automated segmentation method for clinical decision-making.

2.3. MRI phenotype classification

All 59 subjects were grouped by a neuroradiologist (J.S.) into three “MRI phenotypes” based on the degree of hippocampal sclerosis observed on the MRI by the neuroradiologist: “normal” (N = 23, 12 controls and 11 patients), “mild” (N = 17), and “severe” (N = 19). This noninvasive, clinically-focused approach (through a neuroradiologist read) to categorizing atrophy was used solely to ensure that the images inputted to the JLF/CL algorithm could be grouped by severity, a metric which was not consistently included in radiologic reports. Due to concerns about the degree of variation in hippocampal sclerosis for clinical patient scans, it was important to be able to create atlases that could work not only in theory, but also in practice. It was thought that, by creating atlases that more closely mimicked the patient population (an atlas with a larger proportion of severe hippocampal sclerosis subjects to segment a target image with severe hippocampal sclerosis), a better segmentation would be achieved. There is also existing research into the various types of hippocampal sclerosis, resulting in the designation of HS ILAE Types 1–3 based on subfield-predominant loss on pathology and are associated with different outcomes; this provides a further impetus towards tailoring the clinical application of our segmentation techniques to the patient's degree of disease (Blümcke et al., 2013).

2.4. Segmentation

All subjects were manually segmented by trained researchers (C.C. and L.C.) using ITK-SNAP (Yushkevich et al., 2006). Both researchers were trained and certified on the online EADC-ADNI Harmonized Protocol (HarP) for Hippocampal Volumetry, with one researcher conducting the manual segmentations (C.C.) for all 59 subjects and the other confirming the segmentations (L.C.) (Frisoni et al., 2015). All of these segmentations were then confirmed, resolving any discrepancies, by a neurologist. (S.C.). All raters achieved the prescribed threshold for reliability from HarP, indicating their ability to manually segment the hippocampus on MRI according to a standard protocol.

Automatic segmentation of the whole hippocampi included CA1–4, DG, and part of the subiculum (per the designations of the HARP protocol), but the subfields were not individually segmented. The automated segmentation was carried out using JLF/CL, which implemented ASHS for T1-weighted images only (Yushkevich et al., 2015). A set of 50 cross-validation experiments was performed in which the 59-subject dataset was split into testing sets of 15 subjects (five normal, five mild, five severe) and 50 atlas sets of 20 subjects. The proportion of normal subjects in the atlas sets was modulated across the cross-validation experiments as follows, with 10 experiments conducted for each distribution: 0 normal, 10 mild, 10 severe; four normal, eight mild, eight severe; eight normal, six mild, six severe; 12 normal, four mild, four severe; and 16 normal, two mild, two severe. Due to the limited clinical dataset, there were not enough normal subjects to create a 20-subject atlas set of all normal subjects. Additionally, since each subject was tested, but not an equal number of times, weighted averaging was used to compute Dice coefficients per subject to limit subject selection bias.

For each cross-validation experiment, JLF/CL was trained using the atlas set and used to segment the images in the test set. The automated segmentation of the test set was compared to the manual segmentation of the test set in terms of Dice similarity coefficient, a measure of relative overlap (Dice, 1945). In the cross-validation experiments, the MRI scans of TLE patients with right-side TLE were flipped across the midsagittal plane, and segmentation was only performed in the left hemisphere. This flipping is consistent with the goal of the cross-validation experiments, which is to determine the effects of atlas composition on JLF/CL segmentation accuracy (refer to Yushkevich et al., 2015 for more information) (Yushkevich et al., 2015).

As detailed in “Results”, the greatest overall segmentation accuracy in the cross-validation experiments was reached when the atlas set consisted entirely of normal-appearing subjects (epileptic patients with normal hippocampi and the non-epileptic controls). For the subsequent clinical aims of the paper (to evaluate the efficacy of automatic segmentation for lateralizing seizure onset), the JLF/CL atlas set was constructed from 24 control scans (12 original control scans and 12 flipped control scans), which is a number consistent with previous uses of JLF/CL, and trained on this control atlas set. JLF/CL was used to segment the original, un-flipped scans of the 47 TLE patients, with hippocampus labeled in both hemispheres.

The binary voxel-wise segmentation of the left and right hippocampus produced by JLF/CL was used to compute an asymmetry ratio, in which a negative score indicates greater left-sided atrophy and a positive value indicates greater right-sided atrophy.

$$\mathit{Asymmetry\; Ratio}=\frac{\mathit{Left\; Hippocampal\; Volume}-\mathit{Right\; Hippocampal\; Volume}}{\mathit{Left\; Hippocampal\; Volume}+\mathit{Right\; Hippocampal\; Volume}}$$

Our JLF/CL software is available open-source for public use at https://www.nitrc.org/projects/ashs/.

2.5. Surgical outcome

The success of surgical resection was evaluated on the Engel scale, a four-point scale measuring seizure freedom ranging from I (free from disabling seizures) to IV (no worthwhile improvement) (Engel Jr, 1993). Since 90% of relapses occur within two years, and seizure freedom at this timepoint is predictive of long-term seizure freedom, further analysis at the two-year mark in patients with Engel I outcome was conducted (Lindsten et al., 2002). Engel scores were used to define the “gold standard” for clinical lateralization of seizure focus for the 42 subjects who had clinical follow-up at the two-year timepoint: Subjects who had left-side surgical resection and were seizure-free (Engel I) after surgery were designated as left-lateralized, and similarly for the right-side resection. The summary of the demographic and clinical characteristics of patients 2 years post-resection is seen in Table 1.

Table 1.

Summary of demographic and clinical information for temporal lobe epilepsy patients at 2 years post-resection.

	N	Mean Age	Age Range	Female	Nonlesional/Nonlateralized	Left	Right
Patients	42	35.14286	18–66	29 (69%)
MRI Lateralization	42				15 (36%)	13 (31%)	14 (33%)
PET Lateralization	35				7 (20%)	15 (43%)	13 (37%)
EEG Lateralization	42				3 (7%)	19 (45%)	20 (48%)
Resection Lateralization	42					21 (50%)	21 (50%)
Pathology: MTS	31					14 (45%)	17 (55%)
Pathology: Gliosis	11				1 (9%)	2 (18%)	8 (72%)

Of note, all 47 patients in this study had mesial temporal lobe epilepsy, but not all patients had mesial temporal sclerosis on imaging or pathology. When determining surgical outcome, we used seizure-freedom after temporal lobectomy as the primary outcome measure to validate the JFL/CL automated segmentation method given that pathology consistent with mesial temporal sclerosis in a resected hippocampus does not equate to seizure freedom after surgery in all patients. Pathologic correspondence, although helpful, is not necessary for clinical validation and often only partially sampled the hippocampus (because these were clinical, and not research, cases), making it not ideal for our evaluation.

2.6. Statistics

Analysis was conducted in MATLAB and R with the ggplot2, pROC, and BlandAltmanLeh packages. Statistical significance was set at a p-value of 0.05, unless otherwise noted.

2.6.1. Statistics: atlas composition experiments

To evaluate the effects of atlas composition (i.e., proportion of normal subjects) on segmentation accuracy, we performed the following two-tier statistical analysis. Each time a subject i entered a segmentation experiment as part of the cross-validation testing subset, we recorded the proportion of normal subjects in the corresponding cross-validation atlas subset and the Dice coefficient between the JLF/CL segmentation of the subject and the manual segmentation. We then computed a within-subject coefficient of regression ρ_i between the proportion of normal subjects in the atlas and Dice coefficient. For each MRI phenotype (normal, mild, severe as well as a hybrid “diseased” class that combined mild and severe phenotypes), we used a two-sided Student t-test to determine whether the within-subject coefficients ρ_i were statistically different from zero, which would indicate that atlas composition for that phenotype was associated with segmentation accuracy. The two-tiered analysis accounts for the different number of times that different subjects entered into the cross-validation experiments, effectively treating subject as a random effect.

For the experiments where the atlas was composed entirely of subjects with the normal phenotype, we tested whether JLF/CL segmentation accuracy was different between the image severity classes using a Kruskal-Wallis test.

2.6.2. Statistics: lateralization experiments

The subset of epilepsy patients who received surgery and attained Engel I outcome at two years post-surgery (N = 26) was used to define the “gold standard” for seizure site lateralization. In other words, the subject whose left medial temporal lobe was resected and who was seizure-free two years after surgery, was considered to truly have left-lateralized TLE. The 2-year time point was used because seizure freedom at two years after surgery is consistently reported to be predictive of long-term seizure freedom (Lindsten et al., 2002). Among these subjects, we distinguish between a subset for whom the neuroradiologists indicated the apparent site of seizure in the radiological read (“MRI-lateralized” class, N = 17) and a subset where the neuroradiologists were unable to determine the site of seizure (“MRI-nonlateralized”, N = 9).

Receiver Operating Characteristic (ROC) curves were created to test the ability of asymmetry index derived from JLF/CL segmentation to correctly predict the site of seizure in subjects with known clinical outcome-confirmed lateralization. Three ROC curves were constructed for the MRI-lateralized class, MRI-nonlateralized class, and combining all subjects from both classes. Each ROC curve was constructed by recording sensitivity and specificity of lateralization for varying values of asymmetry index threshold. Area under the curve (AUC) was computed for each ROC curve, and 95% confidence intervals on the AUC were computed using the bootstrap method (Robin et al., 2011).

The optimal asymmetry index threshold range (−0.148716, −0.062482) was determined from the ROC containing all Engel I patients, and was then applied to all 2-year timepoint patients (all Engel outcomes) to create left-sided and right-sided JLF/CL lateralization predictions; to determine the lateralization of the non-Engel I patients, the threshold determined from the 26 Engel I patients was used. Subjects that have asymmetry indices more negative than −0.148716 are considered to have larger left hippocampi, subjects that have asymmetry indices more positive than −0.062482 are considered to have larger right hippocampi, and those that fall within these boundaries are considered to be indeterminate in lateralization. Of note, our subject population was clearly delineated between left and right outside of this threshold range. Although the negative threshold range demonstrates bias towards a larger right hippocampus, studies investigating structural asymmetries of the hippocampus have borne this out, indicating that right hippocampi are slightly larger than left in healthy adults (Hou et al., 2013; Pedraza et al., 2004; Woolard and Heckers, 2012).

3. Results

3.1. Effects of atlas composition on segmentation accuracy

The distribution of within-subject Dice coefficient when using a 20-subject control atlas, composed entirely of normal-appearing subjects, is plotted for the normal, mild and severe MRI phenotypes in Fig. 1A. The average Dice coefficient across all groups and all cross-validation experiments was 0.85(±0.05), with the average of 0.87 (±0.03) for the normal group, 0.85 (±0.04) for mild group and 0.84 (±0.08) for the severe group. All three distributions were statistically different (p < .001 on Kruskal-Wallis test).

Fig. 1.

JLF/CL Atlas Optimization. A) Segmentation Accuracy differs by MRI Phenotype. Normal atlases were tested against all subjects, and Dice scores were grouped by MRI phenotype. The average Dice coefficients were 0.87 (±0.03) for Normals, 0.85 (±0.04) for Mild, and 0.84 (±0.08) for Severe. A Kruskal-Wallis test by ranks was statistically significant and indicates that segmentation accuracy differs by MRI phenotype, with an increase in accuracy for normal subjects. B) Correlation between Dice Coefficient and Percentage Normal Atlas Composition. Atlases of varying MRI phenotype compositions were tested against all subjects, and within-subject correlation coefficients between the Normal percentage composition of the atlases and Dice coefficient were determined. Two-tailed unpaired t-tests were conducted for the correlation coefficients of the groups by subject MRI phenotype and the results were plotted. The Normal correlations were non-zero with statistical significance, but the Mild, Severe, and Diseased (Mild+Severe) correlations did not achieve statistical significance. This indicates that increasing the proportion of normal subjects in an atlas set improves segmentation accuracy for normal subject target images, but does not change the segmentation accuracy for Mild, Severe, and Diseased phenotype target images.

Fig. 1B plots the distributions of the within-subject coefficient of regression ρ_i between the proportion of normal subjects in the atlas set and hippocampal segmentation accuracy for the normal, mild, severe, and diseased (mild+severe) phenotypes. Interestingly, the regression coefficient does not significantly differ from zero for mild, severe and diseased phenotypes (p = .74, p = .91, and p = .77 respectively), indicating that changing the proportion of normal subjects in the atlas set does not significantly impact segmentation accuracy for these subjects, on average. However, it does differ from zero significantly in the normal phenotype (p = .023), indicating that decreasing the proportion of normal subjects in the atlas set reduces segmentation accuracy for normal subjects, on average. From this, we conclude that the optimal atlas set composition is achieved when only normal subjects are included in the atlas set. Further depiction of this trend can be seen in Supplementary Fig. 1.

3.2. Segmentation with optimal atlas set

A comparison of the asymmetry indices derived from the manual and automated segmentations of the 47 patients indicates overall good agreement, demonstrating that JLF/CL-derived asymmetry measures are consistent with human-derived measures (Fig. 2). Only two of the 47 patients (4.3%) fell slightly outside of the 95% confidence interval, in which the difference between the asymmetry indices, derived from manual and automated segmentations, was at most approximately 0.1, and the mean of the of the differences in measurements was near zero. Sample segmentations are also provided for a control and patient (Fig. 3)

Fig. 2.

Agreement between Manual and Automated Segmentations for TLE patients. For automated segmentation, the asymmetry indices were calculated with 24-subject atlases composed entirely of Normal-appearing subjects. A) Bland-Altman plot comparing the asymmetry indices determined from manual and automated segmentations. B) Bland-Altman Plot comparing the left-sided volumes determined from manual and automated segmentations. C) Bland-Altman plot comparing the right-sided volumes determined from manual and automated segmentations.

Fig. 3.

Segmentations of Controls and Patients based on Median Dice. Axial, Sagittal, and Coronal Images from the control subject with the median Dice score among controls (0.877843) and the patient with the median Dice score among patients (0.860361) are displayed, organized based on side (Left v. Right) and segmentation (Non-segmented v. Segmented).

3.3. Seizure lateralization

Post-resection Engel outcomes at the two-year timepoint were available in 42 of the 47 patients (the remaining five patients were lost to follow-up). Of these 42 patients, 26 (62%) had Engel I score at two years. Subsequent analysis focused on these 26 gold-standard,“clinically-based lateralization” patients.

The ROC curves plotting the sensitivity and specificity of seizure side lateralization based on the JLF/CL-derived asymmetry index vis-à-vis clinically-based lateralization are plotted in Fig. 4A. The ROC is plotted for all 26 subjects, as well as separately for the MRI-lateralized and MRI-nonlateralized subsets. The AUC is 0.93 for all 26 subjects, 1.0 for the MRI-lateralized subjects (N = 17) and 0.63 for MRI non-lateralized subjects (N = 9). The optimal asymmetry ratio threshold range separating left and right-sided patients was determined through the “All Patients” ROC curve at (−0.148716, −0.062482). This takes into account non-pathological asymmetry; indeed, several studies have indicated that right hippocampi are slightly larger than left in healthy adults, and such asymmetry is manifested in the dentate gyrus (Hou et al., 2013; Pedraza et al., 2004; Woolard and Heckers, 2012; Shah et al., 2018). Only two patients were incorrectly lateralized based on the JLF/CL asymmetry ratio threshold (Fig. 4B). Of note, the 95% confidence interval is wide (79%–100% for all patients, 11%–100% for MRI nonlateralized), but this is likely due to the small sample of 26 Engel I patients.

Fig. 4.

2-Year Engel I Lateralization distribution based ROC-determined threshold. A) In Engel I (seizure-free) TLE patients, ROC curves were created for seizure site lateralization using JLF/CL-derived asymmetry ratios against gold standard lateralization based on surgical outcome for all patients, those who were lateralized by a neuroradiologist, and those who were not lateralized by a neuroradiologist. B) In Engel I TLE patients at the 2 year time-point, epilepsy site lateralization was compared between JLF/CL-derived asymmetry ratios, lateralization in the radiologists' MRI reports, and lateralization based on surgically resected side. The region between the dashed lines indicates the asymmetry index threshold range of (−0.148716, −0.062482).

The radiological reports lateralized 17 of 26 patients (65%), correctly lateralizing seizure onset in all 17 of those cases, leading to a positive predictive value of 100% (Table 2). Of the 11 left-sided patients, the neuroradiologist lateralized, and lateralized correctly, seven of them (64%); of the 15 right-sided patients, the neuroradiologist lateralized, and lateralized correctly, 10 of them (67%). For the 9 patients that the neuroradiologist designated as nonlateralized, four (44%) were ultimately designated as left-sided following resection and five (56%) were designated as right-sided.

Table 2.

Lateralization at 2 years based on surgical resection, neuroradiology reports (MRI), and JLF/CL asymmetry indices.

Surgery	MRI	JLF/CL	Engel I	Engel II	Engel III	Engel IV
Left	Left	Left	7	4
		Right		1	1
	Right	Left
		Right
	Nonlateralized	Left	2
		Right	2	2	1	1
Right	Left	Left
		Right
	Right	Left
		Right	10	2	1	1
	Nonlateralized	Left
		Right	5	2
Total			26	11	3	2

In comparison, JLF/CL was successfully implemented in all 26 patients, correctly lateralizing seizure onset in 24 of the 26 patients (92%), and correctly lateralizing all 17 of the neuroradiologist-lateralized patients (100%). Of the 11 left-sided patients, JLF/CL correctly lateralized nine (82%); and of the 15 right-sided patients, JLF/CL correctly lateralized all 15 of them (100%). For the nine patients that the neuroradiologist designated as nonlateralized, JLF/CL correctly lateralized seizure onset in seven (78%), only incorrectly designating two out of the four left-sided lateralizations (50%) as right-sided, while correctly designating all five right-sided lateralizations as right-sided (100%).

For the poor outcome patients (Engel II, III, and IV), correct lateralization cannot be determined, since the surgery did not achieve seizure freedom. When looking at the neuroradiology reads of all patients in Table 2, including both good and poor outcomes, 10 of the 27 lateralized patients (37%) had poor outcomes, as opposed to six of the 15 nonlateralized patients (45%) who had poor outcomes.

4. Discussion

4.1. Automated segmentation

We demonstrated through the initial atlas composition experiments that the atlas performed best across all subjects when trained using entirely normal-appearing hippocampi on MRI scans, maximizing the Dice coefficient for normal-appearing patients and having no effect on atlas performance in diseased patients. This is a new and somewhat surprising finding, as intuition and some of the current rationale behind multi-atlas segmentation suggest that subjects with hippocampal atrophy would be best segmented using atlases that also have hippocampal atrophy, by matching the atlas to the population (Wolz et al., 2010). This result might indicate that there is substantial heterogeneity in the appearance of hippocampal sclerosis, such that patients with sclerosis included in the atlas do not necessarily have appearance similar enough to the target subject to aid segmentation. The observation that normal-appearing patients could produce a better segmentation overall, and when focused on segmenting normal-appearing patients, is likely due to the relative homogeneity among the normal-appearing hippocampi patient population; this would create more similarities between the atlas and the subject image, and lead to better segmentations. However, since patients with diseased hippocampi can have more variability in their imaging relative to that of patients with normal-appearing hippocampi, an atlas of diseased patients will not necessarily be representative for a given diseased patient in this heterogenous population, creating a potential mismatch between the atlas and target image and a poorer segmentation. In a similar vein, applying an atlas composed of normal-appearing subjects would likely create a poor segmentation when looking at the subject image of a diseased patient. It is possible that with a larger atlas size, we would have found a relationship between the proportion of subjects with abnormal phenotype and segmentation accuracy for subjects with abnormal phenotype, as such larger atlases would be more likely to include images similar in appearance to the target image. More data with manual segmentations would be needed to test this.

Overall, however, the Dice coefficients for normal and abnormal phenotypes (0.84–0.87) are within the same range of 0.8 to 0.9 seen in the multi-atlas and patch-based literature, attesting to the robustness of the JLF/CL method with use of T1-weighted imaging (Winston et al., 2013; Despotovic et al., 2011; Dill et al., 2015; Hogan RE et al., 2015; Caldairou et al., 2016; Hammers et al., 2007). In a recent study involving only healthy subjects, the volBrain method demonstrated better segmentation of the hippocampus using 22 subjects with MP2RAGE (2 T1-weighted images with different inversion times) imaging compared to FSL, FreeSurfer, and SPM with a Dice coefficient of 0.892 (Næss-Schmidt et al., 2016). In other studies investigating automated segmentation in epilepsy without distinction of hippocampal sclerosis severity, the Dice coefficient has ranged from 0.847, with 3 T scans using the STEPS method, to 0.78 with ABSS, 0.74 with LocalInfo, 0.67 with FreeSurfer, and 0.65 with Hammer (Winston et al., 2013; Hosseini et al., 2016).

Our JLF/CL method, although demonstrating a Dice coefficient of 0.87 for the 12 controls and 11 normal-appearing patients, was able to produce a Dice coefficient of 0.85 for all subjects, with 0.85 for mild hippocampal sclerosis and 0.84 for severe hippocampal sclerosis. This indicates that the JLF/CL method performs with similar accuracy to other methods with normal subjects, but also has the ability to better segment hippocampi in patients with temporal lobe epilepsy, including differing hippocampal sclerosis severity. Even with a smaller atlas set (only 20 subjects required) that has been seen in the literature, our method performed similarly to other methods that required 400-subject atlas sets (Winston et al., 2013; Næss-Schmidt et al., 2016). The comparison between manual and automated segmentations indicated overall good agreement for the asymmetry measurements, allowing us to proceed with clinical validation analysis based on the JLF/CL-derived asymmetry indices.

4.2. Clinical significance

It is important to emphasize the translational importance of these findings. We have demonstrated that a 20-subject normal-appearing atlas, which can be composed entirely of controls, is sufficient to automatically segment patients of all HS severities. This novel finding opens the door to easily created template libraries for automated segmentation, potentially enabling a more straightforward and widespread implementation of automated segmentation techniques for clinical applications. Furthermore, the high AUC (0.92) and ability of JLF/CL to correctly lateralize the seizure focus indicates the ability of JLF/CL to assist neuroradiologists in clinical diagnosis and offer epileptologists a valuable biomarker for clinical decision-making. Not only was JLF/CL able to replicate the correct lateralizations that the neuroradiologists had read, but it also lateralized 78% of the scans that the neuroradiologists could not lateralize. This indicates the potential for incorporation of the JLF/CL method into the current multi-factorial clinical decision-making for epilepsy surgery. Clinicians often consider various results in identifying seizure onset, including MRI, PET, and EEG, and the results of the JLF/CL segmentation could eventually add clinical value as well; in uncertain cases, the JLF/CL method could be a useful biomarker to help identify the laterality of seizure onset. While other methods have compared various automated segmentation techniques to some clinical variables (mainly pathology), this study uses a publicly available automated segmentation algorithm (https://www.nitrc.org/projects/ashs/) for both lesional and nonlesional patients on MRI, demonstrating the clinical utility of JLF/CL by matching lateralization for the lesional patients and improving lateralization for nonlesional patients (Caldairou et al., 2016; Sone et al., 2016).

Since it is unclear how to correctly designate correct lateralization in poor outcome patients at the two-year timepoint, in-depth outcome analysis could not be conducted. Therefore, this study focused primarily on Engel I patients, since good surgical outcomes could be better explained by accurate lateralization, while poor surgical outcomes could be due to a larger range of unidentified factors. It is interesting to note that, when comparing all nonlateralized and lateralized patients, regardless of outcome, the nonlateralized cohort had a larger proportion of poor outcome (45%) patients compared that of the lateralized cohort (37%). This stresses the importance of neuroradiologist seizure lateralizations, which the introduction of JLF/CL in the clinical workflow could potentially help improve.

Although pre-surgical planning involves many variables and modalities, including clinical presentation, EEG, and other imaging studies, MR imaging is an important aspect of clinical lateralization, and computational assistance to neuroradiologist reads could positively affect the clinical decision-making pathway (Kini et al., 2016). As a study employing clinical scans with the intent of translation into clinical practice, our focus on post-surgical outcome as the metric of success, as opposed to pathology, can show the potential for improvements in the morbidity and mortality of TLE patients. Our work demonstrates the ability of the JLF/CL automated segmentation method to replicate and assist existing neuroradiologist reads in epilepsy; further work will seek to translate an JLF/CL-assisted MRI read into clinical practice, hopefully allowing for better seizure lateralization and localization, and subsequently improved outcomes following surgical resection.

4.3. Limitations

While there are different methods of automated segmentation, based on our analysis of the accuracy of the JLF/CL compared to that of others (as described previously), the Dice scores were comparable. As a result, the JLF/CL method was implemented for the clinical lateralization experiments. However, this clinical lateralization analysis was also run using the results of FreeSurfer, as a confirmation (Supplementary Fig. 2). FreeSurfer produced a lower AUC of 0.85 (compared to JLF/CL of 0.92). It inaccurately lateralized 2 of the 9 non-lateralized patients, as the JLF/CL method did, but additionally incorrectly lateralized a right-sided patient to the left (which JLF/CL did not do). Since this study was not powered to detect the superiority of one automated segmentation method over another, futures investigations will be needed to answer that question. Regardless of the method ultimately chosen, the analysis of automated segmentation as it relates to clinical outcomes as opposed to pathology is a novel one, and could hopefully be used to improve patient care.

Whereas a neuro-radiologist can indicate lateralization or non-lateralization in the MRI scans, JLF/CL is only able to lateralize seizure onset. The lack of TLE patients with confirmed nonlateralized disease limited our ability to distinguish negative results and produce a negative predictive value. Future work could also investigate the indeterminate asymmetry index boundary between the left and right lateralization ranges; by determining the asymmetry index range of a large set of healthy controls with a confidence interval and setting lateralization thresholds around it, JLF/CL could move beyond solely lateralization and incorporate non-lateralization as well. While the improved seizure lateralization is likely due to the robust JLF/CL method, the binary classification of JLF/CL, when compared to the multiclass classification of a neuroradiologist, could bias the results.

Although the sensitivity of the JLF/CL method allowed for the lateralization of all patients, including 7 of the 9 patients who were non-lateralized by a neuroradiologist, 2 of the 9 non-lateralized patients were incorrectly lateralized. Due to the invasive nature of epilepsy surgery, specificity and the physician's principle of nonmaleficence (“Do No Harm”) serve as a stricter clinical guide, leading to neuroradiologists correctly lateralizing all patients except the ones who were non-lateralized. Future implementation of this JLF/CL method into clinical practice must include these considerations, to minimize the risk of unnecessary or erroneous surgeries.

Additionally, the wide range of asymmetry indices for patients correctly lateralized by neuroradiologists, including patients who had asymmetry indices near the threshold cutoffs, indicate that there may be further considerations for seizure onset besides solely total hippocampal volume asymmetry. Hippocampal subfield volume and shape changes, clinical information, and previous scans are important variables for a neuroradiologist, in addition to many others, and were not incorporated in this JLF/CL method. Future studies dedicated to incorporating more of these variables into machine learning-based approaches could enable these automated techniques to become an invaluable part of clinical practice.

Although we based good outcome on correct lateralization with corresponding surgical resection, it is possible that some of the patients who were listed as poor outcome were correctly lateralized, but incorrectly localized or incompletely resected on the ipsilateral side to the seizure focus. The lack of additional clinical information in our data set, specifically intracranial EEG recordings, prevents finer resolution of the seizure onset zone. Future studies using patient cohorts with intracranial EEG data confirming seizure onset will further validate our work. In addition, the true extent of resections in cases of anterior temporal lobectomies and newer surgical techniques, such as laser ablation, are required to determine volume and subfields of hippocampus removed and adjust the analysis to account for these confounds in surgical technique.

4.4. Conclusions

We have demonstrated the ability of JLF/CL to correctly lateralize seizure onset, using an atlas composed of normal-appearing subjects, and plan to incorporate the JLF/CL-derived volumetric data in our home institution for pre-surgical decision-making pipeline. Our JLF/CL software is available for public use at https://www.nitrc.org/projects/ashs/ under the Penn Temporal Lobe Epilepsy T1-MRI Whole Hippocampus ASHS Atlas: ASHS 1.0 Compatible release entry with the filename ashs_atlas_penntle_hippo_20170915.tar. Further investigations will focus on moving from lateralization to localization, to eventually pinpoint hippocampal subfield-level seizure onset and better identify the epileptogenic focus in nonlesional TLE patients. This refined diagnostic capability will hopefully improve post-resection outcomes and open the door to clinically translatable automated segmentation in all localization-related epilepsies.

The following are the supplementary data related to this article.

Supplementary Fig. 1

The Effect of Increasing Normal Subject Atlas Composition on Dice by Target Image. The Dice indices for all 59 subjects are presented as a result of the 50 cross-validation experiments (Fifty 15-subject target image sets of 5 Normal, 5 Mild, and 5 Severe against Fifty 20-subject atlases of differing MRI phenotype composition), separated by the MRI phenotype of the target image (Normal, Mild, Severe, Diseased) and on increasing percentage of normal subjects in the 20-subject atlas. Of note, a 100% normal atlas was not possible due to dataset limitations and, due to the different proportions of subjects in each experiment, weighted averaging was employed for the Dice coefficient per subject to limit subject selection bias. This is another visual depiction of the trends seen in Fig. 1, where increasing normal composition of the atlas improved segmentation accuracy of normal target images but had no effect on segmentation accuracy of other MRI phenotype target images.

Supplementary Table 1

Demographic and Clinical Information for all subjects. Detailed demographic and clinical details for all subjects and controls including age, sex, MRI clinical report results, PET clinical report results, scalp EEG lateralization, pathology, and resection lateralization are provided.