Predict Seizure Freedom after Epilepsy Surgery, a Challenge in Clinical Practice.

Abstract

Objective:

To compare the accuracy of clinical judgment in predicting seizure outcome after resective epilepsy surgery relative to two recently published statistical tools [the Epilepsy Surgery Nomogram (ESN) and the modified Seizure-Freedom score (m-SFS)].

Methods:

Details of 20 epilepsy surgery patients’ pre-surgical evaluations were presented to 20 epilepsy experts. The final surgical treatment was also disclosed. The clinicians were asked to predict the likelihood of a good outcome (Engel 1) at 2 and 5 years in each case. The ESN and the m-SFS predictions were calculated with the data provided to the clinicians. The discriminative ability of clinical judgment, ESN, and m-SFS was assessed by calculating a concordance index (C-index). Expert opinion, the m-SFS and the ESN performances were compared using a Receiver Operating Characteristic curve analysis (ROC).

Results:

The mean age at surgery was 29 years (SD=14), 40% were male, 70% were right handed and thirteen (65%) had an Engel outcome 1 at 2 and 5 years. The mean C-index for the mean physician’s prediction was 0.478 with a variance of 0.012. The ESN had an area under the curve (AUC) of 0.528 and 0.533 for the 2 year and 5 year predictions in comparison with the clinicians’ predictions that was 0.476, and 0.466, respectively. For the m-SFS, the AUC at 2 years and 5 years was 0.539 and 0.539, respectively. No statistical difference was noted between the ESN and the clinicians or between m-SFS and the ESN, but there is a moderate statistical difference favoring the m-SFS to the clinicians (p 0.0960 and 0.0514, for 2 and 5 years).

Significance:

Clinical judgment was not superior to the ESN nor to the m-SFS. Together with the inter-physician’s prediction variability, our findings reinforce the need for better tools to predict postoperative outcomes.

1. Introduction:

Resective brain surgery is the gold standard therapy for focal drug resistant epilepsy.[1,2] However, postoperative seizure freedom rates vary widely from 30% to 80% in different series.[3–5] The success depends on intrinsic (pathology, location, type of epilepsy) and extrinsic factors (completeness of resection, advances in neuroimaging technology) that play roles as “positive” and “negative” factors affecting the outcome. The prediction of success has been elusive to clinicians and may be as subjective as the interpretation of pre-surgical test findings in the evaluation of patients with intractable epilepsy.[4] The lack of a uniform, objective and effective instrument to evaluate and predict success complicates decision-making prior to surgery. With half of patients failing surgery, there is an unmet need for accurate outcome prediction tools to facilitate pre-operative patient counseling,[5] and potentially minimize healthcare costs through early identification of challenging patients who need referral to a tertiary care center.

Most outcome studies have focused on the identification of isolated outcome predictors.[6–9] We recently developed algorithms such as the modified Seizure Freedom Score (m-SFS)[10] and an Epilepsy Surgery Nomogram (ESN)[11] to create individualized outcome predictions based on a multifactorial synthesis of patient characteristics. However, the clinical usefulness of such tools is inherently dependent on a “needs assessment” based on the accuracy of the current standard of care, which is clinical judgment. Some have suggested that multidisciplinary groups are better at predicting outcomes than individual physicians, [12] regardless of the expertise touting the benefits of multidisciplinary patient management conferences in guiding decision-making as opposed to individual “expert” opinions. Yet, to date, no studies have comprehensively measured the predictive accuracy of clinical judgment taking into account potentially critical factors in building clinical acumen such as the physician’s age, experience, exposure, and seniority. Similarly, no studies have directly compared the performance of statistical predictive models such as the m-SFS or the ESN to clinical judgment.

In this study, we aimed to measure the discriminative ability of expert opinion in predicting postoperative seizure outcomes in patients undergoing epilepsy presurgical evaluation and to compare it with the ESN and SFS, using a well-characterized cohort of patients who underwent surgery in our center and have an ascertained long-term outcome. The goals are to: 1)- measure how accurate are we as epilepsy experts in predicting surgical outcomes? And 2)- compare our performance to the m-SFS and ESN.

2. Methods:

2.1. Case characterization:

20 cases were selected from our epilepsy surgery outcomes database. Patients operated between 2000 and 2007 at Cleveland Clinic were screened to identify a mix of pediatric and adult patients, temporal and extra-temporal resections, lesional and non-lesional patients, invasive and non-invasive EEG performed. The resulting case mix is summarized in supplementary table 1. The medical history included age, gender, handedness, physical examination, prior history of generalized seizures, age at seizure onset, epilepsy risk factors, pre-operative seizure frequency, and seizure semiology. The ictal and interictal findings in the scalp EEG were included as well as the results of the brain MRI, positron emission tomography (PET), single-photon emission tomography (SPECT) and neuropsychological evaluation. All the data was provided as it was interpreted in the multidisciplinary management conference. This closely mimics actual clinical care where the multidisciplinary consensus discussion may modify the initial interpretation of the tests (supplementary table 1). If the patient required an invasive evaluation (9 patients), results were also provided with the final recommendation and surgical treatment. All patients had at least 5 years of follow up. Surgical success was measured at 2 years and at 5 years, and defined as an Engel score of 1 (free of disabling seizures).

2.2. Definition of expert opinion:

24 epilepsy experts from the United States, Brazil, Australia and India participated in the study. Participants were directors or staff physicians in comprehensive epilepsy surgery centers. All available data from the cases characterized as detailed above were de-identified and then provided to the expert participants. The clinicians were asked for each one of the cases: “If you had 100 patients just like this one, how many do you think would have an Engel 1 outcome at 2 and 5 years after surgery?” Information about how many surgeries per year were performed at their respective epilepsy centers and the years of practice after training was also obtained. The information about the outcome was not disclosed to the clinicians.

2.3. ESN prediction:

The ESN is a statistical tool developed in Cleveland clinic and retrospectively validated with the support of collaborators at four comprehensive epilepsy centers: Mayo Clinic- Rochester (Rochester, MN, USA), Niguardia Hospital (Milan, Italy), University of Campinas (Campinas, Brazil), and Aix-Marseille University (Marseille, France). The ESN synthesizes clinical variables including gender, age at onset of seizures, age at time of surgery, mean monthly preoperative frequency of seizures, generalized tonic– clonic seizure history, MRI findings (normal vs abnormal), surgical side of the brain, type of surgery (temporal lobectomy, frontal lobectomy and posterior quadrant resection) and pathological cause (defined by histological analysis or MRI) to provide an individualized prediction of seizure outcomes at 2 and 5 years.[11] For this study, two investigators (MG.K. and L.C ) who were blinded to the actual outcomes entered the relevant characteristics of the 20 cases into the ESN and recorded the tool’s predicted chance of Engel 1 at 2 years and 5 years.

2.4. SFS prediction:

Similar to the ESN, the modified Seizure Freedom Score (m-SFS) takes into account different risk factors: MRI (lesional or not lesional), preoperative seizure frequency (>20/month), epilepsy duration (> 20 years), history of generalized seizure, need of invasive evaluation, and the surgical region (temporal vs extra-temporal).[10] In contrast to the ESN, only categorical variables could be included, and the model cannot adjust for a varying contribution of different outcome determinants. For this study, two investigators (MG.K and L.C) who were blinded to the actual outcomes entered the relevant characteristics of the 20 cases into the m-SFS and recorded the tool’s predicted chance of Engel 1 at 2 years and 5 years.

2.5. Statistical methods:

The discriminative ability of clinical judgment, ESN, and m-SFS was assessed by calculating a concordance index (C-index) which measures the probability that predicting the outcome is better than chance. By definition C-index is used to compare the goodness of fit of logistic regression models. Values for this measure range from 0.5 to 1.0. A value of 0.5 indicates that the model is no better than chance and a value of 1.0 indicates that the model perfectly identifies those within a group and those not. Models are typically considered reasonable when the C-index is higher than 0.7 and strong when the C-index exceeds 0.8. [13] Finally, the expert opinion, the m-SFS and the ESN performances were compared using a Receiver Operating Characteristic curve analysis (ROC). The area under the curve (AUC) of two ROC curves were compared by a bootstrap empirical significance test. Brier score was also used to verify the accuracy of probability of forecast for the 3 predictive models, m-SFS, ESN and expert opinion. The Brier Score is probably the most commonly used verification measure for assessing the accuracy of probability forecast. The Brier score is a statistical model used to compare predicted with observed outcomes. Scores range from 0 (best) to 1 (worst).

The study was approved by the Cleveland Clinic institutional review board and procedures abided by the contributing center’s ethics requirements.

3. Results:

The characteristics of the patient cohort and physician experts are summarized in table 1. Doctors were 63% male, practicing for an average of 12.4 years (range of 5– 40 years) in 9 comprehensive epilepsy surgery centers (including 5 centers that perform >50 epilepsy surgeries/year). Forty two percent of surveyed physicians were 41–50 years of age; 21% were 50–60 years old; 10% were older than 60 years, and the rest were 30–40 years old.

Table 1.

Patient characteristics:

Variables	n (%) SD
PATIENT CHARACTERICTICS
Age at onset (years of age)	13 (SD=10.4)
Epilepsy duration	15.9 (SD=12.27)
History of GTC (Present)	(15) 75%
Seizure frequency per month	13 (SD=10.4)
Abnormal neurological exam	5 (25%)
Abnormal MRI	15 (25%)
PET findings (localizable)	15 (25%)
Ictal SPECT (done)	2 (10%)
Invasive EEG	9 (45%)
Surgical side (Right)	7 (35%)
Temporal resection	15 (75%)
Engel 1 at 2 years of follow up	13 (65%)
Engel 1 at 5 years of follow up	13 (65%)

Figure 1 shows considerable variation among the clinicians’ predictions on each particular patient at 2 and 5 years. The analysis of each doctor’s predictions on all 20 patients as well as the ESN and m-SFS predictions are shown in Figure 2 with the respective C-index. C-index of the prediction at 2 years for the ESN was 0.528 compared with 0.539 for the m-SFS and the mean physician’s 0.478 (var 0.012). C-index values for the prediction at 5 years were 0.539 for the m-SFS, 0.523 for the ESN and 0.478 for the physicians (var 0.012). Note the high variability seen in C-index of different physicians with the lowest being 0.297 and 0.275 and the highest 0.742 and 0.709 at 2 and 5 years respectively. Physician-related factors including years of training, age, and surgeries per year at the epilepsy center were also depicted in different plots with the corresponding C-index (Figures 3–5), and were of no relevance to the predictive performance of the clinicians.

Figure 1.

Scatter plot of each individual doctor’s predictions on all 20 patients. A. Plot for predicting at 2 years. B. Plot for predicting at 5 years. Each dot represents a predicted probability of seizure-freedom from the doctors in black circles, the ESN in red, and the m-SFS score in blue. The green boxes represent the patients who were actually seizure-free.

Figure 2.

Analysis of each doctor’s predictions on all 20 patients as well as the ESN and m-SFS predictions. A. Prediction at 2 years. B. Prediction 5 years. Each bar represents the C-index of each doctor’s prediction on all 20 patients, where the green bar is the ESN’s predictions and the blue bar is the SFS score predictions. The dotted horizontal line is the overall concordance index for all doctors. The mean concordance index for all doctors is 0.478 with a variance of 0.012.

Figure 3.

Analysis of each doctor’s predictions on all 20 patients as well as the ESN and m-SFS predictions, stratified by age groups. A. Plot for predicting 2 years. B. Plot for predicting 5 years. Each bar represents each doctor’s prediction on all 20 patients stratified by age groups 30–40, 41–50, 51–60, and >60, where the green bar is the ESN’s predictions and the blue bar is the m-SFS score predictions. No significant significant difference was noted among predictive models. The dotted horizontal line is the overall concordance index for all doctors.

Figure 5.

Analysis of each doctor’s predictions on all 20 patients as well as the nomograms predictions stratified by number of surgeries per year. A. Plot for predicting 2 years. B. Plot for predicting at 5 years. Each bar represents each doctor’s prediction on all 20 patients stratified by number of surgeries per year, 10–24, 25–50, and >50, where the green bar is the ESN’s predictions and the blue bar is the m-SFS score predictions. No significant significant difference was noted among predictive models. The dotted horizontal line is the overall concordance index for all doctors.

The performance of the physician’s prediction, m-SFS scale and the ESN were compared using a Receiver Operating Characteristic curve analysis (ROC). No statistical difference was noted between the ESN and the clinicians or between m-SFS and the ESN, but there is a moderate statistical difference favoring the m-SFS to the clinicians (p 0.096 and 0.051, for 2 and 5 years) (Figure 6). Brier score of the prediction at 2 years for the ESN was 0.234 compared with 0.265 for the m-SFS and the mean physician’s 0.265. Brier score values for the prediction at 5 years were 0.234 for the ESN, 0.281 for the m-SFS, and 0.302 for the physicians.

Figure 6.

The ROC curve for 2 year predictions (A) and the curve for 5 year predictions (B). The ESN for 2 year and 5 year predictions has an AUC of 0.528 and 0.533, respectively, the doctors’ predictions have an AUC at 2 years and 5 years of 0.476, and 0.466, respectively, and the m-SFS scores have an AUC at 2 years and 5 years of 0.539 and 0.539, respectively. There is no statistical difference between the ESN and the doctors or between m-SFS and the ESN, but there is a moderate statistical difference between m-SFS and the doctors (p-value:0.0960 and 0.0514, for 2 and 5 years). Toward the higher end of Sensitivity and Specificity, both the ESN and the doctors are very similar and the SFS score seems to be very similar to the doctors.

4. Discussion:

With ongoing development of novel technologies, data mining and health information technology allow the creation of new statistical algorithms to provide personalized medical treatment for patients [14]: epilepsy surgery should not be the exception. Nomograms have already been developed in other fields of medicine and more recently in epilepsy to help support clinical decision-making. [15] Our study tried to compare these new tools -the ESN[11] and m-SFS [10] - with clinician’s expertise, and to evaluate the variability in predicting success of epilepsy surgery at 2 and 5 years.

4.1. Clinician performance:

As shown in detail in Figures 1–2, less than half of the clinicians outperformed the ESN and the m-SFS, and considerable variation was noted among the predictions made by the clinicians. The clinical decision-making process in medicine is complex and is elaborated by either an analytical or an intuitive model.[16] The intuitive model is automatic, unconscious, fast, and effortless or routine whereas the analytical model is effortful, usually slow, non-programmed and conscious.[17] The intuitive model, which is characterized by heuristics, appears to be most commonly used in clinical practice and is more susceptible to biases or systematic errors.[16,17] Unfortunately, cognitive bias is inherent to human judgment and, physicians are susceptible to it in the decision making process.[17–19] Physicians’ biases and personality traits may explain some medical errors and in the evaluation of epilepsy surgery; neurologists and neurosurgeons are not exempt. The neurophysiologist may not be the only responsible for errors but also the neurosurgeon, neuropsychologist, radiologist, or any member of the team. The multidisciplinary epilepsy patient management conference is probably the best way to decrease medical errors and is a great example of a more analytical model, which in theory should decrease cognitive biases.[16,20] However, this analytical and thoughtful process is not fully exempt of them either. Some possible biases related to epilepsy surgery evaluation are anchoring, availability, confirmation, diagnosis momentum and overconfidence. For instance, in our study the data provided to the clinicians was an interpretation at the multidisciplinary group meeting, which being a consensus is also susceptible of the above mentioned limitations. To add to the biases; age of the physician, type of training, years of practice and experience of the practitioner are also factors that can directly or indirectly affect the diagnostic and therapeutic decision process.[21] In our study, age of the physician, years of practice or experience did not make a significant correlation with the outcome.

Prognosis in medicine means to know beforehand, which in epilepsy surgery is translated in to prediction of the future course or outcome after surgery.[22] For epilepsy surgery, the possible outcome is difficult to assess given the multiple factors that the clinician faces in the evaluation of a patient with epilepsy. In our study, the large variability noted in different analyses highlights the lack of a common ground and consensus in the evaluation of patients with intractable epilepsy and proves that multiple biases influence the medical decision and therefore the prognosis. The evaluation and the prognostication are susceptible of multiple biases that create uncertainty in patients with intractable epilepsy who undergo surgical evaluation. Therefore, objective tools are required to better interpret the data supporting and guiding the clinician’s expertise.

Prognostic models in medicine are used in multiple settings including, healthcare policy by generating global predictive scenarios, determining study eligibility of patients for new treatments, defining inclusion criteria for clinical trial and, selecting appropriate tests and therapies in individual patient management including supporting decision such as withholding or withdrawing therapy.[22] With the use of biomedical informatics and data mining, predictive medicine may help to adapt guidelines and to customize diagnosis and treatments to the characteristics of the patient.[14,23] The ESN and m-SFS are examples of predictive models that help to counsel the patient and adjust the treatment according to the possible outcome using different factors that adjust to the patient’s features.

Even though the ESN and m-SFS seemed to perform slightly better than physicians, the difference was not statistically significant. Moreover, the large variability among physicians and among different patients stresses the need of other tools to improve the prediction of epilepsy surgery success. The ESN and m-SFS, which are far of being perfect, could still help to guide when the clinician is assessing the possible outcome. Multiple different variables have been already investigated in the evaluation of patients with intractable epilepsy. Some have been proven to be part of factors that may affect the outcome such as lesional MRI, seizure frequency and duration of epilepsy.[6] Some of these variables have been used as part of the ESN and m-SFS. However, other clinical and non-clinical variables should be also included in predictive models in order to create a more comprehensive, personalized and accurate model. For instance, presence of aura, certain semiological findings, genetic and molecular data, networks involvement, time to secondary generalization, findings in interictal and ictal EEG, structural abnormality in the MRI, pathology/microscopic findings and results of ancillary tests (PET, SPECTs, MEG, VBM) are all factors that need to be taken into account for a better decision making process that incorporated in new models or algorithms could help us to understand and manage better the disease. The clinical success of epilepsy surgery cannot be separated from possible complications and side effects from the procedure. Other variables such as possible complications could be included in the models. All these data could be incorporated in computerized models that automatically can guide us through the treatment and prognosis.

The human factors along with their inherent biases are far too complex to be taken into account in a model to predict success of epilepsy surgery. For instance, the surgeon’s and clinician’s expertise and the subjective interpretation of the history and ancillary tests are variables that may affect the outcome and are extremely difficult to evaluate. For instance, as number of surgeries per year may increase the expertise in different epilepsy centers, our analysis did not show the “linear” expected correlation. This may be explained by the complexity of the cases that the largest surgical centers handle and the possible cognitive biases.

Also, there is a clear need for new technologies, novel evaluation methods and probably computerized algorithms that may add to the clinical expertise in the pre-surgical evaluation of patients with intractable epilepsy.

4.2. Limitations:

Our study has certain limitations, the assessment between physicians/experts was done only with 20 different experts most of them from tertiary centers. A better assessment including small and large comprehensive centers is also needed in a future analysis. Second, the information provided to the physicians may seem to be limited for some since only the description of the findings was provided. Actually, the data provided to the clinicians was already influenced by biases in the interpretation by the group of experts in the multidisciplinary group meeting. Third, most of the patients in the analysis underwent temporal lobe resection and the number of cases, twenty, may limit the assessment.

5. Conclusion:

In our study clinical judgment was not superior to the ESN nor to the m-SFS. The inter-physician’s prediction variability reinforces the need for better tools to predict postoperative outcomes. More comprehensive models or algorithms which include more clinical and non-clinical variables and possible complications are needed to guide the clinician and the patient in the decision making process. As of now, the best strategy suggested is to use the available technology, the clinician’s knowledge, the presentation of cases in comprehensive multidisciplinary group conferences along with the available statistical analyses to open a discussion with the patient as a possible candidate of epilepsy surgery.

Figure 4.

Analysis of each doctor’s predictions on all 20 patients as well as the ESNs predictions stratified by years out of training. A. Plot for predicting 2 years. B.Plot for predicting at 5 years. Each bar represents each doctor’s prediction on all 20 patients stratified by years out of training, where the green bar is the ESN’s predictions and the blue bar is the m-SFS score predictions. No significant significant difference was noted among predictive models. The dotted horizontal line is the overall concordance index for all doctors.

Highlights:

Prediction of success of resective surgery for intractable epilepsy has been elusive to clinicians and depends on the interpretation of pre-surgical evaluation.

The best current strategy to predict success of epilepsy surgery is to use the available technology and the clinician’s knowledge.

In our study, clinical judgment was not superior to our 2 developed tools to predict surgical outcomes.

Inter-physician’s prediction variability highlights the need of an effective instrument to predict success in epilepsy surgery to support the decision-making process prior to surgery.

Comprehensive models or algorithms which include clinical and non-clinical variables are needed to guide the decision making process.