Abstract
Objectives:
To examine the applicability of applying an online tool to determine the appropriateness of referral for an epilepsy surgical evaluation and to determine whether appropriateness scores are concordant with the clinical judgment of epilepsy specialists.
Methods:
We prospectively applied the tool in 107 consecutive patients with focal epilepsy seen in an epilepsy outpatient clinic. Variables collected included seizure type, epilepsy duration, seizure frequency, seizure severity, number of antiepileptic drugs (AEDs) tried, AED-related side effects, and the results of investigations. Appropriateness ratings were then compared with retrospectively collected information concerning whether a surgical evaluation had been considered.
Results:
Thirty-nine patients (36.4%) were rated as appropriate for an epilepsy surgical evaluation, all of whom had adequately tried 2 or more appropriate AEDs. The majority of patients (84.6%) rated as appropriate had previously been considered or referred for an epilepsy surgical evaluation. Tool feasibility of use was high, with the exception of assessing whether previous AED trials had been adequate and discrepancies between physician and patient reports of AED side effects.
Conclusions:
Our evidence-based, online clinical decision tool is easily applied and able to determine whether patients with focal epilepsy are appropriate for a surgical evaluation. Future validation of this tool will require application in clinical practice and assessment of potential improvements in patient outcomes.
Despite more than a decade of Class I evidence1,2 and clinical practice guidelines3 stating that patients with drug-resistant focal temporal lobe epilepsy should be considered for a surgical evaluation, significant delays remain between epilepsy diagnosis and referral for surgical evaluation.4 Reasons for surgical delays and underutilization are multifold, including resource limitations, racial disparities,5,6 physician failure to adequately inform their patients about the procedure,7,8 and patient apprehension because of misconceptions of surgical risk.7,9,10
Another significant barrier to surgery is suboptimal surgical referral patterns.11,12 In a changing field where patients and practitioners may become accustomed to ongoing seizures, or overoptimistically embrace new antiepileptic drugs (AEDs), novel tools are needed to optimize patient care while reducing the complexities of determining surgical candidacy.
Using the RAND-UCLA appropriateness methodology,13 we recently developed a freely available, 8-question, Web-based clinical decision tool, the Canadian Appropriateness of Epilepsy Surgery (CASES) tool (www.epilepsycases.com),14 to assist physicians in easily identifying patients who might benefit from an epilepsy surgery evaluation.15 Our aim was to examine the feasibility of use of the CASES tool by prospectively applying it in clinical practice in a large Canadian health region, and to determine whether appropriateness scores generated using the CASES tool were concordant with the clinical judgment of epilepsy specialists.
METHODS
Patient recruitment.
Consecutive adults with focal epilepsy were recruited from an outpatient epilepsy clinic at a large tertiary care center. These patients participated in a 2-part study examining (1) patient knowledge and barriers to epilepsy surgery,10 and (2) feasibility of use of the CASES tool. The tool was developed using the RAND-UCLA appropriateness methodology.14 Briefly, this consisted of a systematic review about epilepsy surgery, the development of a clinical matrix including every possible scenario for patients with focal epilepsy, and then rating of each scenario based on the evidence by a panel of clinical experts (see below). Adults with focal epilepsy were eligible to participate unless they had an undetermined epilepsy syndrome, a progressive or end-stage neurodegenerative disease, or a severe psychiatric comorbidity precluding ability to participate in the knowledge and barriers questionnaire. Neurologists who followed patients in the Calgary Epilepsy Clinic received a letter outlining the inclusion and exclusion criteria for our study. Neurologists were asked to indicate on their weekly clinic list which patients met eligibility criteria to complete the questionnaire. A research assistant then reviewed the charts in the event of any eligibility uncertainties. At the clinic, participating patients completed a written informed consent form and questionnaire.
Standard protocol approvals, registrations, and patient consents.
The study was approved by the Conjoint Health Research Ethics Board at the University of Calgary.
Data abstraction/chart review.
The standardized data abstraction form (appendix e-1 on the Neurology® Web site at Neurology.org) contained variables required to assess appropriateness for an epilepsy surgical evaluation using the CASES tool,15 including epilepsy duration, seizure type(s), seizure frequency, seizure severity, number of adequate AEDs tried, AED side effects, and the results of EEG and/or MRI investigations. If a patient had a contraindication to receiving an MRI of the brain, a CT of the head demonstrating a clear abnormality associated with epilepsy was substituted as evidence of an abnormal MRI.
A sheet defining the variables included in the CASES tool15 was provided to clinicians with the data abstraction form (see table 1). Additional relevant information included epilepsy diagnosis, seizure focus, and age/date of seizure onset. This sheet was completed by a clinician (e.g., senior epilepsy nurse or epileptologist) at the time of the clinic visit. This allowed for the most up-to-date information to be collected. For the investigations, the latest MRI was reviewed. All EEGs (including video EEG monitoring) available at the time of the visit were reviewed, except in patients who already had epilepsy surgery, in which case only the findings from EEGs done after surgery were considered. Any missing data were abstracted by the trained research assistant with guidance from the study epileptologist. Self-reported side effects from AEDs were also determined from the original patient-completed questionnaire.10
Table 1.
Key variables and their definitions for clinical scenarios
CASES tool.
Patient information obtained from the physician-patient consultation and chart review was entered into the online CASES tool (http://epilepsycases.com)14 to obtain an appropriateness score for each patient. As outlined in the RAND Appropriateness Methodology, scores ranged from 1 to 9.13 Each patient was hence classified as appropriate (7–9), uncertain (4–6), or inappropriate (1–3) for an epilepsy surgery evaluation. According to the CASES tool Web site, patients deemed appropriate should be referred for an epilepsy surgery evaluation, those rated uncertain should be strongly considered for referral (hence probably appropriate), and those deemed inappropriate may not need to be referred at the time, but should be re-evaluated at each visit.15
Tool testing and tool validation.
A qualitative approach was taken to assess tool feasibility of use. Study investigators were asked to report any difficulties they experienced while obtaining information for the tool. Physician reporting of AED side effects were compared with patient self-report obtained through the questionnaire. To determine the validity of the tool in our setting, we compared the appropriateness rating in the CASES tool and whether a surgical evaluation had been performed, offered, or considered. Charts were examined in detail for all patients, and a system of “rules” was constructed in order to classify the surgical evaluation status of each individual patient. A surgical evaluation was judged to have been performed, offered, or considered if: (1a) there was a record of a seizure monitoring unit (SMU) admission or requisition with the goals of admission including establishing eligibility for epilepsy surgery or determining seizure focus; (1b) there was a record of an SMU admission along with a note about the patient's determined surgical candidacy in the telemetry report, consensus conference, or consult notes; (1c) the treating neurologist mentioned in the chart consult notes that they had discussed the possibility of epilepsy surgery or an SMU admission with the patient but the patient was not interested; or (1d) the treating neurologist mentioned that they were considering surgery but awaiting more information (e.g., a seizure diary). Additional criteria required that the surgical evaluation be performed, offered, or considered before May 31, 2012 (the recruitment end date), after the patient's most recent epilepsy surgery (if applicable), and within a 5-year period of the recruitment date (unless the patient had previously been determined be a poor surgical candidate).
Data analysis.
Descriptive statistics were obtained for all variables. Chi-square and Fisher exact tests were used to investigate associations between CASES variables and appropriateness scores. SPSS Statistics version 19 (IBM Corp., Armonk, NY) was used for all analyses.
RESULTS
Patient demographic and clinical characteristics.
In total, 107 eligible patients consented to participate during the study recruitment period (March 12, 2012 to April 20, 2012) for a response rate of 83.0%. Although nonresponders were slightly older than study participants (mean = 44.4 years vs 42.4 years) and a larger proportion of nonresponders were male (55.6% vs 46.7%), neither of these differences were statistically significant (p = 0.488 and p = 0.682). Characteristics of our study population are provided in table 2.
Table 2.
Clinical characteristics of study participants
Tool feasibility of use.
We were able to obtain a CASES appropriateness score for all patients, but some information was difficult to find. For example, it was challenging to determine whether a particular trial of an AED was “adequate” because of insufficient documentation of dosages and therapeutic levels, treatment adherence, and reasons for drug discontinuation. Physician reporting of AED side effects was not fully concordant with results from the patient questionnaire. While the overall agreement between patients and physicians was fair (κ = 0.316), agreement was much higher in situations where patients reported the absence of side effects (94.1%) compared with situations where patients reported that side effects were present (45.6%).
Tool validation.
Table 3 lists the proportion of patients with each of the tool clinical characteristics (e.g., number of adequate AED trials) stratified by appropriateness score. Approximately half of participants were rated as inappropriate for a surgical evaluation (table 3). Among patients deemed as inappropriate for referral by the CASES tool, nearly all patients had less than 2 adequate trials of AEDs, were currently seizure-free, or had nondisabling seizures.
Table 3.
Frequency of clinical variables for CASES appropriateness scoresa and relation to evaluation for surgery
Fifteen participants (14.0%) were rated as uncertain and 39 patients (36.4%) were rated as appropriate for an epilepsy surgical evaluation. Seizure type (simple partial vs complex partial/generalized tonic-clonic) was the only CASES variable found not to differ significantly between patients who were rated as appropriate and those who were not (Fisher exact, p = 1.00). The proportion of patients that received a particular appropriateness score according to individual variables in the CASES tool can be viewed online (figure e-1).
In total, a surgical evaluation was performed, offered, or considered in 39.3% (42/107) of our study population (table 3). Among these patients, 8 (19.0%) were rated as “inappropriate” for evaluation by the CASES tool, 1 (2.4%) as “uncertain,” and 33 (78.6%) as “appropriate” (table 3). Of the 8/53 “inappropriate” patients in whom an epilepsy surgical evaluation was considered, the primary goal of SMU admission was to rule out nonepileptic seizures in 3 cases, to look for frequent epileptiform discharges or electrographic seizures as an explanation for cognitive decline in 1 case, and the patient had responded to medication adjustments after an SMU admission in 2 cases (hence these patients were previously appropriate). However, in 2 cases, the treating physician considered the patient to be appropriate for an epilepsy surgical evaluation despite the CASES tool rating them as inappropriate. In the first case, the patient had only tried one AED and as such did not yet meet the definition of drug-resistant epilepsy. In the second case, the patient had a breakthrough seizure just before the tool was administered; however, although this patient had tried 2 AEDs and had an abnormal MRI, the seizures were nondisabling.
Of the 6/39 “appropriate” patients in whom an epilepsy surgical evaluation was not considered, 2 patients were stated to be satisfied with their current seizure control (despite ongoing seizures) and as such were unlikely to be interested in surgery (although it was not offered or discussed). One patient was enrolled in a drug trial instead of receiving a surgical workup, and another was treated with medication while waiting for a recent subdural hematoma (etiology of his new-onset epilepsy) to resolve. One patient required characterization of a new spell type before consideration of surgery. Finally, one patient was seizure-free after epilepsy surgery but still received a rating of appropriate because of an abnormal EEG, medication side effects, and having trialed 2 AEDs.
Excluding the 21 patients who had previously undergone epilepsy surgery from the validation portion of the study had minimal impact on the validity of the tool. When these patients were eliminated, we found that 81% of patients who were deemed appropriate by the tool had been offered a surgical evaluation (vs 85% with all patients included).
DISCUSSION
We demonstrated the feasibility of use of the CASES tool through successfully determining an appropriateness score for all study participants. We established tool validity by finding that 85% of patients rated appropriate for an epilepsy surgery evaluation were considered for or received an evaluation. Thus, the CASES tool selected the same group of patients to be appropriate for an epilepsy surgical evaluation as a team of epilepsy specialists who did not participate in tool development.
Among patients rated inappropriate for an epilepsy surgery evaluation, all but one patient was correctly classified by the tool. Although in reality this patient was not appropriate (they had a single breakthrough seizure after 20 years), it remains possible that a similar patient (who had tried 2 AEDs and had ongoing seizures every 2 years) could be missed by the tool. The lower rating assigned to this clinical scenario by expert panelists likely occurred because the seizures were occurring at a frequency of less than 1 per year, and were nondisabling.
In agreement with the original CASES tool publication,15 appropriateness scores were found to be higher in those with greater seizure frequency, disabling seizures, a higher number of AED trials, and abnormal investigations (especially MRI). None of the scenarios in which the patients had tried one or no AEDs was rated as appropriate, which is consistent with the new definition of drug-resistant epilepsy.16
Although we found the tool to be feasible and valid, some challenges were reported. Classifying the number of adequate AED trials received by a patient was intermittently difficult even for experienced epileptologists. The International League Against Epilepsy recently defined drug-resistant epilepsy as “failure of adequate trials of two tolerated and appropriately chosen and used AED schedules (whether as monotherapy or in combination) to achieve sustained seizure freedom.”16 However, because of incomplete medical records and/or patient difficulty recalling past AED dosages and reasons for discontinuation, clinicians may be uncertain whether previous AED trials were truly “adequate.” As a result, it is important that the CASES tool be applied prospectively in clinical practice. Patients with low tolerability for medication side effects may never complete an adequate trial; given the large number of seizure medications available, patients could continuously cycle through drugs without ever being considered eligible for a surgical evaluation. We thus emphasize the importance of systematic, accurate, and prospective documentation of medication-related information by physicians in their day-to-day practice.
Recent American Academy of Neurology epilepsy quality-improvement measures state that all patients should be queried and counseled about medication side effects at each visit.17 Because there is always the concern that patients and physicians may differ in their reports of AED side effects, we asked both patients and physicians to rate their perceptions of AED adverse effects (although only the physician's rating was input into the CASES tool). Clinicians' report of AED side effects was often discordant with patients' self-report of AED side effects. Although clinicians' judgment is superior regarding the ability to make differentiations between established AED side effects and patient reporting of AED side effects (potentially attributing side effects to AEDs when they are due to their other causes), patient-physician discussion remains crucial for completion of the tool. One limitation of our study is that we did not use a formal tool to assess AED side effects from the patient's perspective (e.g., The Liverpool Adverse Events Profile).18 Using a validated AED side effect screening tool should be considered in future studies.
Patient-physician discussion is also important when assessing seizure-related disability, which has not been rigorously defined and remains generally individual.19 Of note, the tool addresses disability specifically related to the patients' seizures when they occur, but does not take into account overall epilepsy-related disability. It has been shown that epilepsy-related stigma has a similar impact on quality of life as seizure freedom.20
One major limitation of our validation study is that information on whether patients had been considered for epilepsy surgery was collected retrospectively. As a result, we may not have accurately captured whether treating physicians believed that their patients were surgical candidates at the time we determined appropriateness ratings. Future studies could improve our study design by simply asking physicians whether they think their patient is currently appropriate for an epilepsy surgical evaluation. While the ultimate goal would be to apply this tool in general practice and neurology clinics, our results should not yet be extrapolated to general populations. Despite having a catchment area of more than 1 million individuals, participants in this study were ascertained from a single Canadian regional epilepsy clinic. Nearly all patients had completed EEG and MRI investigations, which may differ from other centers where investigations are less readily available—this is noteworthy because incomplete investigations were a major determinant of expert disagreement during tool development.15 Lastly, our decision to include some patients who had already undergone epilepsy surgery increased the complexity of tool application, particularly regarding inclusion of presurgical test results and failed AED trials. None of the expert cases examined during tool development took previous operations into consideration, and current guidelines are unclear regarding whether a patient must fail 2 additional AED trials before being considered for further epilepsy surgery.
The RAND-UCLA appropriateness methodology can distill epilepsy clinical complexities into a concise and practical referral device. Now that the CASES tool has been validated in an epilepsy specialty clinic, it is closer to providing a comprehensive guide for determining candidacy for epilepsy surgery evaluations in general practice. It is important to emphasize that those patients who are rated “appropriate” for an epilepsy surgical evaluation may or may not be found to be surgical candidates. While the CASES tool still requires validation with community physicians and general neurologists before widespread application, the current data foster optimism. Usability and feasibility testing of the tool with the target users (i.e., pediatricians, child and adult neurologists, family medicine specialists, and trainees in these specialties) is ongoing and will provide further information regarding the barriers and facilitators to implementing the tool in practice. As part of this latter work, knowledge translation, dissemination, and sustainability of the tool are being explored. The tool appears to be usable and valid, and has the potential to improve clinical decision-making more effectively than clinical practice guidelines alone. However, it is critical that this tool be validated through prospective application in different settings.
Supplementary Material
ACKNOWLEDGMENT
The authors thank the patients who participated in this study, as well as the nurses and administrative staff in the University of Calgary Epilepsy Clinic for their assistance with recruitment and data collection. J. Roberts holds a Frederick Banting and Charles Best Canada Graduate Scholarship from the Canadian Institutes of Health Research and an Alberta Innovates Health Solutions (AIHS) Graduate Studentship. N. Jetté is the holder of an AIHS Population Investigator Award and a Canada Research Chair in Neurological Health Services Research. S. Wiebe holds the Hopewell Professorship of Clinical Neurological Research from the University of Calgary. K. Sauro holds an AIHS Graduate Studentship and a Western Regional Training Centre Graduate Scholarship.
GLOSSARY
- AED
antiepileptic drug
- CASES
Canadian Appropriateness of Epilepsy Surgery
- SMU
seizure monitoring unit
- UCLA
University of California, Los Angeles
Footnotes
Supplemental data at Neurology.org
AUTHOR CONTRIBUTIONS
Jodie I. Roberts: design/conceptualization of the study, analysis and interpretation of the data, drafting and revising the manuscript for intellectual content. Chantelle Hrazdil: design/conceptualization of the study, interpretation of the data, drafting and revising the manuscript for intellectual content. Samuel Wiebe: design/conceptualization of the study, interpretation of the data, revising the manuscript for intellectual content. Khara Sauro, Alexandra Hanson, Paolo Federico, Neelan Pillay, William Murphy, and Michelle Vautour: revising the manuscript for intellectual content. Nathalie Jetté: design/conceptualization of the study, interpretation of the data, revising the manuscript for intellectual content.
STUDY FUNDING
This study was funded in part by an Alberta Innovates Health Solutions research prize to Dr. Nathalie Jetté.
DISCLOSURE
J. Roberts has a graduate studentship from Alberta Innovates Health Solutions (AIHS) and a Canadian Institutes of Health Research (CIHR) Banting and Best Graduate Scholarship. C. Hrazdil reports no disclosures relevant to the manuscript. S. Wiebe serves on the editorial boards of Neurology, Epileptic Disorders, Epilepsy & Behavior, and Canadian Journal of Neurological Sciences and receives research support from AIHS, CIHR, and University of Calgary (Hopewell Professorship of Clinical Neurosciences Research). K. Sauro has a graduate studentship from AIHS and a Western Regional Training Centre Graduate Scholarship. A. Hanson reports no disclosures relevant to the manuscript. P. Federico serves on the editorial board of the Canadian Journal of Neurological Sciences and receives research support from CIHR and the University of Calgary. N. Pillay, W. Murphy, and M. Vautour report no disclosures relevant to the manuscript. N. Jetté is an associate editor of Epilepsia and is the recipient of an AIHS Population Health Investigator Award and a Canada Research Chair Tier 2 in Neurological Health Services Research. She has operating funds from CIHR, Alberta Health Services, and the University of Calgary Hotchkiss Brain Institute and Faculty of Medicine. Go to Neurology.org for full disclosures.
REFERENCES
- 1.Wiebe S, Blume WT, Girvin JP, Eliasziw M. A randomized, controlled trial of surgery for temporal-lobe epilepsy. N Engl J Med 2001;345:311–318 [DOI] [PubMed] [Google Scholar]
- 2.Engel J, Jr, McDermott MP, Wiebe S, et al. Early surgical therapy for drug-resistant temporal lobe epilepsy: a randomized trial. JAMA 2012;307:922–930 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Engel J, Jr, Wiebe S, French J, et al. Practice parameter: temporal lobe and localized neocortical resections for epilepsy: report of the Quality Standards Subcommittee of the American Academy of Neurology, in association with the American Epilepsy Society and the American Association of Neurological Surgeons. Neurology 2003;60:538–547 [DOI] [PubMed] [Google Scholar]
- 4.Haneef Z, Stern J, Dewar S, Engel J., Jr Referral pattern for epilepsy surgery after evidence-based recommendations: a retrospective study. Neurology 2010;75:699–704 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Englot DJ, Ouyang D, Garcia PA, Barbaro NM, Chang EF. Epilepsy surgery trends in the United States, 1990–2008. Neurology 2012;78:1200–1206 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.McClelland S, III, Guo H, Okuyemi KS. Racial disparities in the surgical management of intractable temporal lobe epilepsy in the United States: a population-based analysis. Arch Neurol 2010;67:577–583 [DOI] [PubMed] [Google Scholar]
- 7.Swarztrauber K, Dewar S, Engel J., Jr Patient attitudes about treatments for intractable epilepsy. Epilepsy Behav 2003;4:19–25 [DOI] [PubMed] [Google Scholar]
- 8.Benbadis SR, Heriaud L, Tatum WO, Vale FL. Epilepsy surgery, delays and referral patterns: are all your epilepsy patients controlled? Seizure 2003;12:167–170 [DOI] [PubMed] [Google Scholar]
- 9.Prus N, Grant AC. Patient beliefs about epilepsy and brain surgery in a multicultural urban population. Epilepsy Behav 2010;17:46–49 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Hrazdil C, Roberts JI, Wiebe S, et al. Patient perceptions and barriers to epilepsy surgery: evaluation in a large health region. Epilepsy Behav 2013;28:52–65 [DOI] [PubMed] [Google Scholar]
- 11.Kumlien E, Mattsson P. Attitudes towards epilepsy surgery: a nationwide survey among Swedish neurologists. Seizure 2010;19:253–255 [DOI] [PubMed] [Google Scholar]
- 12.de Flon P, Kumlien E, Reuterwall C, Mattsson P. Empirical evidence of underutilization of referrals for epilepsy surgery evaluation. Eur J Neurol 2010;17:619–625 [DOI] [PubMed] [Google Scholar]
- 13.Fitch K, Bernstein S, Aguilar MD, et al. The RAND/UCLA Appropriateness Method User's Manual. Santa Monica, CA: RAND Corporation; 2001 [Google Scholar]
- 14.C.A.S.E.S. Canadian Appropriateness Study of Epilepsy Surgery. Available at: www.epilepsycases.com. Accessed June 1, 2013
- 15.Jette N, Quan H, Tellez-Zenteno JF, et al. Development of an online tool to determine appropriateness for an epilepsy surgery evaluation. Neurology 2012;79:1084–1093 [DOI] [PubMed] [Google Scholar]
- 16.Kwan P, Arzimanoglou A, Berg AT, et al. Definition of drug resistant epilepsy: consensus proposal by the ad hoc Task Force of the ILAE Commission on Therapeutic Strategies. Epilepsia 2010;51:1069–1077 [DOI] [PubMed] [Google Scholar]
- 17.Fountain NB, Van Ness PC, Swain-Eng R, et al. Quality improvement in neurology: AAN epilepsy quality measures: report of the Quality Measurement and Reporting Subcommittee of the American Academy of Neurology. Neurology 2011;76:94–99 [DOI] [PubMed] [Google Scholar]
- 18.Baker GA, Jacoby A, Francis P, Chadwick DW. The Liverpool adverse events drug profile. Epilepsia 1995;36:S59 [Google Scholar]
- 19.Langfitt JT, Mathern GW. The future is here: web-based eight-question tool to determine epilepsy surgery evaluation. Neurology 2012;79:1074–1075 [DOI] [PubMed] [Google Scholar]
- 20.Grant AC, Prus N, Nakhutina L. Factors affecting quality of life in epilepsy in a multi-ethnic urban population. Epilepsy Behav 2013;27:283–285 [DOI] [PubMed] [Google Scholar]
- 21.World Health Organization. International drug monitoring: the role of national centres. Report of a WHO meeting. World Health Organization Technical Report Series 1972;498:1–25 [PubMed] [Google Scholar]
- 22.Rosenow F, Luders H. Presurgical evaluation of epilepsy. Brain 2001;124:1683–1700 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.