Abstract
Reliable change index scores (RCIs) and standardized regression-based change score norms (SRBs) permit evaluation of meaningful changes in test scores following treatment interventions, like epilepsy surgery, while accounting for test-retest reliability, practice effects, score fluctuations due to error, and relevant clinical and demographic factors. Although these methods are frequently used to assess cognitive change after epilepsy surgery in adults, they have not been widely applied to examine cognitive change in children with epilepsy. The goal of the current study was to develop RCIs and SRBs for use in children with epilepsy. Sixty-three children with epilepsy (age range 6–16; M=10.19, SD=2.58) underwent comprehensive neuropsychological evaluations at two time points an average of 12 months apart. Practice adjusted RCIs and SRBs were calculated for all cognitive measures in the battery. Practice effects were quite variable across the neuropsychological measures, with the greatest differences observed among older children, particularly on the Children’s Memory Scale and Wisconsin Card Sorting Test. There was also notable variability in test-retest reliabilities across measures in the battery, with coefficients ranging from 0.14 to 0.92. RCIs and SRBs for use in assessing meaningful cognitive change in children following epilepsy surgery are provided for measures with reliability coefficients above 0.50. This is the first study to provide RCIs and SRBs for a comprehensive neuropsychological battery based on a large sample of children with epilepsy. Tables to aid in evaluating cognitive changes in children who have undergone epilepsy surgery are provided for clinical use. An excel sheet to perform all relevant calculations is also available to interested clinicians or researchers.
Keywords: epilepsy, reliable change indices, standardized regression-based change score norms, children, neuropsychology
1.1 Introduction
Neuropsychological assessment is an essential component of epilepsy surgery programs. These evaluations help determine the cognitive risks associated with epilepsy surgery and assess postsurgical neurobehavioral outcomes. Change in cognitive abilities across time or in response to interventions has historically been evaluated by: 1) examining differences in cognitive outcome between groups of patients or 2) comparing change scores in individual patients to some predetermined, yet often arbitrary, difference believed to reflect actual change based on conventional practice (e.g., 10 or 15 standard score points). These procedures for assessing change are confounded by methodological artifacts (e.g., imperfect test reliability, measurement error, practice effects, regression toward the mean) that are likely to lead to erroneous conclusions regarding cognitive outcome.
Beginning in the 1990s, two methods for assessing postsurgical cognitive change while controlling these confounding factors emerged in the adult epilepsy literature: reliable change index scores (RCIs) and standardized regression-based change score norms (SRBs) [1–3]. These methods have been developed for a wide range of cognitive measures and are now routinely applied to assess cognitive outcome in adults following epilepsy surgery [1–4]. Despite the clear benefits, RCIs and SRBs have not been developed to examine cognitive change in children after epilepsy surgery across a wide range of cognitive measures.
Our prospective, longitudinal study was designed to address this gap in the literature. Specifically, this study provides RCIs and SRBs for children with epilepsy across a comprehensive neuropsychological battery using the same methods employed by Martin and colleagues [4] for adults with epilepsy. These data allow clinical neuropsychologists to objectively assess cognitive change after pediatric epilepsy surgery. Moreover, clinicians can use these RCIs and SRBs to monitor the effect epilepsy has on the cognitive development of children who do not undergo surgery and to examine both the efficacy and potential side effects of non-surgical medical treatments.
2.1 Materials and Methods
2.1.1 Participants
This prospective study was approved by the Cleveland Clinic Institutional Review Board. Children who were being evaluated and/or treated for epilepsy within the Cleveland Clinic Epilepsy Center were approached regarding study participation during an outpatient appointment if they met the following inclusion criteria: 1) ages 6 to 16 years; 2) confirmed history of seizures as evidenced on EEG recordings; 3) history of seizures for at least one year; 4) maintained on a stable AED regimen; 5) fluent in English; 6) no past neurosurgical intervention; 7) no history of neurodegenerative disorder; and 8) no neuropsychological testing within the previous 6 months.
A total of 76 children met inclusion criteria, agreed to participate, and completed the initial assessment. Caregivers provided written informed consent, and children over the age of 12 provided assent for the study. Repeat neuropsychological evaluations were scheduled approximately 9 months following the initial evaluation, whenever possible. This test-retest interval was selected to approximate the average interval between pre and postoperative neuropsychological assessments of children who undergo epilepsy surgery at our center. Participants received a $40 gift card after completing their first assessment and a $60 gift card at follow-up. A copy of the test results was also provided to interested caregivers. A total of 13 (17.1%) children did not complete the second neuropsychological assessment. This resulted in a final sample size of 63 children who were an average of ten years old and had completed approximately five years of education. The mean age at seizure onset was 6.5 years (SD = 3.1), and the average duration of epilepsy was 3.7 years (SD = 2.6). Eighty-four percent of the sample was Caucasian, and just over half of the participants were female (57.1%).
Given the wide age range of participants and age-specific differences in test items and developmental factors, participants were stratified into 2 age groups: 6–10 years old (younger group; n=36) and 11–16 years old (older group; n=27). Additional demographic and seizure information for the participants is presented in Table 1.
Table 1.
Demographic and Seizure Data for Study Patients
| Variable | Younger Group Ages 6–10 N=36 |
Older Group Ages 11–16 N=27 |
|---|---|---|
| M (SD) | M (SD) | |
| Age (years) | 8.36 (1.38) | 12.63 (1.60) |
| Education (years) | 2.69 (1.31) | 7.00 (1.66) |
| Full Scale IQ (standard score) | 86.61 (17.27) | 83.67 (16.06) |
| Age at Seizure Onset (years) | 5.24 (2.24) | 8.26 (3.18) |
| Duration of Epilepsy (years) | 3.12 (2.13) | 4.48 (3.03) |
| Inter-test Interval (months) | 13.03 (8.47) | 10.93 (2.97) |
| Sex | Male = 17 (47.2%) | Male = 10 (37.0%) |
| Female = 19 (52.8%) | Female = 17 (63.0%) | |
| Race | Caucasian = 32 (88.9%) | Caucasian = 21 (77.8%) |
| African American = 2 (5.6%) | African American = 3 (11.1%) | |
| Other = 2 (5.6%) | Other = 3 (11.1%) | |
| Handedness | Left = 6 (16.7%) | Left = 4 (14.8%) |
| Right = 29 (80.6%) | Right = 23 (85.2%) | |
| Ambidextrous = 1 (2.8%) | Ambidextrous = 0 (0%) | |
| Type of Seizures | Generalized = 16 (44.4%) | Generalized = 13 (48.1%) |
| Focal = 20 (55.6%; 9 left, 11 right) | Focal = 14 (51.9%; 10 left, 4 right) | |
| Seizure Focus | Temporal = 4 | Temporal = 5 |
| Frontal = 4 | Frontal = 2 | |
| Parietal = 2 | Parietal = 0 | |
| Occipital = 1 | Occipital = 1 | |
| Multilobar = 9 | Multilobar = 6 |
M = mean; SD = standard deviation
2.1.2 Measures
Participants completed a comprehensive neuropsychological evaluation that included measures of intelligence, memory, language, visuospatial skills, executive functioning, and academic achievement. The following instruments were administered on two separate occasions: Wechsler Intelligence Scale for Children – Fourth Edition[5], Children’s Memory Scale[6], Expressive One-Word Picture Vocabulary Test - Revised[7], The Beery-Buktenica Developmental Test of Visual-Motor Integration – Fourth Edition[8], Test of Visual Perceptual Skills – Third Edition[9] (Visual Discrimination, Visual Memory, and Spatial Relations subtests), Wisconsin Card Sorting Test[10], Delis-Kaplan Executive Function System[11] (Trail Making Test, Verbal Fluency Test, and Tower Test), and the Woodcock-Johnson III Tests of Achievement[12] (Letter-Word Identification, Reading Fluency, Calculation, Math Fluency, Spelling, Writing Fluency, Passage Comprehension, Applied Problems, Writing Samples, Word Attack, and Punctuation & Capitalization subtests). All measures were administered according to standardized instructions provided in the respective test manuals and scored using age-adjusted norms.
2.1.3 Analyses
Reliable Change Indices
Practice adjusted RCI cutoff scores were calculated for each of the neuropsychological measures in the test battery according to the methods outlined by Jacobson & Truax[13]. First, test-retest reliability coefficients were computed for each of the neuropsychological measures. Then RCIs were developed for the two separate age ranges (i.e., 6–10 and 11–16). The standard error of measurement was used to calculate the standard error of the difference (SEdiff), where SEdiff = √2(SEM)2. As noted by Jacobson and Truax [13], the SEdiff describes “the spread of the distribution of change scores that would be expected if no actual change had occurred” (p. 14), that is, based solely on chance fluctuations in test scores across time. Next, confidence intervals were established at 80% and 90% by multiplying the SEdiff by ±1.28 and ±1.64, respectively. This provided two different distribution ranges of change scores, with the 90% confidence interval offering a more conservative estimate of test-retest change and the 80% confidence interval a more liberal estimate. The resulting cut-off score ranges were then adjusted for practice effects[3, 14]. Average practice effects were determined by calculating the mean change (i.e., Time 2 mean minus Time 1 mean) for each cognitive measure. Finally, these practice effects were added to the confidence interval in order to center the range of cut-off scores around the average test-retest practice. Score changes outside of these confidence intervals are considered uncommon in children with epilepsy who have not undergone surgery during the test-retest interval since they occur in less than 80% (80% CI) of these children or 90% (90% CI) of these children in the absence of surgical intervention.
Standardized Regression – Based Change Score Norms
A series of multiple regression equations were used to predict retest scores for each neuropsychological measure using the baseline test score and potential modifying factors (e.g., age, age at seizure onset, and test-retest interval). Because age was included as a predictor in the regression equations, SRBs were calculated using data from the full sample of nonsurgical children (N=63). Linear regression analyses were conducted for each neuropsychological test score using the methods outlined by McSweeney et al.[2] Specifically, variables were entered into the regression equation in a stepwise fashion. A probability level of .05 was used to determine variable entry and .10 was used to determine variable removal at each step.
3.1 Results
A summary of baseline and retest mean scores and standard deviations along with mean change scores and test-retest reliabilities for each test is presented in Table 2. Paired t-tests were used to examine differences between baseline and retest performances. Mean change scores were quite variable across the neuropsychological measures, with the greatest differences observed among older children, particularly on the standard scores associated with the Children’s Memory Scale Indices (mean improvement of 8–12.62 points on five of the eight indices) and the Wisconsin Card Sorting Test (mean change scores of 8 on Total Errors and 9.82 on Perseverative Responses). There was also notable variability in test-retest reliabilities across measures in the battery, with coefficients ranging from 0.14 to 0.92. Given the very low reliability of some of these measures, RCI intervals and regression analyses are only reported for those measures with test-retest reliabilities above 0.50.
Table 2.
Test-Retest Means, SD, Mean Difference Scores, and Test-Retest Reliability
| Index/Subtest | Younger Group | Older Group | Test- Retest Reliability | ||||
|---|---|---|---|---|---|---|---|
| Time 1 Mean (SD) | Time 2 Mean (SD) | Mean Change | Time 1 Mean (SD) | Time 2 Mean (SD) | Mean Change | ||
| Intelligence | |||||||
| Wechsler Intelligence Scale for Children – Fourth Edition | |||||||
| Full Scale IQ | 86.94 (17.40) | 86.71 (16.84) | −0.23 | 83.67 (16.06) | 84.56 (14.86) | 0.89 | 0.92 |
| Verbal Comprehension Index | 92.06 (16.06) | 91.00 (15.91) | −1.06 | 90.52 (17.80) | 88.63 (13.83) | −1.89 | 0.88 |
| Perceptual Reasoning Index | 90.89 (14.55) | 93.57 (17.08) | 2.68 | 87.11 (14.72) | 88.67 (13.73) | 1.56 | 0.78 |
| Working Memory Index | 85.23 (17.71) | 82.31 (14.37) | −2.92 | 85.70 (20.03) | 88.41 (19.28) | 2.71 | 0.80 |
| Processing Speed Index | 89.37 (16.02) | 87.91 (16.02) | −1.46 | 83.78 (14.46) | 84.22 (12.46) | 0.44 | 0.76 |
| Similarities | 9.29 (3.39) | 9.29 (3.13) | 0.00 | 8.63 (2.95) | 8.26 (2.44) | −0.37 | 0.82 |
| Vocabulary | 8.21 (3.07) | 8.18 (3.40) | −0.03 | 8.52 (3.82) | 8.41 (3.19) | −0.11 | 0.87 |
| Comprehension | 8.06 (2.98) | 8.09 (2.80) | 0.03 | 7.78 (3.12) | 7.56 (2.68) | −0.22 | 0.71 |
| Word Reasoning | 9.19 (3.30) | 9.00 (3.44) | −0.19 | 8.92 (3.35) | 9.08 (3.43) | 0.16 | 0.78 |
| Block Design | 8.18 (2.96) | 9.18 (3.14) | 1.00* | 6.89 (2.47) | 7.78 (2.90) | 0.89** | 0.79 |
| Picture Concepts | 8.94 (3.42) | 9.44 (3.17) | 0.50 | 9.04 (3.24) | 8.22 (3.07) | −0.82 | 0.54 |
| Matrix Reasoning | 8.26 (2.85) | 8.18 (3.57) | −0.08 | 7.78 (2.85) | 8.44 (2.49) | 0.66 | 0.51 |
| Digit Span | 7.68 (3.52) | 7.71 (3.18) | 0.03 | 7.96 (3.76) | 8.56 (3.94) | 0.60 | 0.73 |
| Letter-Number Sequencing | 6.76 (3.55) | 6.09 (3.22) | −0.67 | 7.19 (4.19) | 7.48 (3.98) | 0.29 | 0.69 |
| Coding | 7.15 (3.12) | 7.09 (3.03) | −0.06 | 6.81 (3.18) | 6.81 (2.79) | 0.00 | 0.76 |
| Symbol Search | 8.75 (3.79) | 8.41 (3.29) | −0.34 | 6.93 (3.09) | 7.59 (2.72) | 0.66 | 0.61 |
| Cancellation | 10.25 (2.72) | 10.03 (2.78) | −0.22 | 8.88 (2.79) | 9.24 (3.14) | 0.36 | 0.73 |
| Memory | |||||||
| Children’s Memory Scale | |||||||
| Visual Immediate | 91.45 (13.45) | 93.24 (14.92) | 1.79 | 90.30 (12.97) | 100.15 (14.08) | 9.85* | 0.53 |
| Visual Delayed | 89.53 (13.61) | 97.00 (15.00) | 7.47* | 87.15 (13.84) | 98.56 (15.43) | 11.41* | 0.54 |
| Verbal Immediate | 87.67 (23.07) | 89.82 (25.96) | 2.15 | 88.41 (22.29) | 94.07 (22.59) | 5.66* | 0.90 |
| Verbal Delayed | 82.50 (22.40) | 88.38 (22.97) | 5.88** | 87.78 (22.30) | 95.85 (21.28) | 8.07* | 0.83 |
| General Memory | 85.31 (22.45) | 89.47 (23.66) | 4.16 | 83.92 (21.08) | 96.54 (22.37) | 12.62* | 0.85 |
| Attention/Concentration | 91.86 (21.66) | 89.00 (20.70) | −2.86 | 90.40 (16.80) | 89.76 (18.73) | −0.64 | 0.89 |
| Learning | 89.55 (18.12) | 91.45 (19.00) | 1.90 | 86.52 (13.88) | 94.52 (17.97) | 8.00* | 0.69 |
| Delayed Recognition | 91.25 (21.42) | 87.00 (19.26) | −4.25 | 89.26 (22.91) | 93.22 (18.39) | 3.96 | 0.66 |
| Dot Locations - Learning | 9.18 (2.89) | 9.18 (3.32) | 0.00 | 7.92 (2.48) | 9.58 (2.89) | 1.66** | 0.19 |
| Dot Locations - Total Score | 9.48 (2.92) | 9.70 (3.39) | 0.22 | 8.33 (2.63) | 10.04 (2.65) | 1.71** | 0.21 |
| Dot Locations - Short Delay | 10.00 (3.10) | 10.48 (2.82) | 0.48 | 8.88 (2.73) | 10.08 (2.66) | 1.20 | 0.15 |
| Dot Locations - Long Delay | 9.41 (3.00) | 10.78 (2.57) | 1.37** | 8.56 (2.79) | 9.70 (3.04) | 1.14 | 0.17 |
| Stories - Immediate | 8.88 (3.70) | 8.79 (4.34) | −0.09 | 8.56 (3.81) | 9.93 (4.09) | 1.37* | 0.82 |
| Stories - Delayed | 8.34 (4.24) | 8.56 (4.33) | 0.22 | 8.11 (4.42) | 9.52 (4.20) | 1.41** | 0.83 |
| Stories - Delayed Recognition | 9.12 (3.78) | 7.81 (2.79) | −1.31** | 7.92 (4.44) | 8.92 (3.76) | 1.00 | 0.60 |
| Faces - Immediate | 7.73 (2.96) | 8.12 (2.74) | 0.39 | 8.48 (3.09) | 10.00 (3.14) | 1.52** | 0.52 |
| Faces - Delayed | 7.19 (2.76) | 8.25 (3.40) | 1.06** | 7.26 (3.35) | 9.85 (3.39) | 2.59* | 0.56 |
| Word Pairs - Immediate | 7.52 (3.65) | 8.64 (4.15) | 1.12** | 8.32 (3.68) | 7.60 (3.65) | −0.72 | 0.71 |
| Word Pairs - Learning | 7.42 (4.48) | 8.06 (4.51) | 0.64 | 7.46 (4.09) | 8.31 (4.01) | 0.85 | 0.82 |
| Word Pairs - Total Score | 7.12 (4.20) | 7.91 (4.73) | 0.79 | 7.67 (4.09) | 8.19 (3.98) | 0.52 | 0.82 |
| Word Pairs - Long Delay | 5.97 (3.85) | 7.66 (3.79) | 1.69** | 7.93 (3.75) | 9.11 (3.69) | 1.18** | 0.60 |
| Word Pairs - Delayed Recognition | 8.06 (4.09) | 8.28 (4.18) | 0.22 | 8.27 (4.14) | 8.73 (3.54) | 0.46 | 0.68 |
| Numbers - Total Score | 8.53 (3.84) | 7.87 (3.30) | −0.66 | 7.88 (3.18) | 7.60 (3.70) | −0.28 | 0.79 |
| Sequences - Total Score | 8.45 (3.80) | 8.23 (3.79) | −0.22 | 8.92 (2.97) | 9.04 (3.12) | 0.12 | 0.84 |
| Family Pictures - Immediate | 9.57 (3.09) | 9.73 (3.03) | 0.16 | 9.21 (3.18) | 10.21 (2.75) | 1.00 | 0.52 |
| Family Pictures - Delayed | 9.28 (3.25) | 9.21 (3.06) | −0.07 | 9.50 (3.40) | 9.88 (3.42) | 0.38 | 0.57 |
| Word List - Learning | 7.75 (5.37) | 6.96 (3.93) | −0.79 | 6.14 (2.88) | 6.77 (3.28) | 0.63 | 0.62 |
| Word List - Delayed | 6.67 (3.53) | 6.74 (3.46) | 0.07 | 6.41 (3.33) | 6.82 (3.62) | 0.41 | 0.57 |
| Word List - Delayed Recognition | 7.74 (3.81) | 5.89 (3.25) | −1.85* | 6.68 (3.21) | 7.23 (3.69) | 0.55 | 0.44 |
| Language | |||||||
| Expressive One-Word Picture Vocabulary | |||||||
| 96.59 (17.49) | 97.06 (16.50) | 0.47 | 93.68 (16.08) | 94.48 (19.41) | 0.80 | 0.86 | |
| Visuospatial | |||||||
| Test of Visual Perceptual Skills | |||||||
| Visual Discrimination | 7.33 (3.08) | 6.89 (2.81) | −0.44 | 5.82 (3.03) | 6.50 (3.22) | 0.68 | 0.48 |
| Visual Memory | 7.07 (4.12) | 7.48 (3.27) | 0.41 | 4.77 (3.44) | 6.45 (3.08) | 1.68 | 0.16 |
| Spatial Relations | 9.52 (4.42) | 9.81 (4.29) | 0.29 | 8.14 (3.31) | 7.77 (3.89) | −0.37 | 0.45 |
| Visual Motor Integration | 89.77 (10.01) | 88.20 (11.18) | −1.57 | 85.11 (12.36) | 84.96 (14.28) | −0.15 | 0.70 |
| Executive Functioning | |||||||
| Wisconsin Card Sorting Test | |||||||
| Categories Completed | 3.75 (1.62) | 4.79 (1.53) | 1.04** | 3.80 (1.83) | 4.60 (1.63) | 0.80** | 0.32 |
| Total Errors | 89.92 (13.10) | 95.71 (15.23) | 5.79 | 85.91 (12.00) | 94.00 (17.21) | 8.09** | 0.48 |
| Perseverative Responses | 93.08 (14.35) | 98.50 (13.51) | 5.42 | 87.22 (14.81) | 97.04 (18.06) | 9.82* | 0.50 |
| Delis-Kaplan Executive Function System | |||||||
| Tower Test | |||||||
| Total Achievement | 9.24 (2.20) | 10.71 (2.20) | 1.47* | 7.45 (3.24) | 9.90 (3.63) | 2.45* | 0.67 |
| Move Accuracy Ratio | 8.47 (3.56) | 9.53 (2.27) | 1.06 | 9.05 (2.70) | 9.60 (2.39) | 0.55 | 0.66 |
| Rule Violations Per Item Ratio | 8.06 (3.03) | 10.41 (1.18) | 2.35* | 8.65 (2.21) | 10.10 (1.07) | 1.45** | 0.42 |
| Verbal Fluency | |||||||
| Letter Fluency | 8.67 (2.93) | 8.39 (3.40) | −0.28 | 7.77 (2.84) | 8.82 (3.38) | 1.05** | 0.74 |
| Category Fluency | 10.11 (3.23) | 9.94 (3.62) | −0.17 | 8.86 (3.71) | 9.23 (3.70) | 0.37 | 0.63 |
| Category Switching Total Correct | 10.00 (3.18) | 9.44 (3.05) | −0.56 | 8.82 (2.94) | 9.14 (3.06) | 0.32 | 0.51 |
| Category Switching Accuracy | 10.11 (3.16) | 9.83 (2.31) | −0.28 | 8.86 (2.59) | 9.56 (2.58) | 0.70 | 0.53 |
| Set-Loss Errors | 9.78 (2.88) | 11.00 (1.94) | 1.22 | 10.45 (2.04) | 11.45 (1.74) | 1.00** | 0.30 |
| Repetition Errors | 7.28 (2.05) | 6.89 (2.25) | −0.39 | 7.32 (2.68) | 7.64 (2.01) | 0.32 | 0.14 |
| Trails | |||||||
| Visual Scanning | 10.33 (2.64) | 10.11 (3.23) | −0.22 | 9.77 (2.53) | 10.50 (2.41) | 0.73 | 0.48 |
| Number Sequencing | 7.94 (3.57) | 9.56 (3.45) | 1.62 | 8.50 (2.96) | 8.59 (2.67) | 0.09 | 0.17 |
| Letter Sequencing | 7.78 (4.01) | 8.28 (4.10) | 0.50 | 8.41 (3.38) | 8.14 (4.04) | −0.27 | 0.68 |
| Number-Letter Switching | 7.06 (4.33) | 8.00 (4.56) | 0.94 | 6.86 (4.28) | 7.64 (3.53) | 0.78 | 0.73 |
| Motor Speed | 10.00 (2.03) | 10.00 (2.57) | 0.00 | 8.45 (3.16) | 8.91 (2.93) | 0.46 | 0.43 |
| Total Errors | 8.56 (4.32) | 10.12 (2.28) | 1.56** | 8.95 (3.63) | 10.50 (1.44) | 1.55 | 0.46 |
| Academic Achievement | |||||||
| Woodcock Johnson | |||||||
| Broad Reading | 98.83 (21.65) | 97.04 (19.70) | −1.79 | 92.76 (15.68) | 95.84 (17.51) | 3.08* | 0.95 |
| Broad Math | 94.58 (15.53) | 95.42 (15.19) | 0.84 | 93.30 (17.60) | 89.78 (14.96) | −3.52** | 0.87 |
| Broad Written Language | 97.65 (21.13) | 97.00 (18.61) | −0.65 | 99.74 (14.10) | 100.26 (14.07) | 0.52 | 0.91 |
| Academic Fluency | 94.36 (20.79) | 95.16 (20.72) | 0.80 | 91.92 (16.26) | 95.12 (17.60) | 3.20* | 0.90 |
| Letter-Word Identification | 99.00 (18.44) | 95.00 (18.19) | −4.00* | 95.15 (13.52) | 95.96 (11.17) | 0.81 | 0.91 |
| Reading Fluency | 97.96 (22.48) | 98.36 (23.41) | 0.40 | 93.46 (15.98) | 97.96 (19.71) | 4.50* | 0.90 |
| Calculations | 94.25 (16.11) | 95.71 (13.17) | 1.46 | 93.56 (17.88) | 87.96 (16.07) | −5.60** | 0.74 |
| Math Fluency | 85.32 (16.59) | 85.92 (19.53) | 0.60 | 89.96 (17.93) | 91.00 (17.30) | 1.04 | 0.85 |
| Spelling | 96.41 (20.96) | 96.37 (19.17) | −0.04 | 102.36 (14.36) | 102.72 (15.33) | 0.36 | 0.92 |
| Writing Fluency | 96.38 (19.80) | 94.62 (18.38) | −1.76 | 94.84 (15.17) | 95.32 (13.37) | 0.48 | 0.79 |
| Passage Comprehension | 94.59 (17.28) | 91.59 (15.45) | −3.00** | 90.42 (13.59) | 90.19 (12.59) | −0.23 | 0.88 |
| Applied Problems | 95.31 (17.27) | 95.85 (17.16) | 0.54 | 93.80 (15.15) | 91.40 (12.46) | −2.40 | 0.83 |
| Writing Samples | 96.88 (15.99) | 98.96 (16.19) | 2.08 | 93.84 (14.85) | 93.92 (15.49) | 0.08 | 0.69 |
| Word Attack | 98.26 (15.61) (16.00) | 96.22 (13.49) | −2.04 | 98.12 (10.21) | 95.85 (7.58) | −2.27 | 0.85 |
| Punctuation and Capitalization | 87.58 (19.43) | 84.31 (22.32) | −3.27 | 85.26 (21.11) | 88.87 (19.18) | 3.61 | 0.71 |
p < 0.01,
p < 0.05
Adjusted reliable change cut scores at both 80% and 90% confidence intervals are provided in Table 3 along with the correction value used to adjust for practice effects. Separate RCIs are provided for younger and older children. The adjusted reliable change scores reported in Table 3 represent cutoff values at or beyond which an observed change score would represent a clinically meaningful change after adjusting for test-retest reliability and practice effects. For example, a 7 year-old whose Working Memory Index improved by 12 standard score points from baseline to retest would show a clinically meaningful change falling outside of the 80% confidence interval, but not outside of the 90% confidence interval. This same 12-point improvement would be considered unremarkable in a 12 year-old patient, but rather would be thought to reflect unreliability of the measure and typical practice effects.
Table 3.
Adjusted Reliable Change Indices (80% and 90% Confidence Intervals)
| Index/Subtest | Younger Group | Older Group | ||||
|---|---|---|---|---|---|---|
| Practice | Adjusted RC 80% | Adjusted RC 90% | Practice | Adjusted RC 80% | Adjusted RC 90% | |
| Intelligence | ||||||
| Wechsler Adult Intelligence Scale for Children – Fourth Edition | ||||||
| Full Scale IQ | 0 | −9, 9 | −12, 11 | 1 | −8, 9 | −10, 12 |
| Verbal Comprehension | −1 | −11, 9 | −14, 12 | −2 | −13, 9 | −16, 12 |
| Perceptual Reasoning | 3 | −10, 15 | −13, 18 | 2 | −11, 14 | −14, 17 |
| Working Memory | −3 | −17, 11 | −21, 15 | 3 | −13, 19 | −18, 23 |
| Processing Speed | −1 | −16, 13 | −20, 17 | 0 | −12, 13 | −16, 17 |
| Similarities | 0 | −3, 3 | −3, 3 | 0 | −3, 2 | −3, 3 |
| Vocabulary | 0 | −2, 2 | −3, 3 | 0 | −3, 2 | −3, 3 |
| Comprehension | 0 | −3, 3 | −4, 4 | 0 | −3, 3 | −4, 4 |
| Word Reasoning | 0 | −3, 3 | −4, 3 | 0 | −3, 3 | −3, 4 |
| Block Design | 1 | −1, 3 | −2, 4 | 1 | −1, 3 | −2, 3 |
| Picture Concepts | 1 | −4, 5 | −5, 6 | −1 | −5, 3 | −6, 4 |
| Matrix Reasoning | 0 | −4, 4 | −5, 5 | 1 | −3, 4 | −4, 5 |
| Digit Span | 0 | −3, 3 | −4, 4 | 1 | −3, 4 | −4, 5 |
| Letter-Number Sequencing | −1 | −4, 3 | −5, 4 | 0 | −4, 5 | −5, 6 |
| Coding | 0 | −3, 3 | −4, 3 | 0 | −3, 3 | −4, 4 |
| Symbol Search | 0 | −5, 4 | −6, 5 | 1 | −3, 4 | −4, 5 |
| Cancellation | 0 | −3, 2 | −3, 3 | 0 | −2, 3 | −3, 4 |
| Memory | ||||||
| Children’s Memory Scale | ||||||
| Visual Immediate | 2 | −15, 19 | −20, 23 | 10 | −6, 26 | −11, 31 |
| Visual Delayed | 7 | −9, 24 | −14, 29 | 11 | −6, 28 | −10, 33 |
| Verbal Immediate | 2 | −11, 16 | −15, 19 | 6 | −7, 19 | −11, 22 |
| Verbal Delayed | 6 | −11, 23 | −16, 28 | 8 | −9, 25 | −14, 30 |
| General Memory | 4 | −12, 20 | −16, 24 | 13 | −2, 27 | −6, 32 |
| Attention/Concentration | −3 | −16, 10 | −20, 14 | −1 | −11, 9 | −14, 12 |
| Learning | 2 | −16, 20 | −22, 25 | 8 | −6, 22 | −10, 26 |
| Delayed Recognition | −4 | −27, 18 | −33, 25 | 4 | −20, 28 | −27, 35 |
| Stories - Immediate | 0 | −3, 3 | −4, 4 | 1 | −2, 4 | −2, 5 |
| Stories - Delayed | 0 | −3, 3 | −4, 4 | 1 | −2, 5 | −3, 6 |
| Stories - Delayed Recognition | −1 | −6, 3 | −7, 4 | 1 | −4, 6 | −6, 8 |
| Faces - Immediate | 0 | −3, 4 | −4, 5 | 2 | −2, 5 | −3, 7 |
| Faces - Delayed | 1 | −2, 4 | −3, 5 | 3 | −1, 7 | −3, 8 |
| Word Pairs – Immediate | 1 | −2, 5 | −3, 6 | −1 | −4, 3 | −5, 4 |
| Word Pairs - Learning | 1 | −3, 4 | −4, 5 | 1 | −2, 4 | −3, 5 |
| Word Pairs- Total Score | 1 | −2, 4 | −3, 5 | 1 | −3, 4 | −4, 5 |
| Word Pairs - Long Delay | 2 | −3, 6 | −4, 7 | 1 | −3, 5 | −4, 7 |
| Word Pairs - Delayed Recognition | 0 | −4, 4 | −5, 6 | 0 | −4, 5 | −5, 6 |
| Numbers – Total Score | −1 | −4, 3 | −5, 3 | 0 | −3, 2 | −4, 3 |
| Sequences – Total Score | 0 | −3, 3 | −4, 3 | 0 | −2, 2 | −3, 3 |
| Family Pictures - Immediate | 0 | −4, 4 | −5, 5 | 1 | −3, 5 | −4, 6 |
| Family Pictures - Delayed | 0 | −4, 4 | −5, 5 | 0 | −4, 4 | −5, 6 |
| Word List - Learning | −1 | −7, 5 | −8, 7 | 1 | −3, 4 | −4, 5 |
| Word List - Delayed | 0 | −4, 4 | −5, 5 | 0 | −4, 4 | −5, 6 |
| Language | ||||||
| Expressive One Word Picture Vocabulary | ||||||
| 0 | −11, 12 | −15, 16 | 1 | −10, 12 | −13, 15 | |
| Visuospatial | ||||||
| Visual Motor Integration | −2 | −12, 8 | −14, 11 | 0 | −13, 12 | −16, 16 |
| Executive Functioning | ||||||
| Wisconsin Card Sorting Test | ||||||
| Perseverative Responses | 5 | −13, 24 | −18, 29 | 10 | −9, 29 | −14, 34 |
| Delis-Kaplan Executive Function System | ||||||
| Trails | ||||||
| Letter Sequencing | 0 | −4, 5 | −5, 6 | 0 | −4, 3 | −5, 4 |
| Number-Letter Switching | 1 | −3, 5 | −4, 6 | 1 | −3, 5 | −4, 6 |
| Verbal Fluency | ||||||
| Letter Fluency | 0 | −3, 2 | −4, 3 | 1 | −2, 4 | −2, 4 |
| Category Fluency | 0 | −4, 3 | −5, 4 | 0 | −4, 4 | −5, 6 |
| Category Switching Total Correct | −1 | −5, 3 | −6, 5 | 0 | −3, 4 | −4, 5 |
| Category Switching Accuracy | 0 | −4, 4 | −5, 5 | 1 | −3, 4 | −3, 5 |
| Tower Test | ||||||
| Total Achievement | 1 | −1, 4 | −1, 4 | 2 | −1, 6 | −2, 7 |
| Move Accuracy Ratio | 1 | −3, 5 | −4, 6 | 1 | −2, 3 | −3, 4 |
| Academic Achievement | ||||||
| Woodcock-Johnson III | ||||||
| Broad Reading | −2 | −10, 7 | −13, 9 | 3 | −3, 9 | −5, 11 |
| Broad Math | 1 | −9, 11 | −12, 14 | −4 | −15, 8 | −18, 11 |
| Broad Written Language | −1 | −12, 11 | −16, 14 | 1 | −7, 8 | −9, 10 |
| Academic Fluency | 1 | −11, 13 | −14, 16 | 3 | −6, 12 | −9, 15 |
| Letter-Word Identification | −4 | −14, 6 | −17, 9 | 1 | −7, 8 | −9, 10 |
| Reading Fluency | 0 | −12, 13 | −16, 17 | 5 | −5, 14 | −7, 16 |
| Calculations | 1 | −13, 16 | −17, 20 | −6 | −22, 11 | −27, 15 |
| Math Fluency | 1 | −11, 12 | −14, 16 | 1 | −12, 14 | −15, 17 |
| Spelling | 0 | −11, 10 | −14, 13 | 0 | −7, 8 | −9, 10 |
| Writing Fluency | −2 | −18, 15 | −23, 19 | 0 | −12, 13 | −16, 17 |
| Passage Comprehension | −3 | −14, 8 | −17, 11 | 0 | −9, 8 | −11, 11 |
| Applied Problems | 1 | −13, 14 | −16, 17 | −2 | −14, 9 | −17, 12 |
| Writing Samples | 2 | −14, 18 | −19, 23 | 0 | −15, 15 | −19, 19 |
| Word Attack | −2 | −13, 9 | −16, 12 | −2 | −9, 5 | −12, 7 |
| Punctuation and Capitals | −3 | −22, 16 | −28, 21 | 4 | −17, 24 | −23, 30 |
RC = reliable change; Results are only reported for cognitive measures with test-retest reliabilities above .50. When interpreting an individual patient’s pattern of scores, test-retest differences falling at or below the lower limit of the RC interval or at or above the upper limit of the RC interval would be indicative of clinically meaningful change.
Results of regression analyses for all neuropsychological measures in the battery are provided in Table 4. Specifically, the multiple R value, standard error of the estimate, and constant are included for each measure along with beta weights for the preoperative test score and any relevant modifiers retained in the equation (e.g., age, onset, or test-retest interval). Preoperative test score was a significant predictor of postoperative test score for all cognitive measures. Modifiers only entered the equation for select cognitive measures, accounting for 1% to 12% of the variance in postoperative test score. For those measures in which a modifier was retained in the equation, two equations are reported – one that includes the modifier and indicates the percentage of variance accounted for by the modifier and one that does not – for clinicians who may prefer to use a simpler equation in clinical practice. Using baseline test scores and modifiers, the SRB equation predicts re-test performance. The difference between the predicted score and the actual score is then transformed into a z-score by dividing it by the standard error of the estimate. Z-scores that are ≧ 1.28 exceed the 80% confidence interval and those that are ≧ 1.64 exceed the 90% confidence interval and represent clinically meaningful changes.
Table 4.
Regression Equations for Neuropsychological Measures
| Domain / Measure | R | SEest | C | Bbaseline | Bage | Bonset | Btestinterval |
|---|---|---|---|---|---|---|---|
| Intelligence | |||||||
| Wechsler Intelligence Scale for Children – Fourth Edition | |||||||
| Full Scale IQ | .916 | 6.537 | 12.299 | .858 | |||
| Verbal Comprehension Index | .873 | 7.402 | 18.519 | .781 | |||
| Perceptual Reasoning Index | .790 | 9.916 | 15.837 | .847 | |||
| Working Memory Index | .813 | 10.035 | 23.765 | .723 | |||
| Processing Speed Index | .769 | 9.531 | 21.854 | .737 | |||
| Processing Speed Index w/ Modifier (4%) | .797 | 9.087 | 28.281 | .731 | −.493 | ||
| Similarities | .826 | 1.627 | 2.224 | .727 | |||
| Vocabulary | .869 | 1.669 | 1.167 | .852 | |||
| Comprehension | .687 | 1.973 | 3.060 | .613 | |||
| Word Reading | .788 | 2.125 | 1.715 | .804 | |||
| Block Design | .788 | 1.906 | 1.997 | .858 | |||
| Picture Concepts | .536 | 2.681 | 4.393 | .501 | |||
| Picture Concepts w/ Modifier (5%) | .581 | 2.608 | 7.261 | .490 | −.270 | ||
| Matrix Reasoning | .534 | 2.716 | 3.580 | .589 | |||
| Digit Span | .730 | 2.538 | 2.500 | .720 | |||
| Letter-Number Sequencing | .695 | 2.559 | 2.491 | .623 | |||
| Coding | .754 | 1.904 | 2.049 | .693 | |||
| Symbol Search | .649 | 2.367 | 3.504 | .565 | |||
| Symbol Search w/ Modifier (5%) | .687 | 2.285 | 4.853 | .559 | −.107 | ||
| Cancellation | .705 | 2.075 | 2.628 | .733 | |||
| Memory | |||||||
| Children’s Memory Scale | |||||||
| Visual Immediate | .525 | 12.238 | 45.132 | .566 | |||
| Visual Immediate w/ Modifier (7%) | .585 | 11.770 | 31.308 | .559 | 1.404 | ||
| Visual Delayed | .569 | 12.400 | 42.511 | .628 | |||
| Verbal Immediate | .897 | 10.673 | 8.967 | .943 | |||
| Verbal Delayed | .822 | 12.764 | 23.078 | .806 | |||
| General Memory | .854 | 11.875 | 17.395 | .891 | |||
| General Memory w/ Modifier (2%) | .867 | 11.501 | 4.532 | .888 | 1.266 | ||
| Attention/Concentration | .900 | 8.752 | 8.223 | .896 | |||
| Learning | .699 | 13.147 | 25.044 | .773 | |||
| Delayed Recognition | .685 | 13.663 | 38.397 | .586 | |||
| Stories - Immediate | .823 | 2.385 | 1.344 | .913 | |||
| Stories - Delayed | .843 | 2.261 | 2.318 | .802 | |||
| Stories - Delayed Recognition | .535 | 2.786 | 4.585 | .447 | |||
| Stories - Delayed Recognition w/ Modifier (6%) | .590 | 2.689 | 1.055 | .480 | .314 | ||
| Faces - Immediate | .432 | 2.430 | 5.798 | .398 | |||
| Faces - Delayed | .501 | 2.837 | 4.985 | .557 | |||
| Word Pairs - Immediate | .687 | 2.877 | 2.317 | .733 | |||
| Word Pairs - Immediate w/ Modifier (8%) | .743 | 2.676 | 6.407 | .761 | −.419 | ||
| Word Pairs - Learning | .828 | 2.404 | 2.106 | .825 | |||
| Word Pairs - Total | .821 | 2.512 | 1.700 | .866 | |||
| Word Pairs - Long Delay | .607 | 3.074 | 4.222 | .594 | |||
| Word Pairs - Delayed Recognition | .701 | 2.753 | 3.373 | .652 | |||
| Numbers – Total Score | .794 | 2.147 | 1.448 | .769 | |||
| Sequences – Total Score | .851 | 1.887 | 1.354 | .856 | |||
| Family Pictures - Immediate | .527 | 2.485 | 5.475 | .481 | |||
| Family Pictures - Delayed | .591 | 2.581 | 4.337 | .560 | |||
| Word List - Learning | .594 | 2.879 | 3.521 | .468 | |||
| Word List - Delayed | .553 | 2.929 | 2.988 | .577 | |||
| Language | |||||||
| Expressive One Word Picture Vocabulary | |||||||
| .851 | 9.316 | 11.825 | .885 | ||||
| Visuospatial | |||||||
| Visual-Motor Integration | .701 | 9.419 | 17.028 | .789 | |||
| Executive Functioning | |||||||
| Wisconsin Card Sorting Test | |||||||
| Perseverative Responses | .487 | 14.170 | 52.027 | .516 | |||
| Delis-Kaplan Executive Functioning System | |||||||
| Tower Test | |||||||
| Total Achievement | .669 | 2.279 | 4.564 | .678 | |||
| Total Achievement w/ Modifier (9% & 12%) | .812 | 1.846 | −1.160 | .797 | .304 | .228 | |
| Move Accuracy Ratio | .661 | 1.784 | 5.307 | .488 | |||
| Verbal Fluency | |||||||
| Letter Fluency | .748 | 2.258 | 1.578 | .874 | |||
| Category Fluency | .640 | 2.664 | 3.239 | .703 | |||
| Category Switching Correct | .474 | 2.495 | 5.311 | .435 | |||
| Category Switching Accuracy | .451 | 1.958 | 6.568 | .348 | |||
| Trails | |||||||
| Letter Sequencing | .618 | 3.061 | 2.559 | .695 | |||
| Number-Letter Switching | .761 | 2.630 | 2.816 | .746 | |||
| Academic | |||||||
| Woodcock Johnson – Third Edition | |||||||
| Broad Reading | .953 | 5.807 | 7.742 | .927 | |||
| Broad Reading w/ Modifier (1%) | .957 | 5.600 | −.981 | .945 | .656 | ||
| Broad Math | .864 | 7.907 | 16.242 | .818 | |||
| Broad Math w/ Modifier (3%) | .880 | 7.551 | 27.636 | .807 | −.998 | ||
| Broad Written Language | .912 | 7.122 | 15.850 | .840 | |||
| Academic Fluency | .899 | 8.639 | 9.308 | .923 | |||
| Letter-Word Identification | .904 | 6.588 | 14.424 | .835 | |||
| Reading Fluency | .903 | .811 | 3.366 | .993 | |||
| Calculation | .733 | 10.313 | 30.876 | .652 | |||
| Calculation w/ Modifier (4.1%) | .761 | 9.962 | 44.585 | .639 | −1.183 | ||
| Math Fluency | .840 | 10.266 | 8.552 | .912 | |||
| Spelling | .923 | 7.042 | 11.014 | .889 | |||
| Writing Fluency | .798 | 9.940 | 24.469 | .738 | |||
| Passage Comprehension | .873 | 6.993 | 17.549 | .791 | |||
| Applied Problems | .830 | .681 | 18.958 | .793 | |||
| Writing Samples | .706 | 11.163 | 29.876 | .708 | |||
| Word Attack | .854 | 5.924 | 26.723 | .707 | |||
| Punctuation and Capitals | .713 | 14.938 | 23.861 | .725 | |||
Results are only reported for cognitive measures with test-retest reliabilities above .50. Numbers in parentheses after equations with modifiers represent the percent of variance accounted for by that particular modifier. R = reliability coefficient; SEest = standard error of the estimate; C = constant; Bbaseline = unstandardized beta (slope) for baseline test score; Bage = age at baseline testing; Bonset = age at recurrent seizure onset; Btestinterval = interval between baseline and repeat testing measured in months.
Case Example
To demonstrate the utility of reliable change indices and standardized regression-based change scores in clinical practice, a brief case example follows. At the time of his preoperative neuropsychological evaluation, the patient was 13 years-old and in the 8th grade. He began experiencing staring spells at age 11, which were subsequently diagnosed as seizures. Video-EEG monitoring showed interictal sharp waves and spikes in the left posterior temporal-occipital regions. Brain MRI revealed a focal area of abnormal morphology and signal intensity in the left temporal lobe posteriorly and inferiorly that involved the fusiform gyrus, parahippocampal gyrus, and medial aspect of the inferior temporal gyrus. PET studies revealed hypometabolism in the left posterior basal temporal, temporal occipital junction, and posterior hippocampus. The patient underwent a left temporal lobectomy approximately three months after his preoperative neuropsychological assessment. The resection extended posteriorly, inferiorly, and mesially to include the lesion visualized on MRI. Pathology was suggestive of low grade glial/glioneuronal neoplasm (WHO grade I/II). The patient was seizure free at the time of his last clinical follow-up, which was two years following his resection.
The patient completed a postoperative neuropsychological assessment approximately 6 months following surgery. A summary of a small subset of his test-retest scores is provided in Table 5 for illustrative purposes. Examination of his change scores against the 90% confidence interval RCIs provided for older children in Table 3 reveals a significant decline in Verbal Memory. In contrast, a significant improvement was apparent on the Visual Motor Integration Test. All other test scores remained unchanged per RCIs. That is, neither the two Standard Score point decline in the Verbal Comprehension Index nor the three Standard Score point improvement in Visual Delayed Memory exceeded the amount of change expected based on typical practice and error in these measures.
Table 5.
Select Neuropsychological Test Results for Case Example
| Cognitive Measure | Pre-op | Post-op | Change Score | RCI Cutoffs | SRB Predicted Post-op Score | SRB Difference as z-score* |
|---|---|---|---|---|---|---|
| WISC- IV | ||||||
| Verbal Comprehension Index | 81 | 79 | −2 | −16, 12 | 82 | −0.38 |
| Perceptual Reasoning Index | 98 | 98 | 0 | −14, 17 | 99 | −0.84 |
| Processing Speed Index | 103 | 94 | −9 | −16, 17 | 98 | −0.40 |
| Children’s Memory Scale | ||||||
| Verbal Delayed | 94 | 77 | −17 | −14, 30 | 99 | −1.72 |
| Visual Delayed | 109 | 112 | +3 | −10, 33 | 111 | 0.08 |
| EOWPVT | 70 | 70 | 0 | −13, 15 | 74 | −0.41 |
| Visual Motor Integration | 67 | 91 | +24 | −16, 16 | 70 | 2.24 |
z-scores exceeding ±1.28 are significant with an 80% confidence interval and those exceeding ±1.64 are significant with a 90% confidence interval
When using SRBs to assess cognitive change on these measures, similar results emerged. Specifically, if we use the SRB equation for Verbal Delayed Memory provided in Table 4, this patient’s predicted postoperative score on this index is calculated as follows: Predicted Postoperative Score = Constant + (T1 Score* Bbaseline) = 23.078 + (94*.806) = 98.84. The difference between his actual postoperative score and his predicted postoperative score can then be calculated and translated into a z-score as follows: z-score = (Actual Score − Predicted Score) / SEest = (77 − 99) / 12.76 = −1.72. This z-score exceeds the 90% confidence interval (i.e., ± 1.64) for identifying statistically significant and clinically meaningful change, providing further support that this patient experienced a meaningful decline in Verbal Delayed Memory following his left temporal lobe resection. SRBs have the added advantage of transforming all of the test scores in the battery to the same metric (i.e., z-scores) allowing for a simple and direct comparison of the magnitude and direction of changes observed across the measures of the battery as depicted in Figure 1.
Figure 1.
Plot of the z-scores demonstrating magnitude of postoperative cognitive change on each of the cognitive measures for the child in the case example. Note that bars exceeding the red lines (z±1.28 for 80% CI or z±1.64 for 90% CI) reflect those postoperative measures for which the change scores reflect significant and meaningful cognitive change.
These calculations have been programmed into a Microsoft Excel spreadsheet to enable clinicians to effortlessly determine meaningful change in scores following pediatric epilepsy surgery. This document requires entering some basic information about the patient (age, age at onset, test-retest interval) and the standardized pre and postoperative test scores. It then compares patients’ change scores to the appropriate RCI intervals and calculates SRBs for each test at both 80% and 90% confidence intervals. This excel document is available from the corresponding author upon request.
4.1 Discussion
This research provides clinicians with the tools necessary to objectively evaluate change in cognitive functioning in their pediatric epilepsy patients. It includes a common metric that researchers can use in future studies to more accurately characterize cognitive outcomes following epilepsy surgery in children. Importantly, the methods for assessing cognitive change provided here allow differentiation between changes in cognition due to epilepsy versus epilepsy surgery, which cannot be accomplished using traditional methods (e.g., change scores). Although the primary goal of this study was to develop RCIs and SRBs for use in evaluating cognitive change in children following epilepsy surgery, it is important to note that these methods can easily be applied to assess change in children with epilepsy in other circumstances such as after a medical event (e.g., seizure cluster) or treatment intervention (e.g., medication change).
We have provided three different metrics for assessing cognitive change – RCIs, SRBs with modifiers, SRBs without modifiers – so that clinicians and researchers can choose the method that best suits their needs. RCI methodology calculates the degree of individual change associated with test imprecision and practice effects and identifies the amount of test-retest change necessary to conclude that clinical change has occurred independent of measurement error. Because RCIs provide cut-off scores to identify meaningful change, they require no additional calculation beyond test-retest difference scores. This permits quick and easy application to patients’ test results. However, RCIs do not correct for regression to the mean or other potential modifiers and, in the case of our study, are based on smaller samples than the SRBs because they were calculated for two separate age groups.
While SRB methodology is more complicated to use, it corrects for multiple confounding factors that RCIs do not. Statistically, SRBs correct for practice by using an individual’s baseline score as a predictor of postoperative performance. This provides more accurate adjustment of practice effects than RCIs because practice can be estimated differently at different levels of baseline performance. SRBs also allow for correction of demographic and disease-related variables that could potentially impact cognitive performance over time. Finally, SRBs convert changes in test scores to a common metric (i.e., z-scores) permitting direct comparison of cognitive change across a wide range of neuropsychological measures.
Studies that have compared RCI and SRB methodologies head-to-head suggest that predictive accuracy is similar for both measures [15, 16]. This has led many clinicians to utilize the easily employed RCI cutoffs rather than calculating SRBs for individual patients. Nevertheless, we have created a Microsoft Excel calculator that calculates SRBs to facilitate the interpretation of cognitive change in individual patients for clinicians and researchers who prefer this more rigorous methodology derived with a larger sample of children. This calculator is available from the corresponding author upon request.
One interesting and unanticipated finding in our study was the negative practice effects observed on a number of cognitive measures. Rather than showing the typical practice effects demonstrated by healthy children, children in our epilepsy sample achieved lower test scores during repeat testing on some cognitive measures. Interestingly, Hermann and colleagues [17] also observed a lack of typical practice effects in adults with temporal lobe epilepsy compared to controls. In our study, this may indicate that children with epilepsy are not developing along the expected trajectory. All of the measures in the neuropsychological battery are age-normed; therefore, if children with epilepsy are not gaining skills at a rate comparable to healthy standardization samples, their scores on these measures will decline over time. Alternatively or additionally, there may be some potential negative effects associated with having taken these tests previously (e.g., overconfidence, trying to remember rather than figure out answers, using a new strategy that is less ideal). While the reason for negative practice effects needs to be investigated, this was typical for some measures like those evaluating working memory, processing speed and delayed recognition amongst the younger age group and those assessing academic achievement regardless of age. This was accounted for in our development of RCIs and SRBs by centering intervals around typical changes in scores, regardless of whether the practice effects were positive or negative.
It is also interesting to note that some of the measures in our neuropsychological battery had very poor test-retest reliability (<.50) in this pediatric epilepsy sample. For example, Dot Locations from the Children’s Memory Scale had test-retest reliability coefficients ranging from .14–.21 and Word List Delayed Recognition had a test-retest reliability coefficient of .44. Other measures in the battery with low test-retest reliability included subscores from the Wisconsin Card Sorting Test (categories, failure to maintain set, total errors), the Delis-Kaplan Executive Function System (Trail Making – visual scanning, number sequencing, motor speed, total errors; Verbal Fluency – set loss errors, repetition errors; Tower Test – rule violation/item ratio), and Test of Visuoperceptual Skills (Visual Discrimination, Visual Memory, Visual Spatial Relationships). Restricted score range (e.g., WCST number of categories) may have contributed to this for some measures. Regardless, the very low test-retest reliability of these measures in this sample raises the question of whether these measures should be used in the repeated neuropsychological assessment of pediatric epilepsy patients. Given the poor psychometric properties in this sample, RCIs and SRBs for these measures are not reported.
Several limitations of the current study deserve discussion. First, the ideal control group for determining meaningful cognitive change in children who undergo epilepsy surgery would be a group of surgical children tested twice prior to surgery, as this group would most closely approximate the demographic and seizure characteristics of children who undergo epilepsy surgery. In the United States, including our Epilepsy Center, children generally proceed to surgery as soon as they are deemed suitable candidates. It is neither feasible, nor ethical, to delay epilepsy surgery in order to obtain neuropsychological testing on two occasions 12 months apart. While we were able to obtain a small subset of surgical patients who were tested twice prior to surgery due to surgical delays or other factors, most of the patients in our control group had a confirmed diagnosis of epilepsy for at least one year but were not surgical candidates. Second, our study did not include any children under the age of 6. Therefore, our results should not be applied to younger children. Only children aged 6 to16 were included primarily to enable the use of the same neuropsychological battery with age-appropriate normative data for all children. The neuropsychological measures for children younger than 6 and older than 16 are different and use different normative groups. Third, to control for age-specific differences in test items and developmental factors, we stratified patients into younger (ages 6–10) and older (ages 11–16) age groups before developing the RCIs. This resulted in reduced samples sizes (n=36 and 27, respectively) for the RCI calculations compared to SRB calculations (n=63) with relatively small numbers of patients with different seizure foci. Finally, despite attempts to have a rather consistent 9 month test-retest interval for all children, there was some variability in the length between baseline and repeat neuropsychological testing due to limited availability of the families to return for testing and/or other scheduling limitations. While not ideal, this is likely to reflect clinical practice as test-retest intervals often vary among our surgical patients as well. Further, the vast majority of patients (80%) in our sample completed both assessments within a 5–12 month interval.
5.1 Conclusions
The establishment of RCIs and SRBs for use in evaluating cognitive change in children with epilepsy is an important step towards improving their care. These methods allow neuropsychologists to more accurately assess cognitive change after surgery or to evaluate other treatment interventions while taking into account the effects of epilepsy on the developing brain. These methodologies also permit researchers to compare cognitive outcomes across studies in order to further our understanding of the “true” cognitive changes associated with epilepsy surgery in children. Finally, use of these methodologies can aid clinicians in advising parents and physicians about the potential cognitive outcomes of epilepsy surgery on an individual level and may result in more targeted and relevant cognitive and school interventions for children with epilepsy regardless of whether or not they undergo surgery.
Highlights.
RCIs and SRBs were developed to evaluate cognitive change in children with epilepsy
These methods control for test-retest reliability, practice effects, error, and clinical factors
These methods allow for accurate assessment of cognitive change related to treatments
Tables for calculating RCIs and SRBs are provided for clinical use
Acknowledgments
This project was supported by the Epilepsy Foundation through the Partnership for Pediatric Epilepsy Research (Award ID 189351 and 122693) and the Targeted Research Initiative for Youth (Award ID 246507) (to J.S.H and R.M.B.).
Additional support for this research was provided by the Cleveland Clinic Epilepsy Center and the Clinical and Translational Science Collaborative of Cleveland (KL2TR000440) from the National Center for Advancing Translational Sciences (NCATS) component of the National Institutes of Health and NIH roadmap for Medical Research (to R.M.B.).
Footnotes
Conflict of Interest
None of the authors has any conflicts of interest to disclose.
The contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Contributor Information
Robyn M. Busch, Email: buschr@ccf.org.
Tara T. Lineweaver, Email: tlinewea@butler.edu.
Lisa Ferguson, Email: fergusl3@ccf.org.
Jennifer S. Haut, Email: hautj@ccf.org.
References
- 1.Hermann BP, Seidenberg M, Schoenfeld J, Peterson J, Leveroni C, Wyler AR. Empirical techniques for determining the reliability, magnitude, and pattern of neuropsychological change after epilepsy surgery. Epilepsia. 1996;37:942–950. doi: 10.1111/j.1528-1157.1996.tb00531.x. [DOI] [PubMed] [Google Scholar]
- 2.McSweeney J, Naugle RI, Chelune GJ, Luders HO. “T Scores for change”: an illustration of a regression approach to depicting in clinical neuropsychology. The Clinical Neuropsychologist. 1993;7:300–312. [Google Scholar]
- 3.Sawrie SM, Chelune GJ, Naugle RI, Luders HO. Empirical methods for assessing meaningful neuoropsychological change following epilepsy surgery. Journal of the International Neuropsychological Society. 1996;2:556–564. doi: 10.1017/s1355617700001739. [DOI] [PubMed] [Google Scholar]
- 4.Martin R, Sawrie S, Gilliam F, Mackey M, Faught E, Knowlton R, Kuzniekcy R. Determining reliable cognitive change after epilepsy surgery: development of reliable change indices and standardized regression-based change norms for the WMS-III and WAIS-III. Epilepsia. 2002;43:1551–8. doi: 10.1046/j.1528-1157.2002.23602.x. [DOI] [PubMed] [Google Scholar]
- 5.Wechsler D. Wechsler Intelligence Scale for Children. 4. San Antonio: The Psychological Corporation; 2003. [Google Scholar]
- 6.Cohen MJ. Children’s Memory Scale. San Antonio: The Psychological Corporation; 1997. [Google Scholar]
- 7.Gardner M. Expressive One-Word Picture Vocabulary Test - Revised. Novato, CA: 1990. [Google Scholar]
- 8.Beery KE, Beery NA. The Beery-Buktenica Developmental Test of Visual-Motor Integration. Minneapolis: NCS Pearson, Inc; 2004. [Google Scholar]
- 9.Martin NA. Test of Visual Perceptual Skills. 3. Novato, CA: Academic Therapy Publications; 2006. [Google Scholar]
- 10.Heaton RK, Chelune GJ, Talley JL, Kay GG, Curtiss G. Wisconsin Card Sorting Test. Psychological Assessment Resources, Inc; 1993. [Google Scholar]
- 11.Delis DC, Kaplan E, Kramer JH. Delis-Kaplan Executive Function System. San Antonio, TX: The Psychological Corporation; 2001. [Google Scholar]
- 12.Woodcock RW, McGrew KS, Mather N. Woodcock-Johnson III Tests of Achievement. Itasca, IL: Riverside Publishing; 2001. [Google Scholar]
- 13.Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol. 1991;59:12–9. doi: 10.1037//0022-006x.59.1.12. [DOI] [PubMed] [Google Scholar]
- 14.Chelune GJ, Naugle RI, Luders H, Sedlak J, Awad IA. Individual change after epilepsy surgery: Practice effects and base-rate information. Neuropsychology. 1993;7:41–52. [Google Scholar]
- 15.Heaton RK, Temkin N, Dikmen S, Avitable N, Taylor MJ, Marcotte TD, Grant I. Detecting change: a comparison of three neuropsychological methods, using normal and clinical samples. Archives of Clinical Neuropsychology. 2001;16:75–91. [PubMed] [Google Scholar]
- 16.Temkin NR, Heaton RK, Grant I, Dikmen S. Detecting significant change in neuropsychological test performance: a comparison of four models. Journal of the International Neuropsychological Society. 1999;5:357–369. doi: 10.1017/s1355617799544068. [DOI] [PubMed] [Google Scholar]
- 17.Hermann BP, Seidenberg M, Dow C, Jones J, Rutecki P, Bhattacharya A, Bell B. Cognitive prognosis in chronic temporal lobe epilepsy. Ann Neurol. 2006;60:80–87. doi: 10.1002/ana.20872. [DOI] [PubMed] [Google Scholar]