Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2026 Apr 26.
Published in final edited form as: Epilepsia. 2025 Apr 26;66(8):2916–2929. doi: 10.1111/epi.18421

Cognitive Screening Informs Referrals for Neuropsychological Evaluation in Children with Epilepsy

Kristina E Patrick 1,2,3, Allison N Shields 2, Holly A Dustin 1,4, Anup D Patel 5,6, Kelly McNally 1,6
PMCID: PMC12353113  NIHMSID: NIHMS2074772  PMID: 40286280

Abstract

Objective:

This study aimed to evaluate the effectiveness of current neuropsychology referral methods for children with epilepsy and develop data-informed recommendations for use of performance-based cognitive screening measures to improve these processes.

Methods:

Children with epilepsy who had been referred to neuropsychology (n=51) or had never been referred (n=34) completed four brief tablet-based screening tests from the NIH Cognitive Toolbox along with a comprehensive neuropsychological test battery. Demographics, medical information, and parent questionnaires were gathered.

Results:

Mean performance on the neuropsychological test battery was worse in the referral group (p=.008, d=.52), but percentage of patients who presented with cognitive impairment (at least two scores 1.5 standard deviations below the mean) did not differ. Demographics did not predict performance on the comprehensive neurocognitive battery (p=.46, R2change =.06). Medical variables added some predictive value (p =.004, R2change =.25). Parent questionnaires added minimal value (p=.066, R2change =.05) beyond the previous variables. Performance on the cognitive screening battery added significant predictive value (p<.001, R2change =.31) above demographics, medical variables, and parent questionnaires, explaining 31% additional variance in performance on the comprehensive neuropsychological battery. Stepwise analysis suggested that only three screening tests, totaling 15-minutes of administration time, were necessary. A cutoff score of .70 standard deviations below the mean on any of those screening tests had high sensitivity (.90) while maintaining specificity above .50. A cutoff score of 1 standard deviation below the mean provided better balance of sensitivity (.74) and specificity (.70).

Significance:

Brief and easy to administer performance-based cognitive screening may add value and reduce bias when making decisions about neuropsychology referrals for children with epilepsy. An ideal clinical model could include neuropsychology consultation with chart review, clinical interview, questionnaires, and brief cognitive screening to inform referrals for more comprehensive evaluation. In settings where this is not possible, cognitive screening may be a useful and minimally resource intensive method for informing referral decisions.

Introduction

Given risks of neurodevelopmental, emotional-behavioral, and learning difficulties among children with epilepsy (Dagar & Falcone, 2020; Nickels et al., 2016; Plioplys, Dunn & Caplan, 2007), screening for neuropsychological deficits is recommended as an epilepsy quality measure by the American Academy of Neurology (AAN; Clary et al., 2022). Furthermore, the International League Against Epilepsy ILAE) Neuropsychology Task Force recommends routine screening in children newly diagnosed with epilepsy and on an ongoing basis for identification of people who require more comprehensive neuropsychological evaluation (Wilson et al., 2015). However, practical guidance on how to effectively and efficiently implement neuropsychological screening has not been proposed. Neuropsychological evaluations are uniquely suited to inform the impact of epilepsy and its treatment on cognitive and behavioral functioning and provide important information related to medical management (Jones-Gotman et al., 2010). While it is crucial to ensure that children with epilepsy who are at risk for cognitive difficulties are referred for a neuropsychological evaluation, access to care is limited. Factors that impact access to care include the length and detail of these assessments limiting the volume of children who can be seen, difficulty accurately triaging referrals to the appropriate specialty service, and time, cost, and travel-related burden for families. It is important to find ways to determine which children are in greatest need of neuropsychological evaluations to help reduce unnecessary utilization of healthcare services, manage resources efficiently, and increase access to care. Currently, there are no empirically supported methods to differentiate children who are likely to benefit from a comprehensive neuropsychological assessment from those who are less likely to benefit, making it challenging for medical providers to appropriately refer patients in need of neuropsychological evaluations while also avoiding unnecessary referrals.

Without standard methods for referring to neuropsychology, this determination differs according to a variety of factors, often resulting in disparities among those who are seen for an evaluation. This includes lower access to care for children who are racial/ethnic minorities, have public insurance, or live in rural areas (Miller et al., 2021; Tran et al., 2023). These disparities stem in part from biases among healthcare providers who are making referrals and families’ knowledge of available resources (Geiger, 2003). Reliance on provider and parent referrals has been shown to lead to referral bias and under-identification of certain populations with neurocognitive deficits (Begeer et al., 2009; Gaub & Carlson, 1997). Objective measures are needed to systematically ensure that access to neuropsychological care is equitable.

As a first step for improving and standardizing the referral process, implementation of parent questionnaires during medical visits has shown promise in flagging patients who may benefit from a neuropsychological evaluation (Lai et al., 2015; Wagner et al., 2016). Approximately 80% of epilepsy programs report conducting these types of screenings, which range from non-standardized informal interviews to broadband parent-report measures of behavioral and emotional symptoms (Wagner et al., 2019). While these methods may provide good information about a child’s general day-to-day functioning, they do not reliably predict performance on task-based measures of cognition (Dang et al., 2020; Toplak et al., 2013) and have limited accuracy in identifying children with developmental delays, even when mediated by physician opinion (Rydz et al., 2006).

To objectively and effectively identify need for neuropsychological evaluation, parent report and physician opinion may need to be supplemented with performance-based screening measures. There has been a recent push for a stepped model of neuropsychological care, in which children at risk for cognitive difficulties undergo regular cognitive screenings at medical visits as part of universal monitoring (Hardy et al., 2017), but this is far from standard practice at present. Research has demonstrated the utility of tablet- or computer-based brief cognitive screening tasks in identifying impairment in pediatric patients with various neurologic disorders (Brooks & Sherman, 2012), cancer (Boulet-Craig et al., 2018), congenital heart disease (Pike et al., 2017), concussion (Meehan et al., 2012), multiple sclerosis (Charvet et al., 2018), and systemic lupus erythematosus (Vega-Fernandez et al., 2015), although many studies examining these research questions do not present adequate psychometric analyses or methodological reporting standards to allow for ease of clinical implementation (Bryce et al., 2021).

We are aware of one study to date that has examined the relationship between tablet-based cognitive screening and performance on a comprehensive neuropsychological test battery in pediatric epilepsy. In a sample of 42 children with epilepsy, Matuska and colleagues (2024) found weak to moderate relationships between the Flanker and Pattern Comparison NIH Toolbox tests, which screen executive function, and performance on several commonly administered standardized neuropsychological tests. Using a cutoff standard score of 85 (1 standard deviation below the mean), overall agreement for classifying impairment in various neuropsychological tests was moderate to substantial, ranging from 0.52 to 0.74. Patients who were not impaired on Toolbox tasks were unlikely to be impaired on clinical measures (negative predictive values ranged from 67% to 100%).

The present study aimed to expand on prior research by developing practical methods for improving the neuropsychology referral process in children with epilepsy. Our first aim was to assess whether the current referral process at a large pediatric medical center was effective in differentiating children in need of a neuropsychological evaluation from those who did not require comprehensive testing. Our sample includes patients who were seen by a neurologist as part of their epilepsy care; some were clinically referred for a comprehensive neuropsychological evaluation and some were not referred based on clinical judgement. We examined differences in neuropsychological test performance between these two groups. Next, to determine factors that may predict neuropsychology referral necessity, we examined whether patient demographics, medical information, parent questionnaires, and scores on performance-based neuropsychological screening measures were associated with neuropsychological test performance. The second aim was to develop preliminary standards for interpreting performance on screening measures in a clinical setting to inform referral decisions. We examined the predictive value of neuropsychological screening subtests on comprehensive neuropsychological test performance to identify the most efficient screening battery that maintained strong predictive value. We also computed cutoff scores for screening performance that can be used to guide clinical practice.

Materials & Methods

Participants

Participants included 85 children 8–16 years of age who were diagnosed with epilepsy and currently being treated with anti-seizure medication. Children with known intellectual disability were excluded because neurocognitive screening would not be necessary to inform referral decisions for these children. In addition, children with severe motor impairment or other disabilities that preclude participation in neuropsychological testing were excluded. Demographic information is presented in Table 1.

Table 1.

Demographic characteristics by referral group

Full Sample
N = 85		 Clinically Referred
N = 51		 Non-Clinically Referred
N = 34
Age: Mean in Years (Range)* 11.38 (8.0–16.8) 10.90 (8.10–16.50) 12.10 (8.0–16.8)
Gender: N (%)*
 Male 33 (38.82%) 25 (49.02%) 8 (23.53%)
 Female 52 (61.18%) 26 (50.98%) 26 (76.47%)
Race: N (%)
 Asian 1 (1.18%) 1 (1.96%) 0 (0.00%)
 Black 11 (12.94%) 5 (9.80%) 6 (17.65%)
 Multi-Racial 5 (5.88%) 2 (3.92%) 3 (8.82%)
 White 68 (80.00%) 43 (84.31%) 25 (73.53%)
Maternal Education: N (%)
 Some High School 5 (5.88%) 1 (1.96%) 4 (11.76%)
 High School Graduate/GED 9 (10.59%) 6 (11.76%) 3 (8.82%)
 Some College 27 (31.76%) 15 (29.41%) 12 (35.29%)
 College Graduate 22 (25.88%) 13 (25.49%) 9 (26.47%)
 Graduate School 16 (18.82%) 10 (19.61%) 6 (17.65%)
 Not reported 6 (7.06%) 6 (11.76%) 0 (0.00%)
Insurance: N (%)
 Public 32 (37.65%) 21 (41.18%) 11 (32.35%)
 Private 53 (62.35%) 30 (58.52%) 23 (67.65%)
Open in a new tab

Note. Asterisks indicate significant differences between the clinically referred and non-referred groups,

*

p < .05

**

p < .01

***

p < .001.

Non-Referral Group:

Thirty-four participants were patients who had been evaluated by a neurologist or nurse practitioner at the hospital’s outpatient epilepsy center and had never been referred for or completed a neuropsychological evaluation. Charts of patients scheduled in the epilepsy center were reviewed weekly and patients were contacted by a research assistant to screen for eligibility and interest in the research study. All patients who met criteria for inclusion were contacted until the enrollment goal was met. One additional patient was originally in the non-referral group but was clinically referred for a neuropsychological evaluation prior to participation in the study and therefore moved to the referral group.

Referral Group:

Fifty-one participants were patients who were completing a comprehensive neuropsychological evaluation following a clinical referral from neurology. Charts of patients scheduled for comprehensive neuropsychological evaluation were reviewed weekly and eligible patients were approached about the study during their neuropsychology testing appointment.

Procedure

The study was approved by the hospital’s institutional review board (IRB). Electronic medical records were reviewed for demographic and medical variables. Parents completed informed consent and participants provided assent. Participants in the non-referred group came into the hospital’s research center. Following informed consent, parents completed questionnaires. Patients completed subtests selected from the NIH Toolbox Cognition Battery on a tablet prior to completing the comprehensive neuropsychological testing battery. See Table 2 for all measures administered. Families were compensated with a $50 gift card and provided with a brief feedback letter. Patients in the referred group completed their standard of care clinical neuropsychological evaluation appointment. The neuropsychological testing battery included tests selected by the patient’s neuropsychologist. These tests were typically consistent with the comprehensive battery given to non-referred participants but sometimes differed slightly based on clinical need. When a different but comparable test was given (e.g., Wechsler Individual Achievement Test, Fourth Edition Word Reading subtest in place of WRAT-4 Word Reading subtest), the score from the administered test was used for analyses. Following informed consent, patients and parents were administered the study questionnaires and NIH Toolbox subtests. Families were compensated with a $10 gift card for study participation.

Table 2.

Measures administered and functions assessed

Measures Functions Assessed
Questionnaires
Strengths and Difficulties Questionnaire (SDQ) 1 Parent-reported emotional, behavioral, attention, and social problems
Pediatric Neuro-QOL System Measure of Cognitive Function 2 Parent-reported cognitive difficulties
NIH Toolbox Cognition3 Subtests
Dimensional Change Card Sort Test (4 minutes) Attention and cognitive flexibility
Flanker Inhibitory Control and Attention Test (3–5 minutes) Attention and inhibitory control
Pattern Comparison Processing Speed Test (3 minutes) Processing speed
Picture Vocabulary Test (5–7 minutes) Receptive vocabulary
Comprehensive Neuropsychological Test Battery
Wechsler Intelligence Scale for Children, Fifth Edition (WISC-V) 4 Intelligence
Word Generation-NEPSY, Second Edition (NEPSY-II) 5 Verbal fluency
Boston Naming Test 6 Confrontation naming
Arrows, NEPSY-II Spatial orientation
California Verbal Learning Test, Children’s Edition (CVLT-C) 7 Verbal learning and memory
Memory for Faces, NEPSY-II Visual memory
Conner’s Continuous Performance Test, Second Edition (CPT-II) 8 Sustained attention and inhibition
Trail Making Test-Delis Kaplan Executive Function System (D-KEFS) 9 Attention, processing speed, cognitive flexibility
Grooved Pegboard 10 Fine motor speed and dexterity
Word Reading-Wide Range Achievement Test, Fourth Edition (WRAT-4) 11 Academic skills (reading)
Math Calculation-WRAT-4 Academic skills (math)
Behavior Rating Inventory of Executive Function, Second Edition (BRIEF-2) 12 Parent-reported executive function
Open in a new tab

Statistical Methods

Standardized neuropsychological test scores were derived from age-based normative data from each of the test manuals. No demographic (e.g., sex, race) corrections were applied. Standard, scaled, and T-scores from each neuropsychological test were transformed into z-scores using normative means and standard deviations. Next, the following scores were computed:

  • Neuropsychological Test Composite: Mean z-score across all neuropsychological tests

  • Individual Domain Composites: Mean z-score across all tests in a given domain (language, nonverbal reasoning, attention/executive function, memory, and academics)

  • Cognitive Impairment: Dichotomous variable coded as 0 or 1 with 1 indicating that the participant had at least two z-scores below −1.5 across all neuropsychological tests given in the comprehensive battery.

Statistical analyses were completed using SPSS Version 28, RStudio (RStudio Team, 2020), and the core, psych, and pROC packages of R statistical software, version 4.2.2 (R Core Team, 2022; Revelle, 2022; Robin et al., 2011).

Results

Aim 1

Comparison of Referred and Non-Referred Patients

Mean test scores for all neuropsychological measures are shown for each group in Table 3. An independent samples t-test was run to compare neuropsychological composite score between patients who were clinically referred for a neuropsychological evaluation and patients that were not referred. Clinically referred patients had a significantly lower mean z-score (M = −.61, SD = .57) compared with patients who were not referred (M = −.30, SD = .43), t(83) = 2.72, p = .008, d = .52. Across the sample, 62 patients were classified as having cognitive impairment (73%), including 39/51 (76%) in the referred group and 23/34 (68%) in the non-referred group. A chi-square analysis indicated that rate of cognitive impairment did not differ between referred and non-referred groups, χ2 = 0.42, p = .517.

Table 3.

Mean z-scores with standard deviations of individual scores on the comprehensive neuropsychological test battery

Domain Neurocognitive Functions Measure/Test Full Sample
M (SD) 		Clinically Referred
M (SD) 		Non-Referred
M (SD)
Intelligence Intellectual Functioning WISC-V Full Scale IQ −0.57 (0.87) −0.74 (0.90) −0.31 (0.78)
Language Confrontation Naming Boston Naming Test Total Score −1.10 (1.36) −1.28 (1.4) −0.82 (1.28)
Verbal Fluency, Phonemic NEPSY-II Word Generation – Initial Letter −0.07 (1.12) −0.32 (1.11) 0.31 (1.03)
Verbal Fluency, Semantic NEPSY-II Word Generation - Semantic −0.87 (1.02) −1.00 (0.91) −0.69 (1.15)
Verbal Reasoning and Lexical Knowledge WISC-V Verbal Comprehension Index −0.30 (0.80) −0.38 (0.79) −0.20 (0.81)
Nonverbal Reasoning Perceptual Organization and Visual Synthesis WISC-V Visual Spatial Index −0.41 (0.90) −0.40 (0.9) −0.42 (0.92)
Abstract Visual and Quantitative Reasoning WISC-V Fluid Reasoning Index −0.36 (1.00) −0.57 (1.03) −0.06 (0.88)
Visual Perception NEPSY-II Arrows −0.52 (1.32) −0.56 (1.33) −0.47 (1.32)
Memory Verbal Memory CVLT-C Total Learning −0.44 (1.00) −0.58 (1.01) −0.24 (0.96)
CVLT-C Long Delay Free Recall −0.38 (1.02) −0.39 (1.08) −0.37 (0.95)
CVLT-C Recognition Discriminability −0.01 (0.88) −0.02 (0.92) 0.01 (0.84)
Visual Memory NEPSY-II Memory for Faces Immediate −0.48 (1.01) −0.54 (1.15) −0.39 (0.78)
NEPSY-II Memory for Faces Delayed −0.40 (1.04) −0.35 (1.14) −0.46 (0.88)
Attention/ Executive Function
 Set-Shifting D-KEFS Number-Letter Switching Time −0.96 (1.27) −1.11 (1.32) −0.76 (1.18)
Processing Speed WISC-V Processing Speed Index −0.53 (1.07) −0.83 (1.09) −0.10 (0.88)
Immediate Attention and Working Memory WISC-V Working Memory Index −0.56 (0.97) −0.63 (0.96) −0.44 (0.99)
Sustained Attention CPT-II Omission Errors −0.73 (1.53) −0.96 (1.62) −0.38 (1.33)
Impulse Control CPT-II Commission Errors −0.22 (0.93) −0.31 (0.67) −0.08 (1.22)
Executive Function Behaviors BRIEF-2 Global Executive Composite −0.88 (1.30) −1.21 (1.26) −0.41 (1.23)
Academic Single Word Reading WRAT-4 Word Reading −0.04 (0.94) −0.09 (1.08) 0.04 (0.70)
Math Calculation WRAT-4 Math Computation −0.44 (0.95) −0.68 (0.94) −0.10 (0.86)
Open in a new tab

Note. WISC-V: Wechsler Intelligence Scale for Children, 5th Edition; BNT: Boston Naming Test; NEPSY-II: A Developmental Neuropsychological Assessment, 2nd Edition; CVLT-C: California Verbal Learning Test, Children’s Version; D-KEFS: Delis-Kaplan Executive Function System; CPT-II: Conners Continuous Performance Test, 3rd Edition; BRIEF-2: Behavior Rating Inventory of Executive Function, 2nd Edition, Parent Version; WRAT-4: Wide Range Achievement Test, Fourth Edition

Predictive Value of Demographic, Medical, Questionnaire, and Screening Variables

See Table 4 for detailed results. Pearson correlations, Spearman correlations, and independent samples t-tests were run to examine relationships between demographic, medical, questionnaire, and cognitive screening variables and the neuropsychological testing composite z-score. Age of seizure onset, duration of epilepsy, number of anti-seizure medications, and parent ratings on the Neuro QOL Cognitive were moderately correlated with neuropsychological test performance. Individual NIH Toolbox measures were also moderately correlated with neuropsychological test performance. The strongest correlation was with the Pattern Comparison Processing Speed Test and the weakest correlation was with the Flanker Inhibitory and Attention Test.

Table 4.

Demographic variables, medical variables, parent questionnaires, and NIH Toolbox test scores relationships with neuropsychological test battery composite z-score

Predictor Variable Domain Hierarchical Regression Individual Variables Mean (SD) Comparisons or Correlations
Demographic Variables F(5, 79) = .95
p = .46
R2change = .06		 Gender Female = −0.48 (0.51)
Male = −0.50 (0.61)		 t(59) = −0.12
p = .906
Race White = −0.51 (0.56)
Nonwhite = −0.41 (0.52)		 t(26) = 0.69
p = .497
Insurance Type Public = −0.61 (0.55)
Private = −0.42 (0.54)		 t(65) = −1.57
p = .121
Maternal Education Some HS = −0.35 (0.48)
HS Graduate = −0.70 (0.76)
Some college = −0.54 (0.52)
College graduate = −0.43 (0.49)
Graduate school = −0.36 (0.59)		 r = 0.10
p = .361
Family History of Learning Problems Yes = −0.59 (0.52)
No = −0.43 (0.56)		 t(70) = 1.34
p = .184
Medical Variables F(8, 71) = 3.22
p = .004
R2change = .25		 Age of Seizure Onset − r = .36
p < .001
Epilepsy Duration − r = −.35
p = .001
Number of Anti-Seizure Medications Trialed − r = −.25
p = .021
Most Recent EEG Normal = −0.60 (0.59)
Abnormal = −0.48 (0.55)		 t(13) = −0.64
p = .536
MRI Findings Normal = −0.44 (0.53)
Abnormal = −0.69 (0.60)		 t(21) = 1.48
p = .154
Seizure Type Focal = −0.45 (0.58)
Generalized = −0.47 (0.45)		 t(73) = 0.13
p = .898
Seizure Frequency Daily = −0.50 (0.12)
Weekly = −0.60 (0.68)
Monthly = −0.58 (0.56)
Yearly = −0.51 (0.59)
<1 per year = −0.34 (0.46)		 r =.15
p = .183
Current Anti-Seizure Medications (ASMs) 1 ASM = −0.41 (0.53)
2+ ASMs = −0.77 (0.52)		 t(30) = 2.66
p = .013
Parent Questionnaires F(2, 69) = 2.82
p = .066
R2change = .05		 SDQ
Total Score − r = −.16
p = .137
Emotional Difficulties − r = −.04
p = .732
Conduct Problems − r = −.12
p = .278
Hyperactivity − r = −.16
p = .153
Neuro QoL Cognitive − r = .29
p = .007
Parent Reported Cognitive Concerns Yes = −0.55 (0.56)
No = −0.45 (0.55)		 t(73) = 0.86
p = .390
NIH Toolbox Cognitive Tests F(4, 65) = 15.17
p < .001
R2change = .31		 Dimensional Change Card Sort Test r =.47
p < .001
Flanker Inhibitory Control and Attention Test r =.36
p < .001
Pattern Comparison Processing Speed Test r =.54
p < .001
Picture Vocabulary Test r =.46
p < .001
Open in a new tab

Note. HS, High School. SDQ, Strengths and Difficulties Questionnaire.

Note. Predictor variable domains were inserted in a hierarchical regression model. Results reflect the added contribution of the variable domain above and beyond the domains already inserted in the model. Neuropsychological test battery composite z-score was the outcome variable. Relationships between individual variables that were included within each domain and composite z-score are also shown.

A hierarchical regression was conducted to analyze predictors of comprehensive neuropsychological testing composite z-score. Predictor variables were inserted in a stepwise fashion: 1) demographics, 2) medical variables, 3) parent questionnaires, and 4) NIH Toolbox scores. Demographic variables did not significantly predict mean performance on the neuropsychological test battery. Addition of medical variables significantly improved the predictive value of the model. Parent questionnaires added a small amount of predictive value that was below the threshold of significance. Adding performance on the 4-subtest NIH Toolbox battery significantly improved the predictive value of the model with a large effect size, explaining 31% of the variance in performance.

Aim 2

Assessment of Unique Contributors in Screening Battery

Next, stepwise regressions were conducted to assess the predictive value of each NIH Toolbox subtest on specific cognitive domains and the overall neuropsychological composite. Mean z-score of tests falling into each domain was calculated to create language, nonverbal, memory, academic, and attention/executive composite scores. All four subtests from the NIH Toolbox battery were entered as stepwise variables to determine which subtests were uniquely associated with neuropsychological test performance. Results are shown in Table 5. Collectively, the Pattern Comparison and Picture Vocabulary subtests were the best predictors of performance on comprehensive neuropsychological testing. The Dimensional Change Card Sort subtest added predictive value for the comprehensive battery and the nonverbal domain. The Flanker Inhibitory Control subtest provided no additional predictive value.

Table 5.

Stepwise regression analyses of NIH Toolbox Cognition subtests’ associations with neuropsychological test battery composite z-score and domain composite z-scores

Cognitive Domain NIH Toolbox Tests Regression Results
Full Battery Included in Model
1-Pattern Comparison Processing Speed Test F(1, 83) = 34.17, p < .001, R2change = .29
2-Picture Vocabulary Test F(1, 82) = 31.15, p < .001, R2change = .20
3-Dimensional Change Card Sort Test F(1, 81) = 8.09, p = .006, R2change = .05
Excluded from Model
Flanker Inhibitory Control and Attention Test
Language Included in Model
1-Picture Vocabulary Test F(1, 83) = 52.09, p < .001, R2change = .39
2-Pattern Comparison Processing Speed Test F(1, 82) = 9.87, p = .002, R2change = .07
Excluded from Model
Flanker Inhibitory Control and Attention Test
Dimensional Change Card Sort Test
Nonverbal Reasoning Included in Model
1-Dimensional Change Card Sort Test F(1, 83) = 24.39, p < .001, R2change = .23
2-Pattern Comparison Processing Speed Test F(1, 82) = 7.64, p = .007, R2change = .07
Excluded from Model
Flanker Inhibitory Control and Attention Test
Picture Vocabulary Test
Memory Included in Model
1-Pattern Comparison Processing Speed Test F(1, 83) = 14.69, p < .001, R2change = .15
2- Picture Vocabulary Test F(1, 82) = 5.50, p = .021, R2change = .05
Excluded from Model
Flanker Inhibitory Control and Attention Test
Dimensional Change Card Sort Test
Attention/
Executive Function		 Included in Model
1-Pattern Comparison Processing Speed Test F(1, 83) = 25.68, p < .001, R2change = .24
2- Picture Vocabulary Test F(1, 82) = 7.22, p = .009, R2change = .06
Excluded from Model
Flanker Inhibitory Control and Attention Test
Dimensional Change Card Sort Test
Academic Included in Model
1- Picture Vocabulary Test F(1, 82) = 23.48, p < .001, R2change = .22
2-Pattern Comparison Processing Speed Test F(1, 81) = 11.20, p = .001, R2change = .09
Excluded from Model
Flanker Inhibitory Control and Attention Test
Dimensional Change Card Sort Test
Open in a new tab

Calculation of Cut-Off Scores for Referral Need Classification

Receiver operating characteristic (ROC) curves were analyzed to examine the sensitivity, specificity, and classification accuracy of various cutoff scores on NIH Toolbox tasks when predicting neuropsychological impairment. The area under the curve (AUC) was also examined, with AUC values near 0.5 indicating no discrimination, AUC values between 0.7 and 0.8 considered acceptable, values between 0.8 and 0.9 considered excellent, and values above 0.9 considered outstanding (Hosmer et al., 2013). Because Pattern Comparison, Picture Vocabulary, and Dimensional Change Card Sort were associated with cognitive domains in previous analyses, only these tasks were examined. The predictor variable was defined as the lowest score across any of these three NIH Toolbox tasks.

Sensitivity, specificity, and accuracy values for various cutoff scores are shown in Table 6. The overall area under the curve was acceptable (AUC = 0.78). True positive, true negative, false positive, and false negative values by referral group are shown in Table 6. A cutoff value of Z = −0.70 yielded .80 accuracy, maximizing sensitivity while keeping specificity above .50. Use of this score would have identified all 23 children who were not referred for a neuropsychological evaluation but performed > 1.5 standard deviations below the mean on at least two tests on the comprehensive clinical battery. A cutoff value of Z = −1.00 was best for maximizing both sensitivity and specificity. Use of this score would have missed 5 (22%) of the patients from the non-referral group who demonstrated neuropsychological impairments but would have reduced the number of children unnecessarily referred for comprehensive testing (3 false positives out of 24 patients) compared with a cutoff score of −0.70 (5 false positives out of 24 patients).

Table 6.

Sensitivity, specificity, and accuracy values for various NIH Toolbox scores

Threshold
(z-score)		 Sensitivity Specificity Accuracy True Positives True Negatives False Positives False Negatives
0 0.97 0.13 0.74 60 3 20 2
−0.1 0.95 0.26 0.76 59 6 17 3
−0.2 0.95 0.3 0.78 59 7 16 3
−0.3 0.95 0.39 0.8 59 9 14 3
−0.4 0.95 0.43 0.81 59 10 13 3
−0.5 0.95 0.43 0.81 59 10 13 3
−0.6 0.94 0.43 0.8 58 10 13 4
−0.7 0.9 0.52 0.8 56 12 11 6
−0.8 0.85 0.52 0.76 53 12 11 9
−0.9 0.76 0.65 0.73 47 15 8 15
−1.0 0.74 0.7 0.73 46 16 7 16
−1.1 0.74 0.7 0.73 46 16 7 16
−1.2 0.71 0.7 0.71 44 16 7 18
−1.3 0.66 0.74 0.68 41 17 6 21
−1.4 0.66 0.74 0.68 41 17 6 21
−1.5 0.58 0.78 0.64 36 18 5 26
−1.6 0.53 0.91 0.64 33 21 2 29
−1.7 0.44 0.91 0.56 27 21 2 35
−1.8 0.44 0.91 0.56 27 21 2 35
−1.9 0.35 0.91 0.51 22 21 2 40
−2.0 0.34 0.91 0.49 21 21 2 41
Open in a new tab

Discussion

This study aimed to assess the effectiveness of current referral methods at a large pediatric medical center and evaluate whether brief performance-based neurocognitive screening might improve those processes. Comparisons of overall performance on a comprehensive neuropsychological test battery between pediatric patients with epilepsy who had been referred for a neuropsychological evaluation and those who had not been referred showed lower neurocognitive scores in the referred group. However, the percentage of patients who had two or more impaired scores (z-scores below −1.5) did not differ between groups. This suggests that referring providers are identifying and appropriately referring many patients for neuropsychological evaluations; however, many patients likely to benefit from an evaluation are being missed. Of note, neuropsychology is highly integrated into the epilepsy program at the center where this study took place, which may improve effectiveness of referral practices. Thus, these results may overestimate how many epilepsy patients are appropriately being referred to neuropsychology at most pediatric medical centers.

Specific factors referring providers are considering when determining whether to make a referral likely varies a great deal by provider. Such factors could include demographic information, medical variables, parent-reported concerns, and developmental/educational test scores. Demographic variables, particularly those associated with social determinants of health, are related to cognitive functioning in several pediatric populations (Xiao et al., 2023). Interestingly, in our study, demographic variables were not associated with performance on neurocognitive testing. Given the high prevalence of cognitive dysfunction in children with epilepsy (Kim & Ko, 2016; Wilson et al., 2015), demographic information may not be as informative in this population. That said, some relationships may have been found with a more thorough analysis of factors related to specific social determinants.

Consistent with prior research (e.g., Lordo et al., 2017), medical variables associated with epilepsy severity, such as epilepsy duration, seizure frequency, seizure type, and abnormalities on EEG or imaging, were predictive of neurocognitive test performance. This is not surprising given that greater epilepsy severity is associated with greater neurological impact. In this study, medical variables explained 25% of the variance in performance on neurocognitive testing above the impact of demographic variables. Given this large effect size, medical variables should be strongly considered when deciding whether to make a referral for neuropsychological testing. Providers may consider referring patients to neuropsychology when they have neurological abnormalities or poorly controlled seizures even in the absence of reported cognitive concerns.

Interestingly, parent reported cognitive, attention, and social-emotional symptoms on questionnaires were not strongly related to neurocognitive test performance. Parent concerns can certainly be beneficial for informing referrals. However, there is a great deal of variability in reporting patterns, and beliefs about what constitutes “normal” can impact parents’ responses. Parents’ responses are likely to be impacted by many different factors such as their level of knowledge about the relationship between epilepsy and cognition, the frequency and degree of communication with school, comparison to other children such as siblings, and knowledge about what to look for. Although parent concerns should be taken seriously, lack of parent concern may not necessarily mean that there are no cognitive problems. Furthermore, targeted clinical parent interview questions may be more informative in determining the need for a neuropsychological evaluation than responses on standardized questionnaires.

The ILAE Neuropsychology Task Force suggests use of computerized assessment measures in conjunction with questionnaires and clinical interview as part of routine screening for cognitive problems in individuals with epilepsy (Wilson et al., 2015). Computerized cognitive screening assessments are becoming more common in clinic settings but are not yet widely utilized as part of the referral decision making process. Matuska and colleagues (2024) found initial evidence of validity of two tests from the NIH Toolbox for predicting performance on clinical neuropsychological tests. Our findings expand on this, suggesting that use of a very brief and easily administered cognitive battery is highly informative in determining the need for a referral to neuropsychology. A 15-minute battery of tests from the NIH Toolbox explained an additional 31% of the variance in neurocognitive test performance beyond the contributions of demographic variables, medical variables, and parent report. The tests that were most beneficial assessed receptive vocabulary, processing speed, and cognitive flexibility. Receptive vocabulary may be tapping into verbal skills more broadly, which are frequently impacted in pediatric epilepsy (Baumer et al., 2019). Executive functions such as processing speed and cognitive flexibility are highly sensitive to impact of neurological abnormalities and common areas of difficulty in children with epilepsy. It is therefore not surprising that tests assessing these domains are sensitive to broader neurocognitive performance.

Our findings demonstrate that brief cognitive screening is an informative part of the neuropsychology referral process. However, in order for these tools to be clinically useful, clinicians need guidelines on cutoff scores that can be used to prompt a referral. Using a standard score of < 90 on Pattern Comparison, Picture Vocabulary, or Dimensional Change Card Sorting as a cutoff, we captured all children in our sample who had not been referred for a comprehensive evaluation but who demonstrated at least two impaired test scores on the comprehensive clinical battery. However, some clinicians may be reluctant to use a cutoff score so close to the average range due to the risk for false positives. Indeed, specificity with this cutoff was only .52. This cutoff score may be best for situations where in-clinic neuropsychology resources are limited, referring providers want to ensure that they do not miss children at risk of cognitive problems, and there is low risk for referring children unnecessarily such as at institutions where neuropsychology services are quickly and readily available. For instance, a medical assistant or nurse could administer this battery during a neurology clinic visit and prompt an automatic referral if the patient scores below the cutoff on any of the three measures. This would mean very few patients with cognitive problems would be missed and patients would likely be seen by neuropsychology more quickly than they would if consultation with neuropsychology or further clinical assessment was required prior to the referral. The downside of this process is that many patients referred to neuropsychology may not actually have any cognitive problems, meaning they would be less likely to benefit from a comprehensive neuropsychological evaluation. Furthermore, a liberal referral process like this would further contribute to lengthy wait times for neuropsychology services for patients who do need to be seen. For a better balance of sensitivity and specificity, a standard score of < 85 may be used. Given the increased rate of false negatives with this value, it may be best to use when combined with other assessment procedures that could also prompt a neuropsychology referral. An ideal model likely includes screening with a neuropsychologist in multi-disciplinary clinic settings using several tools such as clinical interview, consideration of medical and family factors, targeted parent questionnaires, and computer-based cognitive screening (Wilson et al., 2015). With the right tools and targeted interviews, these tasks can be completed in an hour-long clinic visit, which allows for an efficient method for informed decision-making regarding comprehensive neuropsychology referrals along with opportunity for consultative family support.

This study provides preliminary evidence for utility of adding performance-based cognitive screening to current neuropsychology referral procedures for patients with epilepsy. However, interpretation of results and generalizability are limited by the small sample size and limited selection of assessment measures utilized. While our sample size was relatively small, power analyses for ROC curves indicated sufficient power to detect an AUC of .78 with even a much smaller sample size of N = 32. It is important to highlight that 73% of our sample was identified as having cognitive difficulties based on our conceptualization of impairment (at least 2 scores with z <= −1.5 across the neuropsychological assessment battery). This is a higher prevalence of cognitive impairment than has been found in population-based studies of children with epilepsy (Berg et al., 2008). Our threshold was chosen intentionally to promote sensitivity, but overrepresentation of children in the “impaired” group means that the cutoff values for ROC analyses should be interpreted as identifying children with a broader range of cognitive weaknesses, rather than cognitive impairment per se.

In addition, a limited number of tests were selected both from the NIH Toolbox Cognition battery as screening measures and for the comprehensive battery. This was necessary due to time and resource limitations for a research study, but as a result, useful neuropsychological tests may have been missed. The screening battery had the lowest predictive value for the memory domain of the comprehensive battery. Therefore, it may be useful to add a memory screening measure such as the Picture Memory Test from the NIH Toolbox. In addition, while our comprehensive battery did include standard neuropsychological tests often given to children with epilepsy, it is shorter than the batteries that many pediatric neuropsychologists give, particularly in regard to measures of memory and executive function. It is therefore possible that additional deficits would have been identified with a more comprehensive battery. Studies that include larger samples and a more comprehensive test battery may be useful for identifying more reliable screening tests and cut-off scores.

Although the screening battery was highly associated with mean performance on the comprehensive battery and presence of cognitive impairment (at least two scores 1.5 standard deviations below the mean), it cannot serve as a replacement for comprehensive neuropsychological evaluation (Wilson et al., 2015). Performance on a few short cognitive measures can inform the need for full evaluation and provide information that may be useful for brief recommendations. However, more comprehensive testing is necessary to assess the patient’s pattern of neurocognitive strengths and weaknesses, lateralize or localize areas of cognitive dysfunction associated with seizure activity, and provide the amount of information necessary to generate comprehensive recommendations for clinical, educational, and home-based interventions and supports. Furthermore, although performance on testing is an important component, neuropsychological evaluations involve more than just test scores and include integration of information gathered from clinical interview, review of medical and educational records, and observations of behaviors to formulate impressions. Even in patients who do not demonstrate cognitive impairments on comprehensive testing, there are often benefits to completing a neuropsychological evaluation such as providing a discussion of cognitive and psychological risks associated with epilepsy, discussing school performance and supports, and providing recommendations for clinical and educational care. Consequently, there may be benefits to referring for comprehensive neuropsychological evaluation even for patients who did not demonstrate overt deficits on testing and the risks associated with unnecessarily referring are relatively low.

Conclusions

This study provides evidence that brief performance-based cognitive testing can provide value and reduce referral bias as part of neuropsychology screening in children with epilepsy. For high sensitivity, a clinician may consider a referral for more comprehensive neuropsychological evaluation if a patient scores below the average range (SS < 90) on any of the following subtests from the NIH Toolbox: Picture Vocabulary Test, Pattern Comparison Processing Speed Test, or Dimensional Change Card Sort Test. For more balanced sensitivity and specificity – potentially in settings where supplemental assessment procedures are available – a clinician may refer with any score 1 standard deviation below the mean (SS < 85). These three screening measures collectively take less than 15 minutes to administer. They are very easy to administer, score automatically, and require only a tablet, so they can be easily administered in a clinic setting by a neuropsychologist, technician, or medical assistant. Involvement with neuropsychology is likely beneficial for all children with epilepsy. A potentially useful model could include neuropsychology involvement in multi-disciplinary clinic for consultation that includes performance-based screenings to help inform whether more comprehensive testing is likely to be beneficial.

Key Points.

  • Demographic information, epilepsy characteristics, and parent report, which likely influence decisions to refer to neuropsychology, offer limited value in predicting neuropsychology test performance.

  • Standardized cognitive screening is highly predictive of neuropsychological test scores beyond these variables.

  • A 15-minute battery of NIH Toolbox Cognitive screening tests is sufficient to inform neuropsychology referrals.

  • Screening scores 0.70–1.0 standard deviations below the mean (standard scores of 85–89) provide a useful cutoff for risk of neurocognitive impairment.

  • Cognitive screening may be a useful tool for epilepsy providers who refer to neuropsychology or within multi-disciplinary clinics that include neuropsychology services.

Acknowledgments:

This project was funded by an intramural grant through Nationwide Children’s Hospital. This project was supported, in part, by The Ohio State University Clinical and Translational Science Institute (CTSI) and the National Center for Advancing Translational Sciences of the National Institutes of Health under Grant Number UM1TR004548. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Funding:

This project was funded by an intramural grant through Nationwide Children’s Hospital. This project was supported, in part, by The Ohio State University Clinical and Translational Science Institute (CTSI) and the National Center for Advancing Translational Sciences of the National Institutes of Health under Grant Number UM1TR004548. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Footnotes

Disclosures of Conflicts of Interest: None of the authors have conflicts of interest.

Conflict of Interest Statement

None of the authors have conflicts of interest to disclose.

We confirm that we have read the Journal’s position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.

References

  1. Baumer FM, Cardon AL, & Porter BE. (2018). Language dysfunction in pediatric epilepsy. The Journal of Pediatrics, 194, 13–21. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Begeer S, Bouk SE, Boussaid W, Terwogt MM, & Koot HM. (2009). Underdiagnosis and referral bias of autism in ethnic minorities. Journal of Autism and Developmental Disorders, 39, 142. [DOI] [PubMed] [Google Scholar]
  3. Berg AT, Langfitt JT, Testa FM, Levy SR, DiMario F, Westerveld M, & Kulas J. (2008). Global cognitive function in children with epilepsy: a community‐based study. Epilepsia, 49(4), 608–614. [DOI] [PubMed] [Google Scholar]
  4. Boulet-Craig A, Robaey P, Laniel J, Bertout L, Drouin S, Krajinovic M, Laverdière C, Sinnett D, Sultan S, & Lippé S. (2018). DIVERGT screening procedure predicts general cognitive functioning in adult long‐term survivors of pediatric acute lymphoblastic leukemia: A PETALE study. Pediatric Blood & Cancer, 65(9), e27259. [DOI] [PubMed] [Google Scholar]
  5. Brooks BL, & Sherman EM. (2012). Computerized neuropsychological testing to rapidly evaluate cognition in pediatric patients with neurologic disorders. Journal of Child Neurology, 27(8), 982–991. [DOI] [PubMed] [Google Scholar]
  6. Bryce SD, Bowden SC, Wood SJ, & Allott K. (2021). Brief, performance-based cognitive screening in youth aged 12–25: A systematic review. Journal of the International Neuropsychological Society, 27(8), 835–854. [DOI] [PubMed] [Google Scholar]
  7. Charvet LE, Shaw M, Frontario A, Langdon D, & Krupp LB. (2018). Cognitive impairment in pediatric-onset multiple sclerosis is detected by the Brief International Cognitive Assessment for Multiple Sclerosis and computerized cognitive testing. Multiple Sclerosis Journal, 24(4), 512–519. [DOI] [PubMed] [Google Scholar]
  8. Clary HM, Josephson SA, Franklin G, Herman ST, Hopp JL, Hughes I, Meunier L, Moura LMVR, Parker-McFadden B, Pugh MJ, Schultz R, Spanaki MV, Bennett A, & Baca C. (2022). Seizure frequency process and outcome quality measures. Neurology, 98(14), 583–590. [DOI] [PubMed] [Google Scholar]
  9. Conners KC. (2000). Conners' Continuous Performance Test II (CPT II). Multi-Health Systems. [Google Scholar]
  10. Dagar A, & Falcone T. (2020). Psychiatric comorbidities in pediatric epilepsy. Current Psychiatry Reports, 22, 1–10. [DOI] [PubMed] [Google Scholar]
  11. Dang J, King KM, & Inzlicht M. (2020). Why are self-report and behavioral measures weakly correlated?. Trends in Cognitive Sciences, 24(4), 267–269. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Delis DC, Kaplan E, & Kramer JH. (2001). Delis-Kaplan Executive Function System (D–KEFS). APA PsycTests. [DOI] [PubMed] [Google Scholar]
  13. Delis DC & Psychological Corporation. (1994). CVLT-C: California verbal learning test. Psychological Corp., Harcourt Brace Corp. [Google Scholar]
  14. Gaub M & Carlson CL. (1997). Gender differences in ADHD: A meta-analysis and critical review. Journal of the American Academy of Child and Adolescent Psychiatry, 36, 1036–1045. [DOI] [PubMed] [Google Scholar]
  15. Geiger HJ. (2002). Racial and ethnic disparities in diagnosis and treatment: A review of the evidence and a consideration of causes. In Smedley BD, Stith AY, and Nelson AR (Eds.), Unequal treatment: Confronting racial and ethnic disparities in health care (pp. 417–454). National Academies Press. [PubMed] [Google Scholar]
  16. Gioia GA, Isquith PK, Guy SC, Kenworthy L. (2015). Behavior Rating Inventory of Executive Function, Second Edition (BRIEF-2). Lutz, FL: PAR Inc. [Google Scholar]
  17. Goodman R. (1999). The extended version of the Strengths and Difficulties Questionnaire as a guide to child psychiatric caseness and consequent burden. Journal of Child Psychology and Psychiatry, 40, 791–799. [PubMed] [Google Scholar]
  18. Hardy KK, Olson K, Cox SM, Kennedy T, & Walsh KS. (2017). Systematic review: A prevention-based model of neuropsychological assessment for children with medical illness. Journal of Pediatric Psychology, 42(8), 815–822. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Heaton RK, Grant I, & Matthews CG. (1991). Comprehensive norms for an expanded Halstead-Reitan Battery: Demographic corrections, research findings, and clinical applications. Odessa, FL: Psychological Assessment Resources. [Google Scholar]
  20. Hosmer DW Jr, Lemeshow S, & Sturdivant RX. (2013). Applied Logistic Regression. New York, NY: John Wiley & Sons. [Google Scholar]
  21. Jones-Gotman M, Smith ML, Risse GL, Westerveld M, Swanson SJ, Giovagnoli AR, Lee T, Mader-Joaquim MJ, & Piazzini A. (2010). The contribution of neuropsychology to diagnostic assessment in epilepsy. Epilepsy Behavior, 18(1–2), 3–12. [DOI] [PubMed] [Google Scholar]
  22. Kaplan E, Goodglass H, Weintraub S, & Goodglass H. (1983). Boston naming test. Philadelphia, PA: Lea & Febiger. [Google Scholar]
  23. Kim EH, & Ko TS. (2016). Cognitive impairment in childhood onset epilepsy: up-to-date information about its causes. Korean journal of pediatrics, 59(4), 155. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Korkman M, Kirk U, & Kemp S. (2007). NEPSY-II (2nd ed.). San Antonio, TX: Harcourt Assessment. [Google Scholar]
  25. Lai JS, Nowinski CJ, Zelko F, Wortman K, Burns J, Nordli DR, & Cella D. (2015). Validation of the Neuro-QOL measurement system in children with epilepsy. Epilepsy & Behavior, 46, 209–214. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Lai JS, Nowinski C, Victorson D, Bode R, Podrabsky T, McKinney N, Straube D, Holmes G, McDonald CM, Henrickson E, Abresch RT, Moy CS, & Cella D. (2012). Quality-of-life measures in children with neurological conditions: Pediatric Neuro-QOL. Neurorehabilitation and Neural Repair, 26(1), 36–47. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Lordo DN, Van Patten R, Sudikoff EL, & Harker L. (2017). Seizure-related variables are predictive of attention and memory in children with epilepsy. Epilepsy & Behavior, 73, 36–41. [DOI] [PubMed] [Google Scholar]
  28. Matuska E, Carney A, Sepeta LN, Zelleke T, Pasupuleti A, & Berl MM. (2024). Clinical Validation of Selected NIH Cognitive Toolbox Tasks in Pediatric Epilepsy. Epilepsy & Behavior, 153, 109684. [DOI] [PubMed] [Google Scholar]
  29. Meehan WP III, d’Hemecourt P, Collins CL, Taylor AM, & Comstock RD. (2012). Computerized neurocognitive testing for the management of sport-related concussions. Pediatrics, 129(1), 38–44. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Miller LE, Kaseda ET, Koop JI, Mau KA, & Heffelfinger AK. (2021). Differential access to neuropsychological evaluation in children with perinatal complications or autism spectrum disorder: Impact of sociodemographic factors. The Clinical Neuropsychologist, 35(5), 988–1008. [DOI] [PubMed] [Google Scholar]
  31. Nickels KC, Zaccariello MJ, Hamiwka LD, & Wirrell EC. (2016). Cognitive and neurodevelopmental comorbidities in paediatric epilepsy. Nature Reviews Neurology, 12(8), 465–476. [DOI] [PubMed] [Google Scholar]
  32. Pike NA, Poulsen MK, & Woo MA. (2017). Validity of the Montreal cognitive assessment screener in adolescents and young adults with and without congenital heart disease. Nursing research, 66(3), 222–230. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Plioplys S, Dunn DW, & Caplan R. (2007). 10-year research update review: Psychiatric problems in children with epilepsy. Journal of American Academy of Child and Adolescent Psychiatry, 46(11), 1389–1402. [DOI] [PubMed] [Google Scholar]
  34. R Core Team (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/. [Google Scholar]
  35. Revelle W. (2022). psych: Procedures for Personality and Psychological Research, Northwestern University, Evanston, Illinois, USA, https://CRAN.R-project.org/package=psych, Version = 2.2.9. [Google Scholar]
  36. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, & Müller M. (2011). pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12(77), 1–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. RStudio Team (2020). RStudio: Integrated Development for R. RStudio, PBC, Boston, MA, USA, http://www.rstudio.com/. [Google Scholar]
  38. Rydz D, Srour M, Oskoui M, Marget N, Shiller M, Birnbaum R, Majnemer A, & Shevell MI. (2006). Screening for developmental delay in the setting of a community pediatric clinic: A prospective assessment of parent-report questionnaires. Pediatrics, 118(4). [DOI] [PubMed] [Google Scholar]
  39. Toplak ME, West RF, & Stanovich KE. (2013). Practitioner review: Do performance‐based measures and ratings of executive function assess the same construct?. Journal of Child Psychology and Psychiatry, 54(2), 131–143. [DOI] [PubMed] [Google Scholar]
  40. Tran T, Swami S, Shin E, Slomowitz R, DeLeon R, Nussbaum N, & Schraegle WA. (2023). 44 Finding the Onramp: Understanding Access to Neuropsychological Evaluation in New Onset Pediatric Epilepsy. Journal of the International Neuropsychological Society, 29(s1), 41–42. [Google Scholar]
  41. Vega-Fernandez P, Vanderburgh White S, Zelko F, Ruth NM, Levy DM, Muscal E, … & Brunner HI. (2015). Cognitive performance scores for the pediatric automated neuropsychological assessment metrics in childhood‐onset systemic lupus erythematosus. Arthritis Care & Research, 67(8), 1119–1127. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Wagner JL, Kellermann T, Mueller M, Smith G, Brooks B, Arnett A, & Modi AC. (2016). Development and validation of the NDDI-E-Y: A screening tool for depressive symptoms in pediatric epilepsy. Epilepsia. Doi: 10.1111/epi.13446. [DOI] [PubMed]
  43. Wagner JL, Modi AC, Guilfoyle SM, Junger KF, Weyand C, Smith G, Griffin M, & Mucci G. (2019). Current behavioral health and cognitive screening practices in pediatric epilepsy. Epilepsy & Behavior, 101, 106214. [DOI] [PubMed] [Google Scholar]
  44. Wechsler D. (2014). Wechsler Intelligence Scale for Children (5th ed.). San Antonio, TX: NCS Pearson. [Google Scholar]
  45. Weintraub S, Bauer PJ, Zelazo PD, Wallner‐Allen K, Dikmen SS, Heaton RK, Tulsky DS, Slotkin J, Blitz DL, Carlozzi NE, Havlik RJ, Beaumont JL, Mungas D, Manly JJ, Borosh BG, Nowinski CJ, & Gershon RC. (2013). I. NIH Toolbox Cognition Battery (CB): introduction and pediatric data. Monographs of the Society for Research in Child Development, 78(4), 1–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Wilkinson GS, & Robertson GJ. (2006). Wide Range Achievement Test 4 (WRAT4). APA PsycTests. [Google Scholar]
  47. Wilson SJ, Baxendale S, Barr W, Hamed S, Langfitt J, Samson S, Watanabe M, Baker GA, Helmstaedter C, Hermann BP, & Smith ML. (2015). Indications and expectations for neuropsychological assessment in routine epilepsy care: Report of the ILAE Neuropsychology Task Force, Diagnostic Methods Commission, 2013–2017. Epilepsia, 56(5), 674–681. [DOI] [PubMed] [Google Scholar]
  48. Xiao Y, Mann JJ, Chow JCC, Brown TT, Snowden LR, Yip PSF, … & Su C. (2023). Patterns of social determinants of health and child mental health, cognition, and physical health. JAMA pediatrics, 177(12), 1294–1305. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES