Questionnaire-Based Assessment of Executive Functioning: Psychometrics

Abstract

The psychometric properties of the Learning, Executive, and Attention Functioning (LEAF) scale were investigated in an outpatient clinical pediatric sample. As a part of clinical testing, the LEAF scale, which broadly measures neuropsychological abilities related to executive functioning and learning, was administered to parents of 118 children and adolescents referred for psychological testing at a pediatric psychology clinic; 85 teachers also completed LEAF scales to assess reliability across different raters and settings. Scores on neuropsychological tests of executive functioning and academic achievement were abstracted from charts. Psychometric analyses of the LEAF scale demonstrated satisfactory internal consistency, parent-teacher inter-rater reliability in the small to large effect size range, and test-retest reliability in the large effect size range, similar to values for other executive functioning checklists. Correlations between corresponding subscales on the LEAF and other behavior checklists were large, while most correlations with neuropsychological tests of executive functioning and achievement were significant but in the small to medium range. Results support the utility of the LEAF as a reliable and valid questionnaire-based assessment of delays and disturbances in executive functioning and learning. Applications and advantages of the LEAF and other questionnaire measures of executive functioning in clinical neuropsychology settings are discussed.

Executive functions (EFs) are top-down neuropsychological processes responsible for the active regulation of controlled attention, emotion, and planned behavior in the service of goal attainment (Banich, 2009; Barkley, 2012). Although there is no single, universally agreed-upon definition of EF, most conceptualizations of EF include several related but separate processing domains primarily mediated by neural circuits in the prefrontal cortex (Barkley, 2012; Luria, 1966). These core EFs include self-directed attention (considered the central executive because it serves a gatekeeping role for subsequent EFs), working memory (simultaneous processing and storage of a stimulus/event), response inhibition (controlled suppression of a prepotent or automatic response to a stimulus/event), cognitive flexibility (shifting between mental states, responses or tasks), and fluency (rapid processing under concentration demands; Awh, Vogel, & Oh, 2006; Barkley, 2012; McAuley & White, 2011; Miyake, Friedman, Emerson, Witzki, & Howerter, 2000). Neuropsychologists have been interested in early identification and interventions for EF deficits for many years, because delays and disturbances in EF are present at elevated rates in many disorders resulting from central nervous system disorder or injury such as spina bifida, cerebral palsy, epilepsy, traumatic brain injury, and cancer (Daly & Brown, 2007; Horton et al., 2010; O’Hara & Holmbeck, 2013; Parrish, Geary, Jones, Seth, Hermann, & Seidenberg, 2007; Weierink, Vermeulen & Boyd, 2013).

Poor EF is a significant clinical health issue not only due to its prevalence in neurological injury and disorder, but also due to the influence of poor EF on academic outcomes such as memory problems, educational failure (Barkley, 2012) and learning disabilities (Jerman, Reynolds, & Swanson, 2012). EF delays, as well as disorders characterized by poor EF, are commonly associated with academic underachievement, learning deficits, and related problems with learning and memory (Barkley, 2012). Because of the significant role of EF for learning, memory, and academic outcomes, assessment of EF delays that might impact significantly on school success is a common component of neuropsychological evaluation of children with significant academic deficits.

Core Executive Functions and Related Cognitive Abilities

In recent years, researchers have attempted to separate skills that constitute core EFs from closely related cognitive abilities such as learning and memory that are dependent on EF but that are not central to the EF construct. Skills such as working memory, inhibition, and flexibility are universally accepted as core EF abilities because they are necessary for focused, goal-directed activity and have been supported by empirical research such as factor analysis (Miyake et al., 2000). Other core domains of EF include controlled attention, sustained sequential processing (planning and execution of goal-directed behavior), novel problem-solving, and organization, all of which are necessary in order to initiate and complete purposeful, planned, goal-directed activities and have been validated as core components of EF in prior research (Barkley, 2011a; Diamond, 2013; Naglieri & Goldstein, 2013). Organization, planning, working memory, flexibility, controlled attention, and inhibition skills, for example, are routinely assessed using questionnaire-based measures of EF such as the BRIEF, CEFI, and BDEFS-CA (Barkley, 2011a; Gioia et al., 2000; Naglieri & Goldstein, 2013).

On the other hand, cognitive processing skills deployed during learning that are not core to EF but are closely related to (and heavily dependent on) EF include concept formation (Barkley, 2012), comprehension (Gathercole & Baddeley, 1993), factual memory (Buckner, 2004), and academic functioning (Diamond, 2016; Gathercole, Pickering, Knight, Stegmann, 2004). These learning-related areas of cognitive functioning are an integral part of academic learning and higher-order cognitive processing that are common weakness areas for individuals with EF delays. For example, factual memory is dependent on the active deployment of working memory (a core EF) in learning situations, while concept formation requires flexibility and novel problem-solving (core EF) skills. Because these learning-related domains of cognitive functioning are dependent on EF and because of the critical role of EF in learning and academic success, these learning-related domains of cognitive functioning are often included in assessments of EF and are considered to be at risk in children who have EF delays (Barkley, 2012).

Assessment of Executive Functioning Using Behavior Checklists

Clinical evaluation of EF typically includes an office-based visit involving administration of a battery of behaviorally-based neuropsychological assessment instruments. Decades of research support the validity, utility, and diagnostic value of these instruments for the measurement of EF in children and adults across a wide range of disorders (Lezak, 2004). Despite their advantages, however, individually-administered neuropsychological measures of EF have two primary limitations: First, in most cases, they must be individually administered and scored by a technician or professional in an office setting, which limits their utility for screening or brief assessment purposes. Second, relations between office-based neuropsychological measures of EF and actual behavior in the daily environment are modest (Barkley, 2012), leading to some caution when applying neuropsychological test results to conclusions about behavioral outcomes. As a result of these limitations of office-based neuropsychological tests of EF, parent- and teacher-report behavior checklist measures of EF have been developed for both screening purposes and to complement the results of performance-based neuropsychological testing by providing reports of EF behavior in daily life (Barkley, 2011a; Gioia, Isquith, Guy, & Kenworthy, 2000; Naglieri & Goldstein, 2013). These checklists have the advantage of good psychometrics, strong ecological validity, and high clinical utility as a result of their ease of administration, scoring, and interpretation. Furthermore, questionnaire-based assessment of EF may be especially valuable in busy applied multidisciplinary settings (such as pediatric specialty clinics) because of the potential importance of EF in coping, adaptive behavior, learning, adjustment, and self-management skills in response to medical and neuropsychological conditions. Therefore, EF behavior checklists offer the potential to enhance clinical practice in pediatric neuropsychology and to promote the application of EF research in the clinical setting.

One of the first questionnaire-based measures of EF was the Behavior Rating Inventory of Executive Functioning (BRIEF; Gioia et al., 2000), recently revised as the BRIEF-2 (Gioia, Isquith, Guy, & Kenworthy, 2016). The BRIEF and BRIEF-2 forms are parent-, teacher-, and self-report behavior checklists for children and adolescents. BRIEF-2 EF subscales assess areas including inhibition, self-monitoring, shifting, emotional control, initiation, task completion, working memory, planning/organizing, task monitoring, and organization of materials. In addition to the BRIEF, other checklist measures of EF exist, including the Comprehensive Executive Function Inventory (CEFI, for children aged 5 – 18 years; Naglieri & Goldstein, 2013) and the Barkley Deficits in Executive Functioning Scale (BDEFS for adults; Barkley, 2011b; BDEFS-CA for children and adolescents; Barkley, 2011a). The CEFI measures EF domains such as attention, emotion regulation, flexibility, inhibitory control, initiation, organization, planning, self-monitoring, and working memory. The BDEFS measures EF in daily life activities such as time management, organization and problem solving, self-restraint, self-motivation, and self-regulation of emotions. These EF checklists have been effectively used in clinical and research settings as screening measures of EF delays and disturbances, as primary measures of EF in clinical populations, and as complementary measures in addition to traditional individually-administered neuropsychological tests (Ebrahimi, Kassani, Menati, Abedi, Yarmohammadian, & Faramarzi, 2015; García, Rodriguez, González-Castro, & Areces, 2014; Naglieri & Goldstein, 2014).

Current behavior checklist measures of EF such as the BRIEF, CEFI, and BDEFS-CA have been demonstrated to be reliable, valid, and useful in both clinical and research settings, but each of these existing EF scales focuses on a specific set of core EF skills that excludes some cognitive functioning domains that are considered to be core or related to EF. For example, the BRIEF and CEFI include subscales measuring flexibility-shifting, whereas BDEFS-CA does not have a flexibility-shifting subscale. Conversely, the BRIEF does not have a controlled attention subscale (most of the attention items on the BRIEF fall in the Working Memory subscale), while the CEFI has an attention subscale. Additionally, the major established behavior checklist measures of EF do not include scales measuring related cognitive domains that are crucial for learning such as comprehension and declarative-factual memory, and in fact, there are very few behavior checklist measures that evaluate both core EF domains and learning-related domains that are dependent on EF. Therefore, there is an unmet need for an EF checklist that evaluates a broader set of EF-related functions (those that are grounded in EF capabilities and prefrontal cortical activity), encompassing both a broad set of core EF domains and a set of learning and higher-order cognitive processing domains that are related (but not core) to EF. Furthermore, a significant need exists for an EF behavior checklist that is based on a highly simplified administration and scoring methodology that allows for very easy use in clinical settings.

A New Behavioral Checklist of Executive Functioning: The Learning, Executive, and Attention Functioning (LEAF) Scale

We sought to create a reliable and valid instrument to meet these needs for comprehensive EF-related assessment and simplified administration and scoring: the Learning, Executive, and Attention Functioning scale (LEAF). The primary purpose of the LEAF is the measurement of executive functioning and related learning skills. Unlike other EF scales, the LEAF was developed primarily to assist with EF assessment when cognitive and learning factors are a core component of concern. As a result, the LEAF falls at the intersection of EF and learning abilities and includes EF components that are closely related to learning, as well as learning domains that are vulnerable to EF delays.

The LEAF fills an important niche in the clinical assessment of EF in neuropsychological settings that is not fully addressed by the few existing questionnaire measures of EF, in the following ways: (1) The LEAF evaluates a broad set of core cognitive EF abilities as well as related cognitive learning and academic abilities, which are frequently seen in clinical populations with central nervous system disease or injury and are not addressed by existing questionnaires. Core cognitive EF areas assessed by the LEAF include attention, processing speed under conditions requiring concentration, organization (including visual-spatial organization skills), sustained sequential processing to achieve goals (e.g., planning and executing goal-directed behavior), working memory, and novel problem-solving (Barkley, 2012; McAuley & White, 2011; Zelazo, Carter, Reznick, & Frye, 1997). Related cognitive learning areas assessed by the LEAF include comprehension and concept formation, declarative/factual memory, and academic functioning. For example, previous research indicates that comprehension and concept formation are heavily dependent on working memory skills (Castellanos, et al., 2015), and retrieving information from long-term declarative/factual memory is dependent on initial attention and working memory skills during encoding and processing (Ragland et al., 1998). Additionally, previous research indicates that core EF skills (working memory and inhibition) predict academic competence and achievement in school-age children (Gathercole et al., 2004). Consequently, the LEAF contains Academic subscales assessing reading, writing, and math fluency, abilities, which are supported in part by the development of core EF skills. (2) The LEAF emphasizes EF domains related to everyday learning and cognitive functioning as opposed to behavioral psychopathology and psychiatric diagnoses, using item wordings that focus more on information processing than on behaviorally-related problems. For example, LEAF working memory items are conceptualized as the child’s ability to retain and process complex information received from the environment under conditions of concurrent cognitive load (“Gets overwhelmed if required to learn or attend to a lot of information,” see Appendix), as opposed to BRIEF Working Memory scale items, which emphasize attention problems (Gioia et al., 2000). (3) The LEAF was constructed to meet all of Levy, Kronenberger, & Dunn’s (2013) characteristics of a clinically useful behavior checklist: brevity in administration, breadth of additional relevant content, efficiency of scoring and interpretation, and ease of availability for use. For example, LEAF items are grouped by subscale, and all subscales have the same number of items, in order to facilitate rapid scoring in the busy clinical setting, without the need for templates or a computer. By addressing these characteristics, the LEAF was constructed to enhance the feasibility of routine questionnaire-based EF assessment in clinical settings.

In this present paper, we describe the development and psychometrics of the LEAF as a questionnaire measure of EF that complements existing measures and can be used to assess a broad and inclusive set of EF components and related learning and academic constructs.

Methods

Participants

Participants were 118 children and adolescents between the ages of 6 and 17 years (M age = 11.80 years, SD = 3.23 years; 71 males, 47 females; 104 White, 7 African-American, 3 Asian, 2 Hispanic, 1 Indian-Asian, and 1 of unknown ethnic background). The sample was obtained from consecutive referrals of patients who were seen for psychological testing at an outpatient clinic at a pediatric hospital in the Midwest region and who provided a LEAF scale completed by one parent. Across primary, secondary, and tertiary referral questions coded by the primary clinician, the most common reasons for referral were attention/concentration problems (N = 76), learning problems (N = 97), aggression, anger, or oppositionality (N = 13), and hyperactivity (N = 8).

Participants in the sample were clinically diagnosed by the primary testing clinician using criteria from the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV; American Psychiatric Association, 1994). Based on psychological testing and interview results, 55 (47%) participants were diagnosed with Attention-Deficit/Hyperactivity Disorder (ADHD) (25 with predominantly Inattentive subtype, 30 with Combined subtype), 44 (37%) participants were diagnosed with a Learning Disorder, 33 (28%) participants were diagnosed with an Anxiety Disorder (including adjustment disorders with an anxiety component), 28 (24%) participants were diagnosed with a Pervasive Developmental Disorder, 22 (19%) participants were diagnosed with a Depressive Disorder (including adjustment disorders with a depression component), 19 (16%) participants were diagnosed with a Disruptive Behavior Disorder (oppositional-defiant disorder or conduct disorder), 7 (6%) participants were diagnosed with an Adjustment Disorder, and 3 (3%) participants were diagnosed with an Elimination Disorder.

Nearly half (N = 53, 45%) of sample participants had a pediatric condition, and in 31 (26%) cases, that condition was closely related to the reason for referral for testing (i.e., cognitive functioning difficulties possibly secondary to the condition). Neurological conditions were the most common (N = 28, 24%; e.g., spina bifida [N = 7, 6%], seizure disorder [N = 7, 6%], brain malformations /tumors/ cysts [N = 5, 4%]; cerebral palsy [N = 3, 3%]; other neurological conditions [N = 6, 5%]). Other physical conditions in the sample were respiratory (apnea, N = 1, 1%; asthma, N = 7, 6%), gastrointestinal (chronic constipation, N = 1, 1%; irritable bowel, N = 3, 3%; recurrent abdominal pain, N = 1, 1%), endocrine (diabetes, N = 1, 1%; hypothyroid, N = 2, 2%), autoimmune (arthritis, N = 1, 1%; lupus, N = 2, 2%), long-term complications of prematurity and very low birthweight (N = 3, 3%), primary pain syndromes (fibromyalgia, N = 1, 1%; chronic headache, N = 1, 1%), cleft palate (N = 1, 1%), hearing loss (N = 1, 1%), and hyperalaninemia (N = 1, 1%) (numbers add to more than the sample size because of comorbidity). Of these latter non-neurological conditions, only hearing loss and lupus (rule out neuropsychiatric lupus) were related to the primary reason for psychological testing.

Primary parent respondents to the LEAF scale were 103 mothers and 15 fathers. The primary focus in this manuscript is on the psychometrics of parent-completed LEAF scales. Teachers of 85 participants also completed the LEAF scale, and their data are included for inter-rater reliability and construct validity analyses comparing LEAF scores and other behavior checklists. Twenty-seven mothers completed two LEAF scales within the same 35-day period (range = 10 – 35 days), providing data to assess test-retest reliability. Children were tested by a psychologist or trainee (psychology intern or graduate student) at a clinic visit that occurred on the same day that the primary LEAF scale and other parent-report behavior checklists were completed; LEAF scales and other behavior checklists completed by teachers were requested by mail or hand delivery by the parent prior to the clinic testing session and were received by mail or fax.

Procedure

Data for the present study were obtained using chart-review methods approved by the university Institutional Review Board. Performance data from neuropsychological tests, demographic data, and clinically assigned diagnoses and reasons for referral (as coded by the clinician responsible for the testing) were abstracted from test reports and other data in the clinical charts. LEAF scales and other behavioral checklists, which are routinely completed by parents and teachers to provide clinical assessment and monitoring information, were also abstracted from the clinical charts of all participants.

Measures

Learning, Executive, and Attention Functioning (LEAF) Scale

The LEAF scale measures executive functioning and related learning skills in children and adolescents aged 6 – 17 years. The content-development process consisted of a literature review pertaining to tests, disorders, and interventions involving executive functioning and the impact of executive functioning and related processes on learning and behavioral adjustment. We identified traditional components of executive functioning and a broad range of abilities related to and/or dependent upon executive functioning, including working memory (Gioia et al., 2000), sustained sequential processing (Conway, Pisoni, & Kronenberger, 2009), organization, comprehension, and novel problem solving (Rourke, 1995). This process generated two Cognitive-Learning content areas reflecting learning, memory, and reasoning skills closely related to EF but not core to the EF construct: (1) Comprehension and Conceptual Learning (tracking and understanding information), and (2) Factual Memory (memorization and retention of facts), and six Cognitive-EF content areas reflecting core EF domains: (3) Attention (sustained focus); (4) Processing Speed (speed of completing cognitive and behavioral tasks that involve a component of focus and concentration); (5) Visual-Spatial Organization (organization and visual-constructive skills); (6) Sustained Sequential Processing (planning and sustaining effort in order to follow and complete multistep directions and sequences); (7) Working Memory (remembering and processing multiple things at the same time); and (8) Novel Problem Solving (initiating effort toward processing new or unfamiliar information). An additional three Academic content areas were added in order to enhance the clinical utility of the scale, because executive functioning deficits are frequently related to academic achievement difficulties (Gathercole et al., 2004): (9) Mathematics Skills (math calculation difficulty or dysfluency); (10) Basic Reading Skills (reading/phonics difficulty or dysfluency); and (11) Written Expression Skills (limited/impoverished or slow/effortful written expression). Based on prior work showing that five-item behavior checklist subscales can be completed, scored, and interpreted rapidly in busy clinical settings while providing good psychometrics (Levy et al., 2013), five items were created for each of these 11 content areas (items are grouped by content area, in the order of the 11 content areas described above), resulting in a total of 55 items (see Appendix).

Individual items are rated on a 0 – 3 scale, and a raw subscale score for each of the 11 content areas is created by summing the 5 constituent items, such that higher scores indicate more problems. Because respondents often have difficulty consistently anchoring response choices such as “Never,” “Sometimes,” and “Often,” behavioral descriptors were provided for each of the response choices: Response choices were anchored by the following descriptors, “0 - Never: Not a problem; Average for age,” “1 - Sometimes: A little more than average; Not a big problem,” “2 - Often: Causes problems; happens almost everyday,” and “3 - Very Often: Major daily problem.” Consistent with other rating scales (Faries, Yalcin, Harder, & Heiligenstein, 2001; Gadow & Sprafkin, 1997), ratings of “2” (Often) are encouraged for behaviors that cause problems, whereas ratings of “1” (Sometimes) are used to reflect behaviors that may occur more than average but that do not cause big problems. Therefore, an average rating of “2” for the five items comprising a subscale would indicate that behaviors for that subscale were rated, on average, as causing problems and happening almost every day.

Three criterion-referenced interpretation ranges (0 – 4 = “No Problem Range”; 5 – 9 = “Borderline Problem Range”; 10 – 15 = “Problem Range”) were created for the LEAF¹. A LEAF raw score of less than 5 indicates that the average item was rated less than “1”, which has the anchor statement of “Sometimes; A little more than average; Not a big problem.” Hence, a raw score of less than 5 indicates that subscale items were rated, on average, as sporadic and not a big problem. Subscale raw scores of 5 – 9 fall into an intermediate range, with some problematic behaviors and some behaviors not rated as problems; scores in this range are therefore characterized as falling in the “Borderline Problem Range.” Subscale raw scores of 10 or higher are likely to indicate moderate to severe problems and fall in the “Problem Range”. Norm-based scores are not yet available for the LEAF.

Neuropsychological Measures

Participants were also administered several gold-standard performance-based measures of executive functioning (attention and concentration) and academic achievement as a part of their clinical testing. All assessments selected have well-validated psychometric properties, and most have published test manuals.

Attention and concentration were assessed using the Stroop Color and Word Test (SCWT; Golden, 1978; N = 103), the Counting Interference Test (CIT; Hummer et al., 2011; N = 107) and the Conners’ Continuous Performance Test (CPT; Conners & MHS Staff, 2000; N = 97). Successful performance on the SCWT and the CIT requires components of controlled attention to information relevant to the task, as well as inhibition of distractors not relevant to the task. The SCWT measures the ability to inhibit a highly overlearned/automatic process (word reading) in favor of a more effortful/controlled process (naming ink color) for a series of color words (i.e., red, blue, green) that are printed in ink colors that are incongruent with the words. The CIT is a counting Stroop-like test for which participants must state the number of numerals present in a series of one-, two-, or three-digit numbers (e.g., 222, 11, 3), suppressing numeral naming in favor of identifying the number of digits present in the display. The CPT is a computer-administered test that measures timing and accuracy of responses to visually-presented targets vs. nontargets. Raw scores on the color–word condition of the SCWT, the number-count condition of the CIT (number of accurate responses in 45 seconds), and the Hit Reaction Time Standard Error (RTSE) score of the CPT (standard deviation of response speed for all correctly answered items, which has been shown to be one of the most sensitive measures of attention problems on the CPT; Conners & MHS Staff, 2000) were used as measures of focused attention and concentration.

Academic achievement was assessed using two subtests and one composite score of the Woodcock-Johnson III Tests of Achievement, Third Edition (WJ-III; Woodcock, McGrew, & Mather, 2001), which corresponded to the three academic subscales of the LEAF. The WJ-III Basic Reading Skills score (a composite of scores from the Letter-Word Identification and Word Attack subtests, reflecting reading phonics and word identification/reading skills; N = 111) was selected to correspond to the LEAF Basic Reading Skills subscale. The WJ-III Calculation subtest (a measure of formal arithmetic knowledge and math calculation skills; N = 112) was selected to correspond to the LEAF Mathematics Skills subscale. The WJ-III Writing Samples subtest (a measure of written expression for sentences that either stand alone or that are embedded within paragraphs; N = 106) was selected to correspond to the LEAF Written Expression Skills subscale.

Behavior Checklists

In addition to the LEAF scale, parents and teachers completed the BRIEF (Gioia et al., 2000) and the Conduct-Hyperactive-Attention Problem-Oppositional Symptom (CHAOS; Levy et al., 2013) scales. The CHAOS scale is clinically useful for the evaluation of children diagnosed with ADHD and includes four subscales, including Attention Problems, Hyperactivity-Impulsivity, Oppositional Behavior, and Conduct Problems (only the CHAOS Attention Problems subscale, which measures carelessness, disorganization, distractibility, and difficulty sustaining and controlling attention, was used in this study; Levy et al., 2013). Similar to the BRIEF, the CHAOS Attention Problems subscale has high internal consistency (ranging from .90 - .91 for teacher and parent forms), medium-to-high inter-rater reliability (r = .58 for mother-father; r = .41 for parent-teacher), and satisfactory test-retest reliability over 10–26 weeks (r = .78; Levy et al., 2013). Furthermore, all items on the LEAF, BRIEF, and CHAOS are rated on a severity scale, such that higher scores indicate greater symptom severity.

In the present study, the LEAF was compared to five BRIEF subscales reflecting domains of EF that overlap in content with LEAF subscales: Initiate (initiation of tasks/activities, generating ideas or problem solving strategies; corresponding LEAF subscale is Novel Problem Solving), Working Memory (short-term working memory for completing tasks; corresponding LEAF subscales are Working Memory and Attention), Plan-Organize (managing current and future tasks demands, setting goals, organizing oral and written material; corresponding LEAF subscale is Sustained Sequential Processing), Organization of Materials (orderliness of belongings; corresponding LEAF subscale is Visual-Spatial Organization), and Monitor (self-monitoring; no single corresponding LEAF subscale, but content overlaps somewhat with LEAF Attention and Sustained Sequential Processing subscales). In addition, LEAF results were compared with the Attention Problems subscale of the CHAOS (carelessness, disorganization, distractibility, inattention; corresponding LEAF subscale is Attention).

Data Analyses

All data analyses for the present study, other than interrater reliability and correlations with other behavior checklists, used parent-reported LEAF scales, because the focus of this study was on parent-reports of EF in daily living. Psychometric evaluation of the LEAF was performed using widely accepted statistical techniques (DeVellis, 1991). Item- and subscale-level analyses and descriptive statistics were reported first, in order to provide information about the distribution of LEAF items and scores in this clinically-referred sample.

Secondly, principal axis factoring was performed on the 5 items of each LEAF subscale in order to evaluate the unidimensionality of each LEAF subscale. The eigenvalue > 1 convention was used to evaluate the number of factors comprising each LEAF subscale, such that subscales producing only a single factor with an eigenvalue > 1 were considered to be unidimensional. For subscales yielding more than 1 factor, oblimin rotation (used when factors are assumed to be correlated) was used to identify item loadings (see Gioia et al., 2000 for a similar example of this approach to investigate unidimensionality of subscales and relatedness of items).

Next, three measures of reliability were obtained for LEAF subscales: internal consistency, parent-teacher inter-rater reliability, and test-retest reliability. Internal consistency of LEAF subscales was evaluated using Cronbach’s $\propto$ > .70 (DeVellis, 2012; DeVon et al., 2007; Hair, Black, Babin, & Anderson, 2010). For consistency with psychometrics reported for other behavior checklists of EF, inter-rater reliability and test–retest reliability were evaluated with Pearson correlational analyses. Test-retest scores at or above .70 were considered satisfactory (Nestor & Schutt, 2015). Construct validity was evaluated with Pearson correlational analyses between LEAF subscales and other behavior checklists completed by the same respondent. Positive correlations between LEAF, BRIEF, and CHAOS subscales suggest that children, across checklists, are similarly rated as having greater problems. Construct validity was also examined using Pearson correlational analyses between LEAF subscales and the performance-based neuropsychological tests. Negative correlations between LEAF subscales and SCWT and CIT scores (higher SCWT and CIT scores indicate better EF) suggest that children who perform better on these neuropsychological assessments are also rated as having fewer problems on the LEAF; in contrast, positive correlations between LEAF subscales and CCPT Reaction Time Standard Error scores (higher CCPT Reaction Time Standard Error scores indicate more variability in response times during the test, which is an indication of poor EF) suggest that children who perform better on the CCPT are rated as having fewer problems on the LEAF. In line with published conventions, correlations (r values) were operationally defined as small (r = .10), medium (r = .30), or large (r = .50; Cohen, 1992).

Finally, the 3 LEAF criterion-referenced interpretation ranges (0 – 4 = “No Problem Range”; 5 – 9 = “Borderline Problem Range”; 10 – 15 = “Problem Range”) were validated using ANOVAs by comparing mean norm-based scores for BRIEF and WJ-III subscales/subtests that corresponded to LEAF subscales across the 3 LEAF interpretation ranges. Significant ANOVA tests were further evaluated with Pairwise t-tests. Mean BRIEF and WJ-III norm-based scores for participants with LEAF scores in the “No Problem Range” were expected to fall near the mean norm value (T = 50 for BRIEF subscales; Standard Score = 100 for WJ-III subtests), while those for participants with LEAF scores in the “Problem Range” were expected to fall at least 1 – 2 SD out of the norm range. Mean BRIEF and WJ-III norm-based scores for participants with LEAF scores in the “Borderline Problem Range” were expected to fall between those for participants with LEAF scores in the “No Problem Range” and “Problem Range”.

Results

Item- and Subscale-Level Descriptive Analyses

The mean scores for all 55 parent-report LEAF items fell between 0.5 and 2.5 on the 0 – 3 rating scale of response choices, and for 47 of the 55 items (85%), the mean score fell between 1 and 2 (see Table 1 for all item-level analyses). Skewness values for all items were between −1 and +1, and kurtosis values for all items were less than 0. All 220 item-level response choices (e.g., 55 items, each of which could be rated 0, 1, 2, or 3) were endorsed by at least 5% of the sample, 211 of the 220 response choices (96%) were endorsed by at least 10% of the sample, and in 97% of cases (213 of the 220 response choices) at least N = 10 observations were present for each LEAF item response choice (Linacre, 2002). These findings demonstrate that all LEAF item response choices were endorsed by a reasonable number (at least 5% or more, and in 97% of cases, N = 10 or more) of participants in the clinical sample and that item distributions were not severely skewed.

Table 1.

LEAF Item-Level Statistics

LEAF Subscale/Item	Mean	SD	Skewness	Kurtosis	% responding ‘0’	% responding ‘3’
Cognitive-Learning:
Comprehension and Conceptual Learning
Item 1	0.94	0.88	0.65	−0.30	36	6
Item 2	1.31	0.97	0.28	−0.85	22	15
Item 3	1.22	1.06	0.30	−1.15	32	14
Item 4	1.27	0.94	0.29	−0.78	22	12
Item 5	1.30	0.95	0.17	−0.89	23	11
Factual Memory
Item 6	1.24	1.00	0.34	−0.94	27	14
Item 7	1.20	0.99	0.38	−0.88	28	13
Item 8	1.04	0.95	0.65	−0.42	32	10
Item 9	1.32	0.94	0.31	−0.75	20	14
Item 10	1.35	0.93	0.10	−0.85	20	11
Cognitive-EF:
Attention
Item 11	1.81	1.01	−0.37	−0.97	13	31
Item 12	1.97	0.96	−0.49	−0.82	8	36
Item 13	1.93	0.92	−0.33	−0.92	6	33
Item 14	2.07	0.90	−0.56	−0.67	5	39
Item 15	1.46	0.84	0.27	−0.51	10	13
Processing Speed
Item 16	1.71	1.02	−0.12	−1.16	13	29
Item 17	1.44	1.02	0.26	−1.04	18	21
Item 18	1.51	1.08	0.02	−1.27	22	24
Item 19	1.92	1.06	−0.50	−1.06	13	40
Item 20	1.94	0.95	−0.36	−1.00	7	36
Visual-Spatial Organization
Item 21	2.07	1.03	−0.71	−0.75	10	46
Item 22	2.06	1.04	−0.69	−0.82	10	46
Item 23	0.85	0.98	0.97	−0.11	47	10
Item 24	1.51	1.17	0.04	−1.49	26	30
Item 25	1.02	0.92	0.63	−0.42	33	9
Sustained Sequential Processing
Item 26	1.77	1.02	−0.27	−1.06	13	30
Item 27	1.26	1.07	0.32	−1.13	30	17
Item 28	2.20	0.94	−0.80	−0.58	5	51
Item 29	1.91	1.00	−0.39	−1.01	9	36
Item 30	1.64	1.02	−0.03	−1.15	14	26
Working Memory
Item 31	1.77	0.96	−0.22	−0.96	10	27
Item 32	2.10	1.03	−0.78	−0.69	10	48
Item 33	1.83	0.97	−0.39	−0.84	11	29
Item 34	1.23	1.08	0.40	−1.11	31	18
Item 35	1.64	1.15	−0.16	−1.42	22	32
Novel Problem-Solving
Item 36	1.58	0.98	−0.05	−1.00	15	20
Item 37	1.77	1.02	−0.25	−1.11	13	31
Item 38	1.14	1.15	0.50	−1.21	40	20
Item 39	1.11	1.05	0.54	−0.92	36	14
Item 40	0.86	1.02	0.89	−0.42	49	10
Academic:
Mathematics Skills
Item 41	1.50	1.19	0.31	−1.52	28	30
Item 42	1.47	1.20	0.07	−1.53	30	29
Item 43	1.65	1.10	−0.16	−1.31	20	30
Item 44	1.19	1.21	0.45	−1.38	41	24
Item 45	1.56	1.17	−0.03	−1.47	25	31
Basic Reading Skills
Item 46	1.26	1.09	0.37	−1.16	30	19
Item 47	1.07	1.02	0.56	−0.84	36	12
Item 48	1.15	1.09	0.45	−1.11	36	15
Item 49	1.17	1.03	0.46	−0.92	31	14
Item 50	1.28	1.07	0.33	−1.13	28	18
Written Expression Skills
Item 51	1.50	1.10	0.03	−1.30	22	24
Item 52	1.71	1.18	−0.28	−1.42	22	35
Item 53	1.88	1.01	−0.38	−1.03	10	34
Item 54	1.85	1.10	−0.50	−1.09	17	36
Item 55	1.83	1.07	−0.39	−1.13	14	34

Note. For all items, range of answers was 0 – 3. SD = Standard Deviation; % responding ‘0’ and % responding ‘3’ = percentage of sample responding “0” (Never: not a problem; average for age) or “3” (Very Often: major daily problem), respectively, to the item. Values are not reported for % responding ‘1’ and % responding ‘2’ because these item responses were endorsed by 10% or more of the sample for all items.

Descriptive data for parent-report LEAF subscales by age (6 – 11 vs. 12 – 17 years) are depicted in Table 2. All LEAF subscales had mean raw scores between 5 and 10 in this clinically referred sample, reflecting an average item endorsement of approximately 1 (sometimes) to 2 (often). Large standard deviations for subscale raw scores (3 to 5 raw score points) suggest considerable variability in LEAF subscale scores within the sample. No differences were found between the age groups (p > 0.10), reflecting the fact that all participants were clinically referred for psychological testing.

Table 2.

Parent-Report LEAF Subscale Scores by Age

LEAF Subscale	6 – 11 Years		12 – 17 Years		t
	Mean	SD	Mean	SD
Cognitive-Learning:
Comprehension and Conceptual Learning	6.33	3.86	5.82	3.93	0.71
Factual Memory	6.02	3.98	6.25	4.12	0.31
Cognitive-EF:
Attention	9.94	4.01	8.72	4.07	1.63
Processing Speed	8.78	3.72	8.33	3.89	0.64
Visual-Spatial Organization	7.57	3.41	7.42	3.49	0.24
Sustained Sequential Processing	9.06	3.86	8.58	4.15	0.64
Working Memory	8.98	3.95	8.27	4.20	0.94
Novel Problem-Solving	6.61	4.03	6.34	4.36	0.34
Academic:
Mathematics Skills	7.41	5.30	7.34	4.99	0.07
Basic Reading Skills	6.45	4.79	5.25	4.84	1.34
Written Expression Skills	8.98	4.18	8.22	4.97	0.88

Note. Values are raw scores. SD = Standard Deviation. T-tests are comparisons of the two age groups; zero t-tests were statistically significant (df = 116, all p > 0.10)

Factor and Reliability Analysis of LEAF Subscales

For 10 of the 11 factor analyses of LEAF subscales (all except the Visual-Spatial Organization subscale), one factor accounted for more than 50% of the variance (e.g., eigenvalue > 2.5), and a single-factor solution was supported by the eigenvalue > 1 convention and inspection of scree plots (table of results available upon request from the authors). Hence, factor analysis results supported the unidimensionality of these 10 LEAF subscales.

However, for the Visual-Spatial Organization subscale, two factors had eigenvalues greater than 1 (eigenvalues for the five factors were 2.27, 1.16, 0.78, 0.53, and 0.26). Oblimin rotation of a two-factor solution resulted in two correlated factors (r = 0.373). Factor 1 consisted of Items 21 (Poor organization; Factor 1 Loading = 0.77, Factor 2 Loading = 0.13) and 22 (Room, desk, locker, work area is very messy; Factor 1 = 0.95, Factor 2 = −0.13). Factor 2 consisted of Items 23 (Not very good with puzzles or putting things together; Factor 1 = −0.08, Factor 2 = 0.59) and 24 (Drawing and/or handwriting poor; Factor 1 = 0.08, Factor 2 = 0.76). Item 25 (Doesn’t pay attention to visual details in the environment) loaded on both factors about equally (0.23 and 0.27, respectively).

These findings suggest that the Visual-Spatial Organization subscale consists of two related groups of items reflecting organization and visual-spatial skills. Although the medium correlation between these groups of items and the medium-to-large corrected item-to-total correlations of all items on the subscale (ranging from 0.31 to 0.60) support the aggregation of items into a single subscale score, the factor analysis result also indicates that the Visual-Spatial Organization subscale consists of two distinct but related factors. Therefore, the total subscale score for Visual-Spatial Organization is included in all of the analyses in this manuscript with the caveat that two related subgroups of items may exist on this subscale.

Table 3 provides values for internal consistency, inter-rater, and test–retest reliability. Internal consistency values were .79 or higher for all LEAF subscales with the exception of Visual-Spatial Organization (.69). Corrected item-to-total correlations for each LEAF item with its respective subscale score (e.g., item correlation with the sum of the other 4 items on each subscale) were large (>0.50) for 50 of the 55 LEAF items (a table with item-total correlations is available from the authors); for 3 items, item-to-total correlations were 0.40 – 0.49, and for 2 items, item-to-total correlations were 0.30 – 0.39. No LEAF item had a corrected item-to-total correlation of less than 0.30, and all corrected item-to-total correlations were statistically significant (p < 0.05).

Table 3.

LEAF Subscale Reliability and Validity

LEAF Subscale	Internal Consistency	Interrater Reliability Parent–Teacher, N = 85	Test-retest Reliability Mother 10 – 35 days, N = 27
Cognitive-Learning:
Comprehension and Conceptual Learning	.87	.46***	.82***
Factual Memory	.90	.47***	.74***
Cognitive-EF:
Attention	.92	.37***	.88***
Processing Speed	.79	.33**	.75***
Visual-Spatial Organization	.69	.54***	.77***
Sustained Sequential Processing	.86	.34***	.82***
Working Memory	.84	.28**	.78***
Novel Problem-Solving	.86	.37***	.83***
Academic:
Mathematics Skills	.92	.57***	.83***
Basic Reading Skills	.95	.51***	.88***
Written Expression Skills	.88	.41***	.88***

Note. Values for internal consistency are Cronbach’s α. Values for interrater and test–retest reliability are Pearson correlation coefficients.

^**p < 0.01;

^***p < 0.001

Parent–teacher inter-rater reliability reached statistical significance for all of the LEAF subscales, and most parent-teacher correlations were in the medium to large effect size range (e.g., correlations fell between 0.33 and 0.57 for 10 of the 11 LEAF subscales). Test–retest correlations for mothers completing two LEAF scales within the same 10 – 35 day period were statistically significant and ranged from 0.74 to 0.88 for all subscales (see Table 3).

Construct Validity

Behavior Checklists

Correlations between the LEAF, CHAOS, and BRIEF are reported in Table 4. The parent-reported LEAF Cognitive-Learning and Cognitive-EF subscales correlated significantly with CHAOS Attention Problems and BRIEF subscales in the expected direction for all correlations except one (LEAF Factual Memory with BRIEF Organization of Materials). Nearly 2/3 of the correlations exceeded r = .50. Importantly, parent-reported LEAF Cognitive-Learning and Cognitive-EF subscales showed especially large correlations with CHAOS or BRIEF subscales measuring corresponding domains of executive functioning: LEAF Attention with CHAOS Attention Problems (r = .84, p < .001) and BRIEF Working Memory (r = .74, p < .001); LEAF Visual-Spatial Organization with BRIEF Plan/Organize (r = .71, p < .001) and BRIEF Organization of Materials (r = .60, p < .001); LEAF Sustained Sequential Processing with BRIEF Plan/Organize (r = .67, p < .001); LEAF Working Memory with CHAOS Attention Problems (r = .70, p < .001) and BRIEF Working Memory (r = .67, p < .001); and LEAF Novel Problem-Solving with BRIEF Initiate (r = .61, p < .001). On the other hand, correlations for LEAF Cognitive-Learning and Cognitive-EF subscales that had no corresponding subscale on the CHAOS or BRIEF were generally much lower, although most were statistically significant: Comprehension and Conceptual Learning (all r < 0.55; median r = 0.50), Factual Memory (all r < 0.52; median r = 0.42), and Processing Speed (all r < 0.56; median r = 0.44). Correlations between the LEAF Academic subscales and CHAOS and BRIEF subscales were also substantially lower than correlations between the corresponding LEAF and CHAOS/BRIEF subscales. Most teacher-reported LEAF Cognitive-Learning and Cognitive-EF subscales also showed large correlations with teacher-reported BRIEF and CHAOS subscales (Table 4). As with parent-reported LEAF subscales, teacher-reported LEAF subscales typically showed the highest correlations with CHAOS and BRIEF subscales measuring the same (or similar) type of cognitive/executive function.

Table 4.

Relationships between LEAF, CHAOS, and BRIEF

LEAF Subscale	CHAOS Subscale		BRIEF Subscale
	Attention Problems		Initiate		Working Memory		Plan-Organize		Organization of Materials		Monitor
N	Parent 117	Teacher 74	Parent 114	Teacher 26	Parent 114	Teacher 26	Parent 114	Teacher 26	Parent 114	Teacher 26	Parent 114	Teacher 26
Cognitive-Learning:
Comprehension and Conceptual Learning	.54***	.49***	.47***	.54**	.52***	.60**	.45***	.38a	.22*	.22	.53***	.43*
Factual Memory	.41***	.39**	.43***	.51**	.51***	.55**	.45***	.35a	.16a	.16	.39***	.29
Cognitive-EF:
Attention	.84***	.79***	.59***	.82***	.74***	.85***	.57***	.80***	.39***	.60**	.61***	.82***
Processing Speed	.55***	.51***	.44***	.56**	.52***	.58**	.43***	.59**	.26**	.45*	.33***	.44*
Visual-Spatial Organization	.72***	.67***	.61***	.72***	.60***	.63**	.71***	.63**	.60***	.64***	.73***	.66***
Sustained Sequential Processing	.78***	.71***	.59***	.77***	.68***	.73***	.67***	.70***	.47***	.58**	.59***	.73***
Working Memory	.70***	.59***	.59***	.69***	.67***	.60**	.58***	.50**	.41***	.35a	.59***	.54**
Novel Problem-Solving	.56***	.47***	.61***	.61**	.49***	.48*	.49***	.33a	.29**	.18	.53***	.44*
Academic:
Mathematics Skills	.18a	.40***	.29**	.58**	.20*	.57**	.23*	.49*	.06	.31	.29**	.38a
Basic Reading Skills	.36***	.30*	.24*	.37a	.31**	.37a	.31**	.20	.20*	.08	.26**	.29
Written Expression Skills	.41***	.47***	.34***	.62**	.34***	.57**	.41***	.52**	.22*	.24	.44***	.52**

Note. CHAOS = Conduct-Hyperactive-Attention Problem-Oppositional Symptom. BRIEF = Behavior Rating Inventory of Executive Function. Complete sample includes 118 parent-completed and 85 teacher-completed LEAF forms. Values are Pearson correlation coefficients between behavior checklist measures completed by the same respondent.

^ap < 0.10;

^*p < 0.05;

^**p < 0.01;

^***p < 0.001

Neuropsychological Tests

Correlations between the LEAF Cognitive-Learning and Cognitive-EF subscales and individually-administered neuropsychological measures of executive functioning (Stroop Color-Word, CIT Number-Count, Conners’ CPT) are reported in Table 5. Five out of eight LEAF Cognitive-Learning and Cognitive-EF subscales (Comprehension and Conceptual Learning, Attention, Processing Speed, Visual-Spatial Organization, and Working Memory) were consistently correlated with Stroop Color-Word and CIT Number-Count scores. Conners’ CPT Reaction Time Standard Error scores were less consistently correlated with LEAF Cognitive-EF subscale scores, with small but significant correlations found only for the LEAF Attention and Working Memory subscales. For the WJ-III Tests of Achievement, LEAF Academic subscales (Mathematics, Basic Reading, and Written Expression) correlated most strongly with corresponding WJ-III composite and subtest scores (Calculation, Basic Reading, and Writing Samples; Table 6).

Table 5.

Relationships between Parent-Reported LEAF and Neuropsychological Measures of Executive Functioning

LEAF Subscale	Stroop	CIT	CCPT
	Color-Word	Number-Count	Reaction Time Standard Error
N	103	107	97
Cognitive-Learning:
Comprehension and Conceptual Learning	−.25*	.21*	.14
Factual Memory	−.18a	−.08	.18a
Cognitive-EF:
Attention	−.33**	−.30**	.22*
Processing Speed	−.24*	−.23*	.02
Visual-Spatial Organization	−.22*	−.25**	.16
Sustained Sequential Processing	−.31**	−.18a	.06
Working Memory	−.28**	−.27**	.25*
Novel Problem-Solving	−.18a	−.17a	.14
Academic:
Mathematics Skills	−.27**	−.22*	.32**
Basic Reading Skills	−.35***	−.33***	−.05
Written Expression Skills	−.43***	−.38***	.12

Note. Stroop = Stroop Color and Word Test. CIT = Counting Interference Test. CCPT = Conners’ Continuous Performance Test. Values are Pearson correlation coefficients.

^ap < 0.10;

^*p < 0.05;

^**p < 0.01;

^***p < 0.001

Table 6.

Relationships between the Parent-Reported LEAF and Measures of Academic Achievement

LEAF Subscale	WJ-III Tests of Achievement
	Basic Reading Skills Composite	Calculation Subtest	Writing Samples Subtest
N	111	112	106
Cognitive-Learning:
Comprehension and Conceptual Learning	−.35***	−.40***	−.48***
Factual Memory	−.43***	−.37***	−.41***
Cognitive-EF:
Attention	−.10	−.10	−.29**
Processing Speed	−.25**	−.12	−.32**
Visual-Spatial Organization	−.19*	−.18a	−.34***
Sustained Sequential Processing	−.15	−.14	−.30**
Working Memory	−.24*	−.28**	−.41***
Novel Problem-Solving	−.28**	−.30**	−.36***
Academic:
Mathematics Skills	−.34***	−.51***	−.35***
Basic Reading Skills	−.48***	−.23*	−.41***
Written Expression Skills	−.36***	−.30**	−.52***

Note. WJ-III = Woodcock-Johnson III Tests of Achievement. The WJ-III Basic Reading Skills score is a composite of the Letter-Word Identification and Word Attack subtests. Values are Pearson correlation coefficients.

^ap < 0.10;

^*p < 0.05;

^**p < 0.01;

^***p < 0.001

BRIEF and WJ-III Scores for LEAF Criterion-Referenced Interpretation Ranges

Table 7 depicts the means and standard deviations for BRIEF and WJ-III scores for participants falling into the three LEAF criterion-referenced interpretation ranges for each LEAF subscale. For each LEAF subscale, a corresponding BRIEF or WJ-III score was selected based on similarity of the construct measured: BRIEF Working Memory for LEAF Attention and LEAF Working Memory; BRIEF Organization of Materials for LEAF Visual-Spatial Organization; BRIEF Plan/Organize for LEAF Sustained Sequential Processing; BRIEF Initiate for LEAF Novel Problem-Solving; WJ-III Calculation for LEAF Mathematics Skills; WJ-III Basic Reading for LEAF Basic Reading Skills; and WJ-III Writing Samples for LEAF Written Expression Skills. Three LEAF subscales (Comprehension and Conceptual Learning, Factual Memory, and Processing Speed) did not have corresponding BRIEF or WJ-III subtest/subscale scores and were excluded from analysis.

Table 7.

BRIEF or WJ-III Scores Corresponding to Parent-Report LEAF Criterion-Based Interpretation Ranges

LEAF Subscale Problem		F	No Problem	Borderline Problem
(Corresponding BRIEF/WJ-III score)
Cognitive-Learning:
Comprehension and Conceptual Learning			N = 41	N = 53
N = 24		-----
(No Corresponding BRIEF subscale)
Factual Memory			N = 42	N = 53
N = 23	-----
(No Corresponding BRIEF subscale)
Cognitive-EF:
Attention			50.6 (9.6)a, N = 16	65.4 (8.4)b, N = 43
74.4 (8.8)c, N = 59	48.1***
(BRIEF Working Memory)
Processing Speed			N = 19	N = 52
N = 47	-----
(No Corresponding BRIEF subscale)
Visual-Spatial Organization			47.8 (11.1)a, N = 24	59.8 (8.8)b, N = 58
65.9 (6.5)c, N = 36	30.2***
(BRIEF Organization of Materials)
Sustained Sequential Processing			53.4 (9.2)a, N = 19	65.1 (10.0)b, N = 46
73.0 (8.9)c, N = 53	30.8***
(BRIEF Plan/Organize)
Working Memory			54.0 (10.4)a, N = 21	67.2 (10.4)b, N = 44
74.2 (8.0)c, N = 53	34.0***
(BRIEF Working Memory)
Novel Problem-Solving			53.8 (10.0)a, N = 43	64.7 (11.1)b, N = 46
70.5 (9.1)c, N = 29	25.0***
(BRIEF Initiate)
Academic:
Mathematics Skills			104.6 (16.2)a, N = 41	90.0 (15.4)b, N = 33
83.6 (17.7)c, N = 44	16.3***
(WJ-III Calculation)
Basic Reading Skills			102.8 (12.6)a, N = 58	94.9 (12.7)b, N = 32
85.7 (13.4)c, N = 28	16.2***
(WJ-III Basic Reading)
Written Expression Skills			107.4 (13.0)a, N = 27	96.5 (12.0)b, N = 36
87.1 (12.4)c, N = 55	21.9***
(WJ-III Writing Samples)

Note. Values (unless otherwise indicated) are mean (SD) for BRIEF or WJ-III subscale/subtest scores. df for F-tests is (2,111) for BRIEF subtests, (2,109) for WJ-III Calculation, (2,108) for Basic Reading, and (2,103) for WJ-III Writing Samples. N’s are for entire sample. Values with different superscripts are significantly different (p < 0.05) between LEAF Interpretation Ranges.

^***p < 0.001 for F-test

ANOVAs comparing participants in the three LEAF interpretation ranges were statistically significant (p < 0.001, Table 7) for all tested LEAF subscales. Follow-up pairwise t-tests showed that participants in the “No Problem Range” scored significantly lower than those in the other two ranges and that participants in the “Problem Range” scored significantly higher than those in the other two ranges (p < 0.05 for all pairwise comparisons). For the “No Problem” group on all LEAF subscales, mean BRIEF T-scores and WJ-III standard scores were within ½ SD of the normative mean, indicating average behaviors and skills. For the “Problem Group” on all LEAF subscales except Visual-Spatial Organization, mean BRIEF subscale scores were over 2 SDs above the normative mean (e.g., T > 70), and mean WJ-III scores were approximately 1 SD below the normative mean (e.g., Standard Score of approximately 85). These findings are indicative of problems in these areas measured by BRIEF and WJ-III norm-referenced scores. As expected, participants scoring in “Borderline Problem” LEAF ranges had BRIEF and WJ-III scores that fell between those for the “No Problem” and “Problem” groups (Table 7).

Discussion

EF is an umbrella term used to describe a broad cluster of related cognitive and emotional abilities necessary for self-directed goal attainment (Banich, 2009; Barkley, 2012). The present study provides evidence supporting the reliability and validity of the LEAF scale as a behavior checklist measure of broad executive and related functioning in children and adolescents. Reliability analyses demonstrated the unidimensionalty of subscales (with the possible exception of Visual-Spatial Organization, which showed a hierarchical structure of two sub-factors contributing to a larger global subscale score), strong internal consistency (Cronbach’s $\propto$ > .70 for all subscales except one with $\propto$= 0.69), adequate inter-rater reliability between parents and teachers (median r = 0.41for the 11 subscales), and strong test-retest reliability (median r = 0.82 for the 11 subscales). Validity analyses showed strong relations within respondents between LEAF subscale scores and corresponding subscale scores from other questionnaire measures of executive functioning (BRIEF and CHAOS). Relations between LEAF subscale scores and corresponding neuropsychological measures of executive and academic functioning were statistically significant in most cases.

Factor analyses of 10 of the 11 subscales produced results indicating that the subscales were reflected by a single dimension. These findings suggest that test items on each subscale measure the same underlying construct. Furthermore, internal consistency values were satisfactory for all LEAF subscales, especially considering the small number of items (5) per subscale. In developing LEAF subscales, we chose to emphasize internal consistency analyses instead of factor analyses for two reasons: First, the LEAF is intended for use as a clinically meaningful instrument. In order to maximize clinical utility, subscale content should match conceptually and clinically meaningful areas of EF and related abilities. As a result, we sought to retain the clinically meaningful subscale content as long as items were sufficiently interrelated (as shown by the Cronbach’s $\propto$ statistic). Second, the sample size was too small for a factor analysis of all 55 test items to be stable. Use of an internal consistency methodology and factor analysis of each subscale separately therefore provided statistical evidence of the homogeneity of subscale item content while also allowing us to retain conceptually and clinically meaningful groupings of items (Comrey, 1988).

For the Visual-Spatial Organization subscale, internal consistency and factor analysis results suggest that two related factors comprise the subscale score. One factor consisted of items reflecting organization skills and behavior in the environment, while a second factor comprised visual-organization skills in writing and puzzle construction. Most EF questionnaire subscales of organization skills ask only about organization of materials in the environment, but we chose to add questions about visual-spatial organization because of the close correspondence of organization of materials and visual-spatial organization in disorders of executive functioning and learning (Rourke, 1995). Our results suggest that these two subdomains of organization are related (e.g., factors correlated r = 0.37, and $\propto$ = 0.69) but distinct. Future research should investigate whether these two domains of organization are present in nonreferred samples and in samples with other clinical concerns such as hearing loss (Kronenberger, Beer, Castellanos, Pisoni, & Miyamoto, 2014). Based on this research, it may be warranted to investigate the two organization domains separately as well as to use the total score. Pending this additional research, scores on the Visual-Spatial Organization subscale should be interpreted with caution.

Correlations between the LEAF and corresponding subscales from the BRIEF and CHAOS indicate excellent construct validity across behavior checklists completed by the same respondents. Five of the eight LEAF Cognitive-Learning and Cognitive-EF subscales contained content that corresponded directly to subscales on the BRIEF and CHAOS; correlations between these corresponding subscales were large. The remaining three LEAF Cognitive-Learning and Cognitive-EF subscales contained content that overlapped much less with BRIEF or CHAOS subscales (Comprehension and Conceptual Learning, Factual Memory, and Processing Speed), and, as expected, correlations of these LEAF subscales with BRIEF and CHAOS subscales were generally weaker.

Most LEAF Cognitive-EF subscales correlated significantly with neuropsychological measures of EF, particularly for Stroop-like tasks (SCWT and CIT). Neuropsychological measures of EF tend to be correlated modestly, if at all, with results obtained from observation, interview, or questionnaire measures of everyday behavior (Barkley, 2012). The correlations found between the LEAF Cognitive-EF subscales and the Stroop scores in this study were in the medium range, consistent with findings for other behavioral observation, interview, or questionnaire measures (Barkley, 2012), supporting the validity of LEAF subscales as satisfactory measures of EF. Correlations between the LEAF Academic subscales and results from the WJ-III Tests of Achievement were also quite high for corresponding academic areas (r = −0.48 to −0.52), providing additional converging support for the value of the LEAF behavior checklist as a screening measure of academic problems in children with suspected EF delays.

In order to facilitate interpretation of LEAF raw scores, three criterion-referenced interpretation ranges were created based on the anchors for response choices of LEAF items. Specifically, LEAF subscale raw scores of 0 – 4 fall within the “No Problem Range” and indicate that the average item was answered with a response choice of less than “1” per item (e.g., with 5 items per scale, a score of 4 or less would indicate that the average item score is 0.80 or less). LEAF subscale raw scores of 5 – 9 fall within the “Borderline Problem Range” and indicate that the average item was answered with a response choice of at least “1” per item but less than “2” per item. Finally, scores of 10 or greater fall within the “Problem Range” and indicate a per-item average rating of “2”. We caution clinicians that interpretation of the LEAF should take into account that different patterns of item endorsement might result in the same subscale raw score, therefore attention to individual item scores is important for subscales that fall within the “Borderline Problem Range” or “Problem Range.”

These three LEAF criterion-referenced interpretation ranges were validated in analyses using BRIEF and WJ-III norm-referenced subscale/subtest scores. Participants in the “No Problem Range” consistently scored very close to the normative mean on BRIEF and WJ-III scores, whereas those in the “Problem Range” scored 1 – 2 SD on average in the direction of problems on BRIEF and WJ-III scores. Participants in the “Borderline Problem” range fell between these extremes and differed significantly from both of the extreme groups on BRIEF and WJ-III scores. Thus, the LEAF criterion-referenced interpretation ranges corresponded as expected to norm-based scores on well-established measures of similar constructs and differed significantly from each other on norm-referenced scores from these measures.

As no behavior checklist is free of error, several limitations should be taken into account when interpreting the present results. First, inter-rater reliability analyses showed moderate agreement (correlations in the medium to large range) between parents and teachers on all LEAF subscales. Correlations between parent and teacher behavioral ratings on behavior checklists are universally found to fall in this range and may reflect how the child’s behavior changes between the home and school environments (Gioia et al., 2000). Future research is recommended to investigate additional factors contributing to parent-teacher differences in rating child EF, factors influencing teacher ratings of EF, and how teacher-reported LEAF scores compare to other behavior checklists and neuropsychological assessments. In the present study, we found large correlations between teacher-ratings on the LEAF and corresponding teacher-ratings on other EF-related behavior checklists, and further investigation of EF behaviors in the classroom may reveal additional clinically-relevant contributors to and sequelae of EF delays as reported by teachers.

Second, although the internal consistency of LEAF subscales was high (Cronbach’s $\propto$ ranged from .69 to .95) and factor analyses supported the unidimensional nature of 10 of the 11 LEAF subscales, the Visual-Spatial Organization subscale was found to consist of two related groups of items. Additional investigation and replication of this result will be important for understanding and supporting this subscale. Third, while factor structure was evaluated at the individual subscale level, the sample was too small to evaluate the factor structure of all 55 LEAF items in a single analysis. Further research is needed with a larger sample of children to determine the factor structure of all LEAF items. Finally, a limitation of the present study was the lack of a nonreferred normative sample. Although the LEAF criterion-referenced interpretation ranges were supported by analyses showing correspondence to BRIEF and WJ-III norm-referenced scores, norms for the LEAF will provide additional information about the degree to which scores are abnormal compared to typically developing children and adolescents. We are currently in the process of obtaining samples of nonreferred children and adolescents to address this question.

In summary, delays and disturbances in executive functioning are associated with several pediatric disorders affecting brain functioning and can contribute to at-risk long-term outcomes, particularly related to learning and higher-order cognitive processing (Kronenberger et al., 2014). Therefore, early identification and intervention is critically important in pediatric clinical settings. The LEAF is a broad behavior checklist of EF and related learning and academic skills that was developed to augment behaviorally-based performance assessments of EF and learning in children and adolescents and to serve as a screening tool for possible problems in EF. As such, it may serve as a helpful clinical instrument for pediatric neuropsychologists to encourage screening for EF and related problems in at-risk populations and to provide additional information to complement the results of conventional neuropsychological assessment of EF, yielding broader multisource-multitrait data. The clinical utility of the LEAF is enhanced by several key factors: it is a freely accessible brief behavior checklist that is easy to administer, score, and interpret. For example, most other EF checklists have varying numbers of items per subscale and have items distributed randomly throughout the checklist. This makes the checklist difficult to score without a scoring key. The LEAF, however, has the same number of items (5) per subscale, does not have a very large number of items per subscale, groups items by subscale, and uses criterion-referenced scores in order to allow for very efficient use without scoring keys or computers (e.g., see Levy et al., 2013). When interpreting results from the LEAF it is important to note the content and goals of the scale – to provide a reliable and valid measure of cognitive EF and related learning skills. Our intent was not to replicate the content of the BRIEF or other broad EF scales; therefore, the LEAF does not include EF measures of emotional control and behavioral inhibition.

Future research should address characteristics of the LEAF in larger and more diverse populations to investigate normative scores and characteristics associated with different pediatric and neuropsychological conditions. As noted earlier, additional research concerning teacher-report LEAF scales may also enhance the use of the LEAF as a teacher-report measure. Finally, in order to better understand the interrelations of LEAF items and subscales, investigation of the higher-order factor structure of LEAF subscales and items, using much larger samples, is an important next-step for understanding LEAF psychometrics.

Appendix

Learning, Executive, and Attention Functioning (LEAF) Scale Items and Instructions

Instructions: Please answer the following questions based on this child’s behavior during the LAST WEEK. Please circle your answers, and answer all questions (make your best guess if you are uncertain).

Never (Not a problem; Average for age)	Sometimes (A little more than average; Not a big problem)	Often (Causes problems; Happens almost everyday)	Very often (Major daily problem)
Cognitive Learning:
Comprehension and Conceptual Learning
0	1	2	3	Doesn’t seem to understand things that are said to him/her
0	1	2	3	Has difficulty following long conversations or explanations.
0	1	2	3	Poor comprehension of reading material.
0	1	2	3	Doesn’t “get the point” of what is being said.
0	1	2	3	Doesn’t really understand new learning materials.
Factual Memory
0	1	2	3	Difficulty memorizing information.
0	1	2	3	Doesn’t retain facts well.
0	1	2	3	Has poor memory.
0	1	2	3	Forgets things that he/she has just learned.
0	1	2	3	Remembers the main idea but forgets details.
Cognitive-EF:
Attention
0	1	2	3	Poor attention span.
0	1	2	3	Mind seems to drift or wander when he/she is suppose to concentrate.
0	1	2	3	Does not stay focused on learning material.
0	1	2	3	Easily distracted.
0	1	2	3	Doesn’t listen when others are teaching or talking to him/her.
Processing Speed
0	1	2	3	Has trouble completing work quickly, even when motivated to do well.
0	1	2	3	Works deliberately and slowly on schoolwork and homework.
0	1	2	3	Writes and/or reads slowly.
0	1	2	3	Needs extra time to complete tests or other work.
0	1	2	3	Is slow to get started on things.
Visual-Spatial Organization
0	1	2	3	Poor organization.
0	1	2	3	Room, desk, locker, work area, etc. is very messy.
0	1	2	3	Not very good with puzzles or putting things together.
0	1	2	3	Drawing and/or handwriting is poor.
0	1	2	3	Doesn’t pay attention to visual details in the environment.
Sustained Sequential Processing
0	1	2	3	Doesn’t plan ahead.
0	1	2	3	Doesn’t learn from punishment or other past experiences.
0	1	2	3	Has trouble with long assignments or multistep directions.
0	1	2	3	Loses track of step-by-step directions.
0	1	2	3	Doesn’t complete tasks in proper order; Haphazard in approaching problems.
Working Memory
0	1	2	3	Can’t do more than one thing at a time.
0	1	2	3	Gets overwhelmed if required to learn or attend to a lot of information.
0	1	2	3	If distracted, loses track of what he/she was doing/learning.
0	1	2	3	Forgets things that he/she knew how to do a few hours or days before.
0	1	2	3	Gets upset or “shuts down” when challenged with learning.
Novel Problem-Solving
0	1	2	3	Struggles when learning new materials.
0	1	2	3	Does not solve problems independently (needs help on new problems).
0	1	2	3	Resists learning anything that is unfamiliar, new, or different.
0	1	2	3	Has difficulty with new situations, new people, or unfamiliar settings.
0	1	2	3	Avoids new experiences.
Academic:
Mathematics Skills
0	1	2	3	Math is difficult for him/her.
0	1	2	3	Slow at math.
0	1	2	3	Makes mistakes during arithmetic calculations or counting.
0	1	2	3	Prefers reading and language subjects to math.
0	1	2	3	Takes a long time to learn new mathematics operations or concepts.
Basic Reading Skills
0	1	2	3	Reading is slow.
0	1	2	3	Has trouble sounding out new words when reading.
0	1	2	3	Reading is hesitant or choppy; doesn’t read smoothly.
0	1	2	3	Makes mistakes when sounding out or pronouncing words in reading.
0	1	2	3	Skips or mistakes words during reading.
Written Expression Skills
0	1	2	3	Writing is slow.
0	1	2	3	Written expression is very simple (basic, immature).
0	1	2	3	Makes errors in grammar, punctuation, or spelling when writing sentences.
0	1	2	3	Has difficulty explaining things or expressing self in writing.
0	1	2	3	Struggles with subjects that require writing.

Note. Subscales are indicated on this version of the LEAF for illustrative purposes. Please contact the authors for the administration version of the LEAF scale.

Criterion-referenced interpretation ranges (0 – 4 = “No Problem Range”; 5 – 9 = “Borderline Problem Range”; 10 – 15 = “Problem Range”)