Complementary Assessments of Executive Function in Preterm and Full Term Preschoolers

Abstract

Executive functions (EFs) are interrelated cognitive processes that have been studied in relation to behavior, attention, academic achievement, and developmental disorders. Studies of EF skills assessed through parent report and performance-based measures show correlations between them ranging from none to modest. Few studies have examined the relationship between EF skills measured through parent report and performance-based measures in relation to adaptive function. The present study included preschool children born preterm as a population at high-risk for EF impairments. Preschool children (N = 149) completed a battery of EF tasks that assess working memory, response inhibition, idea generation, and attention shifting or cognitive flexibility. Parents reported on children’s EF and adaptive skills. Preterm children showed more parent-rated and performance-based EF impairments than did full term children. The combined use of either parent report or performance-based measures resulted in identification of a large number of children at risk for EF impairment, especially in the preterm group. Both parent report and performance-based EF measures were associated with children’s adaptive function. EF skills are measurable in young children, and we suggest that EF skills may serve as targets for intervention to improve functional outcomes. We recommend the use of both parent report and performance-based measures to characterize children’s EF profiles and customize treatment.

Executive function (EF) is an umbrella term referring to complex, interrelated skills used to direct goal-oriented behavior. These skills include inhibitory control (i.e., resisting an automatic response in order to do what is requested or needed), working memory (i.e., holding information on-line despite competing information or while manipulating other information), cognitive flexibility (i.e., adjusting to changed demands; switching between rules or tasks), and organization, sequencing, and planning (P. Anderson, 2002; Miyake, Friedman, Emerson, Witzki, & Howerter, 2000). EF has been associated with other domains, including attention, behavior, academic achievement, and social function (Blair & Razza, 2007; Cameron et al., 2012; Diamantopoulou, Rydell, Thorell, & Bohlin, 2007; Ganesalingam et al., 2011). EF impairments are found in several neurodevelopmental disorders, including attention-deficit/hyperactivity disorder (ADHD), autism, and learning disorders (Gathercole & Alloway, 2006; Gioia, Isquith, Kenworthy et al., 2002). EF skills have been linked to brain structures and neural networks in several neuropsychological and neuroimaging studies of typical and atypical development (P. Anderson, 2002; Durston & Casey, 2006; Luna, Padmanabhan, & O’Hearn, 2010; Warren et al., 2013). Studies of children with neurodevelopmental disorders and known brain injuries have the potential to elucidate the biological basis of EF impairments. Given that there are associations between EF and later outcomes (Blair & Razza, 2007; Cameron et al., 2012; Diamantopoulou et al., 2007; Ganesalingam et al., 2011; Jacobson, Williford, & Pianta, 2011), evaluation of EF at young ages offers the opportunity to identify children at greatest risk for later disorders and possibly to intervene to improve outcomes. Therefore, the evaluation of EF skills at an early age is critically important. Our principal aims in this study were to assess the relationship between parent report and performance-based EF and to determine the contributions, if any, of each type of EF measure to adaptive function in preschool children. We used a group of children born preterm as a biologic model of high-risk for EF impairments in comparison to children born full term.

Sequelae of preterm birth include impairments in EF and related domains (Aarnoudse-Moens, Weisglas-Kuperus, Van Goudoever, & Oosterlaan, 2009; Aylward, 2005). Studies of preschoolers and kindergartners born preterm find impairments in working memory (Baron, Kerns, Muller, Ahronovich, & Litman, 2012; Espy et al., 2002), inhibitory control (Baron et al., 2012; Edgin et al., 2008), and planning, switching and attention (P. J. Anderson, Doyle, & Group, 2004; Edgin et al., 2008; Woodward, Clark, Pritchard, Anderson, & Inder, 2011). One study found that the use of disparate EF measures in preterm preschoolers resulted in limited ability to conduct meta-analysis (Mulder, Pitchford, Hagger, & Marlow, 2009), although deficits on verbal fluency (Baron, Erickson, Ahronovich, Baker, & Litman, 2011) and shifting tasks were identified (Mulder et al., 2009).

Children born preterm are at risk for EF impairments, in part, on the basis of neural injury (Rose, Feldman, Jankowski, & Van Rossem, 2011). The preterm brain is highly vulnerable to insults in the neonatal period, and injuries, such as periventricular hemorrhage and leukomalacia and diffuse white matter injury, likely contribute to multiple adverse outcomes (Volpe, 2001, 2009). Damage may occur in several regions that subserve EF skills, such as prefrontal and parietal lobes, cerebellum, and subcortical regions, as well as in connections among these regions (Volpe, 2001, 2009; Woodward, Edgin, Thompson, & Inder, 2005). A few studies using structural MRI have found associations between white matter injury and EF skills in preterm toddlers and preschoolers (Edgin et al., 2008; Woodward et al., 2011; Woodward et al., 2005).

EF skills in children can be measured using either (1) parent or other caregiver/teacher reports on standardized rating scales and (2) performance-based measures with tasks that tap core EF constructs. Standardized measures such as the Behavior Rating Inventory of Executive Function-Preschool Version (BRIEF-P) show good psychometric properties, with content validity based on factor analysis of clinical and normative samples, convergence/discriminance with preschool rating scale measures, and on the ability to detect EF deficits in children with risk factors or disorder (Gioia, Isquith, Retzlaff et al., 2002). Developmentally sensitive, performance-based EF measures have been specifically designed for use in preschoolers (Carlson, 2005; Garon et al., 2008).

A few studies of preterm children describe both parent rating and performance-based EF measures (P. J. Anderson et al., 2004; Ritter, Perrig, Steinlin, & Everts, 2013; Scott et al., 2012). Two studies of extremely low birth weight/extremely preterm children (< 1000 g, < 28 weeks gestational age) at ages 5 and 8 years found deficits across several EF domains using both types of measures (P. J. Anderson et al., 2004; Scott et al., 2012). A study of very low birth weight/very preterm (< 1500g, < 32 weeks gestational age) school-age children found that parent ratings of working memory were significantly associated only with performance measures of working memory and shifting; parent ratings of inhibition and shifting were not associated with any EF performance measures (Ritter et al., 2013). Studies of typical and other clinical populations that use parent ratings of EF and performance-based measures of EF show correlations between some, but not all EF constructs; in some cases, EF measurements correlate, but are not aligned, i.e., a parent-rating for one EF construct correlated with a performance-based measure of a different EF construct (P. Anderson, 2002; Payne, Hyman, Shores, & North, 2011; Toplak, Bucciarelli, Jain, & Tannock, 2009). Some studies find insignificant (Mahone & Hoffman, 2007) or modest (Payne et al., 2011) correlations between parent ratings and performance-based measures of EF.

Explanations for the discrepancy between parent reports and performance-based EF measures often highlight the differences in ecological validity between the two approaches. Parent ratings provide a global view of children’s behaviors in everyday environments, placing the use of EF skills in the context of naturalistic settings. Performance-based measures often attempt to isolate a single EF construct, in an effort to get a “pure” measurement in a controlled, one-on-one setting. Although EF tasks often demonstrate face validity for the construct being measured and identify group differences in EF performance, discrepancies with parent report may reflect the child’s ability to demonstrate the skill in a controlled environment, but lack of ability in a natural environment filled with distractions and increased task demands. Proponents of rating scales argue that such ratings are more ecologically valid and related to adaptive or occupational function (Barkley & Fischer, 2011; Barkley & Murphy, 2010). Reliance on parent ratings alone, however, may introduce bias. For example, parent ratings of both EF and other functional outcomes may introduce bias due to shared response bias in parent perception, i.e., viewing a child’s skills as uniformly negative or positive. Parent ratings may also be biased or skewed by other factors, such as parental mental health. The majority of authors studying both parent ratings and performance-based measures have argued that each approach is important, capturing divergent but related information (P. Anderson, 2002; Isquith, Roth, & Gioia, 2013; Ritter et al., 2013; Silver, 2014; Toplak et al., 2009).

If EF skills, whether measured by parent ratings or performance-based measures, are related to functional outcomes, then the evaluation of EF skills in children becomes particularly relevant to early identification of developmental disorders. In typical and clinical populations, EF skills have been shown to correlate with academic performance, including reading and math (Blair & Razza, 2007; Cameron et al., 2012; Diamantopoulou et al., 2007; Jacobson et al., 2011), behavior problems/symptoms (Brocki, Eninger, Thorell, & Bohlin, 2010; Wahlstedt, Thorell, & Bohlin, 2008), social skills (Ganesalingam et al., 2011) and overall adaptive function (Cahn-Weiner, Boyle, & Malloy, 2002; Clark, Prior, & Kinsella, 2002; Gilotty, Kenworthy, Sirian, Black, & Wagner, 2002; Loe, Lee, Luna, & Feldman, 2012; Taylor, Klein, Drotar, Schluchter, & Hack, 2006).

To date, studies of school age preterm children have focused primarily on the links between EF skills and academic achievement (Downie, Frisk, & Jakobson, 2005; Mulder, Pitchford, & Marlow, 2010; Rose, Feldman, & Jankowski, 2011), including reading (Frye et al., 2009) and math (Aarnoudse-Moens, Weisglas-Kuperus, Duivenvoorden, Van Goudoever, & Oosterlaan, 2013). Only a few studies of preterm children have examined the links between EF skills and other important functional outcomes, such as overall adaptive function (Loe et al., 2012; Taylor et al., 2006), at school age. Very few of the studies of EF and outcome in preschoolers born preterm have linked EF with functional outcomes. A recent study of extremely preterm/extremely low birth weight kindergarten children found that performance-based measures of EF were related to diagnoses of ADHD-Inattentive, ADHD-Combined, enuresis, and parent-ratings of EF performance (Scott et al., 2012); however, other functional outcomes in academic achievement (Taylor et al., 2011) were reported separate from EF impairments (Orchinik et al., 2011).

Our study had the following goals: (1) to characterize EF skills in preterm compared to full term preschoolers using an age-normed parent rating scale of EF; (2) to characterize age and group differences in EF skills on a performance-based battery of tasks that tap several EF component skills; (3) to examine the strength of associations between parent ratings and performance-based measures and to compare the ability of composite scores from each method to identify children with EF impairments; and (4) to assess whether composite measures of parent-rated and performance-based EF skills contribute to the variance in adaptive skills as a measure of overall function.

We hypothesized that (1) compared to full term children, preterm children would show poorer EF skills on parent ratings of EF; (2) younger children would perform more poorly than older children, and preterm children would perform more poorly than full term children on performance-based measures of EF; (3) parent ratings and performance-based measures would show small but significant correlations; in addition to identifying children with EF difficulty on both sets of measures, each method would identify unique children with EF difficulty undetected by the other method; and (4) both parent rated and performance-based measures of EF skills would each contribute to the variance in children’s adaptive skills or overall function.

METHODS

Participants

Participants were 3 to 5 year old children recruited from Palo Alto, CA, and surrounding communities. A convenience sample, the children were born between 2004–2009. Preterm subjects were recruited by letters sent to families of children who were evaluated at High Risk Infant Follow-up Services at Lucile Packard Children’s Hospital at Stanford in Palo Alto, CA and by postings on local parent message boards. Control children were recruited by postings on parent message boards, by flyers in general pediatric clinics, and by word of mouth. Gestational age, birth weight, and medical complications were gathered from parent report and medical records. Study subjects had a history of preterm birth (≤ 34 weeks gestation) and birth weight < 2500 g (n = 70). Controls were born full term (≥ 37 weeks) and had no major medical complications (n = 79). Exclusion criteria for all participants included sensory impairments (i.e., blind or deaf), identified genetic syndrome or congenital heart disease, and inability to comprehend task instructions. Medical complications at birth in the preterm group included: 11 had abnormal findings on head ultrasound or MRI (at least grade 2 intraventricular hemorrhage or IVH, echodensities, or cystic lesions) and 10 had mildly abnormal findings (grade 1 IVH or choroid plexus cyst); 40 had RDS and 7 developed chronic lung disease; 5 had necrotizing enterocolitis; and 8 were small for gestational age (defined as lying at or below the 3^rd percentile in birth weight for gestational age).

Controls were group-matched to children born preterm for age, gender, ethnicity, and race. See Table 1. Maternal education (3 categories: < 4-year college degree, 4-year college degree, master’s degree or higher) was used as the measure of socioeconomic status (SES); compared to the preterm group, a larger proportion of the full term group was in the highest SES group. Although fewer data were available, paternal education showed the same pattern of results. Forty-one preterm children (60%) received early intervention compared to no full term children. There were no differences between preterm and full term groups in the number of children attending daycare, preschool, or kindergarten. Mean IQ scores were in the average range, but significantly lower in the preterm compared to full term group.

TABLE 1.

Participant characteristics

	Preterm (n = 70)		Full Term (n = 79)
Participant Characteristicsa	M (SD) or #/category	Range or %	M (SD)	Range or %	t or X2	p
Age (years)	4.6 (.76)	3–5.9	4.4 (.81)	3.1–5.9	−1.5	.14
Age category (3-, 4-, 5 years) (number)	20:30:20	29-, 43-, 29%	31:29:19	39-, 37-, 24%	1.9	.39
Perinatal Data
GA (weeks)	29.6 (2.6)	24–34	39.3 (1.4)	37–42	28.4	<.001*
Birthweight (g)	1365 (455)	468–2495	3334 (507)	2239–4510	24.8	<.001*
Demographicsb	Preterm		Full Term
Race, n (%)					.064	.8
White	42 (60)		49 (62)
Nonwhite	28 (40)		30 (38)
Ethnicity, n (%)					1.5	.48
Nonhispanic/Nonlatino	47 (67)		60 (76)
Hispanic/Latino	8 (11)		6 (7)
Mixed	15 (21)		13 (17)
Gender, n (%)					2.4	.118
Male	40 (57)		35 (44)
Female	30 (43)		44 (56)
Maternal Education, n (%)					11.9	.003*
< 4 year college degree	21 (30)		7 (9)
4 year college degree	23 (33)		27 (34)
MA or higher	26 (37)		45 (57)
Paternal Education, n (%)					10.3	.006*
< 4 year college degree	11 (17)		5 (7)
4 year college degree	25 (39)		16 (22)
MA or higher	29 (45)		52 (71)
Maternal Age	39.2 (5.6)		38.6 (4.9)		−.63	.531
Standard Scores	M (SD)	Range	M (SD)	Range	t	p
IQa	101.9 (15)	73–127	110.2 (13)	82–139	3.7	< .001*

Note:

^aData analyzed by t-test.

^bData analyzed by chi-square (asymptotic or exact significance: 2-sided).

^*p < .05.

The study was approved by the Stanford University Institutional Review Board. A parent or legal guardian provided informed consent. Participants were compensated for participation.

Measures and Variables

Participant characteristics

Demographic information included race, ethnicity, maternal and paternal education, household composition, and parent report of services and preschool/kindergarten experience.

Parent rated executive function

Parents completed the Behavior Rating Inventory of Executive Function-Preschool version (BRIEF-P), a standardized parent-rating scale of behavioral manifestations of EF in children 2.0–5.11 years (Gioia et al., 2000; Isquith, Gioia, & Espy, 2004). Sixty-three items measure aspects of executive function and are reported as five scales: Inhibit, Shift, Emotional Control, Working Memory, and Plan/Organize. Three broad indexes comprised from combinations of the scales (Inhibitory Self-Control as the sum of Inhibit and Emotional Control; Flexibility as the sum of Shift and Emotional Control; Emergent Metacognition as the sum of Working Memory and Plan/Organize) and a composite score (Global Executive Composite, GEC, as the sum of all 5 scales) are also reported. Items are rated on a 3-point scale of “never,” “sometimes,” or “often.” Scores are reported as T-scores (mean of 50, SD of 10) in which higher scores indicate greater executive dysfunction. Scores at or above 65 are considered clinically significant. The measure is normed for gender and age by two groupings: 2:0 to 3:11 and 4:0 to 5:11. Internal consistency coefficients are .85 to .9 for clinical scales and .95 for the composite score and test-retest (M = 4.5 weeks) coefficients are .78 to .9. Content validity is based on factor analysis of clinical and normative samples, convergence/discriminance with preschool rating scale measures, and on the ability to detect EF deficits in children with risk factors or disorder (Gioia, Isquith, Retzlaff et al., 2002). Items from the BRIEF-P inquire about the child’s behavior in relation to specific EF skills. For example, to evaluate the child’s inhibitory control, parents are asked whether the child acts out of control; for working memory, whether the child has trouble remembering, even after a short amount of time; and for planning and organization, whether the child puts things away in a random, disorganized way when cleaning up. We used the GEC as a summary measure of parent-rated EF skills.

Parent rated child adaptive skills

Parents completed the Vineland Adaptive Behavior Scales, Second Edition (Vineland II), Parent/Caregiver Rating Form (Sparrow, Balla, & Cicchetti, 2005). This well-known measure of adaptive skills from birth to adulthood generates standard scores (mean=100, SD=15), percentiles, and age equivalents in the following domains (subdomains): Communication (receptive, expressive, written); Daily Living Skills (personal, domestic, community); Socialization (interpersonal relationships, play and leisure time, coping skills); Motor Skills (gross, fine). A composite score, the Adaptive Behavior Composite, is also generated. We used the Adaptive Behavior Composite as a summary measure of children’s overall function.

Child intellectual ability

Full scale IQ was estimated using the two-subtest format of the Stanford-Binet Intelligence Scales, 5^th edition, an assessment of intelligence and cognitive abilities in individuals age 2–85+ years. Vocabulary and object-series/matrices subtests are combined to provide the Abbreviated Battery IQ (ABIQ) (Roid, 2003).

Performance-based EF skills

Children completed a battery of EF tasks. Tasks were selected to represent core EF constructs based on the developmental literature on EF in typical and preterm preschoolers (Aarnoudse-Moens, Smidts, Oosterlaan, Duivenvoorden, & Weisglas-Kuperus, 2009; Aarnoudse-Moens, Weisglas-Kuperus et al., 2009; Carlson, 2005; Garon et al., 2008; Mulder et al., 2009). During one testing session of approximately one hour, children completed a behavioral battery of EF tasks in the following fixed order:

1. 6-Boxes Task measures self-ordered working memory and planning (Landry, Smith, Swank, & Miller-Loncar, 2000). Six stationary boxes with different color lids are baited with a treat in view of the child. The child’s task is to find all treats in the least number of reaches by keeping track of the boxes that have already been searched. A 5-second delay is imposed after each reach. Since boxes remain stationary, the child can use appearance (lid color) or spatial location to keep track of which boxes have already been searched. Three boxes were presented first as a control task to ensure comprehension. Dependent variable: number of reaches to find all treats (continuous).

2. Verbal Fluency measures idea generation or verbal productivity with organizational components (i.e., the use of subcategories as a strategy results in increased productivity) (Lezak, Howieson, & Loring, 2004). The task requires the child to generate as many words as possible within specific categories (i.e., animals, foods) in one minute. Dependent variable: number of correct words (belongs to the category) (continuous).

3. Day/Night measures complex response inhibition, defined as response inhibition coupled with working memory (Gerstadt, Hong, & Diamond, 1994). The child must hold a rule in mind and respond while inhibiting a prepotent response, suppressing competing visual information when responding. The child should respond “night” when shown a picture of the sun and “day” when shown a picture of moon and stars. This test is analogous to the Stroop task. Dependent variable: number of correct responses reverse scored by subtracting the number of practice trials (continuous) (Carlson & Moses, 2001).

4. Bird/Dragon is a measure of complex response inhibition, defined as response inhibition and working memory (must also hold a rule in mind) (Reed, Pien, & Rothbart, 1984). The child should follow verbal instructions of one puppet (bird), but not the other puppet (dragon). The task is a modified Simon Says task. Dependent variable: total number of trials with correct responses reverse scored by subtracting the number of practice trials (continuous).

5. Dimensional Change Card Sort, measures cognitive flexibility/task switching and attention shifting (Zelazo, 2006). The child is shown cards depicting colored shapes that can be sorted according to color or shape. The child must sort according to one dimension and then shift to sort according to the other dimension. Dependent variable: Pass (5 or more of 6 cards correct) or fail on first dimension and second dimension (categorical); number of correct responses on the post-switch phase (continuous) for calculating z-score (see results).

6. Gift Delay (wrap) task measures simple response inhibition and delayed gratification (Kochanska, Murray, Jacques, Koenig, & Vandegeest, 1996). Child is asked not to look while the examiner wraps a present noisily for one minute. Dependent variable: Pass (no peeking) or fail (categorical).

Data Analysis

Data analysis was conducted in SPSS. Data from the BRIEF-P was missing for 4 preterm children. Data from the Vineland was missing for 5 full term and 9 preterm subjects. Data were missing from the EF battery as follows: one preterm child for the bird/dragon task (would not play); one preterm and one full term for the day/night task (technical failure); 14 preterm and 6 full term for the verbal fluency task (change in testing protocol); 3 preterm and 2 full term for the gift wrap (technical failure). There were no significant differences in demographic variables for preterm and full term groups completing the verbal fluency task compared to the larger study sample. Differences in numbers of subjects for each task are reflected in the tables, and degrees of freedom have been adjusted accordingly. Significance was set at p < .05.

RESULTS

Parent-Rated EF

BRIEF-P scores were analyzed with Mann-Whitney U-tests due to nonparametric distribution. As shown in Table 2, the preterm group compared to full term group had significantly higher scores, indicating more EF problems, for all BRIEF-P scores, including the GEC, indexes, and scales.

TABLE 2.

Group Differences in Parent-Rated Executive Function

	Preterm n = 66		Full Term n = 79
BRIEF Indices	M (SD)	Range	M (SD)	Range	Z	p
Global Executivea Composite (GEC)	54.3 (15.1)	31–95	43.6 (7.6)	33–67	−4.5	<.001*
Inhibitory Self-Control Index (ISCI)	52.8 (14.7)	34–99	44.9 (8.4)	35–90	−3.3	.001*
Flexibility Index (FI)	51.2 (11.8)	36–86	45.6 (7.4)	35–72	−3	.003*
Emergent Metacognition Index (EMI)	56 (15.1)	33–93	43.7 (7.8)	36–65	−5.2	<.001*
BRIEF Scales
Inhibit	53.1 (15.2)	34–102	45.5 (8.4)	36–83	−2.8	.005*
Shift	50 (10.7)	37–79	45.8 (7.4)	37–68	−2.3	.022*
Emotional Control	52.1 (12)	36–86	45.5 (7.9)	36–68	−3.5	<.001*
Working Memory	56.8 (14.4)	36–88	44.2 (7.7)	30–66	−5.7	<.001*
Plan/Organize	53.5 (15)	32–94	43.7 (8)	34–64	−4.1	<.001*
Proportionsb

	n (%) with T-score ≥ 60	n (%) with T-score ≥ 60	X2	p
GEC	22 (33)	4 (5)	19.5	<.001*
ISCI	17 (26)	5 (6)	10.5	001*
FI	12 (18)	4 (5)	6.3	.012*
EMI	27 (41)	5 (6)	25	<.001*
Inhibit	18 (27)	4 (5)	13.8	<.001*
Shift	10 (15)	6 (8)	2.1	.148
Emotional Control	14 (21)	6 (8)	5.6	.018*
Working Memory	28 (42)	4 (5)	29	<.001*
Plan/Organize	22 (33)	5 (6)	17.3	<.001*

Note:

^aData analyzed by t-test.

^bData analyzed by chi-square (asymptotic or exact significance: 2-sided).

^*p < .05.

Identification of Poor Performance on Parent-Rated EF

Poor performance on the BRIEF-P was defined as t-score ≥ 60, or 1 SD above the mean. A significantly greater proportion of the preterm compared to full term group had poor performance or impairment on all BRIEF-P outcomes, except for the Shift scale. See Table 2.

Performance-based EF Battery

For EF tasks with continuous outcome variables, we used ANCOVA to compare group differences (preterm vs. full term), controlling for maternal education as a measure of SES given the group differences in SES. Sex and age group (3-, 4-, and 5- years) were entered as between-group variables to examine group-by-sex and group-by-age category interactions. There were no main effects of sex or group-by-sex interactions; therefore, results for group differences are reported with age group as the between-group variable.

There were significant differences between preterm and full term groups for all EF tasks. See Table 3. All children were able to perform the 3-boxes control task. There were significant main effects of age category for the EF tasks as follows: bird/dragon, F(2, 141) = 14.9; day/night, F(2, 140) = 10.1; and verbal fluency, F(2, 122) = 18.6, all p < .001. There was a trend for main effect of age category for the working memory 6-boxes task, F(2, 142) = 2.6, p = .08. There were no group-by-age interactions for any of the tasks: for 6-boxes, F(2, 142) = .63, p = .53; for bird/dragon, F(2, 141) = .36, p = .701; for day/night, F(2,140) = .234, p = .79; and for verbal fluency, F(2, 122) = .227, p = .797. Maternal education was a significant covariate only for 6-boxes, F(1, 142) = 4.1, p = .046. Maternal education was not significant for bird/dragon, F(1, 141) = 1.3, p = .27; day/night, F(1, 140) = 1.6, p = .211; or verbal fluency, F(1, 122) = 2.4, p = .126.

TABLE 3.

Group Differences in Performance-Based Executive Function

EF Construct	Preterm n=70			Full Term n=79			F-tests for group status	Effect Size
Task (Continuous Outcomes)	Age (years)			Age (years)
	3	4	5	3	4	5
	Mean (SD)			Mean (SD)				η2
Working Memory
	n=20	n=30	n=20	n=31	n=29	n=19
3-boxes (No. reaches)	3.05 (.22)	3.03 (.18)	3.05 (.22)	3 (0)	3 (0)	3 (0)	F(1,142) = 3.1, p = .08	.022
6-boxes (No. reaches)	7.4 (1.8)	6.8 (1.1)	7.5 (2.6)	6.7 (1.2)	6.2 (.58)	6.3 (.82)	F(1,142) = 13.6, p < .001	.087
Complex Response Inhibition
	n=19	n=30	n=20	n=31	n=29	n=19
Bird/Dragon (No. Correct, reverse scored)	7 (6.3)	11.2 (4.5)	12.2 (3.3)	10.2 (4.8)	13.3 (2.3)	13.9 (.23)	F(1,141) = 12.8, p < .001	.083
	n=20	n=29	n=20	n=30	n=29	n=19
Day/Night (No. correct, reverse scored)	4.8 (7.6)	7.8 (5.6)	8.8 (5.7)	6.6 (5.6)	10.2 (3.9)	12.3 (2.6)	F(1,140) = 6, p = .012	.041
Idea Generation
	n=16	n=24	n=16	n=27	n=28	n=18	F(1, 129) = 6.53, p = .012	.051
Verbal Fluency	4.4 (3.3)	7.4 (4.3)	10.5 (4)	6.6 (4)	10.4 (3.6)	12.3 (6.6)
EF Construct	Preterm			Full Term			X2, p value	Effect Size
Task (Categorical Outcomes)								ϕ
Simple Response Inhibition	n = 67			n = 77
	No. (%)			No. (%)
Gift Wrap (fail)	29 (43%)			18 (23%)			6.5, p = .011*	0.21
Attention Shifting/Task Switching/Cognitive Flexibility	n = 70			n = 79
Card Sort 1st dimension (fail)	8 (11%)			5 (6%)			1.2, p = .271	.09
Card Sort 2nd dimension (fail)	32 (46%)			25 (20%)			11, p = .001*	.27

Note:

^*p < .05.

For EF tasks with categorical outcome variables, we first used chi-square to examine differences between groups in proportion of children with pass/fail status. A larger proportion of preterm children failed the gift wrap task compared to full term (43% vs. 23%). See Table 3. Logistic regression predicting to failure on the task included maternal education, sex, age, and group status in the model. Children were more likely to fail the gift wrap task (or peek) if they belonged to the preterm group (OR: 3.1; 95% CI 1.4–6.96), were boys (OR: 2.89; 95% CI 1.29–6.48), or younger in age (b = −.793, SE .28, p = .005). Maternal education was not significant (b = .347, SE .27, p = .204). For the card sort task, there were no differences in the proportion of preterm compared to full term children who were able to sort by the first dimension; however, significantly more preterm children failed to sort by the second dimension (46% vs. 20%). Logistic regression predicting to failure on the second dimension showed that the preterm group was significantly more likely to fail (OR: 3.0; 95% CI 1.38–6.35); sex (OR 1.46; 95% CI .691–3.1), age (b = −.442, SE .26, p = .08), and maternal education (b = −.438, SE .25, p = .08) were not significant.

Identification of Poor Performance on Performance-Based EF Measures

To identify poor performance on the EF performance-based measures, we converted continuous outcome measures to z-scores derived from performance (mean, SD) of full term subjects in the three age groups of 3, 4, and 5 years. Poor performance on a task was defined as z-score of 1 SD or more below the mean (≤ 16^th percentile) and assigned a score of −1. For the card sort task, z-scores were calculated from the number of cards sorted correctly by the second dimension. We summed the number of tasks with poor performance resulting in a range of scores from −5 (poor) to 0 (good). The preterm group had more children in poor performance categories than the full term group, overall X²(5) = 22.7, p < .001. Figure 1 shows the preterm curve skewed toward poor performance. There was a significantly greater proportion of preterm children (n = 18 or 26%) than full term children (n = 3 or 4%) with poor performance (defined as poor performance on 3-, 4-, and 5- tasks) compared to good performance (defined as poor performance on 0, 1-, and 2- tasks), X²(1) = 14.7, p < .001. An average z-score for the 5 EF tasks was calculated using a reverse sign for the 6-boxes task since more reaches were associated with poor performance; preterm children (M = −.825, SE .12) had significantly lower z-scores than full term (M = −.0036, SE .058), t(99) = 6.0, p < .001.

Figure 1.

Percent of preterm and full-term children as a function of performance categories.

Correlations within and between EF measures

We used Spearman zero-order correlations to examine within-measure associations. For the BRIEF-P, the GEC and BRIEF-P indexes were significantly correlated in the expected directions. BRIEF-P subscales of Inhibit, Shift, Emotional Control, Working Memory, and Plan/Organize were significantly correlated with each other, ranging from r = .48 to .84, all p < .001. For the performance-based EF measures, there were significant small to moderate correlations among all measures. See Table 4.

TABLE 4.

Zero-order correlations

Variable	1	2	3	4	5	6	7	8
1. GEC	1
2. ISCI	.915***	1
3. FI	.787***	.757***	1
4. EMI	.916***	.758***	.611***	1
5. 6-Boxes	.206*	.176*	.189*	.178*	1
6. Bird/Dragon	−.164	−.085	−.172*	−.198*	−.308***	1
7. Day/Night	−.017	.050	−.045	−.103	−.193*	.442***	1
8. Verbal Fluency	−.187*	−.121	−.151	−.189*	−.245**	.505***	.256**	1

Note:

^*p < .05;

^**p < .01;

^***p < .001

Abbreviations: GEC, global executive composite; ISCI, inhibitory self-control index; FI, flexibility index; EMI, emergent metacognition index

To examine the strength of associations between parent-rated EF and performance-based EF measures, we used Spearman zero-order correlations. The correlations among the GEC, BRIEF-P indexes, and performance-based EF scores are reported in Table 4. The GEC showed small, but statistically significant correlations with the 6-boxes and verbal fluency tasks. The inhibitory self-control index (ISCI) was significantly correlated with the 6-boxes task; the flexibility index (FI) was correlated with the 6-boxes and bird/dragon tasks; and, the emergent metacognition index (EMI) was correlated with the 6-boxes, bird/dragon, and verbal fluency tasks.

Comparison of Parent-Rated and Performance-Based EF Measures in Identification of Poor Performance

Given the significant correlations among BRIEF-P subscales, among performance-based EF measures, and the limited correlations between the BRIEF-P and the performance-based EF measures, we used the GEC from the BRIEF-P and the average z-score for performance-based tasks as summary measures. The correlation between the GEC and average z-score was r = −.176, p = .034. We identified children with poor performance on either the performance-based measures of EF (average z-score, ≤1 SD) or the BRIEF-P (GEC, ≥1 SD), or both. This approach allowed for unique identification of children with poor performance identified by one EF assessment method, but not the other. For the full term group, only 4 children had poor performance on the GEC, 2 children had poor performance based on average z-score, and none had poor performance on both. For the preterm group, 22 had poor performance on the GEC, 26 had poor performance based on average z-score, and 10 had poor performance on both measures.

Using the above cutoffs (1 SD for GEC and z-scores), we calculated sensitivity, specificity, and odds ratios for preterm group membership status as the condition of interest using scales (parent-ratings, GEC) alone, tests (performance-based measures, z-score) alone, both scales AND tests, and either scales OR tests. The odds ratio was not calculated for both scales and tests, since there were no full term participants with poor performance on both. See Table 5. All measures had low sensitivity and high specificity in detecting the condition of interest (preterm group membership). However, the use of scales OR tests approximately doubled the sensitivity in identifying true positive cases over the use of scales or tests alone. Use of scales OR tests also had the highest odds ratio—children with a score 1 SD above or below the mean for either the GEC or the z-score, respectively, were 26.5 times more likely to have the condition of interest.

TABLE 5.

Sensitivity, Specificity, and Odds Ratios for Preterm Status Based on Scores +/− 1SD

	Scales	Tests	Scales and Tests	Scales or Tests
Sensitivity (%)	33	37	15	69
Specificity (%)	95	98	100	92
Odds ratio	9.4	22.8	--	26.5

Associations with Adaptive Function

On the Vineland Adaptive Behavior Composite, a summary measure of overall child function, the preterm group (M = 96.8, SE 1.6) scored almost one standard deviation lower than the full term group (M = 111, SE 1.3), t(133) = 6.91, p < .001. To examine whether EF skills are associated with overall child function in the entire sample, we conducted hierarchical multiple linear regression predicting to the Vineland Adaptive Behavior Composite score. We used the GEC from the BRIEF-P and the average z-score for performance-based tasks as summary variables. In the model, maternal education as a measure of SES was entered in the first step. As shown in Table 6, maternal education was not significant (overall F(1, 131) = .073, p = .787, R² = .001). In the second step, average z-score for the performance-based EF tasks was entered after controlling for maternal education in the first step. Performance-based EF was related to overall child function, accounting for 19.2% of the variance, R² change = .192, F change_{1, 130} = 31, p < .001. The total variance explained significantly increased to 19.3% with overall F(2, 130) = 15.5, p < .001, R² = .193). In the third step, the BRIEF-P GEC score was entered, after accounting for maternal education and average EF z-score. We chose to enter the performance-based EF score and parent-rated EF scores in separate steps to identify the contribution of each to the variance in adaptive function. Both measures of EF were associated with overall child function, with parent ratings of EF contributing an additional 14.1% of the variance, R² change = .141, F change_{1, 129} = 27.3, p < .001. The total variance explained increased significantly to 33.4% with overall F(3, 129) = 21.6, p < .001, R² = .334.

TABLE 6.

Hierarchical Multiple Regression Model Predicting to Vineland Adaptive Behavior Composite Score

	B	SE B	β	p	Δ R2
Step 1
1. Maternal Education	.446	1.6	.024	.787	.001
Step 2
1. Maternal Education	.161	1.5	.009	.914
2. EF Z-score	6.365	1.1	.439	< .001*	.192
Step 3
1. Maternal Education	−.913	1.4	−.048	.507
2. EF Z-score	4.770	1.1	.329	< .001*
3. GEC	−.435	.083	−.396	<.001*	.141

Note:

^*p < .05.

Abbreviations: EF, executive function, GEC, global executive composite.

We repeated the model entering both performance-based and parent-rated EF simultaneously in the second step to account for only unique variance from each in predicting adaptive function. The results were the same as for the above step-wise entry, with EF measures contributing a collective 33.3% to the variance in outcome in the second step, R² change = .333, F change_{2, 129} = 32.3, p < .001. The total variance explained was the same as the stepwise model above – 33.4% with overall F(3, 129) = 21.6, p < .001, R² = .334.

Comparison of Parent-Rated and Performance-Based EF Measures in Identification of Low Adaptive Function

Using the same cutoffs (1 SD for GEC and z-scores) above for detecting the condition of interest, we calculated sensitivity, specificity, and odds ratios for low adaptive function group membership as the condition of interest using scales (parent-ratings, GEC) alone, tests (performance-based measures, z-score) alone, both scales AND tests, and either scales OR tests. We defined low adaptive function as an adaptive behavior composite score ≤ 90, or 1 SD below the mean for the entire sample. See Table 7. The use of scales OR tests increased the sensitivity in identifying true positive cases over the use of scales or tests alone. The use of tests alone had the highest odds ratio—children with a z-score 1 SD below the mean were 17.6 times more likely to be in the low adaptive function group.

TABLE 7.

Sensitivity, Specificity, and Odds Ratios for Low Adaptive Function Status Based on Scores +/− 1SD

	Scales	Tests	Scales and Tests	Scales or Tests
Sensitivity (%)	40	65	30	75
Specificity (%)	87	90	97	80
Odds ratio	4.4	17.6	15.7	11.7

Receiver operator curves (ROC) were computed to plot the sensitivities and specificities for the two conditions of interest described previously: (1) preterm group membership and (2) low adaptive function group membership. ROC curves were computed based on scales only, tests only, and a composite score based on scales and tests (GEC and average z-score). The area under the curve associated with preterm group status using scales only was .717; for tests only, .754; and for composite scales and tests, .81. The area under the curve associated with low adaptive function group status using scales only was .781; for tests only, .829; and for composite scales and tests, .875.

DISCUSSION

Our study of three- to five- year old preterm and full term preschool children went beyond the usual study of parent-rated and performance-based measures of EF by examining not only group and age differences, but also by examining how these EF measurements: (1) correlate with each other, (2) identify children with EF impairments, and (3) are associated with children’s overall function. We found that children born preterm showed impairments on a wide range of EF skills, whether based on parent report or measured directly. We found significant, but modest correlations between parent report and direct measures of EF; in addition, although some children performed poorly on both measures, other children showed impaired performance on one or the other. Another novel finding is that parent report and direct measures of EF both contributed unique variance to children’s overall function, highlighting the importance of using both methods as complementary assessments.

There were several group differences in EF. Using the BRIEF-P, scores on all scales and indexes were higher, indicating more EF problems, in the preterm compared to full term group. Parent ratings of EF also identified a larger proportion with impairments or poor performance in the preterm group on the GEC and all indexes compared to the full term group. Our results were similar to those of a recent study of almost 6-year old kindergarten children (Scott et al., 2012) and another study of 8-year old children with extremely low birth weight/extremely preterm gestation (< 1000 g, < 28 weeks) (P. J. Anderson et al., 2004). Using the same parent rating scale, the studies found significant differences between preterm and full term groups in BRIEF scores across all (Scott et al., 2012) or most (P. J. Anderson et al., 2004) scales. A study of very low birth weight/very preterm (< 1500g, < 32 weeks gestational age) children at age 8 to 12 years focusing on the scales of inhibit, working memory, and shift found that the preterm group differed from the full term group only on the working memory scale, although the preterm group had larger proportions of children than the full term group with both working memory and shifting impairment (Ritter et al., 2013). Our study included children with a wider range of birth weight and gestational age than these studies, indicating that parent rated EF impairments occur across the spectrum of preterm birth.

Using a comprehensive battery of EF tasks, we found significant group differences on all tasks and age effects for most tasks. Preterm children showed poorer performance than full term children on measures of working memory, complex response inhibition, verbal fluency, simple response inhibition, and attention/task shifting. Our findings are consistent with some studies that utilized multiple EF tasks or a comprehensive battery, although many other studies focus on children that are of extremely low birth weight or extremely preterm gestation (< 1000 g, < 28 weeks) (P. J. Anderson et al., 2004; Scott et al., 2012). Studies are heterogeneous in their findings of performance-based EF impairments—in some studies there are more global EF deficits with impairments on several tasks, whereas other studies show isolated EF impairments (Pozzetti et al., 2013). One study of preterm children found impairments only in cognitive flexibility at 24 months corrected age; working memory and inhibitory control were not different from full term children (Pozzetti et al., 2013). Similar to our study, the children had average IQ, no major brain damage, and were born across spectrums of gestational age and birth weight (< 34 weeks and < 2500 g). Disparities in EF impairments across studies may reflect differences in study samples or sensitivity of tasks at different ages and in different populations. Our study suggests that at preschool age, using a developmentally sensitive battery and parent report measures can identify children with EF impairments.

We found modest but statistically significant correlations between a parent-report composite score of EF and two performance-based EF tasks that assess working memory and verbal fluency. The emergent metacognition index, comprised of working memory and plan/organize subscales, correlated with three performance-based tasks. Our results were similar to another study of school age children with history of very low birth weight/very preterm gestation (< 1500g, < 32 weeks), which found that parent ratings of working memory were significantly associated with performance measures of working memory and shifting; parent ratings of inhibition and shifting were not associated with any EF performance measures (Ritter et al., 2013). Studies of other clinical populations, such as children with ADHD (Toplak et al., 2009), neurofibromatosis (Payne et al., 2011), and brain disease, including early treated phenylketonuria, early treated hydrocephalus, and frontal focal lesions (V. A. Anderson, Anderson, Northam, Jacobs, & Mikiewicz, 2002), which correlate parent ratings of EF and performance-based measures, also show mixed results.

Past studies highlight differences in ecological validity as the explanation for discrepancies between parent report and performance-based measures. One novel finding from this study is that both performance-based and parent-rated EF contributed unique variance to overall child function, as measured by the Vineland Adaptive Behavior Composite score. The results are consistent with both cross-sectional and longitudinal studies on associations of EF with functional outcomes in academic, social, and adaptive function domains. We believe that our findings suggest that both types of measures are ecologically valid, despite showing modest or limited correlations among individual EF constructs measured by parent ratings or direct observation.

Other researchers have proposed divisions of EF into “cold” and “hot” functions and have aligned these functions with neuroanatomical correlates to explain differences in EF measurement (V. A. Anderson et al., 2002; Payne et al., 2011). For example, the dorsolateral prefrontal cortex may be involved with cognitive aspects of EF, or “cold” functions such as working memory, attention, and organization skills, whereas the orbitofrontal or dorsomedial regions may be involved in emotional and social skills, and may be associated with “hot” functions, such as impulse control, response inhibition, and social cognition (V. A. Anderson et al., 2002; Payne et al., 2011). If the BRIEF-P is tapping hot aspects of EF and the performance-based measures are tapping cold aspects of EF, then this may result in weak associations between the measures (V. A. Anderson et al., 2002; Payne et al., 2011). The correlations among scales and tests in our study showed that the emergent metacognition index, comprised of “cold” working memory and plan/organize subscales, correlated with the largest number of EF tests. The other two indexes, flexibility and inhibitory self-control, both include the “hot” subscale of emotional control. We believe that the subdivision of EF into “hot” and “cold” functions and alignment with parent report or performance-based EF tasks is inherently limited since there is overlap among EF constructs and among different types of EF measures, whether assessed through parent report or performance-based measures.

Imaging studies have found associations between neuroanatomical structures and both parent-rated and performance-based EF measures. Studies using structural MRI have found associations between brain integrity and ratings of everyday EF, such as smaller frontal lobe volume and poorer ratings of working memory in typically developing children, shorter corpus callosum length in infancy and poorer EF ratings at age 4, and correlations between parent EF ratings and splenium volume in children with 22q11.2 deletion syndrome (Isquith et al., 2013). A recent functional MRI study of inhibition-related functions found that both ratings and tests of inhibition were associated with the left inferior frontal gyrus and bilateral inferior parietal cortex, whereas the dorsolateral prefrontal cortex was also associated with both measures of inhibition, but demonstrated unique patterns in subregions for each type of measure (Warren et al., 2013). Although this study was focused on young adults with varying degrees of anxiety and depression, the findings suggest both separate and shared biologic substrate underlying scales and tests (Warren et al., 2013), similar to the behavioral findings in our study.

Using the parent-rated and performance-based measures to identify poor performance resulted in identification of a small subset of children who performed poorly on both measures, but each measure also identified additional subjects with poor performance on one or the other. Applying these data in an “evidence-based clinical neuropsychological practice” approach as proposed by Chelune (Chelune, 2010) and Youngstrom (Youngstrom, 2012) showed that the use of scales or tests greatly increased the sensitivity and odds ratio in detecting children in the at-risk preterm group. In contrast, the use of tests alone had the highest odds ratio in detecting children in the low adaptive function group, and the use of scales or tests increased the sensitivity only marginally compared to tests alone. We concur with other investigators who advocate for the use of both measures in order to capture divergent but related information (P. Anderson, 2002; Isquith et al., 2013; Ritter et al., 2013; Silver, 2014; Toplak et al., 2009). Using this approach, we propose that children be identified as at-risk if elevated on either scales or tests. Given that both performance-based and parent-rated EF contributed unique function to overall child function, these results also lend support to the use of both measures as complementary assessments in preterm children.

Despite the diversity in EF methods used and the range of gestational age among the preterm participants, the study had limitations. The mean SES of the sample was high, consistent with the geographic area, and may not be representative of the preterm and full term population with lower SES. High SES may have limited the ability to identify associations of SES with outcomes. Nonetheless, even with the resources and access to services accompanying higher SES, the preterm children in our study demonstrated impairments in EF and lower IQ compared to full term children. It is possible that shared method variance (i.e., both the BRIEF-P and Vineland are based on parent report) contributed to the regression analysis showing associations between EF and adaptive function. Although items on the BRIEF-P and Vineland do not show much overlap, except for the problem behaviors sections of the Vineland, which are not included in the Adaptive Behavior Composite, shared method variance is possible due to commonalities in parent response and format of parent-report questionnaires. We did not collect teacher reports of EF skills, as not all children were in school programs. Future work that includes teacher or non-parent caregiver reports would decrease the shared response bias attributable to parent ratings of both EF and functional outcomes. The measurement of additional functional outcomes through direct assessment and standardized observation rather than parent report would also allow further examination with lessened response bias with regard to outcomes, similar to EF in the current study. Longitudinal study of the development of EF and functional outcomes would allow for better understanding of the developmental trajectory of skills in preterm children. Current brain imaging on the children in the study to link deficits in EF skills to specific measures of injury was not available.

EF skills are important measurable skills not only for children born preterm, but also for children at risk for EF impairments due to other risk factors, as well as in typically developing children. Study results suggest that both parent-rated and performance-based EF measures can be used as complementary assessments to characterize the EF skills of preschool children. This approach is especially important for populations at high-risk for EF impairments. The use of both measures identified additional children with EF impairments that could be targeted for intervention. The contribution of both types of EF measures to functional outcome also supports the use of both methods. Although evaluation of our high-functioning, high SES control group of full term children did not identify large numbers of children with EF impairments, many pediatric practices serve children with a variety of risk factors for EF difficulties, including diverse medical and/or developmental conditions and socioeconomic status. We recommend standard screening of all children with EF rating scales with additional monitoring and EF evaluation for high-risk groups, such as children born preterm. Most high-risk infant follow-up programs do not follow children beyond the toddler years, despite adverse long-term outcomes. In the context of a pediatric clinic setting, use of a parent EF rating scale is feasible and provides additional information that goes beyond the developmental screens or behavior rating scales that might be more typically administered in the clinic setting. In contrast, while the administration of a standardized or research oriented EF battery in the general pediatric clinic setting is not feasible, performance-based measures remain desirable in clarifying individual EF profiles, especially when designing or customizing treatment.

Given the links between EF skills and other important functional outcomes, we suggest that EF skills may serve as targets for intervention to improve outcomes. A few studies have started to explore the potential of interventions targeting EF skills in preterm children. A computer-based working memory intervention in children born prematurely resulted in working memory improvements at both preschool age (Grunewaldt, Lohaugen, Austeng, Brubakk, & Skranes, 2013) and adolescence (Lohaugen et al., 2011). Another intervention approach in preterm children called Play and Learning Strategies focused on parent training to enhance parent responsiveness and parent-child interactions (Landry, Smith, Swank, & Guttentag, 2008); although EF skills were not directly targeted per se, outcomes included improvements in child outcomes of social, communication, and problem solving skills (Landry, Smith, & Swank, 2006). Other computerized interventions targeting attention (Rueda, Posner, & Rothbart, 2005), nonverbal reasoning (Bergman Nutley et al., 2011), and working memory and inhibition (Thorell, Lindqvist, Bergman Nutley, Bohlin, & Klingberg, 2009) have focused on typical preschoolers. Preschool curricula targeting multiple EF skills have shown improvements in EF (Bierman, Nix, Greenberg, Blair, & Domitrovich, 2008) and related pre-academic skills in low SES children (Diamond, Barnett, Thomas, & Munro, 2007). Our next steps will focus on interventions targeting EF skills in the preterm population.

Abbreviations

ADHD

attention-deficit/hyperactivity disorder

BRIEF-P

Behavior Rating Inventory of Executive Function-Preschool

EF

executive function

GEC

global executive composite

IQ

intelligence quotient

SD

standard deviation

SES

socioeconomic status