Validity and Reliability of Executive Function Measures in Children with Heavy Prenatal Alcohol Exposure: Correspondence Between Multiple Raters and Laboratory Measures

Abstract

Background:

Rating scales are designed to complement traditional performance-based measures, and both can provide useful information about the functioning of youth with histories of prenatal alcohol exposure. Few studies, however, have compared ratings from multiple informants or the relationship between these subjective rating scale scores and the objective results from laboratory performance-based scales.

Methods:

The current study addressed both of these questions in three study groups: children with histories of prenatal alcohol exposure (AE; n = 47), attention-deficit/hyperactivity disorder (ADHD; n = 41), and typically developing controls (CON; n = 73). All subjects completed a standardized neuropsychological test battery, including laboratory measures of executive functioning and a self-report measure of executive function behaviors. Parents and teachers completed corresponding rating scales of executive function behaviors for each subject. This study assessed the relationship between these behavior rating scales and corresponding neuropsychological tests, and inter-rater agreement among the multiple informants.

Results:

Weak correlations were found between the rating scales and laboratory measures, indicating poor convergent validity for the behavior rating scale. Inter-rater reliability was found but differed by group. Agreement was found between parent and teacher ratings for children with prenatal alcohol exposure, whereas teacher-child agreement was found for those with ADHD.

Conclusion:

Findings from this study indicate that behavior ratings can be used to supplement laboratory measures but may not be measuring cognitive abilities regardless of whether a clinical diagnosis is present. A multi-method approach should be used when measuring skills in this domain. This was one of the first studies to examine cross-informant agreement in a sample of children with prenatal alcohol exposure. Further research is necessary to understand why inter-rater agreement differed for children with prenatal alcohol exposure and those with ADHD.

Introduction

A considerable amount of research has demonstrated the negative effects of prenatal alcohol exposure on the physical, cognitive, and behavioral development of affected individuals (Mattson et al., 2019, Riley et al., 2011). Fetal alcohol spectrum disorders (FASD) is an umbrella term used to refer to the wide range of effects that can occur due to prenatal alcohol exposure and includes fetal alcohol syndrome (FAS) and other diagnoses that do not require facial dysmorphology. This multifaceted range of disorders affects approximately 1.1% – 5.0% of children in the United States (May et al., 2018), with associated global annual costs ranging between $22,810 (children) to $24,308 (adults) per individual, demonstrating that FASD is a significant issue of concern (Greenmyer et al., 2018). Moreover, these economic and societal costs are likely underestimated, given the high rates of missed diagnoses and misdiagnosis due to overlapping symptoms with other clinical disorders, particularly attention-deficit/hyperactivity disorder (ADHD; Chasnoff et al., 2015).

One area of particular interest is executive functioning abilities as impairment in this area is observed in both children with FASD and those with ADHD, but with varying degrees and patterns of deficits reported (Khoury and Milligan, 2016, Kingdon et al., 2016, Vaurio et al., 2008). The two main ways to measure cognitive abilities such as executive functioning skills include the use of performance-based neuropsychological tests and behavior rating scales. While performance-based tests measure the distinct components of executive functioning skills in a laboratory setting, rating scales are designed to assess skills in a more natural context. The Behavior Rating Inventory of Executive Function (BRIEF; Gioia et al., 2000) is a widely used rating scale of everyday behaviors that are related to executive functioning in children and adolescents and was designed to provide complementary and supportive information to traditional performance-based tests (Isquith et al., 2013). However, previous studies in FASD report low or non-significant correlations between the two types of measures, with parents rating their child as more severely impaired compared to deficits found with neuropsychological measures (Gross et al., 2015, Nguyen et al., 2014, Rai et al., 2017, Doyle et al., 2019, Mohamed et al., 2019). This relationship has been assessed in other pediatric clinical samples (e.g., children with brain tumors, clinically-referred youth, and those with ADHD), with no-to-moderate association reported (Howarth et al., 2013, Toplak et al., 2008, McCandless and O’ Laughlin, 2007, McAuley et al., 2010). However, the type of rater could affect this relationship; among nonexposed children, correlations were found between teacher ratings of executive function behaviors and performance-based tests (Tamm and Peugh, 2019, Cho et al., 2011). Previously conducted studies of the FASD population (Gross et al., 2015, Nguyen et al., 2014, Rai et al., 2017, Doyle et al., 2019, Mohamed et al., 2019) relied on reports completed by the parent and did not consider other informant sources such as teacher- and self-reports. Thus, it is not clear whether BRIEF ratings completed by raters other than the primary caregiver could have a stronger relationship with neuropsychological measures than those found using parent reports.

The use of multiple raters in a clinical diagnosis is important as different observers allow for a more complete view of a child, yet almost no research exists analyzing the concordance between raters of children with histories of prenatal alcohol exposure. In children with ADHD, agreement between parent and teacher reports has been mixed with some finding agreement between the two reporters (Soriano-Ferrer et al., 2014, Schneider et al., 2019), while many others report no agreement (Mares et al., 2007, Blake-Greenberg, 2003, Kenealy, 2002). In general, teachers report overall increased severity of executive functioning impairments in children with ADHD compared to parent ratings (Mares et al., 2007, Soriano-Ferrer et al., 2014). Ratings of attention completed by multiple teachers of the same child were in greater agreement relative to parent evaluation (Efstratopoulou et al., 2013), further underlining the clinical usefulness of evaluations that use multi-informant ratings, as children may be exhibiting varying behaviors across differing environments. Additionally, children with ADHD tend to rate themselves as having fewer executive function difficulties as compared to parent ratings (Steward et al., 2017). This finding has also been reported in FASD with ratings of adaptive functioning and social skills (Mariasine et al., 2014). There is a trend of significant disagreement among respondents in other clinical populations (Mahone et al., 2002, McCurdy et al., 2016, Hughes et al., 2009). However, no known studies to date have assessed inter-rater reliability on executive functioning ratings in children with histories of prenatal alcohol exposure. Many children with FASD are brought up in families with non-biological parents (Astley, 2010) which could affect parent behavior reports. Therefore, results may differ compared to those found in other clinical samples, making this an important relationship worth investigating in FASD.

Study Aims

The current study is an extension of a previous study by Nguyen et al. (2014) that found no relationship between parent-reports using the BRIEF and performance-based executive functioning measures in both children with FASD and ADHD. Using a sample derived from the same database, the current study aimed to further expand previous findings with two main aims. The first aim was to assess the convergent validity of parent-, teacher- and self-report ratings on the BRIEF with neuropsychological assessment of executive functioning conducted in a laboratory setting. We hypothesized that similar to previous findings, weak-to-moderate effect sizes would be found between neuropsychological measures and parent, teacher, or child ratings of executive functioning behaviors. The second aim of the current study was to evaluate the inter-rater reliability between parent-, teacher-, and self-ratings of executive function behaviors on the BRIEF among children with prenatal alcohol exposure, those with ADHD, and typically developing controls. We hypothesized that weak-to-moderate effect sizes would be found among raters for both the ADHD and FASD groups, but larger effect sizes would be found with typically developing children.

Materials and Methods

General Methods

Data for the current study were collected as a part of a larger study conducted within the Collaborative Initiative on Fetal Alcohol Spectrum Disorders, Phase Two (CIFASD II). This multisite study had an overarching goal of building a neurobehavioral profile and improving classification accuracy of FASD. All subjects underwent a standardized comprehensive assessment battery and dysmorphology evaluation (for full methodology, see Mattson et al., 2010). Data were collected from five sites: (1) the Center for Behavioral Teratology at San Diego State University, (2) the Fetal Alcohol and Drug Exposure Clinic at Emory University, (3) the Fetal Alcohol and Related Disorders Clinic at University of California, Los Angeles, (4) the Center on Alcoholism, Substance Abuse and Addictions at the University of New Mexico, and (5) seven Northern Plains communities. Data were collected from September 2007 through July 2012. Various recruiting methods were used and differed by site.

A standardized neuropsychological battery was administered by a trained examiner who was blind to subject group. The battery comprised a variety of cognitive measures, including general intellectual functioning and executive functioning. While subjects completed the battery of assessments, primary caregivers completed interviews and questionnaires. More caregivers of subjects in the alcohol-exposed groups were not the child’s biological parent, while caregivers of the nonexposed subjects were primarily biological parents. Before testing, informed consent and assent were obtained from caregivers and subjects, respectively. Teachers were mailed questionnaires. A financial incentive was provided upon completion. The Institutional Review Board at San Diego State University and other CIFASD sites approved the procedures of this study.

Subjects

Subjects recruited for CIFASD II were eligible for the study if they were from a CIFASD research site in the United States, had known alcohol exposure histories, and complete diagnostic data related to ADHD. The total eligible sample included 384 subjects. Subjects were excluded from analysis if the teacher survey data were missing or incomplete. Thus, the study sample (N=161) represented 42% of the CIFASD II sample eligible for the study. Subjects were between 8 and 16 years of age (M = 11.8, SD = 2.5) and were included in one of three groups: children with prenatal alcohol exposure (AE; n = 47), children with ADHD (ADHD; n = 41), and typically developing controls (CON; n = 73). Children in the AE group had confirmed histories of heavy prenatal alcohol exposure, defined as average maternal consumption of more than four alcoholic drinks per occasion or 14 drinks per week during the pregnancy. Alcohol exposure was confirmed by direct report (17%) or, when not available, retrospective report through medical records, birth records, social services records, and when available maternal report (83%). Subjects were only included within the AE group if sufficient documentation of alcohol-use during pregnancy was available or if exposure was suspected in a child with a diagnosis of FAS. All children were examined for FAS using a standardized methodology (Jones et al., 2006, Mattson et al., 2010). Children in the AE group were not excluded for concurrent diagnoses; 76% met diagnostic criteria for ADHD.

Subjects in both comparison groups (ADHD and CON) had minimal or no prenatal alcohol exposure (i.e., no more than one drink per week on average and never more than two drinks on a single occasion throughout gestation); those with more than minimal prenatal alcohol exposure or unknown exposure were excluded. Subjects in the ADHD group met diagnostic criteria for ADHD. Clinical and psychiatric diagnoses were determined based on the Computerized Diagnostic Interview Schedule for Children IV parent interview (C-DISC-4.0; Shaffer et al., 2000). Subjects in the CON group were screened for any significant concerns or conditions on enrollment and were excluded if they exhibited subclinical symptoms of ADHD (i.e., at least four symptoms on the C-DISC-4.0). Subjects in all groups were recruited from the local community; subjects in the ADHD and CON groups were recruited to match by age, race/ethnicity, sex, and socioeconomic status to the AE group at that site. Further exclusionary criteria for all groups included: 1) histories of significant head injury or loss of consciousness (LOC) greater than 30 minutes (no subjects had LOC greater than 5 minutes), 2) primary language other than English, 3) psychiatric or physical disability that would prevent participation, 4) evidence of other known causes of intellectual disability, or 5) adoption from abroad after the age of five years old or less than two years before assessment. Demographic data for all three groups are presented in Table 1.

Table 1.

Demographic Data for Children with Prenatal Alcohol Exposure (AE), Children with Attention-Deficit/Hyperactivity Disorder (ADHD), and Typically Developing Controls (CON)

Demographic variable	AE (n = 47)	ADHD (n = 41)	CON (n = 73)	Group comparisons
CIFASD Site
Atlanta	23	11	19
Los Angeles	1	0	1
Northern Plains	5	3	16
Albuquerque	5	6	4
San Diego	13	21	33
Sex [n (% Females)] *	19 (40.4)	9 (22)	36 (49.3)	ADHD<CON
Age in years [M (SD)]	12.3 (2.75)	11.2 (2.15)	11.9 (2.37)
Handedness [n (% Right)]	40 (85.1)	36 (87.8)	69 (94.5)
Race [n (% White)]	25 (53.2)	30 (73.2)	44 (60.3)
Ethnicity [n (% Hispanic)]	2 (4.3)	9 (22.0)	14 (19.2)
FSIQ [M (SD)] *	78.5 (18.09)	91.4 (20.66)	103.9 (18.62)	AE<ADHD<CON
FAS Diagnosis [n (%)] *	15 (31.9)	0 (0.0)	0 (0.0)	AE>ADHD,CON
ADHD Diagnosis [n (%)] *	36 (76.6)	41 (100.0)	0 (0.0)	AE,ADHD>CON
Biological Parent Rater [n (%)] *	19 (40.4)	36 (87.8)	70 (95.9)	AE<ADHD,CON

CIFASD, Collaborative Initiative on Fetal Alcohol Spectrum Disorders; FAS, fetal alcohol syndrome; FSIQ, Full Scale IQ.

^*Significant (p<.05) differences between groups

Measures

The measures used in this study were selected from the larger CIFASD II battery and were chosen to assess the relationship between cognitive and behavioral measures of executive function, and concordance among multiple raters. Assessments described below measured executive function abilities, general intelligence, attention, and memory.

Behavior Rating Inventory of Executive Function (BRIEF)

The BRIEF (Gioia et al., 2000) is a questionnaire assessing executive function behaviors in children and adolescents between ages 5 – 18 years and can be completed by the primary caregiver (BRIEF-P) or teacher (BRIEF-T). The questionnaire consists of 86 items that use a three-point scale (Never, Sometimes, Often). Raters indicate whether the child has experienced problems with a given behavior that is related to executive functioning within the last six months, as described by each item. The BRIEF-Self Report (BRIEF-SR; Guy et al., 2004) comprises 80 items and measures a child’s own view of their behavior (between 11 – 18 years of age). For all versions of the BRIEF, responses result in eight clinical scales capturing the basic components of behavioral executive functioning including Inhibition, Set Shifting, Emotional Control, Working Memory, Plan/Organize, Organization of Material, Monitor, Initiate (BRIEF-T and BRIEF-P only), and Task Completion (BRIEF-SR only). These clinical scales are then combined into two global index scores (Behavioral Regulation and Metacognition) and an overall composite score (Global Executive Composite). Raw scores are transformed into age- and sex-adjusted T-scores (M = 50, SD = 10) for interpretation (T ≥ 65 considered clinically significant).

Delis-Kaplan Executive Function System (D-KEFS)

The D-KEFS (Delis et al., 2001) is a set of neuropsychological tests that measures various aspects of executive function for ages 8 to 89 years old, including fluency, response inhibition, planning, concept formation, and cognitive flexibility. The following tests were included in the current study to match BRIEF clinical scales: Trail Making, Color-Word Interference, Verbal Fluency, and Tower Test. See Table 2 for descriptions of BRIEF clinical scales and corresponding neuropsychological measures. D-KEFS measures were selected to replicate those used in the Nguyen et al. (2014) study, which were originally included because they both empirically and theoretically relate to the clinical scales from the BRIEF.

Table 2.

Corresponding BRIEF Clinical Scales and Neuropsychological Measures

BRIEF Clinical Scale	Description	Neuropsychological Measure	Description
Inhibit	Ability to demonstrate inhibitory control (i.e., resist impulses)	Color-Word Interference Test Inhibition: Completion Time Color-Word Interference Test Inhibition: Total Errors	Time taken to complete the inhibitory condition of the color-naming task Total number of errors that occurred during the task
Emotional Control	Expression of emotions and ability to modulate emotional responses	N/A
Shift	Ability to switch from one aspect or activity to another as situational demands vary	Trail Making Test Number-Letter Switching: Completion Time Trail Making Test Number-Letter Switching: Set Loss Errors (raw)	Time taken to connect and switch between sequential numbers and letters Number of set loss errors that occurred during the task
Initiate	Capability to independently start activities and generate responses	Verbal Fluency Letter: Total Correct	Total number of correctly produced words during the letter fluency condition
Working Memory	Capacity of holding information in mind in order to complete a task	Digit-Span Backwards	Number of digits that were correctly recalled in reverse order
Plan/Organize	Ability to plan for and manage task demands	Tower Test: Total Achievement Tower Test: Move Accuracy Ratio	Amount of time taken to correctly complete the given arrangement Number of moves made to number of moves required to solve the given arrangement
Organization of Materials	Measure of orderliness	N/A
Monitor	Capability to assess one’s own performance	Tower Test: Rule Violations (raw)	Number of rule violations that occurred during the task

Table was adapted from Nguyen et al. (2014). All BRIEF clinical scores were age- and sex-adjusted T-scores. All scores included for comparison were age-corrected scaled scores except Trail Making Test Switching: Set Loss Errors and Tower Test: Rule Violations, which were both raw scores.

Neuropsychological variables were included if they theoretically corresponded to the items in each BRIEF scale.

BRIEF, Behavior Rating Inventory of Executive Function; N/A, No equivalent neuropsychological measure

Wechsler Intelligence Scale for Children

The Wechsler Intelligence Scale for Children – Fourth Edition (WISC-IV; Wechsler, 2003) was administered to all subjects. The Full-Scale Intelligence Quotient (FSIQ) was included as a measure of IQ. Additionally, scores from the Digit Span Test (backward condition) were included as a working memory measure.

Statistical Analyses

Statistical analyses were run using SPSS v.26 (IBM Corporation, 2019). Data analysis was restricted to those with BRIEF-T data. Subjects with additional missing data (e.g., missing self-report data) were excluded from corresponding analyses. Outlier scores were adjusted to be a maximum of three standard deviations above/below the mean, respectively. Categorical demographic data (sex, race, ethnicity, and handedness) were analyzed using Chi-square, while continuous data (age and FSIQ) were analyzed using analysis of variance (ANOVA).

To determine the relationship between BRIEF clinical scales and performance-based measures (Aim 1), within-group Pearson correlation coefficients between the behavioral and cognitive measures were analyzed. For all correlational analyses, an alpha level of p <.05 was used to determine statistical significance, while alpha levels of .05 < p < .08 were considered marginally significant. Correlations around .10 were considered weak, .30 moderate, and .50 strong (Cohen, 1988).

Aim 2 of the study was to investigate the agreement between parent-, teacher-, and self-ratings. Agreement was examined for each clinical scale of the BRIEF and for the Global Executive Composite score. Comparisons were made between all three raters for each group (AE, ADHD, and CON) using within-group Pearson correlation coefficients. To better understand the effect of rater and group on the BRIEF scales a repeated measures ANOVA was run. Group differences in the neuropsychological measures were analyzed using one-way ANOVA. Significant group differences were followed up with post-hoc analyses using Tukey’s Honest Significant Difference test at an alpha level of .05.

Results

Demographic Data

Demographic data can be found in Table 1. The groups did not differ on race χ² (12) = 16.050, p = .189, ethnicity χ² (4) = 7.244, p = .124, or handedness χ² (4) = 3.806, p = .433, but did differ on subject sex, χ² (2) = 8.221, p = .016 and FSIQ, F(2, 160) = 25.846, p <.001. Groups also differed on FAS diagnosis (only present in the AE group) and ADHD (present in the AE and ADHD groups). As expected there were fewer girls in the ADHD group than CON group as boys are more likely to be diagnosed with ADHD than girls (Arnett et al., 2015). All groups differed on FSIQ (AE<ADHD<CON). Groups also differed on family structure. In the AE group 40.4% of BRIEF-P reporters were the biological parent. In comparison, most reporters for children in the ADHD and CON groups were biological parents (87.8% and 95.9%, respectively).

Relationship of BRIEF to Neuropsychological Measures

Group means for the BRIEF scale scores and neuropsychological measures are presented in Tables 3 and 4. To assess the relationship between each BRIEF scale and the corresponding performance-based neuropsychological assessments (i.e., convergent validity; see Table 2), within-group Pearson correlations were calculated. Relationships were examined separately for each rater (parent, teacher, child) and group (AE, ADHD, CON). There were slight differences in sample sizes due to missing neuropsychological data. For BRIEF-P analyses sample sizes ranged from 41 – 46 (AE), 36 – 40 (ADHD), and 64 – 70 subjects (CON). Sample sizes using BRIEF-T data ranged from 42 – 47 (AE), 37 – 41 (ADHD), and 67 – 73 (CON). Finally, comparisons between BRIEF-SR scores and neuropsychological measures had sample sizes between 27 – 29 (AE), 16 – 18 (ADHD), and 40 – 43 (CON).

Table 3.

Descriptive Data for Selected Behavior Rating Inventory of Executive Function (BRIEF) Scales. Data are Presented as M (SD).

BRIEF Scale	AE	ADHD	CON
BRIEF- Parent	n = 46	n = 40	n = 70
Inhibit	70.8 (13.49)	64.9 (11.85)	45.0 (5.73)
Shift	64.2 (13.89)	62.4 (14.44)	44.9 (7.27)
Initiate	63.7 (11.18)	62.1 (9.91)	45.0 (7.66)
Working Memory	70.3 (11.11)	68.8 (8.51)	44.6 (6.59)
Plan/Organize	67.7 (10.43)	65.7 (8.65)	44.7 (7.98)
Monitor	66.6 (10.89)	64.6 (10.19)	42.4 (8.59)
BRIEF – Teacher	n = 47	n = 41	n = 73
Inhibit	66.9 (19.38)	63.1 (15.04)	49.9 (9.10)
Shift	63.7 (15.93)	60.1 (14.62)	50.1 (9.24)
Initiate	66.9 (12.67)	65.2 (12.30)	51.2 (11.54)
Working Memory	71.5 (16.19)	67.9 (14.25)	51.6 (12.58)
Plan/Organize	68.0 (12.45)	65.8 (12.67)	52.0 (11.78)
Monitor	67.1 (14.47)	64.7 (12.58)	51.6 (11.24)
BRIEF – Self Report	n = 29	n = 18	n = 43
Inhibit	56.6 (11.12)	50.6 (10.51)	44.7 (9.37)
Shift	61.3 (12.65)	54.0 (10.09)	44.2 (10.67)
Working Memory	60.3 (11.24)	53.7 (8.91)	46.3 (10.57)
Plan/Organize	58.3 (10.80)	53.4 (9.93)	43.7 (9.35)
Monitor	58.8 (11.88)	53.0 (9.94)	43.9 (9.07)

Clinical significance on the BRIEF is defined as T ≥ 65, with higher scores indicating greater impairment. Differences in sample sizes across BRIEF reports were due to some missing data and because the BRIEF-Self Report could only be administered to youth between 11 and 18 years of age.

ADHD, attention-deficit/hyperactivity disorder; AE, alcohol-exposed children; CON, control.

Bold text indicates BRIEF scale met clinical significance (T ≥ 65).

Table. 4.

Descriptive Data for Neuropsychological Measures. Data are Presented as M (SD).

Neuropsychological Measures	AE	ADHD	CON	Group Comparison*
Color-Word Interference Test Inhibition: Completion Time	7.6 (3.85)	8.7 (4.21)	10.5 (2.45)	AE, ADHD<CON
Color-Word Interference Test Inhibition: Total Errors	8.3 (3.77)	8.2 (4.02)	9.6 (3.05)	NS
Trail Making Test Number-Letter Switching: Completion Time	6.2 (3.93)	7.0 (4.55)	10.1 (3.25)	AE, ADHD<CON
Trail Making Test Number-Letter Switching: Set Loss Errors (raw)	.8 (1.32)	1.0 (1.61)	.7 (1.26)	NS
Verbal Fluency Letter: Total Correct	7.9 (3.40)	10.0 (3.41)	10.4 (2.93)	AE< ADHD, CON
Digit-Span Backwards	8.1 (3.29)	9.0 (3.63)	10.3 (2.76)	AE<CON
Tower Test: Total Achievement	7.1 (3.36)	8.6 (3.22)	9.9 (2.72)	AE<ADHD, CON
Tower Test: Move Accuracy Ratio	9.5 (3.62)	8.4 (3.24)	9.0 (3.28)	NS
Tower Test: Rule Violations (raw)	4.1 (3.27)	4.1 (3.38)	1.9 (2.34)	AE, ADHD>CON

All scores are age-corrected scaled scores except Trail Making Test Switching: Set loss errors and Tower Test: Rule violations, which are both raw scores.

ADHD, attention-deficit/hyperactivity disorder; AE, alcohol-exposed children; CON, control; NS, not significant.

^*Significant (p < .05) differences between groups.

Correlations between BRIEF and neuropsychological measures are presented in Table 5. On the BRIEF-P, weak-to-moderate effect sizes were found for all relationships tested in the AE (r’s = .020 – .385), ADHD (r’s = .007 – .369), and CON groups (r’s = .050 – .250). Furthermore, few significant correlations were found in all three groups. For correlations using BRIEF-T data few significant relationships were found and effect sizes ranged from weak-to-moderate in all three groups (AE: .024 – .341; ADHD: .035 – .332; CON: .062 – .351). On the BRIEF-SR, the only significant correlation found was between the Inhibit scale and Color-Word Interference Test Inhibition Total Errors (r = −.576, p = .001) in the AE group. Weak-to-moderate effect sizes were found for all other correlations in the AE (r’s = .047 – .264), ADHD (r’s = .054 – .452), and CON groups (r’s = .012 – .286).

Table 5.

Within-Group Correlations (Two-Tailed) Between Performance-Based Neuropsychological Measures and BRIEF Clinical Scales for Parents (BRIEF-P), Teachers (BRIEF-T), and Self (BRIEF-SR).

		BRIEF-P						BRIEF-T						BRIEF-SR
		AE		ADHD		CON		AE		ADHD		CON		AE		ADHD		CON
BRIEF Scale	Neuropsychological Measure	r	p	r	p	r	p	r	p	r	p	r	p	r	p	r	p	r	p
Inhibit	Color-Word Interference Test Inhibition: Completion Time	−.12	.456	.369	.027	−.141	.249	−.284	.069	.035	.839	−.137	.255	.053	.788	.301	.258	.286	.067
Inhibit	Color-Word Interference Test Inhibition: Total Errors	−.385	.013	.083	.632	−.121	.317	−.279	.074	−.151	.371	−.079	.505	−.576	.001	.149	.582	.043	.783
Shift	Trail Making Test Number-Letter Switching: Completion Time	−.202	.188	−.061	.721	−.123	.323	−.279	.063	.108	.517	−.199	.102	−.264	.183	.452	.069	−.051	.755
Shift	Trail Making Test Number-Letter Switching: Set Loss Errors (raw)	.095	.541	−.179	.289	.11	.363	.087	.5701	−.131	.435	.062	.601	.116	.566	−.288	.262	.012	.941
Initiate	Verbal Fluency Letter: Total Correct	−.233	.123	−.049	.776	−.248	.048	−.242	.105	−.281	.092	−.165	.181	.209	.277	−.054	.83	−.156	.318
Working Memory	Digit-Span Backwards	−.277	.062	−.007	.964	−.231	.055	−.341	.019	−.143	.373	−.351	.002	.047	.81	.286	.25	.2	.217
Plan/Organize	Tower Test: Total Achievement	.054	.724	.015	.927	−.25	.041	−.034	.822	−.332	.039	−.299	.012	−.165	.393	−.362	.14	.095	.546
Plan/Organize	Tower Test: Move Accuracy Ratio	−.02	.894	−.289	.078	.05	.679	.024	.872	.119	.47	.067	.574	−.138	.485	−.37	.131	−.215	.172
Monitor	Tower Test: Rule Violations (raw)	−.126	.426	−.242	.149	.075	.55	.217	.1621	.16	.337	.266	.027

ADHD, attention-deficit/hyperactivity disorder; AE, alcohol-exposed children; CON, control.

¹Significant using multiple regression analysis. See text for details.

Bold numbers indicate statistically significant correlation (p<.05).

All BRIEF scale scores were age and sex-adjusted T-scores and all neuropsychological scores were age-corrected scaled scores except Trail Making Test Switching: Set Loss Errors and Tower Test: Rule Violations, both of which were raw scores. Therefore, to assess whether age played a role for these two comparisons, a multiple regression between the BRIEF scale and the neuropsychological score while holding age constant, was run. For the most part, when age was included in the model, results remained the same. Only 2 of the 18 analyses differed: in the AE group, Trail Making Test: Number-Letter Switching Errors was significantly related to BRIEF-T Shift (F (2,42) = 3.870, p = .029) and Tower Test: Rule Violations was significantly related to the BRIEF-T Monitor scale (F (2,40) = 3.603, p = .036).

Inter-rater Agreement on the BRIEF

To address Aim 2, Pearson correlations were used to assess the inter-rater agreement on each of the BRIEF scales and the Global Executive Composite (GEC) score generated. For each group, comparisons were made between parent and teacher ratings, parent and child ratings, and teacher and child ratings.

Inter-rater correlations on the BRIEF are presented in Table 6. Significant moderate correlations between parent and teacher ratings were found with the BRIEF Inhibit (r =.346, p = .019), Shift (r = .317, p = .032), Working Memory (r = .318, p = .031) and Monitor (r = .334, p = .023) scales for children in the AE group. In comparison, in the ADHD group only the Shift scale (r = .398, p = .011) displayed a significant and moderate correlation, with all other non-significant correlations displaying weak effect sizes. In the control group, significant but weak correlations were found with the Inhibit (r = .284, p = .017), Plan/Organize (r = .266, p = .026), and GEC scores (r = .264, p = .027). When comparing parent and self-report ratings on the BRIEF, no significant correlations were found for the AE group, although the Inhibit scale correlation was moderate in size and marginally significant (r = .361, p = .054). Correlations in the control group were non-significant and weak. In the ADHD group, significant and large correlations were found between the Inhibit (r = .663, p = .003), Monitor (r = .538, p = .021), and GEC (r = .472, p = .048) scores. No significant relationships were found when comparing teacher and self-report ratings for all groups, and most were weak in size.

Table 6.

Inter-Rater Agreement (Pearson Correlations) Across Parent, Teacher, and Self-Report Ratings on BRIEF

	Parent vs. Teacher						Parent vs. Self-Report						Teacher vs. Self-Report
	AE (n = 46)		ADHD (n = 40)		CON (n = 70)		AE (n = 29)		ADHD (n = 18)		CON (n = 40)		AE (n = 29)		ADHD (n = 18)		CON (n = 43)
BRIEF Scale	r	p	r	p	r	p	r	p	r	p	r	p	r	p	r	p	r	p
Inhibit	.346	.019	.258	.108	.284	.017	.361	.054	.663	.003	−.138	.396	.361	.054	.152	.548	−.081	.604
Shift	.317	.032	.398	.011	.087	.472	.222	.246	.275	.269	−.245	.128	−.095	.624	−.456	.057	−.019	.904
Initiate*	.272	.068	−.201	.213	.194	.107
Working Memory	.318	.031	−.127	.436	.224	.062	.147	.445	.169	.502	−.275	.086	.076	.697	−.064	.801	.023	.881
Plan/Organize	.27	.07	.097	.551	.266	.026	.091	.64	.425	.079	−.117	.471	−.091	.64	−.11	.665	−.028	.858
Monitor	.334	.023	.094	.564	.179	.139	.123	.524	.538	.021	−.147	.365	−.046	.811	.083	.744	−.153	.329
GEC	.289	.051	.102	.532	.264	.027	.167	.388	.472	.048	−.002	.988	−.064	.74	−.325	.189	.011	.945

ADHD, attention-deficit/hyperactivity disorder; AE, alcohol-exposed; BRIEF, Behavioral Rating Inventory of Executive Function; CON, control; GEC, Global Executive Composite

^*Initiate scale is not available on the BRIEF-SR

Bold numbers indicate statistically significant correlation (p<.05)

To further explore the relationship between raters on the BRIEF, a 3 × 2 × 6 repeated-measures ANOVA with Greenhouse-Geisser correction was run with group (AE/ADHD/CON) and rater (parent/teacher) as between-subjects factors and scale as the repeated measure. Only parent and teacher data were included in this follow up analysis because the BRIEF-SR included different scales. Detailed results are presented in Table 7. The three-way interaction between group, rater, and scale was not significant. The 2-way interactions (group x rater, group x scale, and rater x scale) were all significant and the significant pairwise comparisons are listed in Table 7. Briefly, collapsed over scale, teachers reported greater executive dysfunction on the BRIEF than parents, but only in the control group. Similarly, when collapsed across group, teachers reported greater executive dysfunction than parents on 3 of the 6 scales. Finally, when collapsed across raters, group differences varied by scale. There were also significant main effects for group (CON<AE,ADHD), rater (Parent<Teacher), and scale; significant pairwise comparisons are listed in Table 7.

Table 7.

Repeated Measures ANOVA Comparing BRIEF-T and BRIEF-P Data

Factor	Statistic	Pairwise comparisons*
Group	F(2,153) = 113.6, p < .001	CON<AE, ADHD
Rater	F(1,153) = 4.3, p = .039	Parent<Teacher
Group × Rater	F(2,153) = 6.4, p = .002	CON: Parent<Teacher
Scale	F(3.3, 501.5) = 11.4, p < .001	Shift<Inhibit, Monitor WM>Inhibit, Shift, Initiate, Plan/Organize, Monitor Plan/Organize>Shift, Initiate, Monitor
Group × Scale	F(6.6, 501.5) = 3.4, p = .002	Inhibit: CON<ADHD< AE Shift: CON<AE, ADHD Initiate: CON<AE, ADHD WM: CON<AE, ADHD Plan/Organize: CON<AE, ADHD Monitor: CON<AE, ADHD
Rater × Scale	F(3.9, 592.7) = 4.1, p = .003	Initiate: Parent<Teacher Plan/Organize: Parent<Teacher Monitor: Parent<Teacher
Group × Rater × Scale	F(7.8, 592.7) = 1.4, p = .212	NS

ADHD, attention-deficit/hyperactivity disorder; AE, alcohol-exposed; BRIEF, Behavioral Rating Inventory of Executive Function; CON, control; NS, not significant; WM, working memory.

^*Significant (p < .05) differences between groups

Discussion

Relationship of BRIEF to Neuropsychological Measures

The first aim of this study examined the convergent validity of the BRIEF. To accomplish this, we used correlational analyses to assess the relationship between the BRIEF ratings completed by the parent, teacher, and child with corresponding neuropsychological measures. Consistent with our hypothesis, weak-to-moderate correlations were found between the measures for both the alcohol-exposed and the ADHD group across all three raters. The findings of the current study are consistent with previous reports of non-significant relationships between performance-based measures and parent-report measures in children with FASD (Gross et al., 2015, Mohamed et al., 2019, Nguyen et al., 2014, Rai et al., 2017) and expand the existing literature to include teacher ratings and self-report ratings that show a similar trend. Similar to a previous report derived from the same database (Nguyen et al., 2014), we found a few weak correlations between the BRIEF clinical scales and performance-based measures in the control group. There was evidence of known-group validity of the BRIEF, as the control group had significantly lower BRIEF scores compared to the AE and ADHD groups, which did not differ from each other. The overall lack of strong significant correlations across the groups may be because we analyzed domain-specific correlations between the BRIEF clinical scales and matching neuropsychological tests of executive function, whereas previous studies that reported significant relationships between the two measures did not (Toplak et al., 2008, McCandless and O’Laughlin, 2007, Parrish et al., 2007).

Findings from our study provide further support that behavior rating scales and performance-based assessments are measuring different constructs, regardless of the presence of a clinical diagnosis. While the BRIEF is useful as a measure of behavioral dysfunction, it does not appear to be a measure of neurocognitive ability. Conversely, neuropsychological measures are standardized assessments administered in highly structured environments which do not always assess the real-world application of skills. The lack of significant relationships between the BRIEF clinical scales and corresponding neuropsychological assessments underscores the difficulty with predicting real-world functioning in children using performance-based measures, a problem that is also found within the adult literature (Marcotte et al., 2010).

A strength of the BRIEF is that it provides an ecologically valid measure of everyday executive function abilities (Gioia and Isquith, 2004). Previous studies have reported relationships between the BRIEF and measures of problematic behaviors (Doyle et al., 2019), social skill impairment (Schonfeld et al., 2006), attention deficits (McAuley et al., 2010), and increased involvement in the criminal justice system (Ogilvie et al., 2011). Thus, there is support of its use as a general measure of behavioral impairment. In the current study, scores on the BRIEF were able to accurately distinguish typically developing children from those in the clinical groups. Other studies have reported similar strengths of the BRIEF (McAuley et al., 2010; McCandless & O’ Laughlin, 2007) as well as its usefulness as a screening tool for prenatal alcohol exposure (Nguyen et al., 2014). In a clinical setting, the administration of the BRIEF could aid in identifying individuals with behavioral difficulties and those who are at risk of experiencing long-term difficulties. Clinicians may be more inclined to utilize the BRIEF due to its simple and quick administration, however, the findings from our study demonstrate that the BRIEF should not be administered in isolation or in place of performance-based measures. The utilization of both types of instruments appears optimal to gain a broader understanding of executive dysfunction.

Inter-rater Agreement on the BRIEF

This study is one of the first to examine the inter-rater reliability of behavioral executive functioning ratings among multiple informants in a sample of children with prenatal alcohol exposure, ADHD, and typically developing controls (Aim 2). We hypothesized that similar to the majority of findings in ADHD (Mares et al., 2007, Blake-Greenberg, 2003, Kenealy, 2002) and those conducted in FASD (Mariasine et al., 2014), inter-rater agreement would be poor for children with prenatal alcohol exposure. However, this was not the case as results from the correlational analyses reveal that parents and teachers show a high level of agreement on executive function ratings of children with prenatal alcohol exposure. Four out of the seven BRIEF scales tested were moderately correlated across parent and teacher ratings (correlations ranging from .317 – .346), with the other three clinical scales being marginally significant in this group (see Table 6). Within the FASD population, no known studies have assessed inter-rater agreement of executive function abilities. However, significant agreement has been shown with ratings of externalizing behaviors in children with prenatal alcohol exposure (Werlinger, 2017) and results from the current study indicate that this pattern of agreement also exists with behavioral executive functioning. Although inter-rater agreement was not found in the ADHD group among parent and teacher reports, there was evidence of parent-child agreement in this group as a majority of the BRIEF scales displayed moderate-to-large effect sizes. There was little evidence of agreement between parent and teacher ratings within the control group, as all correlations displayed weak effect sizes. A weak significant correlation was found for the Global Executive Composite score in the control group, although it is unclear whether this composite score represents the best summary given the different degrees of correlation between raters on the BRIEF subscales. The overall lack of agreement found within the control group may be due to the restricted range of scores on the BRIEF.

The general literature on inter-rater agreement on the BRIEF clinical scales indicates large discrepancies between ratings (Steward et al., 2017, Mahone et al., 2002, McCurdy et al., 2016, McCandless and O’ Laughlin, 2007). However, a few have reported significant agreement both in samples of children with ADHD (Soriano-Ferrer et al., 2014, Schneider et al., 2019) and typically developing children (Gioia et al., 2000). Several factors may explain the discrepancies between parent and teacher reports. First, the raters observe children in environments that differ greatly (i.e., classroom versus home), and the varying demands of each setting may explain the differences seen between parent and teacher ratings. For instance, young children are more likely to display behavioral problems at home than in school settings (Strickland et al., 2012). Second, teachers may only have contact with a child for a few hours a day and/or may have known the child for less than a year, which may limit their familiarity with an individual child. Conversely, teachers interact with a variety of students daily, which allows them to view behaviors compared to a normative group. Finally, rater bias may be playing a role in behavior ratings. A previous study of children with ADHD reported that parent behavior ratings displayed increased signs of rater bias compared to ratings completed by the teacher (Hartman et al., 2007).

Findings from our study indicate that parent and teacher ratings agree with regards to children with prenatal alcohol exposure, but not ADHD. Several rater and child characteristics may influence a child’s behavior and informant’s perception of that behavior and as such, may account for this group difference. For instance, biological relation to the child could impact parent ratings on the BRIEF. Biological parents of children with prenatal alcohol exposure report higher levels of parent-related stress compared to adoptive parents (Paley et al., 2006). Within our sample, parent reporters within the ADHD and CON groups were primarily biological parents, whereas fewer children in the alcohol-exposed group were living with a biological parent. Additionally, previous studies have indicated that the social and cultural context that a child lives in may play a role in rater disagreement. Children from lower SES backgrounds are rated as having increased inattentive behaviors by teachers, but not parents, and increased reports of hyperactive/impulsive behaviors by both raters (Lawson et al., 2017; Phillips & Lonigan, 2010). Race also seems to play a role. African-American race was related to increased endorsement of hyperactive/impulsive symptoms by teachers compared to parents (Lau et al., 2004; Lawson et al., 2017; Youngstrom et al., 2000).

Overall in our study, parents and teachers seemed to endorse greater executive dysfunction compared to deficits measured using neuropsychological measures, consistent with previous findings in FASD (Gross et al., 2015, Nguyen et al., 2014, Rai et al., 2017, Doyle et al., 2019, Mohamed et al., 2019). The use of the BRIEF in isolation therefore could lead to more children being rated as impaired compared to other types of measures. This underscores the need for results to be interpreted with caution and the inclusion of multiple measures in a clinical setting. Additionally, we found that teacher ratings were significantly higher on the BRIEF than parent ratings only in the control group, although both ratings were well below the cutoff for clinical significance. Importantly, teacher and parent ratings of children with prenatal alcohol exposure or ADHD were similar suggesting concurrent validity between raters and supporting the use of either rater as adjuncts to direct measures in future research. Although formal analyses were not conducted, it appears that parents and teachers indicated greater executive dysfunction compared to child ratings as no BRIEF-SR scales reached clinical significance across all three groups. This is consistent with previous findings of children rating themselves as having fewer executive functioning impairments compared to parent ratings in several different clinical populations, including ADHD (Steward et al., 2017, Mahone et al., 2002, McCurdy et al., 2016) and expands the existing literature to include children with prenatal alcohol exposure. A lack of self-awareness of cognitive deficits could have lasting effects on an individual. For instance children with traumatic brain injury who experience poor self-awareness of deficits often experience decreased motivation, impaired judgement, and are less likely to benefit from therapy (Robertson & Schmitter-Edgecombe, 2015). Future clinical interventions should focus on improving upon the self-awareness of executive function difficulties in children with prenatal alcohol exposure so that they can learn to better advocate for themselves.

Limitations, Strengths, and Future Directions

The most significant limitation of the current study was the number of participants included. For certain analyses, only 87 participants were included primarily due to age restrictions for the BRIEF-SR, which could only be completed by children between 11 and 18 years of age. Furthermore, data were restricted to only those with completed BRIEF-T forms (161 of 384), as many teachers did not return their completed forms for participants. However, these sample sizes are equivalent to or greater than most studies in the FASD literature. For some of the analyses the small sample size may explain why several correlations displayed moderate effect sizes but were not significant. In particular, several analyses comparing BRIEF-SR and neuropsychological measures in the ADHD group may have reached statistical significance with a larger sample. Exploratory analyses comparing the analysis group to the larger eligible sample indicated that the smaller sample was similar in most important ways to the larger eligible sample with a few exceptions. The larger eligible sample showed group differences in FSIQ scores, sex, age, race, and ethnicity. In the smaller analysis sample, group differences were only apparent for FSIQ and sex, although means for the other variables followed the same pattern as in the larger eligible sample. We also compared the BRIEF-P data for subjects with and without teacher data in the larger sample and found no differences. The means scores were very similar, and the presence of teacher data did not affect the average parent scores for any of the groups or the combined sample. These exploratory analyses indicate that the restricted sample analyzed in the current study was representative of the larger group.

Given the complexity of this study and previously reported methodological considerations (Dennis et al., 2009), we did not include IQ in our analyses. However, we conducted exploratory correlation analyses to examine the relation between FSIQ and BRIEF composite scores for each rater. FSIQ was significantly (p< .05) negatively correlated with teacher ratings for the AE and CON group and with parent ratings for the CON group only. Future studies might examine this relation in more detail, perhaps using study groups that are matched on IQ.

Despite its limitations, the study has many strengths, and the results significantly add to the literature on FASD. This study examined both subjective and objective measures of executive function and, importantly, included multiple raters (i.e., parent, teacher- and self-reports). Furthermore, it is also one of the first to assess inter-rater reliability in a sample of children with prenatal alcohol exposure. Another significant strength of this study is that data were collected from five different sites across the United States, allowing for more generalizability of the results.

This study demonstrated that the BRIEF does not measure the same aspects of executive function as performance-based neuropsychological assessments and emphasizes the importance of using a multi-method approach when determining deficits in this domain. Future studies should continue to address the ecological validity of the BRIEF, in which comparisons should be made to other behavioral measures of executive function. One important finding from this study was the inter-rater agreement between parent and teacher ratings of children with prenatal alcohol exposure. Future research should examine rater characteristics, including biological relation to child, demographics, experience, and training, which may influence the ability of parents and teachers to recognize and report specific symptoms. Additionally, child characteristics such as age, IQ, and degree of impairment may also affect the ability of raters to identify behavioral deficits.