The Behavior Rating Inventory of Executive Function (BRIEF) to Identify Pediatric Acute Lymphoblastic Leukemia (ALL) Survivors At Risk for Neurocognitive Impairment

Abstract

Neurocognitive problems, including executive dysfunction, are potential late effects of pediatric acute lymphoblastic leukemia (ALL) treatment. Surveillance for neurocognitive impairment in a timely and efficient manner is imperative to ongoing clinical care. We sought to determine if the Behavior Rating Inventory of Executive Function (BRIEF) Parent Form identified leukemia survivors with cognitive impairment. In this 28-site cross-sectional study, parents of 256 children, a mean of 8.9 ± 2.2 years after treatment for standard-risk precursor-B ALL and in first remission, completed the BRIEF. We used a multivariate logistic regression to calculate the association between elevated scores on three composite BRIEF indices (Behavioral Regulation Index (BRI), Metacognition Index (MI), Global Executive Composite (GEC)) and special education and Attention-Deficit/Hyperactivity Disorder (ADHD) outcomes. All BRIEF index scores were significantly associated with receipt of special education services or ADHD. The BRI was most strongly associated with ADHD (OR = 4.33; 95% confidence interval (95% CI) = 1.72 – 10.9). The GEC was most strongly associated with ADHD (OR = 4.46, 95% CI 1.77 – 11.22). Elevated scores on the BRIEF GEC were associated with low sensitivity (24.1 – 39.1) for detecting the outcomes but better specificity (range 87.7 – 89.3). These results suggest that the parent-completed BRIEF is associated with clinical outcomes but is not a sensitive tool to identify leukemia survivors that require a comprehensive neuropsychological assessment.

Introduction

Over 90% of children with acute lymphoblastic leukemia (ALL) will be long-term survivors [1]. Even after eliminating cranial radiation from most modern leukemia regimens, systemic and intrathecal chemotherapy continues to place pediatric patients at risk for neurocognitive impairment [2, 3]. The Children’s Oncology Group Long-Term Follow-Up Guidelines recommend routine surveillance for all pediatric ALL survivors [4], as do other international pediatric oncology organizations. [5, 6] [7] Traditional neuropsychological assessment, however, is time- and cost-intensive for patients and their families. Further, it typically requires expertise in psychological testing that is not readily available at all pediatric cancer treatment centers. [8] Efficient and accessible surveillance measures to identify children with neurocognitive problems who need comprehensive assessment is important to high quality survivorship care. [9]

Problems in the domain of executive function have been identified in pediatric cancer survivors. [10–12] Executive function is an umbrella term describing inter-related cognitive processes necessary for goal-directed behavior that includes such processes as emotional regulation, working memory, planning, and task flexibility. [13] Executive function is strongly associated with such life goals as independent living, marrying, achieving higher education, and employment. [10, 14–18]

The Behavior Rating Inventory of Executive Function (BRIEF) Parent Form is a standardized instrument that allows the observer to rate the executive function of the 5- to 18-year-old child. The BRIEF is psychometrically sound, [19] takes approximately 10 to 15 minutes to complete and an additional 10 to 15 minutes to score. Parent assessment with the BRIEF has been well validated and used to assess the executive function of pediatric patients at risk for dysfunction such as those with ADHD, Tourette’s Disorder, and Fetal Alcohol Syndrome. [20, 21] Recent studies with pediatric brain tumor survivors have shown that the BRIEF identified working memory deficits and was an effective surveillance tool used to identify those with diminished intellectual ability. [22, 23] Brain tumor survivors generally have more severe neurocognitive effects, including both global cognitive and executive functioning, as compared to survivors of other pediatric cancers. [12] It has not been determined whether the BRIEF is a reasonable surveillance measure in pediatric leukemia patients treated without cranial radiation and who generally have meaningful, but less severe, impairment.

Evidence is needed to demonstrate if the BRIEF has benefit as a surveillance tool specifically useful in the pediatric ALL population. We sought to determine this by 1) Assessing whether BRIEF scores were associated with the presence of “real-life” cognitive problems as measured as receipt of special education services, diagnosis of ADHD and/or use of attention enhancing medication since completing therapy; and 2) Determine whether the BRIEF is a measure sensitive enough to correctly identify those in need or not in need of comprehensive assessment. We had a unique opportunity to address this question in a large patient sample from a multi-site long-term follow-up study of ALL patients treated with regimens similar to contemporary therapy without cranial radiation.

Materials and METHODS

Participants

We conducted a cross-sectional study of patients with ALL previously enrolled and randomized in legacy Children’s Oncology Group therapy protocols (i.e., closed to enrollment), CCG 1922 and 1952, which were open from March 1993 to August 1995 and May 1996 to February 2000, respectively. [24, 25] Patients were eligible for participation in this neurocognitive follow-up study if they were diagnosed and enrolled on one of these protocols for standard risk precursor B-ALL at one of the participating 28 sites. The National Cancer Institute Criteria [26] defines standard risk pre-cursor B-ALL as peripheral white blood count < 50,000 and age between 1.0 and 9.99 years. Participating institutions were chosen from the total 250+ Children’s Oncology Group sites based on their interest and availability of a licensed psychologist, as well as the support staff necessary to implement the study. An effort was made to include both community and tertiary care programs from all major regions of the nation to maximize ethnic and geographic diversity. Patients were off-therapy and in first remission. All patients received central nervous system-directed chemotherapy with intrathecal medications; however, no patient had a history of central nervous system leukemia. Thus, no child received cranial radiation. Additional eligibility requirements included completion of all therapy at least one year previously, age at evaluation of 6 to16.99 years, no history of developmental disorders (e.g., trisomy 21, developmental delay) or very low birth weight (<1500 grams). Patients were not diagnosed with or treated for ADHD prior to the leukemia diagnosis or during treatment. They had not been retained in school or received special education services before or during treatment. Patients were excluded if they had been non-randomly assigned to more intensive therapy because of unfavorable cytogenetic findings or a slow response after induction. The methodology has been previously reported. [27–30]

A total of 746 patients were enrolled on the two therapeutic studies at the participating sites and were potentially eligible. Of these, 263 patients consented and completed the evaluation and 256 had complete data. Of the remaining 483 patients, 219 were lost to follow-up and could not be traced, 236 refused, and 28 were never offered participation due to accidental oversight by study staff. There were no differences between participants and non-participants with respect to age at diagnosis, elapsed time since treatment, or gender. [27, 29] Also see supplementary materials for additional information.

Procedures

The institutional review board of each participating center as well as the Yale University Human Investigation Committee approved the current study. Informed consent and assent were obtained for all participants in accordance with the Declaration of Helsinki. Participants underwent a comprehensive, half-day, neurocognitive assessment supervised by a licensed psychologist at no cost to the patients. Parents of participants completed a questionnaire inquiring about marital status, education, and income. The questionnaire also assessed the medical and educational history of the child including whether the child had been diagnosed with ADHD and/or was taking medication for attention related issues, or had received special education services only since completing treatment. Additionally, parents completed the Behavior Rating Inventory of Executive Function (BRIEF), a norm-referenced parent-report measure of the child’s executive function.

Measures

The BRIEF Parent Form is an 86-item behavior rating Likert scale that assesses the frequency (Never, Sometimes, Often) with which the child demonstrates difficulty with the item described [19]. It consists of eight clinical scales assessing dimensions of executive function: inhibit, shift, emotional control, initiate, working memory, plan/organize, task monitor, and organization of materials. Subscale scores contribute to three summary scores: Behavioral Regulation Index (BRI), Metacognition Index (MI), and Global Executive Composite (GEC) (Table 1). Age and sex-specific norms were used to transform raw scores to T-scores. The BRIEF is sensitive to executive function in children with a range of disorders, [31–33] including those with cancers. [10, 23] The BRIEF has excellent psychometric properties that include appropriate construct validity. Internal consistency is strong (Parent and Teacher, r = .82–.98) and the 2-week test-retest reliability is also high (Parent, r = .81). The BRIEF is an appropriate measure of the designated outcomes as problems with executive functioning are associated with learning disabilities and those receiving special education services [34] as well as frequently observed in children with ADHD. [35]

Table 1.

Description of Behavior Rating Inventory of Executive Function (BRIEF) Parent Form composite scales

Measure	Description
Behavior Regulation Index (BRI)	Ability to shift cognitive set and modulate emotions and behavior via appropriate behavioral control. Includes the Inhibit, Shift, and Emotional Control subscales.
Metacognition Index (MI)	Ability to cognitively self-manage tasks and monitor one’s performance. Includes the Initiate, Working Memory, Plan/Organize, Organization of Materials, and Monitor subscales.
Global Executive Composite (GEC)	Summary measure of all subscales.

Statistical Analysis

Univariate analysis and stepwise regression were completed to assess the role of gender, age at diagnosis, and time elapsed since treatment as potential confounders. Multivariate logistic regressions were used to assess the association between an elevated score on a BRIEF composite index and three outcomes of interest related to the child: 1) enrollment in special education services; 2) diagnosis of ADHD/medication; and 3) enrollment in special education and diagnosis of ADHD/medication. We used a cut-off score 1.5 standard deviation above the mean (i.e., T ≥ 65) consistent with the manual directions to indicate clinical significance. [19] Odds ratios (ORs) and their 95% confidence intervals were calculated. We calculated sensitivity, specificity, positive predictive power, and negative predictive power statistics of each BRIEF composite score for the outcomes of special education, diagnosis of ADHD/medication and the combined outcome. All analyses were conducted using the software package SAS (version 9.3, The SAS Institute, Cary, North Carolina).

Results

Participants

The participants were a mean age of 12.5 years (SD = 2.4, range = 7.0 – 16.9) at the time of evaluation. The elapsed time since diagnosis of ALL was 8.9 years (SD = 2.2, range = 4.25–14.73) (Table 2).

Table 2.

Demographic characteristics of participants (n = 256)

	Mean (SD) or N (%)
Age at evaluation (years)	12.8 (2.5)
Age at diagnosis (years)	3.9 (1.8)
Time elapsed since diagnosis (years)	8.9 (2.2)
Sex
Female	136 (53)
Male	120 (47)
Race and ethnicity
White, non-Hispanic	213 (83.2)
Hispanic	19 (7.4)
Black, non-Hispanic	6 (2.3)
Native American/ Alaskan Native	1 (0.4)
Asian	5 (2.0)
Other	11 (4.3)
Unknown	1 (0.4)
Identification*
Special education services	59 (23.5)
ADHD/medication	23 (9.3)
Special education and ADHD/medication	13 (5.1)

^*There are missing data for one patient.

Association between the BRIEF and clinical outcomes

Gender, age at diagnosis, and time elapsed since diagnosis were not associated with receipt of special education services, ADHD, or the combined variable (patients with special educations services and ADHD) as indicated by univariate analysis. Additionally, a backwards stepwise regression was completed which did not identify any significant confounders.

A multivariate logistic regression indicated that all three composite BRIEF scales were significantly associated with use of special education services and diagnosis of ADHD/medication (Table 3). History of special education services was significantly associated with the Behavioral Regulation Index (BRI) (OR = 2.44; 95% CI = 1.18 – 5.05; p = 0.016) the Metacognition Index (MI) (OR = 2.31; 95% CI = 1.01 – 4.83; p = 0.027), and the Global Executive Composite (GEC) (OR-2.27; 95% CI= 1.08 – 4.74). The Global Executive Composite (GEC) was most strongly associated with diagnosis of ADHD/medication (OR = 4.45; 95% CI = 1.77, 11.22; p = 0.002). All BRIEF composite scores were significantly associated with special education and diagnosis of ADHD/medication (OR = 3.13 – 3.53; 95% CI = 1.54 – 7.21). Using a cutoff of 1.5 SD above the mean, elevated scores on the GEC were associated with poor sensitivity for all outcomes (24.1 – 39.1) but better specificity (range 87.7 – 89.3). Likewise, positive predictive value for all outcomes was low (range 23.7 – 47.4) but negative predictive value was stronger (range 77.7 – 93.5).

Table 3.

Association of BRIEF scales and patient outcomes^*

	Special Education Services			ADHD			Special Education Services and ADHD
	Odds Ratio	95% CI	p-value	Odds Ratio	95% CI	p-value	Odds Ratio	95% CI	p-value
BRI	2.443	1.183, 5.045	0.016	4.329	1.723, 10.873	0.002	3.325	1.643, 6.730	0.001
MI	2.306	1.101, 4.830	0.027	4.411	1.752, 11.109	0.002	3.532	1.730, 7.210	0.001
GEC	2.267	1.084, 4.741	0.03	4.456	1.770, 11.219	0.002	3.131	1.535, 6.389	0.002

^*Age at diagnosis, gender, and time elapsed since therapy completion were not identified as covariates

Odds ratios based on scores > 1.5 SD above the mean; BRI = Behavioral Regulation Index; MI = Metacognition Index; GEC = Global Executive Composite; CI = Confidence Interval

DISCUSSION

In this large cross-sectional study, we examined whether the BRIEF, a short norm-referenced parent-report instrument, was an appropriate tool for surveillance of executive dysfunction in childhood ALL survivors. The results indicate that impairment on the BRIEF composite scales was associated with patients who were identified with special education needs or ADHD. With relatively high specificity and negative predictive power as compared to low sensitivity and positive predictive power, most patients with impaired scores on the BRIEF truly will have meaningful impairment. Although the BRIEF may be helpful in avoiding unnecessary testing, it is not particularly useful in identifying most of those in need of a comprehensive neuropsychological evaluation.

Despite less pronounced executive functioning deficits as compared to those in pediatric brain tumor survivors, the BRIEF parent report identified those leukemia survivors with neurocognitive difficulties in the real-world setting. These findings are consistent with other recent studies that have demonstrated some efficacy in using the BRIEF to detect impaired cognitive functioning in survivors of central nervous system tumors. [22, 23] The present study, however, used the BRIEF as the only surveillance measure and related these observations to some documented level of day-to-day deficiency, either in academic achievement or allocation of attentional resources. This study adds to the literature that parents are a reliable fund of information and their report may spot difficulties that affect daily functioning in the home, school, and community settings that might otherwise go undetected. [31, 36–38] This study included a relatively large number of childhood ALL survivors who all were treated with regimens very similar to current protocols. Due to the size of the population, multiple potential confounders could be examined and controlled for in the analyses. Age at diagnosis, gender, and number of years elapsed since therapy completion were not associated with deficiencies in executive function as identified by scores on the BRIEF. Although this finding is not in alignment with some work, [39–41] it is consistent with Campbell et. al, [10] analysis of executive function among pediatric ALL survivors.

The clinical usefulness of the BRIEF as a time-efficient method of identification of children at risk for neurocognitive problem may be limited, however. As strong test specificity suggests, those with elevated scores truly need the comprehensive assessment, which avoids over-testing. By correctly identifying those children, the financial burden can be reduced for families, particularly as these evaluations are costly and insurance companies may not provide reimbursement in some countries. [42] Further, this allows institutions to divert potentially limited resources to those patients with the most need. On the other hand, no cutoff yields good enough sensitivity to indicate that the BRIEF is a sufficient stand-alone surveillance tool to identify those who may benefit from neuropsychological assessment.

We acknowledge some possible limitations of this study. First, this was a cross sectional study and parent’s knowledge of their child’s learning needs or diagnoses may increase attunement to their child’s difficulties, thus affecting item endorsement on the BRIEF and creating a stronger predictive relationship. The questions asked on the BRIEF, however, are specific to daily functioning (e.g., needs help from an adult to stay on task, is not a self-starter, keeps room messy) and do not necessarily have an obvious connection to the outcomes that were used. Moreover, the BRIEF has been used as an effective measure to monitor change in executive function after an intervention which suggests its valid use among parents who know their child’s condition. [43] Second, the use of special education services and ADHD diagnosis are only two measures of ecological validity and there can be variability in how these children come to the attention of professionals. [44] For example, parental education attainment and advocacy can affect referrals to school and mental health professionals. Among patients in this study, however, there was no significant association between household income or parental education and the use of special education services or ADHD diagnosis. Moreover, these outcomes are reasonably objective as there are criteria for each that need to be met before a designation is made. [45] Additionally, symptoms of ADHD are often non-specific and can overlap with other conditions such as mood disorders. Moreover, the BRIEF was not designed to discriminate among these issues and so elevated scores may erroneously indicate those in need of cognitive testing, but who may benefit from a focused mental health assessment instead. These outcomes are self-reported with which there are inherent difficulties; however, similar questions were asked several ways in the questionnaires in order to assure accuracy. The scope of the BRIEF is limited to executive function so it can potentially miss deficits in other cognitive domains that impact functioning. Although scores on this measure were associated with children with cognitive weaknesses, identifying another brief scale to capture areas outside of executive function may be useful. Finally, in this analysis the BRIEF was used as the sole source of information about the patient. In reality, collateral information would likely be collected. If a parent or patient endorsed school problems, despite non-clinical findings on the BRIEF, this patient would likely be referred for additional comprehensive neuropsychological testing. Since we did not test this systematically, the additive value of collateral information is not known. Finally, long-term outcomes such as employment are not available in a pediatric population but could be the focus of future investigations.

With increasing numbers of survivors, it will be important to have a quick, valid surveillance tool, particularly in resource-limited communities, to identify those patients in need of comprehensive neurocognitive assessment. Detecting neurocognitive impairment that affects functioning in everyday life is important among this growing number of pediatric leukemia survivors who may present with more subtle cognitive challenges. Although the BRIEF is associated with negative cognitive outcomes in leukemia survivors, results suggest that using the parent-completed BRIEF as the sole measure to detect patients at risk for neurocognitive problems may not be effective.