Behavioral and cognitive outcomes for clinical trials in children with neurofibromatosis type 1

Abstract

Objective:

To evaluate the appropriateness of cognitive and behavioral outcome measures in clinical trials in neurofibromatosis type 1 (NF1) by analyzing the degree of deficits compared to reference groups, test-retest reliability, and how scores correlate between outcome measures.

Methods:

Data were analyzed from the Simvastatin for cognitive deficits and behavioral problems in patients with neurofibromatosis type 1 (NF1-SIMCODA) trial, a randomized placebo-controlled trial of simvastatin for cognitive deficits and behavioral problems in children with NF1. Outcome measures were compared with age-specific reference groups to identify domains of dysfunction. Pearson r was computed for before and after measurements within the placebo group to assess test-retest reliability. Principal component analysis was used to identify the internal structure in the outcome data.

Results:

Strongest mean score deviations from the reference groups were observed for full-scale intelligence (−1.1 SD), Rey Complex Figure Test delayed recall (−2.0 SD), attention problems (−1.2 SD), and social problems (−1.1 SD). Long-term test-retest reliability were excellent for Wechsler scales (r > 0.88), but poor to moderate for other neuropsychological tests (r range 0.52–0.81) and Child Behavioral Checklist subscales (r range 0.40–0.79). The correlation structure revealed 2 strong components in the outcome measures behavior and cognition, with no correlation between these components. Scores on psychosocial quality of life correlate strongly with behavioral problems and less with cognitive deficits.

Conclusions:

Children with NF1 show distinct deficits in multiple domains. Many outcome measures showed weak test-retest correlations over the 1-year trial period. Cognitive and behavioral outcomes are complementary. This analysis demonstrates the need to include reliable outcome measures on a variety of cognitive and behavioral domains in clinical trials for NF1.

Neurofibromatosis type 1 (NF1) is a common autosomal dominant disorder with a birth incidence of 1:2,700, caused by mutations or deletions in the NF1 gene.^1–3 Learning disabilities, cognitive deficits, and behavioral problems in various domains are reported in up to 80% of children with NF1,^4–7 and several promising therapeutic options are emerging from preclinical studies. Lovastatin, a cholesterol-lowering agent, improved learning deficits of Nf1 mice.⁸ Other candidate drugs are l-dopamine and methylphenidate, both improving attention in Nf1 mice in which neurofibromin was selectively knocked out in neuroglial progenitor cells.^9,10 Lamotrigine improves cognitive deficits in 2 separate Nf1 mouse models, through regulation of the excitability of inhibitory interneurons.¹¹

Six clinical trials aimed at treating cognitive deficits in NF1 are reported in the literature (table 1).^12–17 Clearly, there is a lack of consensus on the set of outcome measures used. This article analyzes the scores on outcome measures used in the Simvastatin for cognitive deficits and behavioral problems in patients with neurofibromatosis type 1 (NF1-SIMCODA) trial, which evaluated the effect of 12-month simvastatin treatment on cognitive functioning and behavioral problems in children with NF1. Simvastatin did not have any positive effect on cognitive functioning, behavioral problems, or school performance.¹⁶ We investigated 3 main questions important to the future selection of appropriate outcome measures: (1) Which outcome measures were most affected in comparison to normative reference groups? (2) What is the 1-year test-retest reliability of scores on the outcome measures used? (3) How are the scores on the various outcome measures correlated? In addition, what outcome measures are associated with health-related quality of life?

Table 1.

Cognitive clinical trials in neurofibromatosis type 1 and their outcome measures

METHODS

Patient population.

The data from 84 children who participated in the NF1-SIMCODA trial were analyzed.¹⁶ The children had a median age of 11.5 years (range 7.9–16.0), and 45 were girls (54%). Exclusion criteria for the trial were use of neurotropic medication, including stimulant, antipsychotic, antiepileptic, or antidepressant drugs, or current simvastatin use; symptomatic CNS abnormalities; insufficient comprehension of the Dutch language; severely impaired vision or deafness; segmental NF1; or an IQ below 48.

Standard protocol approvals, registrations, and patient consents.

The standard protocol was approved by the Central Committee on Research involving Human Subjects (the Hague, the Netherlands) and the Ethical Committee of University Hospital Leuven (Belgium) and performed in agreement with the Declaration of Helsinki (2008 version). The trial is registered with The Netherlands Trial registry www.trialregister.nl, number NTR2150. Informed oral and written consent was obtained from parents and assent was obtained from adolescents aged 12 and older.

Outcome measurements.

The primary outcome measures in NF1-SIMCODA were full-scale intelligence measured with the Wechsler Intelligence Scale for Children–III (WISC-III) and the Attention Problems and the Internalizing Behavioral Problems scales of a parent-rated questionnaire, the Child Behavioral Checklist (CBCL).¹⁶ Regarding secondary outcomes, we included more specific neuropsychological tests: the Rey Complex Figure Test (RCFT), the Stroop Color Word Interference task (Stroop CWT), and the Grooved pegboard test. Other secondary outcome measures were teacher-rated school performance, psychosocial quality of life by Child Health Questionnaire (CHQ), and self-reported internalizing behavioral problems. In addition to the prespecified outcome measures,¹⁶ we have added to the current analysis: (1) a distinction between verbal and performance intelligence on the WISC-III-NL; (2) the copy part of the RCFT, since large deficits on the copy might explain large deficits in the delayed recall scores; and (3) behavioral data from parent, youth self-report, and teacher questionnaires on total problems, attention problems, social problems, and internalizing emotional and behavioral problems (e.g., anxiety or depression-like behavior).

Data analysis.

Baseline scores were used from all included participants (n = 84). To allow comparisons of the magnitude of baseline deficits between different scoring systems, outcomes with a standardized scoring system (e.g., IQ score, z scores, or t scores) were converted into SD scores by dividing the effect by the SD in the reference group, such that 0 equals reference group average with a SD of 1. Normality was checked using Shapiro-Wilk tests, and visual inspection of histograms. If any of these violated assumptions of normality, nonparametric tests were used. Differences between the NF1 group and the reference group were tested using independent t tests for normally distributed data and 1-sample Wilcoxon signed-rank tests for non-normally distributed data. Within the placebo group, paired sample t tests were used to calculate significant before-after differences in normally distributed data and Wilcoxon signed-rank tests for non-normally distributed data. Differences were considered significant when p < 0.05. Data are presented as mean ± SD, unless otherwise indicated. Pearson product moment correlation analyses were used to examine associations between first and second measurements in the placebo group, as an indication of test-retest reliability or stability. Bivariate normality (linearity) was assessed by inspecting scatterplots.¹⁸ In this exploratory analysis, we did not correct the significance level for multiple testing, given the high a priori probabilities of multiple significant findings in this study and the interdependency of cognitive measures. Methods to control for multiple testing would unreasonably increase the risk of type II error. Statistical analyses were performed with SPSS version 21 (SPSS Inc., Chicago, IL).

Principal component analysis.

Since it is likely that scores on certain outcome measures are differentially correlated to other outcome measures, we performed a principal component analysis (PCA). This way, the correlation between outcome measures can be grouped in components and the internal structure of the outcome data can be revealed. We performed PCA for those outcome measures that had complete cases in more than 75% of the sample. We excluded variables from the PCA for which Kaiser-Meyer-Olkin (KMO) measures were low: Stroop CWT (KMO = 0.50) and Grooved Pegboard test (KMO = 0.42). The final PCA therefore included outcome measures for behavioral problems (internalizing behavioral problems, attention problems, social problems), cognitive functioning (total performance IQ, verbal IQ, RCFT delayed recall), and health-related quality of life (psychosocial quality of life summary scale from CHQ). Principal component analysis was conducted on these 7 selected outcome measures with oblique rotation (direct oblimin rotation). The KMO verified the sampling adequacy for the analysis, KMO = 0.73, and all KMO values for individual items were at least 0.59 (well above the acceptable limit of 0.5). An initial analysis was run to obtain eigenvalues for each component in the data. Two components had eigenvalues over Kaiser criterion of 1 and in combination explained 68% of the variance. The scree plot showed an inflexion that would justify retaining 3 components. Because this third component had an eigenvalue below 0.7 (Jollefei criterion) and the sample size is small, we retained 2 components.

RESULTS

Children with NF1 (8–16 years) had a mean full-scale intelligence (WISC-III) of 83.3 IQ points (SD 15.6), which is 1.1 SD below population mean (table 1). The largest deviations from the reference group in the cognitive domain were seen on performance IQ and the RCFT delayed recall. Stroop CWT, Grooved Pegboard test, and teacher ratings have incomplete or no Dutch reference groups, therefore the raw scores are displayed. Behavioral problems, as measured by the CBCL, were most prominent for attention problems rated by parents. Also, social problems were commonly reported by parents. Behavioral problems that were less pronounced than could be expected from literature were internalizing behavioral problems rated by parents and internalizing behavioral problems self-rated by children. All baseline group averages were significantly different from reference groups, except for psychosocial quality of life.

One-year test-retest effects in the placebo group.

The placebo group of the trial (n = 39) provides a unique opportunity to examine 1-year longitudinal changes in scores on the outcome measures. Full-scale intelligence was significantly higher after 12 months of placebo use (table 2). This is due to an average 5.5 IQ point increase in performance intelligence. Verbal intelligence did not improve over time. Improvements were seen on Stroop CWT scores, which were lower after 12 months, and on the Grooved Pegboard test. This is not unexpected, as these 2 tests have no adequate age-corrected reference data. Psychosocial health-related quality of life was significantly higher after 12 months. Small reductions in behavioral problems were observed on most domains, but these were not statistically significant.

Table 2.

Degree of cognitive deficits and behavioral problems in the total study population at baseline and test-retest effects in the placebo group

Correlations between before and after measurements give an indication of test-retest reliability or the within-subject stability of the underlying construct.¹⁸ Poor before-after correlation coefficients would indicate that a higher number of subjects would be needed to find significant treatment effects in a randomized trial. Before-after correlations were good to excellent for Wechsler scales, poor for RCFT delayed recall, acceptable for Stroop CWT, and good for Grooved Pegboard test. Behavioral problems reported by parents had questionable correlation coefficients for total problems and social problems but acceptable correlations for internalizing behavioral problems. Of note, before-after correlations were low for attention problems, a scale that was used as an outcome measure in NF1-SIMCODA. Youth self-report before-after correlations (available for 24 subjects in the placebo group) were low for internalizing behavioral problems and social problems, and questionable for total problems and attention problems. Behavioral problems reported by the teacher showed a poor before-after correlation. Importantly, the majority of children changed teachers over the 1-year period. Psychosocial health-related quality of life had questionable before-after correlation.

Analysis of the correlation structure within the outcome data.

Next we analyzed the correlation structure between the scores at baseline on various outcome measures. We had to consider the total sample size of 83 complete cases, allowing for a limited number of variables to be included. We included those outcome measures that were evaluated in a sufficient number of children (see Methods for selection criteria). Table 3 shows the correlation matrix of the selected outcome measures, together with the significance levels. Notably, parent-rated attention problems showed no significant correlation with scores on the Stroop CWT, which we included as a measure of attention. The principal component analysis of these correlations resulted in 2 components. Table 4 shows the component loadings after rotation. The outcome measures that cluster on the same factor suggest that component 1 represents behavioral problems, and that quality of life clusters together with these behavioral problems. Component 2 represents cognitive functioning. The correlation between the 2 components is low (−0.123). These data suggest that the performance of children in the NF1-SIMCODA trial on the cognitive outcome measures was independent from the behavioral problems reported by parents. In addition, it shows that psychosocial quality of life scores as reported by parents correlate much higher with behavioral problems than with cognitive deficits.

Table 3.

Pearson correlations

Table 4.

Summary of principal component analysis for the 7 outcome measures included (n = 83)

DISCUSSION

We analyzed the performance of a range of cognitive, behavioral, and quality of life outcome measures in a real-life clinical trial population of children with NF1 in order to improve clinical trial design of future studies.

This report confirms the broad range of cognitive deficits and behavioral problems in patients with NF1.^4,6 As expected, the participants scored poorly on full-scale intelligence and RCFT delayed recall. In the behavioral domain, parents frequently reported attention problems, but internalizing behavioral problems were less prevalent in our sample than expected from previous studies.^6,19,20 Interestingly, social behavioral problems were a major issue in our study population, and were also correlated to loss of quality of life. Recently, much attention has been directed at the prevalence and characterization of autism spectrum disorder within the NF1 population.^21–24 One ongoing clinical trial is specifically designed to evaluate the effect of simvastatin on autism in children with NF1 (Simvastatin in Neurofibromatosis Type 1–Autism [SANTA] trial, EUDRACT number 2012-005742-38).

The test-retest reliability analysis we performed indicated that the most reliable tests include full-scale intelligence and performance intelligence, followed by neuropsychological tests. Performance intelligence increased by an average 5.5 points in the placebo group, which might be explained by training effects, but placebo effects or regression to the mean are other possible explanations. We found evidence for a poor test-retest reliability of scores on attention problems rated by parents. This finding indicates that the symptoms of attention problems in a given child may fluctuate more strongly than do other outcomes. Our analysis cannot discriminate between test-retest reliability (the test being imprecise) and a true high variability of attention problems over time.¹⁸ Attention problems are considered a key aspect of the neurocognitive profile of NF1 and have been targeted in all therapy trials so far. The low stability of attention problems parent rating might reflect a poorly understood natural variability within the NF1 population that is not observed within the reference population. Test-retest correlations for the attention problems scale measured weeks apart in the reference population are as high as 0.90,²⁵ and long-term stability spanning 2–3 years during school age are around 0.70.²⁶ In a clinical sample of patients with attention deficit/hyperactivity disorder (ADHD), stability of attention problems over 4 years was 0.53.²⁷ Interestingly, Stroop CWT in our study has high stability over 1 year but was not correlated to questionnaire-based attention problems. Apparently, the abilities captured with Stroop CWT such as inhibition, reading proficiency, and speed of information processing are different from the long-term behavioral constructs that are asked for in the CBCL parental questionnaire. The indication that attention problems ratings have high intraindividual variability in NF1 warrants further prospective evaluation.

In addition, we found that behavioral problems, but not cognitive deficits, were strongly associated with psychosocial quality of life on the CHQ-PF50. This is in agreement with a previous study, where behavioral problems were associated with reduced quality of life, but school performance was not.¹⁹ There might be an overlap in constructs measured between the behavioral questionnaires and the quality of life questionnaire, explaining this correlation. Therefore, the CHQ-PF50 seems insensitive to issues these children face in cognition and academic achievement. Future research might focus on developing an NF1-specific health-related quality of life scale to include all items that are relevant to children with NF1, including academic achievement and cognitive performance.

The correlation between cognitive outcome measures and behavioral problems in this population of children with NF1 was poor. Some children displayed mainly behavioral problems, while some only had cognitive problems. A previous study on the correlation of scores on neuropsychological tests for attention with parent-rated questionnaires on working memory and attention found at best moderate correlations between tests and questionnaires.²⁸ In addition, a behavioral diagnosis of ADHD predicted poorer academic achievement.²⁹ It is unclear what causes this heterogeneity. There might indeed be multiple pathways responsible for cognitive deficits and behavioral problems in children with NF1, with interindividual differences in the relative contribution of these mechanisms.³⁰ Since it is unpredictable which neurocognitive substrates or behavioral problems will respond to treatment, it seems justified to include a broad set of outcome measures in early clinical trials, covering cognition, behavior, and quality of life.

Other researchers have commented that outcome measures should focus on transferring tasks from mouse models to humans.^31–33 Paired-associate learning, a subtest of the Cambridge Neuropsychological Test Automated Battery, has been implemented in a running clinical trial of lovastatin in NF1, based on its assumed similarity to the visuospatial learning in the mouse model.³² Although the NF1 population is indeed affected on this particular task, it remains unclear what the clinical relevance is of such tasks. In a similar fashion, it might be tempting to translate the Morris Water Maze task to a human maze-like test, as was done in one study.³³ It is important to respect the ecological differences in behavior between mice and children. Drugs that improve performance of children on a visuospatial learning task, but do not improve clinically relevant patient-reported outcomes such as behavioral problems, academic achievement, and quality of life, are of mostly academic interest.

This study highlights the variability of the neurocognitive profile of NF1 and demonstrates the need to include outcome measures on a variety of cognitive and behavioral domains in clinical trials. Although the results of the current analysis are specific for NF1, the approach of validating outcome measures for cognitive research can be used for other disorders. It is important to scrutinize the few larger trials that have been completed in order to guide future clinical trial design in this field.

GLOSSARY

ADHD

attention deficit/hyperactivity disorder

CBCL

Child Behavioral Checklist

CHQ

Child Health Questionnaire

CWT

Color Word interference task

KMO

Kaiser-Meyer-Olkin

NF1

neurofibromatosis type 1

NF1-SIMCODA

Simvastatin for cognitive deficits and behavioral problems in patients with neurofibromatosis type 1

PCA

principal component analysis

RCFT

Rey Complex Figure Test

WISC-III

Wechsler Intelligence Scale for Children–III

AUTHOR CONTRIBUTIONS

All authors had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Dr. van der Vaart: study concept and design, acquisition, analysis, and interpretation of the data, drafting of the manuscript, critical revision of the manuscript for important intellectual content. A.B. Rietman: acquisition, analysis, and interpretation of the data, critical revision of the manuscript for important intellectual content. Dr. Plasschaert: acquisition, analysis, and interpretation of the data, critical revision of the manuscript for important intellectual content. Dr. Legius: study concept and design, critical revision of the manuscript for important intellectual content. Dr. Elgersma: study concept and design, critical revision of the manuscript for important intellectual content. Dr. Moll: study concept and design, critical revision of the manuscript for important intellectual content.

STUDY FUNDING

Supported by NWO-ZonMW (translational research) to Y.E. and H.A.M., an NWO-ZonMW VICI grant to Y.E., and the Marguerite-Marie Delacroix Foundation to E.L.

DISCLOSURE

The authors report no disclosures relevant to the manuscript. Go to Neurology.org for full disclosures.