Alternative Approaches to Outcomes Assessment: Beyond Psychometric Tests

Abstract

Outcomes assessments in clinical trials involving cognition and behavior rely upon IQ and neuropsychological assessments. These procedures provide limited evaluations of everyday functions. Some participants cannot perform cognitive tasks because they are low functioning or may represent missing data because of inability to travel. Interview-based assessments of adaptive behavior yield results that reflect everyday functions and can be done by telephone regardless of level of cognitive functioning. The design of the neurobehavioral component of the follow-up study for the Management of Myelomeningocele fetal surgery trial is as an example of a clinical trial that incorporates these alternatives.

Introduction

Outcome assessments of children in clinical trials involving cognition and behavior have traditionally relied upon IQ tests, often with supplementation from neuropsychological measures. Such assessments are viewed as key outcomes because they either broadly predict general levels of performance (e.g., IQ) or assess domains, such as executive functions and attention that are often affected by a treatment or other agent of interest. These assessments are reliable and valid for assessing cognition and brain function, and are sensitive to variation in outcomes. However, IQ tests may not be highly sensitive to some cognitive domains impacted by brain dysfunction in specific disorders (1). In addition, IQ tests often have different forms depending on the age level of the child. In a clinical trial, varying forms can be a problem because the age range of children involved in the trial may extend the range of a single IQ test. Although each test yields a norm referenced composite score, the components of this score may vary even if the standardized quotient is used. Finally, while IQ tests are moderately correlated with achievement, there have been long term issues concerning the degree to which an IQ test is sensitive to variations in the everyday, habitual functioning of the child (2).

Neuropsychological tests are often more sensitive to variations in outcomes depending on the disorder and brain function involved. However, many of the same issues emerge that may be more general characteristics of the use of psychometric tests. A specific test may not span the age range of the child and the issue of generalization to everyday functioning can be tenuous. The relation of cognitive assessments of key domains, such as executive functions, to the child’s observed capacity for organizing and regulating their behavior has been frequently questioned, with low correlations reported between psychometric assessments and ratings of executive behaviors (3). Thus, rating scale assessments of executive functions have emerged as supplements and/or alternatives to psychometric assessments of executive functions (4).

A potential problem with any psychometric procedure is that children may be too young or too impaired to perform at a level that makes performance on a test feasible or meaningful. Many IQ and neuropsychological assessments have relatively few items at the bottom of the distribution, so children just emerging into the normative age range of a test or who are simply low performing may obtain scores at the bottom of the distribution that are not discriminating. If there are many children with these characteristics in a clinical trial, scores will accumulate around the lowest possible score, creating skewed distributions that are not suitable for parametric analyses and leading to decisions (e.g., categories scores according to level of impairment) that are costly in terms of statistical power.

One additional issue is that some parents are not willing or able to return to the setting where the treatment and baseline assessments were obtained because they have moved or are to overwhelmed and lack time. Although some reading tests have been validly administered over the telephone (e.g., Test of Word Finding Efficiency, a short, timed word reading fluency assessment) (5), this provides a limited assessment of cognitive proficiency. Examiners can go to the home, but this approach is costly and the quality of the assessment conditions may be uneven across cases. The result is missing data, which creates problems for any clinical trial, especially from an intent to treat perspective because the impact of the missing data cannot be gauged.

This paper discusses the use of adaptive behavior assessments as a supplement to IQ and neuropsychological assessments. Many trials will benefit from the use of adaptive behavior assessments, especially given the age range covered by these assessments and the opportunity to obtain data by interview and ratings. It is reasonable to use adaptive behavior assessments as the primary end point of a clinical trial. Secondary end points should include direct assessments of cognitive performance as well as alternatives to performance-based measures, such as rating scales of cognitive behavior. Adaptive behavior per se, however, is not just another way of assessing cognitive ability, behavioral adjustment, or quality of life. The follow-up of the management of myelomingocele (MOMS2) clinical trial of fetal surgery for spina bifida illustrates the use of adaptive behavior assessment as a primary end point and how it fits in with other assessments.

Adaptive Behavior Assessments

Adaptive behavior is a person’s every day, habitual capacity for independent functioning (6). Behaviors under this rubric represent daily activities needed in order to communicate, relate to other people, and take care of personal needs. The focus is not on what a child can do, but whether the child habitually performs the behavior on a daily basis. What is considered normative adaptive behavior changes with age as a child becomes less dependent on adults to provide care and support. Thus, adaptive behavior assessments typically cover a broad range in terms of age and development. There are daily activities that fall into the domain of adaptive behavior at any age. Adaptive behavior is grouped into domains involving communication or language, social or interpersonal, and daily living or practical skills. The performance of physical or motor activities is assessed. In addition to an overall composite score, each large domain has items that can be broken into smaller domains (e.g., in communication: receptive, expressive, and academic skills).

The three primary assessments of adaptive behavior are the Scales of Independent Behavior-Revised (SIB-R) (7), the Vineland Adaptive Behavior Scales-II (Vineland) (8), and the Adaptive Behavior Assessment System-2 (ABAS-2) (9). Each of these assessments involves a composite for overall adaptive functioning, assessment of the four primary domains of adaptive behavior (E.g., Communication, Socialization, Daily Living, Motor), and subdomain scores (e.g., in Communication: expressive, receptive, and written language), the latter typically criterion referenced. The major scales of each measure are highly reliable. All three scales are based on reports from parents or primary caregivers, or teachers. Each can be done as rating scales, and the Vineland and SIB-R as interviews, although only the Vineland is actually normed for a nondirective, semi-structured interview procedure. The ABAS-2 has separate forms for different age levels, but the Vineland and SIB-R are designed for broad levels of age and capabilities.

Adaptive behavior is a mandatory component of the three pronged criteria used to identify intellectual disabilities (10): multifactorial IQ score two standard deviations below average; adaptive behavior composite (or one of three primary domains) two standard deviations below average; age of onset before 18 years of age. Adaptive behavior assessments are widely used to evaluate individuals with other disabilities characterized by significant difficulties with independence, such as autistic spectrum disorders. Adaptive behavior assessments also provide important dimensions for assessing children with brain injury as illustrated below.

Traumatic Brain Injury

In traumatic brain injury (TBI), adaptive behavior assessments are especially useful because they can be completed early post-injury with the caregiver specifically instructed to provide responses that address the child’s capacity for completing daily, habitual activities prior to injury, thus providing an indicator of premorbid levels of functioning. Subsequent assessments are very sensitive to the severity of TBI and show recovery, with clearly lower scores relative to the baseline (premorbid) assessment in the severely injured group with TBI.

Figure 1 provides findings from a study in which the Vineland was obtained at baseline as a premorbid assessment, and then at 6, 12, and 24 months post injury for children with mild and severe TBI (11). This figure shows clear differentiation of severe versus mild TBI, with a significant reduction at 6 months only in the group with severe TBI. The group with severe TBI shows improvement in the adaptive behavior composite and 12 and 24 months, with relatively stable composites in the mild group. Given that the severe TBI is typically characterized by a diffuse axonal injury (and usually multifocal injury in the frontal and temporal regions), Figure 1 shows that adaptive behavior is primarily affected by severe, generalized injury.

Figure 1.

Vineland Adaptive Behavior Scales composite scores by severity of head injury and follow up interval in children with traumatic brain injury (TBI)

Pediatric Brain Tumors

Adaptive behavior is also assessed in different areas of pediatric oncology and is emerging as part of clinical trials in this area. Ris et al. (12) obtained the Vineland scales, IQ and neuropsychological tests, and rating scales of behavioral adjustment in a large sample of children an average of 111 days after surgery for tumors. Over half the sample showed significant difficulties with adaptive behavior, compared to about 40% with below average intelligence; 25% had evidence of significant difficulties on behavior rating scales and 40% had reading difficulties.

Spina Bifida

Children with spina bifida, myelomeningocele and hydrocephalus are infrequently intellectually disabled and have relative preservation of some language, reading, and social skills. Weaknesses are apparent on measures of language and reading comprehension, motor, mathematics, and nonverbal processing measures (13)., These children have a modal pattern of adaptive behavior strengths and weaknesses on the SIB-R (Figure 2) that parallels results from cognitive and behavioral assessments: a relative strength in the social/communication domain, severe problems with motor development due to the partial to complete paraplegia secondary to the spinal lesion (myelomeningocele) and the fine motor difficulties associated with the Chiari II malformation. Note that the subgroup with thoracic level spinal lesions shows a similar pattern of performance relative to lumbar and sacral lesions, but is more impaired in motor functions.

Figure 2.

Profiles of children with spina bifida myelomeningocele (SBM) and normal comparison children on the Scales of Independent Behavior-Revised (SIB-R) subscales

There are also impairments in both groups in the Personal and Community Living domains, partly due to the urinary complications of spina bifida, but also because of the lack of adequate development in a variety of domains involving self-care, daily living, and community domains. However, this pattern is not just a reflection of their motor or cognitive difficulties, especially across domains. Table I predicts each domain from demographic, cognitive, environmental (e.g., SES), and medical factors that are highly variable in spina bifida. Demographic, environmental, and behavioral adjustment ratings show weak relations with each adaptive behavior domain. Cognitive variables are weak predictors of the Motor domain, and account for less than half the variance in Social/Communication and Community Living domains, although the relations are large and significant. The relation with Community living is largely because of the mathematics difficulties common in people with spina bifida that are required for community independence. Fine motor skills predict the Personal Living domain, which assesses a variety of self-care skills. IQ is generally weakly related to adaptive behavior. Medical variables predict only the Motor domain, largely because of the effect of lesion level. Thus, the SIB-R is sensitive to variations in outcomes of children with spina bifida.

Table I.

Prediction of adaptive behavior domains from the Scales of Independent Behavior-Revised with demographic, cognitive, environmental, and medical variables in children with spina bifida myelomeningocele and hydrocephalus

	Motor	Social/Communication	Personal Living	Community Living
Demographic (R2)	0.03	0.10	0.05	0.12
Cognitive (R2)	0.24	0.40	0.34	0.48
Stanford Binet IQ	0.23**	0.17*	0.06	0.24**
Reading	−0.21*	0.15	−0.02	0.00
Math	0.19	0.36**	0.25**	0.57**
Fine Motor	0.29**	0.08	0.38**	0.10
Environmental (R2)	0.09	0.13	0.10	0.17
Medical (R2)	0.26	0.07	0.15	0.09
Lesion Level	−0.40**	−0.12	−0.16*	−0.16*
Shunt Revisions	−0.05	−0.05	−0.09	−0.21*
Seizures	−0.13	−0.20**	−0.23*	−0.13
Oculomotor	−0.15*	0.00	−0.13	−0.06

^*p< 0.05

^**p< .0001

Management of Myelomeningocele Trial

The MOMS2 study is a follow-up of a prenatal surgery trial for myelomingocele (14). One hundred eighty-three of a planned 200 mothers with fetuses having myelomeningocele below the thoracic level and hindbrain herniation were randomly assigned to standard postnatal repair or prenatal repair of the spinal lesion between 19 and 26 weeks gestation. The trial was terminated because analyses of the primary endpoints at 12 months of age showed a significant reduction in shunt placement for the prenatally repaired group (40%) versus the standard postnatal repair group (82%). In addition, the prenatal repair group had significantly higher composite scores on the Bayley Scales of Infant Development-2 (15) (especially on the Motor scale) at 12 and 30 months, reduced hindbrain herniation at 12 months, and better ambulation at 30 months. However, there was also a higher rate of maternal and fetal complications in the prenatal repair group.

The MOMS2 follow-up will evaluate this cohort at 5–9 years post repair. Table II provides the primary and secondary outcome assessments. Because the follow-up involves multiple assessments, including brain and spine MRI, functional, motor and quality of life assessments, tests of urinary functions, and other interdisciplinary outcomes, the neurobehavioral assessment had to fit into a three hour time period. Note that the primary endpoint is the Vineland, partly because of its sensitivity to variations in outcomes after spina bifida. However, the Vineland can be completed on every child regardless of their level of functioning and capacity for completing psychometric assessments. It does not involve multiple forms. Because of the wide geographic variability of the three participating clinical sites (San Francisco, Philadelphia, Nashville), the Vineland can be completed by phone if the parent is not able or willing to return to the primary clinical site. The procedure will be completed by the nondirective interview in which questions are not read to the caregiver to avoid biased or socially desirable responses that might occur with rating scales.

Table II.

Neurobehavioral assessment procedures for the MOMS2 follow-up

With child

Kaufman Brief Intelligence Test 2

Woodcock Johnson Test of Achievement 3rd Edition Math Fluency Reading Fluency Word Identification Calculation Passage Comprehension

California Verbal Learning Test for Children

Children’s Memory Scale Dot Locations Memory for Faces

Beery Visuomotor Integration Test

Purdue Pegboard

Handedness Inventory

Word Generation

With parents

Vineland Adaptive behavior Scales- II (interview administration)

Behavior Rating Inventory of Executive Function (BRIEF) Parent Form

Swanson Nolan Achenbach Pelham Attention Problems Scale-IV

Child Behavior Checklist

The secondary endpoints involve an abbreviated IQ test that reduces demands for motor-based responses and speed. Fine motor and perceptual-motor tasks, commonly impaired in children with spina bifida and hydrocephalus, which should be sensitive to changes in hindbrain herniation, will be obtained. Reading and mathematics achievement, which are classically dissociated in spina bifida, will be assessed. Verbal and nonverbal tests of memory and learning are administered because these domains are usually very impaired in spina bifida and hydrocephalus (13), partly because of severe damage to the hippocampus in many because of hydrocephalus. Executive functions are assessed by questionnaire partly because of time and also because such assessments may be more sensitive to dysexecutive problems in spina bifida (4, 16). The MRI assessment includes structural and diffusion tensor imaging that will be qualitatively evaluated by pediatric neuroradiologists and also quantitatively analyzed. These assessments may account for variations in neurobehavioral outcomes and also provide direct assessments of the mechanisms whereby fetal surgery reduces the need for shunting.

Conclusions

As the MOMS2 follow-up demonstrates, primary and secondary assessments of outcomes should be interdisciplinary. The neurobehavioral assessment procedures should include a variety of assessments, including cognitive performance and ratings, academic achievement, behavioral adjustment, and quality of life. Adaptive behavior assessments are direct evaluations of the child’s capacity for every day, habitual execution of behaviors that reflect functional independence. As such, these assessments do not simply duplicate other domains. Adaptive behavior assessments also have flexible administrations that can enhance the power of an outcomes study or clinical trial by adequately measuring the lower end of the distribution and ensuring that more children can be evaluated. In addition, these assessments have characteristics in terms of their capacity to capture broad levels of age and development that reflect important functional outcomes involving communication, socialization, and daily living skills.