Parent-reported cognition of children with cancer and its potential clinical usefulness

Abstract

Purpose

Cognitive dysfunction is a common concern for children with brain tumors (BTs) or those receiving central nervous system (CNS) toxic cancer treatments. Perceived cognitive function (PCF) is an economical screening that may be used to trigger full, formal cognitive testing. We assessed the potential clinical utility of PCF by comparing parent-reported scores for children with cancer with scores from the general US population.

Methods

Children (n = 515; mean age =13.5 years; 57.0 % male) and one of their parents were recruited from pediatric oncology clinics. Most children (53.3 %) had a diagnosis of CNS tumor with an average time since diagnosis of 5.6 years. PCF was evaluated using the pediatric PCF item bank (pedsPCF), which was developed and normed on a sample drawn from the US general pediatric population. Children also completed computer-based neuropsychological tests. We tested relationships between PCF and clinical variables. Differential item functioning (DIF) was used to evaluate measurement bias between the samples.

Results

No item showed DIF, supporting the use of pedsPCF in the cancer sample. PedsPCF differentiated children with (vs. without) a BT, p < 0.01, and groups defined by years since diagnosis, p < 0.01. It significantly (p < 0.05) correlated with computerized neuropsychological tests in 40 of 60 comparisons. Children with BTs were rated as having worse pedsPCF scores than the norm, regardless of years since diagnosis.

Conclusions

PCF significantly differentiated cancer survivors with various clinical characteristics. It is brief and easy to implement. PCF should be considered for routine care of pediatric cancer survivors.

Introduction

While cancer is the leading cause of death by disease in children under 15 years old, its overall cure rate for children and adolescents in the United States is approaching 90 % [1]. As a result of treatment progress over the past three decades, it is estimated that one of every 640 US citizens is a survivor of childhood cancer [1], a proportion that continues to grow. Extended remission and cure come with considerable sequelae such as cognitive impairment, which extends into adulthood [2–4]. While late effects of central nervous system (CNS) therapy typically emerge within 1–2 years of treatment, they may be delayed up to 7 years [5–7]. Children with CNS tumors are at particular risk for cognitive difficulties. A recent investigation of childhood cancer survivors found that those with CNS tumors reported the poorest functioning in all aspects of cognition and showed poorer adaptation to adult life, as demonstrated by lower educational, occupational, and financial attainment, as well as decreased likelihood of marriage when compared to non-CNS cancer survivors [8]. These results are consistent with other literature [5, 9–11] showing that although many childhood cancer survivors demonstrate coping and psychosocial adjustment similar to their non-cancer peers [both general population (GP) and children with other chronic conditions], those with academic or other cognitive problems experience worse overall adjustment. Physician awareness of these adverse effects can facilitate surveillance, enhancing early identification and intervention.

Effective post-treatment surveillance requires periodic assessment with standardized measures of cognition. Neuropsychological evaluations have typically been used to estimate a respondent’s cognitive capabilities; however, they are not always feasible due to their length, financial burden to families, and the limited availability of neuropsychological examiners during routine clinical visits. Perceived cognitive function (PCF), whether reported by self or proxy (parent), is a promising screen for impairment with the goal of identifying individuals requiring formal cognitive testing. PCF also correlates with neuroimaging results [12, 13], supporting its validity. To track PCF from childhood through adulthood, a validated measure is needed that accounts for developmental changes. This capability was developed in the GP as a parent-report questionnaire for PCF in their children, referred to as the pedsPCF [14]. The pedsPCF employs an “item bank,” comprised of questions about cognitive functioning, each of which is calibrated using item response theory (IRT) [15, 16]. Among other advantages, an item bank provides a foundation for the development of highly efficient assessment through computerized adaptive testing (CAT) platforms. With CAT, only the most informative items are presented, based on the respondent’s responses to previously presented items [17]. Using this approach, a precise estimate of PCF can be obtained with the presentation of only a few items; such brevity is well-suited for busy clinical practice. The peds-PCF consists of 43 items which show satisfactory psychometric properties as evaluated using both classical test theory and IRT approaches. It produces reliable scores which can discriminate children with (vs. without) significant symptoms of attention, social, and thought problems [14]. US GP-based norms are available to serve as a reference when pedsPCF is used with clinical populations such as children with cancer or adult survivors of childhood cancer. In this paper, we report the first validation study of this parent-report instrument, comparing ratings of cognitive function of children with cancer, including brain tumors (BTs), to children in the general US population.

Methods

This study was approved by Institutional Review Boards at all participating sites.

Sample

A total of 515 cancer patients and for each, one of their parents, were recruited from the Ann & Robert H. Lurie Children’s Hospital of Chicago (formerly, Children’s Memorial Hospital, Chicago), Boston Children’s Hospital, and St. Jude Children’s Research Hospital between July 2009 and December 2010. Eligibility criteria for patients included a diagnosis of BT or another form of childhood-onset cancer [non-brain tumor (non-BT)], and age between 7 and 17 years. Children with BTs who received any type of treatment were eligible for participation, as were children without BTs who received any cognitively toxic treatment such as chemotherapy and brain irradiation. Children at all stages of the disease continuum were recruited for participation. Both patients and parents were required to understand English in order to sign the assent/consent forms and complete the study questionnaires. Demographic information (shown in Table 1) was provided by one parent of each patient. In brief, the average age was 13.5 years (SD = 4), 57.0 % were male, and 70.3 % were White. Half (53.3 %) had a diagnosis of a CNS tumor, followed by leukemia (22.8 %) and Hodgkin’s lymphoma (6.4 %). We over-recruited children with CNS tumors in order to overcome a potential ceiling effect upon PCF ratings. The average time since diagnosis was 5.6 years and the average number of years since last treatment was 3.3; the majority had undergone surgery (71 %) and chemotherapy (71.3 %). Most children had good (22.1 %), very good (38.0 %), or excellent (32.7 %) quality of life as reported by their parents. Parents completed the pedsPCF (described below) in clinics using tablet computers. Additionally, patients completed a computer-based neuropsychological testing battery, CogState™ (CogState Ltd., Melbourne, Australia).

Table 1.

Sample demographic and clinical information

Variable	Categories with the variable
Age (in years)		Mean =13.5 (SD = 4.0)
Years since diagnosis		Mean = 5.6 (SD = 4.7)
Years since last treatment		Mean = 3.3 (SD = 3.9)
Gender	Male	57.0 %
Cancer type	Brain	53.3 %
	Leukemia	22.8 %
	Hodgkin’s disease	6.4 %
	Non-Hodgkin’s lymphoma	3.4 %
Ethnicity	Hispanic origin (yes)	18.7 %
Race	White	70.3 %
	African American	9.5 %
Attending school	Yes	93.7 %
Type of classrooma	Regular classroom; no IEP	55.5 %
	Regular classroom; with IEP	27.5 %
	Special education	7.2 %
	Other	9.7 %
Repeated grade	Yes	11.3 %
Education (father)	High school grad or less	35.0 %
	Some college	23.3 %
	College degree	24.8 %
	Advanced degree	17.9 %
Education (mother)	High school grad or less	26.9 %
	Some college	28.0 %
	College degree	29.1 %
	Advanced degree	15.9 %
Current extent of diseaseb	Local	41.6 %
	Regional	7.9 %
	Metastasis	4.9 %
	Not evidence of disease	45.6 %
Treatment	Chemotherapy	71.3 %
	Radiotherapy	34.2 %
	Surgery	71.0 %
	Both chemotherapy and radiation	26.4 %
Radiation typec	Limited field/localized	32.9 %
	Craniospinal	25.9 %
	Proton beam	15.9 %
	Whole brain	4.7 %
	Gamma knife	2.4 %
	Intrabeam	1.2 %
	Others	11.8 %
Karnofsky or Lansky performance statusd	100	75.9 %
	90	15.5 %
	70–80	7.1 %
	50–60	1.5 %
Parent-rated child’s quality of life	Excellent	32.7 %
	Very good	38.0 %
	Good	22.1 %
	Fair or Poor	7.2 %

^aOnly those attending school were included. IEP: Individualized educational program

^b% was calculated using non-missing data (n = 305). Disease severity was not documented in a consistent manner across recruitment sites as well as across cancer types and thus not reported here

^c% was calculated based on children who received radiotherapy (n = 170)

^dClinician rated

CogState is a computerized battery of tasks designed to measure various aspects of cognitive functioning. Tasks from the CogState battery administered in the current study included measures of processing speed, attention, learning, and working memory. CogState was chosen for this study because it is easily administered and relatively brief, decreasing the response burden of patients. Though the CogState is a relatively new tool, it has been used in a variety of populations such as posterior cranial fossa lesions [18], pediatric attention deficit disorder [19], pediatric developmental coordinator disorder [20], concussion [21], and HIV-related dementia [22].

Pediatric perceived cognitive function item bank (PedsPCF)

The development of the pedsPCF is documented elsewhere [14, 23]. In brief, the pedsPCF was developed from the perspectives of children, parents, teachers, and clinicians using qualitative approaches. Its psychometric properties were evaluated using data collected from 1,409 children aged 7–17 drawn from the US general pediatric population and their parents. The final pedsPCF item bank consisted of 43 items, in which unidimensionality of items and stability of measurement properties between sub-samples were supported. The pedsPCF significantly differentiated samples defined by characteristics such as medication use for attention deficits, children who had repeated a grade, special education status, presence versus absence of a neurologic diagnosis, and relevant symptom clusters, with large effect sizes (>0.8), and predicted symptom accuracy rates ranging from 79 to 89 %. In this study, a higher pedsPCF score indicates better function.

Analysis

A two-stage analysis flowchart is shown in Fig. 1. The first stage was to ensure acceptable psychometric properties of the 43 pedsPCF items within the cancer sample. Once it was confirmed, we compared the measurement properties of the pedsPCF between cancer and GPs using the IRT model. GP-based data as described in Lai et al. [24] were used for this purpose. A finding of comparable measurement properties supports the use of the GP-based T score system [14] in the clinical sample. Otherwise, direct comparisons between the two populations would not be recommended. The second analysis stage evaluated the clinical utility of the pedsPCF among children with cancer.

Fig. 1.

Analysis flowchart

Stage 1 analysis

Confirmatory factor analysis was used to evaluate the unidimensionality of items within the cancer sample (criteria: comparative fit index, CFI > 0.9; root mean squared error of approximation, RMSEA<0.08; R² >0.3; modification index< 10) [25, 26]. Item fit was evaluated using S-χ² and S-G² (criterion: p >0.01) [27], and item parameters were estimated using an IRT model—Samejima’s [28] graded response model (GRM). Details about IRT models and their applications to patient-reported outcomes are documented elsewhere [16, 29].

Item parameters were used to estimate information functions at the level of individual items and at the level of the entire item bank and to characterize the precision of items and the overall scale on the measurement continuum. Items with higher information functions are more likely to be chosen by CAT, given their high precision and reduced error rate at measuring PCF. We evaluated the stability of an item’s measurement properties using differential item functioning (DIF) [30, 31] analyses and IRT-scaled scorebased ordinal logistic regression [32] both within the cancer sample as well as between the cancer sample and the GP. Variables examined within the cancer sample were as follows: child’s age (<12 vs. ≥12 years), race (white vs. non-white), paternal education (high school graduate vs. lower), maternal education (high school graduate vs. lower), gender, and diagnosis (BT vs. non-BT cancer). We evaluated uniformed (analogous to a significant group effect and conditional on the latent trait) and non-uniformed (equivalent to a significant interaction of group and trait) as well as overall DIF. Items that showed significant DIF (criterion: p < 0.01) with non-negligible magnitude (R² > 0.02) [33, 34] in more than one comparison were candidates for removal from the pedsPCF due to potential measurement bias. Items without DIF within cancer sample were then compared to the GP to evaluate DIF between cancer and GPs. IRT-scaled scores were then generated using the GRM model using both GP-based as well as cancer population-based parameters. If the scaled scores resulted from these two sets of parameters were similar, the GP-based parameters were used for remaining analyses.

Stage 2 analysis

Analysis of variance (ANOVA) and t tests were used to examine the discriminative validity of the pedsPCF in relation to clinical variables, including diagnosis (BT vs. non-BT), years since diagnosis (<1, 1–4.99, or ≥5 years), years since last treatment (<1, 1–4.99, or ≥5 years), treatment modality (no radiation or chemotherapy, radiation only, chemotherapy only, or both radiation and chemotherapy), and educational placement [without individualized education program (IEP) vs. with IEP, including any other form of special education]. We expected a better pedsPCF score for children with non-BT, shorter length since diagnosis, non-radiation therapy, and regular classroom attendance. Multiple regression analysis was then used to identify significant predictors of pedsPCF scores. Associations between PCF and CogState variables were examined using Pearson’s correlations. Discriminant function analysis (DFA) [35] was used to estimate cut-off scores between BT and non-BT groups and between the BT group and the GP.

Finally, we evaluated the relative clinical utility of the full-length pedsPCF, CAT, and a short-form PedsPCF by comparing scores obtained from each. The CAT stopping rule used in this study was as follows: standard error of measurement<0.3 or numbers of items exceed 12, whichever came first. The CAT scores were from a CAT simulator developed by the Patient-Reported Outcomes Measurement Information System (PROMIS, www.nihpromis.org) initiative. We constructed a short-form to serve as an example by selecting the PCF items with maximum information functions (described in Stage 1 analysis).

Results

Stage 1 analysis

Acceptable fit indices were found in the factor analysis (CFI = 0.95, RMSEA = 0.082). Although the RMSEA value is only slightly higher than the preset threshold (<0.08) and was considered “adequate” by MacCallum et al. [36], we thus considered it to be acceptable and to confirm unidimensionality of the 43 PCF items. All items had an acceptable IRT fit index [27] (p > 0.01), and no items showed significant DIF in the cancer sample or between the cancer and GP samples. This finding confirms comparability of PCF measurement properties between cancer and GPs and supports the use of the GP-based scoring system in the current clinical samples.

Stage 2 analysis

Results showed that BT had significant worse PCF scores than non-BT, t = 5.65, p< 0.01, and those who did not receive IEP had better PCF scores than those who did, t = −4.25, p < 0.001. PedsPCF significantly predicted “years since diagnosis,” F(2,512) = 8.94, p < 0.01, and “years since last treatment,” F(2, 490), F = 4.22, p = 0.02; but not treatment modality, F(3, 502) = 0.87, p = 0.46. Specifically, for “years since diagnosis,” “<1 year” had significantly (p< 0.05) better PCF scores than “1–4.99” and “ ≥5 years” but there was no significant difference between “1–4.99” and “≥5 years.” For “years since last treatment,” “<1 year” had significantly (p < 0.05) better PCF scores than “≥5 years.” However, diagnosis was the only significant predictor in multiple regression analysis including years since diagnosis, years since last treatment, treatment modality, and interaction effects. For children who received craniospinal or whole brain radiation (n = 52), a moderate correlation was found between years since last treatment and PCF, r = −0.43, p<0.01. As shown in Fig. 2, BT group scored consistently lower than GP without neurological conditions (mean = 54.19) regardless of years since last treatment, F = 8.84, p < 0.001.

Fig. 2.

Deviation of the pedsPCF T scores from the US general population who did not report any neurological condition (mean = 54.19)

Table 2 shows that 40 out of 60 correlations between PCF and CogState were significant (p< 0.05). Specifically, PCF scores were significantly (p < 0.001) correlated with CogState indices of processing speed, attention, learning, and working memory, r = −0.28, −0.25, 0.17, and 0.30, respectively. When evaluating this relationship by diagnostic groups, for the BT group, the PCF was significantly correlated with processing speed across all years, with correlation coefficients ranging from 0.25 (>5 years since diagnosis) to 0.66 (< 1 years since diagnosis), and with attention and working memory, with correlation coefficients ranging from 0.24 (>5 years since diagnosis for both attention and working memory) to 0.41 (1–5 years, working memory). For the non-BT group, PCF was significantly correlated with cognitive domains for children at least 1 year post-treatment with correlation coefficients ranging from 0.25 to 0.49. PCF was significantly correlated with all cognitive domains for children >5 years postdiagnosis (r ranged from 0.29 to 0.38) and with working memory for all time periods. One exception was for children >5 years post-treatment; no significant correlation between PCF and working memory was found.

Table 2.

Correlations between PCF and computerized neuropsychological tests (CogState) by either “years since last treatment” or “years since diagnosis”

	n	Processing speed (reaction time)	Attention (reaction time)	Learning (accuracy)	Working memory (accuracy)
All sample	515	−0.28**	−0.25**	0.17**	0.30**
BT only	269	−0.33**	−0.29**	0.08	0.30**
Non-BT only	246	−0.23**	−0.18**	0.18**	0.28**
Years since last treatment * cancer type
BT
<1 year	74	−0.36**	−0.30**	0.18	0.35**
1–4.99 years	101	−0.38**	−0.38**	0.13	0.41**
5+ years	80	−0.27*	−0.21	−0.12	0.09
Non-BT
<1 year	95	−0.03	0.00	0.00	0.17
1–4.99 years	76	−0.32**	−0.39**	0.25*	0.39**
5+ years	67	−0.45**	−0.29*	0.35*	0.27
Years since diagnosis * cancer type
BT
<1 year	43	−0.66**	−0.40*	0.29	0.39*
1–4.99 years	84	−0.31**	−0.31**	0.10	0.38**
5+ years	142	−0.25**	−0.24**	0.00	0.24**
Non-BT
<1 year	56	−0.13	−0.08	0.06	0.28*
1–4.99 years	94	−0.14	−0.21	0.05	0.24*
5+ years	96	−0.38**	−0.29**	0.35**	0.38**

BT Brain tumor

^*p<0.05;

^**p<0.01

^aFor reaction time (i.e., processing speed and attention), lower scores represent better performance

^bFor accuracy (i.e., learning and working memory), higher scores represent better performance

^cTotal numbers of comparisons (correlations) = 60

DFA results showed that pedsPCF significantly (p < 0.001) predicted children’s clinical status, with canonical correlations of 0.25 and 0.33, and correct prediction rates of 60.2 and 76.7 %, respectively, for the BT versus non-BT as well as for the BT versus the GP without reported neurological conditions, respectively. Accordingly, cut-off scores were 40.3 and 51.2 for GP without reported neurological condition versus BT, and between BT and non-BT, respectively.

Applications: CAT simulation and short-form construction

In CAT simulation, the average number of items needed to estimate PCF was 5.5 (SD = 2.8). Individual PCF scores obtained by using all 43 items were highly correlated with CAT results, r = 0.94, supporting the utility of a CAT testing platform in this clinical population. Using the information functions produced by IRT analyses and ranks of items frequently selected during CAT simulation, we created a 7-item short-form (shown in Table 3). We purposely excluded school-related items in this version in order to maximize its generalizability to children who do not attend mainstream classrooms at the time of the assessment. Figure 3 compares PCF scores generated from CAT and from this sample short-form to the scores obtained when participants completed all 43 items.

Table 3.

Items included in the sample pedsPCF short-form

	None of the time	A little of time	Some of the time	Most of the time	All of the time
1. Your child has to work really hard to pay attention or he/she makes mistakes	5	4	3	2	1
2. Your child forgets things easily	5	4	3	2	1
3. Your child has trouble remembering to do things like school projects or chores	5	4	3	2	1
4. Your child has trouble keeping track of what he/she is doing if he/she gets Interrupted	5	4	3	2	1
5. Your child has to use written lists more often than other people his/her age so he/she will not forget things	5	4	3	2	1
6. Your child has to read things several times to understand them	5	4	3	2	1
7. It is hard for your child to pay attention to one thing for more than 5–10 min	5	4	3	2	1

School-specific items are purposefully excluded from this short-form in order to maximize its generalizability

Fig. 3.

Comparisons between scores from full-length 43-item peds-PCF item bank, computerized adaptive testing (CAT), and short-form (SF)

Discussion

PCF is a valuable source of information about a child’s cognitive functioning that is relevant to pediatric cancer disease and treatment variables. This information can be very useful to clinicians and can be an important complement to extensive neuropsychological testing in clinical settings. PCF assessment systematizes much of what clinicians in such settings informally ask patients and parents during follow-up care. A formal PCF measure offers the benefit of standardizing such questions, allowing monitoring of changes over time, and further serving as a reference for timely referral for comprehensive cognitive evaluation. The comprehensive and psychometrically sound PCF item bank enables the development of brief-yet-precise assessments, which are easily implemented in routine clinical follow-up. PCF also has the advantage of ecological validity and easy accessibility, circumventing the obstacles of financial burden to families and limited availability of trained professionals needed for neuropsychological testing [37–39].

Neuropsychological testing provides precise measures of specific cognitive components. However, its ecological validity as a predictor of everyday functioning has been a focus of concern [37, 40], because such testing is conducted within a highly structured environment that can deviate significantly from everyday life. The pedsPCF was designed to serve as a complementary source of information about everyday cognition-related behaviors and to facilitate efficient and cost-effective referral for comprehensive neuropsychological testing. We believe that PCF is sensitive to children’s everyday functioning and that it represents the interplay between a child’s cognitive capability and the external environment. We found significant correlations between pedsPCF and CogState testing.CogState was chosen for its brevity and its sensitivity to disruption of components of information processing known to be affected by neurological and neurodevelopmental disorders. Though CogState has been used in clinical populations including BTs (posterior cranial fossa lesions [18]), its reliability and validity in children with cancer are not well documented. Future studies comparing pedsPCF with other conventional comprehensive neuropsychological testing are warranted, not only to replicate the study results but also to establish the specificity and sensitivity of PCF in relation to specific components of cognition.

Our ultimate goal is to implement applications of the PedsPCF, such as CAT, in routine follow-up of longterm cancer survivors. PedsPCF methodology offers various ways to accomplish this goal. Parents and patients could complete the pedsPCF at home by internet at designated follow-up intervals, with data automatically stored into an electronic medical record accessible by health care providers. An automated e-mail alert could be generated to providers when a pedsPCF score exceeds a preset threshold. For example, we found a T score of 40 (1 SD worse than the mean of the GP) to be a reasonable screening cut-off score [14]. A clinician receiving an alert of a T score tripping this threshold could then follow-up with a patient and their parents for a possible clinic visit.

In this study, we took a rigorous psychometric approach to ensure that the measurement properties of the pedsPCF were adequately examined and potential biases minimized. Results indicate that the same scoring system developed in the US pediatric GP can be applied with pediatric cancer survivors. Furthermore, although the physiological basis of cognitive concerns may be different in other conditions with cognitive comorbidity such as epilepsy or hydrocephalus, these conditions appear to impact children’s daily cognitive functioning in a manner resembling pediatric cancer. Further studies validating the pedsPCF in other chronic conditions are therefore recommended to determine whether the same scoring system could be used, allowing comparison of cognition-related behaviors of children with other conditions in a psychometrically sound manner.

We acknowledge limitations of this study. Although data were collected from three major pediatric oncology clinics in the United States, the current clinical sample cannot be considered nationally representative. The types, locations, and grades of tumors all contribute to different treatment protocols, which result in various treatment outcomes and late effects. As a result, the extent to which the current results can be generalized to other clinical populations needs to be determined. A major criticism of self- or parent-reported cognition is the nature of the underlying trait being measured. Multiple factors such as emotional well-being and personality can also contribute to perceived cognition. Though this study has paved the way for better measurement of PCF, future studies evaluating the relationships among these factors are warranted. We focused on parent-report cognition due to the concern about children’s immature metacognition. Metacognition reflects the experience and knowledge of an individual has about his/her own cognitive processes [41, 42]. This includes knowledge of their own information-processing skills, knowledge about the nature of cognitive tasks, and knowledge about strategies for coping with such tasks, executive skills related to monitoring and self-regulation of an individual’s cognitive activities [43]. To understand the predictivity of the pedsPCF in children’s real life, future studies should be conducted which PCF is studied as a predictor of real-world outcomes such as special education utilization; academic, employment, and financial attainment; and independent living.

In conclusion, PCF as measured using the pedsPCF discriminated patients with presence versus absence of a BT. It was moderately correlated with years since diagnosis for children who received craniospinal or whole brain radiation, and moderately correlated with computerized neuropsychological testing. Children with cancer showed poorer PCF compared to the US pediatric GP regardless of years since diagnosis or treatment. CAT and short-form applications of the PedsPCF can provide brief-yet-precise PCF estimates with the potential to aid clinicians in identifying patients at neuropsychological risk and to facilitate timely referral for comprehensive cognitive assessment while minimizing response and economic burdens to families. We believe that our results support the importance of monitoring PCF of children with cancer and that such monitoring should be included in their long-term follow-up care to facilitate appropriate referral for childhood cancer survivors.