Why Computerized Adaptive Testing In Pediatric Brain Tumor Clinics

Abstract

CONTEXT

Monitoring of health-related quality of life (HRQOL) and symptoms of patients with brain tumors is needed yet not always feasible. This is partially due to lack of brief-yet-precise assessments with minimal administration burden that are easily incorporated into clinics. Dynamic computerized adaptive testing (CAT) or static fixed-length short-forms, derived from psychometrically-sound item banks, are designed to fill this void.

OBJECTIVE

This study evaluated the comparability of scores obtained from CATs and short-forms.

METHODS

Patients (ages 7–22) were recruited from brain tumor clinics and completed PROMIS CATs and short-forms (Fatigue, Mobility, Upper Extremity, Depressive Symptoms, Anxiety, and Peer Relationships). Pearson correlations, paired t-tests, and Cohen’s d were used to evaluate the relationship, significant differences and the magnitude of the difference between these two scores, respectively.

RESULTS

Data from 161 patients with brain tumors were analyzed. Patients completed each CAT within 2 minutes. Scores obtained from CATs and short-forms were highly correlated (r=0.95 – 0.98). Significantly different CAT versus short-form scores were found on 4 (of 6) domains yet with negligible effect sizes (|d| < 0.09). These relationships varied across patients with different levels of reported symptoms, with the strongest association at the worst or best symptom scores.

CONCLUSIONS

This study demonstrated the comparability of scores from CATs and short-forms. Yet the agreement between these two varied across degrees of symptom severity which was a result of the ceiling effects of static short-forms. We recommend CATs to enable individualized assessment for longitudinal monitoring.

INTRODUCTION

Advances in medical treatment have enabled increasing numbers of children and adolescents with brain tumors to become survivors, such that the 5 year overall survival rate for pediatric brain tumor patients is now 75% (source: Surveillance, Epidemiology, and End Results [SEER] Program; seer.cancer.gov). However, these same lifesaving treatments can also have significant short and long term negative effects on brain tumor patients’ lives. As survival has increased, the scope of cancer treatment has broadened to include prevention and amelioration of both tumor and treatment-related acute and chronic problems. Compared to both healthy controls and pediatric survivors with other types of cancer, brain tumor patients and survivors show worse social, emotional, cognitive, physical and school functioning and health-related quality of life (HRQOL) (1–10) along with greater likelihood of unemployment, financial challenges and legal difficulties (e.g., workplace discrimination, disability insurance denial, being a victim of theft, fraud or assault, etc.) during adulthood.(11, 12) We believe that monitoring health status and HRQOL of children with brain tumors during routine follow-up may at least partially ameliorate these consequences through provision of appropriate and timely intervention. For example, physical exercise training interventions may improve physical fitness for some types of cancer patients and survivors.(13) Mathematics abilities and visual working memory of childhood cancer survivors could be improved by an effective early intervention.(14)

Patients’ reports of their subjective experience (patient reported outcomes) are increasingly accepted as valid means of ascertaining the varied effects of diseases and their treatments on their lives. Standardized assessment of patient-reported outcomes (PROs) has been shown to improve patient-clinician communication, influence clinical decision-making, result in better symptom management(15–20) and be acceptable to both patients and clinicians.(21) Yet such assessments are not widespread, and there are multiple challenges to implementing routine PRO assessment into the clinical setting (22, 23) including concerns about increased patient and clinician burden, uncertainty about interpreting and using PRO results, and lack of widely accepted measurement tools. The Patient Reported Outcome Measurement Information System (PROMIS)(24, 25) is a set of valid and reliable measures that can be used to assess symptoms, function and multiple aspects of health-related quality of life. Most PROMIS measures are calibrated using item response theory models, allowing them to be administered using a variety of formats, including computerized adaptive tests (CATs) and short, fixed length forms (SFs).(24, 26, 27) CAT uses iterative processes to obtain brief-yet-precise estimations, using the following steps: all participants will first complete a screening item; an initial score will be estimated based on the response using the pre-programmed algorithm; the next most informative item around the estimated score on the measurement continuum will be selected by the algorithm for participants to complete; and the score is re-estimated based on the participant’s response to that item. This iterative estimation process continues until the stopping rule is met. As a result, precise estimation can be achieved by using just a few items since only the most informative item will be chosen.(28–30) SFs, on the other hand, consist of a fixed set of items that are the same for every participant. When SFs are created from a calibrated item bank, like PROMIS, the desired measurement properties of the SF can be taken into consideration. For example, a researcher or clinician may select items from the item bank that target precise estimation of more severe symptoms or a set of items that provide moderate precision across the full spectrum of severity.

The brevity and ease of use of CAT and SFs can enhance implementation in clinic settings. Indeed, several studies have demonstrated that pediatric cancer patients and survivors find PROMIS measures to be acceptable and feasible to complete in both inpatient and outpatient settings.(31, 32) Varni et al(33) compared CAT and SF versions of pediatric PROMIS measures in a mixed sample from the general population and asthma clinic patients. They found that scores on the PROMIS pediatric measures are highly correlated regardless of the scoring or administration technique. However, the CAT stopping criterion (minimum and maximum number of items allowed) used impacted the desired precision levels of the scores, especially for those who were at extreme ends of the measurement continuum (i.e., either ceiling or floor).

CAT administration makes individualized assessment possible as items are selected iteratively according to patients’ responses. As a result, CAT is ideal for monitoring patients’ HRQOL at different occasions since scores are comparable regardless of whether the same items are administered at each assessment.(24, 34) We believe that children with brain tumors can benefit from this feature as most of them require long-term follow-ups throughout their life. Yet since CAT administration relies on the use of an electronic device, it may not be feasible to some clinics with limited resources. In this situation, SFs are considered reliable alternatives to obtain patients’ HRQOL.(34–36) To our knowledge no studies have been conducted to evaluate implementation of CAT administration in pediatric neuro-oncology settings. Nor has anyone compared the properties of fixed-length short-forms versus CAT versions in a pediatric brain tumor population. The purpose of this study was to compare scores on pediatric PROMIS measures obtained by using CAT and SF. Understanding the potential commonalities among and differences between these two administration formats for PROMIS measures will allow investigators to select the most appropriate applications to meet their needs.

METHODS

All study procedures were approved by each institution’s review board (IRB).

Participants and Procedures

Patients with brain tumor between the ages of 5 and 22 and their parents were recruited from the Ann and Robert H. Lurie Children’s Hospital of Chicago, the Northwestern Medicine Chicago Proton Center, and Boston Children’s Hospital. Patients could be at any stage of their treatment continuum (including pre- and post-treatment and long-term survival) and undergoing or have undergone any type of cancer treatment (chemotherapy, radiation, and/or surgery.) Patients and parents were excluded from the study if they lacked sufficient literacy to read and understand consent/assent forms in English or respond to the questions.

Once consented, patients (ages 7–22 years) completed the baseline study questionnaires using self-report of PROMIS CATs and short-forms measuring domains of Fatigue, Mobility, Upper Extremity Function, Depressive Symptoms, Anxiety, and Peer Relationships. While pediatric PROMIS was validated on children ages 8–17, we included children with an age of 7 (n=5) based on our previous experiences showing 7 year olds are able to complete self-reported symptom and HRQOL measures.(37) We also included patients with ages 18–22 (n=15) to capture patients who were transitioning into adult clinics. These items have been used on patients ages 18–25 years,(38) supporting the inclusion of these young adults. Parents completed demographic information and other symptom and HRQOL measures (data were not reported in this manuscript). Parents of patients ages 5–7 completed proxy versions of the pediatric PROMIS measures. Measures were either completed in-clinic using a tablet computer or, if not convenient for the participant, from home or any other place with an internet connection. Each participant first took the CAT version of a given pediatric PROMIS measure and then any remaining items in the associated SF that were not presented during the CAT. A stopping rule for CAT testing included the use of a minimum of 5 items with a maximum of 15 items (outside of Peer Relationship all other CATs had a maximum of 13 items) and/or a standard error of 3, whichever came first. Short Form versions contain the following number of items: Fatigue (10), Mobility (8), Upper Extremity Function (8), Depressive symptoms (8), Anxiety (8), and Peer relationships (8). Clinical data were obtained via medical chart review. Only data from patient self-reported versions were presented in this manuscript.

Statistical Analyses

Descriptive statistics were calculated for time and number of items to complete PROMIS CATs. To avoid interrupting clinical flow and participant exhaustion, participants were allowed to take a break during the assessment, thereby extending the time to complete an assessment. Thus, times to complete a CAT of greater than one hour were considered unrealistic and were coded as missing. All PROMIS measures were scored using IRTPRO and established item parameters and reported using a T-score metric, in which the general population based mean=50 and standard deviation=10. We used Pearson correlations (criterion: >0.7)(39) to describe the relationship between scores obtained from CAT and SF; paired t-tests to evaluate the significant differences (criterion: p<0.05) and effect size (ES; Cohen’s d) (criterion: >0.2) to describe the magnitude of the difference between these two scores. For each pediatric PROMIS measure, participants were classified into three groups based on their T-scores: <45 (1/2 SD below norm), 45–55 (1/2 SD within the norm), and >55 (1/2 SD above the norm). Analyses were conducted with all patients together as well as within these severity groups. Higher scores represent worse Fatigue, Anxiety, and Depression. Conversely, higher scores represent better functioning on Mobility, Upper Extremity Function and Peer Relationship.

RESULTS

Data from 161 children with brain tumors were analyzed. The mean age of the children was 13.9 years (range = 7 to 22). The sample was gender-balanced (53.9%), primarily white (78.9%) and just over half (55.7%) attended regular classrooms. Mean time since tumor diagnosis was 5.2 years, with an average of 3.7 years since last treatment. Yet about a third of patients received their last treatment within one year. Almost all children (94.7%) had been treated for their tumor (chemotherapy, radiotherapy, and/or surgery) with 20.3% receiving all three forms of treatment. More demographic and clinical characteristics of the sample are shown in Table 1.

Table 1.

Sample demographic and clinical information

Variable	Categories with the variable
Age (in years)		Mean=13.9 (SD=3.7)
Years since diagnosis		Mean=5.2 (SD=4.6)
Years since last treatment		Mean=3.7 (SD=3.4)
	<= 1 year; > 1 year	36% (64%)
Years since last chemotherapy		Mean=5.1 (SD=12.0)
	<= 1 year; > 1 year	39.8% (60.2%)
Years since last radiation		Mean=3.6 (SD=3.3)
	<= 1 year; > 1 year	37.9% (62.1%)
Years since last surgery		Mean=5.4 (SD=11.5)
	<= 1 year; > 1 year	25.3% (74.7%)
Gender	Male	53.9%
Tumor Type	Embryonal tumors Medulloblastoma	23.0%
	Ganglioma	18.3%
	Pilocytic Astrocytoma	13.5%
	Astrocytoma (diffuse, infiltrative, fibrillary)	11.1%
Status of tumor	Initial diagnosis	85.8%
	Recurrence	14.2%
Treatment	Chemotherapy	77.5%
	Radiotherapy	54.2%
	Surgery	69.9%
	Surgery, chemotherapy, and radiation	20.3%
	No surgery, chemotherapy and radiation	5.3%
Radiation type c	Limited field/localized	7.4%
	Craniospinal	20.6%
	Proton beam	52.9%
Race	White	78.9%
	African American	5.5%
Attending School	Yes	95.4%
Type of Classroom a	Regular classroom; no IEP	55.7%
	Regular classroom; with IEP	32.8%
	Special education	7.4%
	other	4.1%
Karnofsky or Lansky performance statusd	100	57.0%
	90	32.0%
	70–80	9.4%
	50–60	1.6%
Parent-rated child’s quality of life	Excellent	28.4%
	Very good	37.0%
	Good	27.6%
	Fair or Poor	7.1%

^aOnly those attending school were included. IEP: Individualized educational program

^b% was calculated using non-missing data (n=305). Disease severity was not documented in a consistent manner across recruitment sites as well as across cancer types, and thus not reported here.

^c% was calculated based on children who received radiotherapy (n=170)

^dClinician rated

Selected features of CAT and SF administration are shown in Table 2. Average number (standard deviation; SD) of items to complete each CAT was 9.7 (2.9), 8.7 (2.8), 8.1 (3.3), 10.4 (2.7), 8.3 (3.4), 8.1 (3.2) for Anxiety, Fatigue, Mobility, Upper Extremity Function, Depression, and Peer Relationships, respectively. Patients completed each CAT within 2 minutes, ranging from 1.3 to 2.0 across measures. Strong correlations between CAT and SF scores were found across all domains, ranging from 0.95 (Depression) to 0.98 (Peer Relationships). However, as shown in Figure 1, in general, more skewed distributions were observed in short-form scores than in CAT scores.

Table 2.

Comparisons between CAT and Short-form

Item bank	Computerized Adaptive Testing								Short-Form (T-Score)				CAT vs. SF T-score
	Number of items administered					T-Score
	N	Time (SD)a	Mean (SD)b	Min	Max	Mean (SD)	Min	Max	N	Mean (SD)	min	max	Changec (CAT-SF)	Paired ttest	Effect size d	Pearson r
Anxiety	140	1.38 (1.69)	9.7 (2.9)	5	13	42.7 (10.7)	31.3	72.2	128	41.9 (10.5)	32.3	72.9	0.8	p=0.035	0.07	0.966
Fatigue	161	2.01 (3.96)	8.7 (2.8)	5	13	43.7 (12.9)	25.4	73.8	147	44.8 (11.6)	30.3	73.0	−1.1	p<0.001	0.08	0.976
Mobility	157	1.46 (0.98)	8.1 (3.3)	5	13	48.1 (9.4)	21.2	61.7	144	47.6 (9.1)	25.0	58.5	0.5	p=0.004	0.05	0.972
Upper Extremity	148	1.3 (0.97)	10.4 (2.7)	5	13	48.6 (9.2)	25.7	57.2	136	48.6 (9.7)	21.6	56.7	0	p=0.974	0.00	0.962
Depression	145	1.31 (2.46)	8.3 (3.4)	5	13	45.2 (11.2)	31.8	72.0	130	45.0 (10.6)	35.2	75.7	0.2	p=0.704	0.02	0.947
Peer relationship	137	1.49 (1.95)	8.1 (3.2)	5	15	49.8 (10.5)	17.0	66.0	125	49.4 (10.7)	17.7	64.4	0.4	p=0.010	0.03	0.978

^aTime to complete CAT, in minutes

^bNumber of CAT items administered

^cMean differences between CAT and SF T-scores

^dMean difference / standard deviation of differences

Figure 1.

T-Score Distributions and Correlations between CAT and SF

Paired t-test results showed significant (p<0.05) differences between CAT and SF scores on Anxiety, Fatigue, Mobility, and Peer Relationship (Table 2). However, their effect sizes were all less than 0.2 (range between 0 and 0.08) indicating these differences were negligible. Distributions of the differences between CAT and SF scores are shown in Figure 2, in which majorities of differences were around zero. When comparing CAT and SF by T-score groups (i.e., <45, between 45 and 55 (inclusive), and >55), as shown in Table 3, most correlation coefficients remained at acceptable levels (>0.7) except for Depression. An unexpected low correlation of 0.18 was found for those whose T-scores were between 45 and 55. Significant differences between CAT and SF T-scores of non-negligible magnitude (ES>0.2) were identified at the better functioning end of the continuum on all measures, except for Anxiety T-scores, which showed larger differences at the worse functioning end (p=0.021, ES=0.60). However, all differences between CAT and SF were 2.5 T-score points or less and most were less than 1 T-score unit. An extremely high ES (39.19) was found on patients with Upper Extremity Function CAT T-scores greater than 55. This was because this group had low variance (standard deviation=0.01) of T-scores resulting in a large ES, even though the mean difference was minimal (mean=0.50).

Figure 2.

Distribution of Differences and Agreement between T-Scores Obtained from CAT versus SF

Table 3.

Comparisons between CAT and Short-form by groups

Group (by T-score)	Fatiguea				Anxietya				Depressiona				Mobilitya				Upper Extremitya				Peer Relationshipa
	n	Corrb	pc	ESd	n	Corrb	pc	ESd	n	Corrb	pc	ESd	n	Corrb	pc	ESd	n	Corrb	pc	ESd	n	Corrb	pc	ESd
<45	76	0.88	<0.001	−0.76	81	0.79	0.309	0.11	67	0.83	<0.001	−0.61	58	0.92	0.175	0.18	43	0.84	0.801	0.04	39	0.94	0.111	0.26
45–55 (inclusive)	40	0.79	0.128	−0.25	29	0.74	0.583	0.10	35	0.18	0.007	0.49	45	0.73	0.333	−0.15	32	0.67	0.038	−0.38	44	0.86	0.958	0.01
>55	31	0.90	0.080	−0.33	18	0.88	0.021	0.60	28	0.95	0.634	−0.09	41	NA	<0.001	0.87	61	1	<0.001	39.19	42	0.88	0.002	0.51

^aScoring direction is reflected by the score names. For Fatigue, Anxiety, and Depression, higher scores mean more symptomatic (i.e., more fatigue, anxious and depressive). For Mobility, Upper Extremity Function and Peer Relationships, higher scores mean better functioning.

^bPearson correlation

^cp-value for paired t-test

^dEffect size = mean difference / standard deviation of differences

DISCUSSION

Results of this study support the use of the pediatric PROMIS CATs in pediatric brain tumor clinics. If the CAT administration is not feasible, SFs targeting to patients’ severity levels can be used. Inferior health-related quality of life due to disease and/or treatments has been a concern for childhood brain tumor patients and survivors throughout their lifespan and should be monitored closely.(11) HRQOL measures that can provide brief-yet-precise estimation with minimal administration burden should be included in routine clinical visits from on-therapy to long-term follow-ups. Both CATs and SFs derived from comprehensive and psychometrically-sound item banks serve as appropriate means to achieve this goal. The dynamic CATs enable individualized assessment by administering the most informative items based on patients’ previous responses. However, when using a computer is not feasible, fixed-length SFs may be more practical. As long as items from CATs and SFs are from the same IRT-calibrated item banks, scores from CATs and SFs are comparable.(28, 40) Our results, showing high correlations and negligible effect sizes between scores on these two forms, further supports the score comparability of CAT and SF versions of PROMIS measures. This conclusion is consistent with previous findings.(33)

However, we also noted that concordance between CATs and SFs varied across the continuum. As shown in Table 3, higher correlation coefficients were found on the more symptomatic end of the measurement continuum with two exceptions, Mobility and Upper Extremity Function. For patients with better functioning of Mobility, CAT was able to capture various degrees of Mobility (scores ranging between 56.4 and 61.7) while these patients reported the same SF score (58.5) (see Fig 1.d). Therefore, no reliability coefficient could be estimated due to the lack of variance on the Mobility SF. For patients who reported Upper Extremity Function T-scores >55, a perfect correlation between these two forms was found. This low variation contributed to an effect size of 39 (mean=0.5 and SD=0.01).

More than one SF can be constructed from a given item bank. As demonstrated by Lai and her colleagues,(29) investigators can develop SFs based on their needs; such as targeting patients with worse symptoms only, patients with minimal symptoms only, or a SF covering various degrees of symptom severity across the whole continuum. As investigators typically do not know patients’ HRQOL/symptom levels prior to assessments, we recommend using CATs given its dynamic and individualized nature as described in INTRODUCTION. The advantage of using CAT over a short-form was demonstrated by a low correlation coefficient (r=0.18) for patients with depression CAT T-scores between 45 and 55 (inclusive). As shown in Fig 1.c., these patients reported the same lowest possible SF score of 35 but various CAT scores, implying individualized CAT was able to capture a more varied degree of (lack of) depressive symptoms for these patients.

The diagnosis of childhood brain tumors is associated with significant life disruption due to both acute disease effects and the increasingly recognized adverse long-term treatment consequences. There is a need to better understand HRQOL of pediatric brain tumor patients and how it compares to that of children with other chronic conditions. However, progress in this area has been limited by a number of factors. For example, children with brain tumors were often excluded from many pediatric HRQOL studies because their experiences were considered atypical to that of the majority of pediatric survivors. Furthermore, pediatric brain tumor patients are a particularly challenging group to study because of the relatively small number of patients, the diversity of brain tumors, the varying functional impact of the tumors, and the range of surgical and treatment effects that can occur.(41) And, finally, there is a lack of accepted brief-yet-precise and psychometrically-sound measures. The PROMIS measurement system allows for tailored and brief-yet-precise assessments, was validated on children with cancer, including brain tumors,(31) and is available in electronic health record (EHR) software.(42) These characteristics make it a strong candidate for use as an HRQOL measure in clinical settings.

We note several limitations to our study. First, patients were allowed to pause assessments when needed and complete them at a later time. This was done to avoid interrupting clinical flow and to reflect the reality of patients’ daily lives. Yet this flexibility resulted in some participants taking an unusually long time to complete an assessment. Second, we did not include non-English speaking families in this study, which might have unintentionally excluded some economically disadvantaged families without internet access at home and those with low health literacy. Future studies should be conducted to evaluate the feasibility of implementing pediatric PROMIS in these disadvantaged populations.

In conclusion, this study demonstrated that PROMIS CATs and SFs produce comparable scores, at the level of group comparisons, for children with a brain tumor. The agreement between scores obtained by using CATs and SFs varied at individual level across symptom severity levels, which resulted from ceiling or floor effects introduced by fixed-length short-forms. We thus recommend CATs to enable individualized assessment for longitudinal monitoring. If the CAT administration is not feasible, multiple SFs which target different severity levels can be considered to minimize ceiling or floor effects. Low respondent burden was evidenced by brief time required to complete a CAT. Including pediatric PROMIS CATs or SFs in routine clinical visits should be considered.