Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Apr 1.
Published in final edited form as: Appl Neuropsychol Child. 2017 Nov 21;8(2):113–122. doi: 10.1080/21622965.2017.1394853

Disseminability of Computerized Cognitive Training: Performance across Coaches

Ashley S Fournier-Goodnight 1, Jason M Ashford 2, Kellie N Clark 2, Karen Martin-Elbahesh 2, Kristina K Hardy 3, Thomas E Merchant 2, Sima Jeha 2, Robert J Ogg 2, Hui Zhang 2, Lei Wang 2, Heather M Conklin 2
PMCID: PMC5962364  NIHMSID: NIHMS914987  PMID: 29161113

Abstract

Cogmed® is a computerized cognitive intervention utilizing coaches who receive standardized instruction in analyzing training indices and tailoring feedback to remotely monitor participant’s performance. The goal of this study was to examine adherence, satisfaction and efficacy of Cogmed across coaches. Survivors of pediatric brain tumors and acute lymphoblastic leukemia (N=68) were randomized to intervention (Cogmed) or waitlist control. The intervention group was matched with one of two coaches. Cognitive assessments were completed before and after intervention, and participants and caregivers in the intervention group completed satisfaction surveys. T-tests showed no differences in adherence across coaches (number of sessions completed p=.38; d=.32). Noninferiority statistics were not consistently equivalent for satisfaction, but equivalence was supported for caregiver perceptions of pragmatic utility and participant perceptions of logistical ease of Cogmed. Equivalence was not consistently suggested for cognitive outcomes, but was supported on measures tapping relevant cognitive domains (attention, working memory, processing speed, academic fluency). This study suggests adherence can be maintained across coaches. While aspects of satisfaction and cognitive outcomes were equivalent, the possible influence of coach-based variables cannot be ruled out. Findings highlight challenges in standardizing the coaching component of multicomponent computerized interventions and the need for ongoing research to establish dessiminability.

Keywords: cognitive rehabilitation, working memory training, Cogmed, computer-based intervention, cognitive late effects

Individuals diagnosed with malignancies of the central nervous system (CNS) including brain tumors (BT) and acute lymphoblastic leukemia (ALL) during childhood are at increased risk for the development of cognitive late effects (Butler & Mulhern, 2005; Mulhern & Butler, 2004). More specifically, this population often exhibits deficits in attention, working memory (WM) and processing speed that emerge 1 to 5 years following diagnosis (Butler & Mulhern, 2005; Mulhern & Butler, 2004). In association with improvements in survival rates for children treated for BTs and ALL, there has been an increased focus upon improving quality of life by mitigating cognitive late effects (Castellino, Ullrich, Whelen, & Lange, 2014). Interventions aimed at remediating neurocognitive deficits include those that are pharmacological, therapist-delivered, and computerized. While psychostimulants such as methylphenidate have demonstrated some efficacy with this population, survivors of childhood cancer more often have medical contraindications and are less likely to respond to these medications in comparison to patients with other diagnoses (i.e., Attention-Deficit/Hyperactivity Disorder [ADHD]; Conklin et al., 2010a; Conklin et al., 2007; Conklin et al., 2010b). Therapist-delivered cognitive interventions have also shown utility with pediatric patients with BTs and ALL. These interventions include strategies from cognitive rehabilitation (e.g., massed practice of divided and sustained attention tasks), special education (e.g., teaching metacognitive strategies such as task preparedness and task monitoring), and cognitive-behavioral therapy (e.g., reframing of cognitive struggles).; however, this requires intensive, face-to-face sessions, which demand close proximity to a facility providing such services (Butler et al., 2008; Butler & Copeland, 2002; Patel, Katz, Richardson, Rimmer, & Kilian, 2009). Given the limitations associated with these interventions, computerized interventions have received much recent attention in the literature.

Computerized interventions are software-based programs that target specific cognitive skills and allow for tailored feedback regarding a patient’s performance in the context of a structured, interactive, and personalized program (Kesler, Lacayo, & Jo, 2011; Palermo & Wilson, 2009). These interventions are generally advantageous in that they promote treatment adherence among clinical groups who may otherwise have limited access to care (La Greca & Mackey, 2009). Computer- or internet-based treatment approaches have been used successfully in pediatric clinical populations including patients with recurrent pain, asthma, and traumatic brain injury (TBI; Hicks, von Baeyer, & McGrath, 2006; Krishna et al., 2003; Wade, Carey, & Wolfe, 2006). These interventions are considered advantageous over pharmacological and therapist-delivered interventions given that computerized programs have limited side effects and can be delivered in the home (Kesler et al., 2011; Palermo & Wilson, 2009).

Cogmed® (Pearson Education, Inc.) is a specific computerized intervention designed to improve WM (www.cogmed.com). This program has demonstrated efficacy with children with neurodevelopmental and acquired attention problems (Holmes, Gathercole, & Dunning, 2009; Klingberg et al., 2005; Westerberg et al., 2007). A recent pilot study with childhood cancer survivors showed a compliance rate of 85% indicating adequate feasibility and acceptability of Cogmed when used with this population (Hardy, Willard, Allen, & Bonner, 2013). Though preliminary findings of this study suggested improvements in visual WM, it was not adequately powered and consisted of a homogeneous sample in terms of geographic location and socioeconomic status (SES). A larger study comprising a more diverse sample showed a similar adherence rate of 88% (Cox et al., 2015). This study also revealed that both participants and caregivers were generally satisfied with Cogmed. The efficacy of this program with pediatric cancer survivors has recently been examined (Conklin et al., 2015). Participants who completed training with Cogmed showed statistically significant improvements in WM, attention, and processing speed as well as declines in executive dysfunction. Collectively, this research demonstrates that overall adherence to, satisfaction with and efficacy of Cogmed are adequate among pediatric cancer survivors.

In addition to benefits offered by a computer-based platform such as Cogmed, adherence to this program is facilitated by the provision of individualized feedback regarding patients’ performance and suggestions for improving outcomes in the form of coaching (La Greca & Mackey, 2009). Cogmed coaching consists of phone- and email-based interactions based on real-time, internet data that occur throughout training. Coaches receive standardized instruction from Cogmed in analyzing training indices and tailoring feedback for patients and their families. More specifically, the Cogmed training for coaches focuses on examining and interpreting indices produced by data collected from the program designed to measure and detail adherence as well as aspects of participants’ performance. Adherence is measured and detailed by generating indices available for the coach’s review related to number of weekly sessions completed and the duration of those sessions. Participants’ performance is measured and detailed by generating indices available to the coach related to scores across sessions (a built in training index). The training for coaches also focuses on providing feedback to families/participants via phone or email about the nature of the patient’s performance and adherence. Coaches receive instruction designed to assist the participant in improving adherence as well. The approaches utilized are cognitive-behavioral including reinforcement schedules, reordering of exercises, implementing breaks, breaking training into smaller parts and increasing parent monitoring. These strategies are suggested via phone or email to caregivers or participants and individualized based upon analysis of training indices and interactions with caregivers/participants. Coaches are also trained through Cogmed to engage in troubleshooting with families about other that may impact adherence (e.g., scheduling conflicts, time management).

Phone-based coaching directed toward adherence has been used successfully in pediatric patients with asthma and Human Immunodeficiency Virus (HIV)/Acquired Immunodeficiency Syndrome (AIDS; Garbutt, Yan, Highstein, & Strunk, 2014; Puccio et al., 2006; Smith, Jaffe, Fisher, Highstein, & Strunk, 2004). More specifically, phone-based coaching increased compliance with emergency care physicians’ recommendations to follow up with participants’ primary care physicians and routine asthma management (Garbutt et al., 2014; Smith et al., 2004). Phone-based contacts with families of pediatric participants with HIV/AIDS improved adherence to medication regimens (Puccio et al., 2006).

More consistent with the Cogmed intervention, other studies have combined computer-based treatment with phone-based coaching (Nguyen et al., 2013; Wise et al., 2007). These multicomponent interventions were also successful in increasing treatment adherence among pediatric patients with asthma (Wise et al., 2007); however, phone-based coaching did not increase the benefits of a computer-based weight-loss intervention (Nguyen et al., 2013). Though improvements in adherence have been generally apparent, multicomponent studies utilizing phone- and computer-based treatments have not provided information related to participants’ and caregivers’ overall satisfaction with the intervention or differences in adherence, satisfaction, and efficacy across coaches, which is an important step in establishing the disseminability of an intervention.

Given these limitations in the literature, the goal of the current study was to examine adherence to, satisfaction with and the efficacy of Cogmed across coaches for the purpose of further investigating the clinical utility and disseminability of this intervention. Previous studies have shown that compliance, satisfaction and efficacy were adequate for participants overall; however, differences in these factors across coaches have not been previously examined (Conklin et al., 2015; Cox et al., 2015). Investigation of these differences is a preliminary step in establishing the effectiveness and external validity of Cogmed, which is the logical investigational progression given the evidence of its efficacy with diverse populations (Conklin et al., 2015; Holmes et al., 2009; Klingberg et al., 2005; Westerberg et al., 2007). It was hypothesized that there would be similar outcomes between coaches in terms of adherence, satisfaction and efficacy based upon their standardized training through Cogmed.

Method

Participants

This intervention trial has been described in detail elsewhere and is summarized here (Conklin et al., 2015; Cox et al., 2015). Participants were pediatric patients diagnosed with BTs of the infratentorium or ALL treated with cranial radiation therapy and/or intrathecal chemotherapy who were off treatment for at least one year with no disease recurrence. These participants were required to be between the ages of 8 and 16 years, speak English, and have an intelligence quotient (IQ) of at least 70. Individuals with a history of CNS injury/disease (i.e., TBI, stroke) or ADHD predating cancer diagnosis were excluded. Additional exclusion criteria included use of psychotropic medications within two weeks of enrollment, sensory or motor dysfunction preventing valid assessment or completion of intervention, or a psychological condition preventing or taking priority over cognitive intervention. This study was approved by the institutional review board and written, informed consent was obtained prior to participation.

Procedure

Patients meeting inclusion and exclusion criteria were contacted in the order of their scheduled medical appointments to avoid potential recruitment biases. A screening assessment of cognitive functioning was initially conducted to determine eligibility for intervention, which was based upon the presence of WM deficits. This was defined by performance on the Digit Span, Letter-number Sequencing, or Spatial Span subtests of the Wechsler Intelligence Scale for Children, Fourth Edition Integrated (WISC-IV Integrated) that was more than one standard deviation below the normative mean or the participant’s IQ as measured by the Wechsler Abbreviated Scale of Intelligence (WASI; Kaplan et al., 2004; Wechsler, 1999). Participants found eligible for intervention were randomized to computerized training with Cogmed or a waitlist control group.

Participants in the intervention group were paired with one of two Cogmed coaches who had received standardized instruction in the analysis of training indices and provision of individualized feedback to participants and their families. These coaches were one male and one female with master’s degrees who are research assistants at St. Jude Children’s Research Hospital. An attempt was made to gender-match participants to coaches for comfort of participants given that they were largely children. Standardized instruction in coaching was provided via web-based training and a coaching manual. The focus of this instruction is on maximizing motivated and consistent training by recommending cognitive-behavioral strategies. These strategies were individualized, and specific components of these strategies depended upon the coach’s analysis of training indexes and interactions with caregivers and/or participants. Consistent with the Cogmed model, coaches contacted participants by phone at least weekly and spoke with participants and/or their caregivers for 10 to 15 minutes; however, frequency and duration of contact varied according to participant need. Coaching notes from these contacts were reviewed and discussed as needed by the same supervising psychologist.

The participants were asked to complete 25 training sessions at home within 5 to 9 weeks. Game-like exercises oriented toward visuospatial and verbal WM comprised training sessions. Each lasted 30 to 45 minutes, and the difficulty level adjusted based upon the participant’s performance. Program-based incentives to facilitate motivation and continued participation in training included playing a racing game following completion of daily sessions. In line with recommendations of Cogmed, coaches assisted in implementing individualized reinforcement schedules (e.g., sticker charts, allowing participants to select movies to rent or special meals to cook following completion of a specific number of sessions) as needed. Furthermore, participants were provided with additional incentives consisting of $10 gift cards following completion of 9, 17, and 25 training sessions and after completion of assessments. Participants in both the intervention and control conditions received equivalent incentives (i.e., controls received gift cards at 2, 4, and 6 weeks and following completion of assessments).

Measures

All participants completed brief neuropsychological assessments before and after training to assess attention, WM, processing speed and academic fluency. The primary, a priori outcome measure was the change score (i.e., post-training standardized score minus pre-training standardized score) from the Spatial Span Backward subtest on the WISC-IV Integrated (Kaplan et al., 2004). Additional outcome measures (all representing change scores) for WM consisted of the Digit Span Backward and Letter-number Sequencing subtests as well as the Working Memory Index (WMI) from the WISC-IV Integrated. The Metacognition Index (MI) and Working Memory subscale from the Behavior Rating Inventory of Executive Function (BRIEF) served as parental report outcome measures of WM (Gioia, Isquith, Guy, & Kenworthy, 2000). Relevant outcome measures for attention included the Spatial Span Forward and Digit Span Forward subtests of the WISC-IV Integrated, Omissions scores from the Conners’ Continuous Performance Test II (CPT-II), and the Inattention and Executive Function subscales from the Conners’ Third Edition, Parent (Conners’ 3-P; Conners, 2004; Conners, 2008; Kaplan et al., 2004). The Hit Reaction Time (RT) scores from the CPT-II served as the outcome measure for processing speed. The Reading Fluency and Math Fluency subtests from the Woodcock-Johnson Tests of Achievement, Third Edition (WJ-III Ach) were the outcome measures for academic fluency (Woodcock, McGrew, & Mather, 2001).

Participants in the intervention group and their caregivers completed questionnaires created locally by the researchers to assess satisfaction with the intervention. These surveys presented items in a Likert-type scale with response options ranging from 1 (strongly disagree) to 5 (strongly agree). The caregiver version of the questionnaire included 20 Likert-type items, while the participant version was comprised of 11 Likert-type items. These questionnaires were completed halfway through and following conclusion of the intervention.

Statistics

Descriptive statistics for demographic and clinical variables for the intervention group were calculated for each coach in order to characterize the associated sample. Chi-square tests of independence were conducted to evaluate the relation between categorical clinical and demographic variables (i.e., gender, diagnosis, treatment intensity) and coach. For the purpose of this analysis, treatment intensity was dichotomized into participants who were treated with cranial radiation therapy and those whose treatment did not involve radiation to the brain. Independent samples t-tests (two way; α = .05) were used to examine possible differences between coaches among continuous demographic and clinical variables. Given the small sample size, effect sizes (i.e., Cramer’s V/φ, Cohen’s d) were calculated for demographic and clinical variables (Cohen, 1988).

Two one-sided t-tests (TOST; α = .01) were used to examine equivalence or noninferiority between coaches for caregiver as well as participant satisfaction and primary outcome measures for WM, attention, processing speed and academic fluency (Schuirmann, 1987). The equivalence bound for caregiver and participant satisfaction data (five-point Likert items) was one. Given that no standards were found in the literature for typical equivalence bounds for Likert items by these authors, it was agreed that a change in rating of one on Likert items would be clinically meaningful (Walker & Nowacki, 2011). The equivalence bounds for WM, attention, processing speed and academic fluency corresponded to one standard deviation for measures used within these neurocognitive domains (e.g., equivalence bound of three for scaled scores, 10 for t-scores and 15 for standard scores).

Results

Overall Sample

Among 128 patients initially screened, 80 exhibited deficits in WM sufficient for qualification. Of those who qualified, five were excluded and seven declined participation leaving 68 for randomization. Overall, participants were balanced by gender (53% male), but were chiefly Caucasian (78%). The majority of the sample (69%) was treated for ALL typically with chemotherapy alone (87%). Participants with BTs were primarily treated with radiation therapy (73%). On average, participants were 12 years of age and five years from conclusion of treatment.

Among participants randomized to intervention (n = 34), 30 (88%) completed at least 20 out of 25 sessions, which was the initial criterion for compliance. No significant differences were found between participants who completed the intervention and those who discontinued early in terms of demographic and clinical variables including pre-intervention cognitive performance. Those participants who showed slow progress (i.e., score gain less than 20 following 20 sessions on a built-in Cogmed training index) were offered five additional sessions. On average, participants completed 26 sessions over 47 days. Because one participant was coached by neither Coach 1 nor Coach 2, data from this participant were excluded from analyses.

Sample by Coach

About half (51%; n = 15) of the participants were paired with Coach 1. Given the intentional attempt to gender match participants to coach, Coach 1 worked primarily with male participants and Coach 2 worked predominantly with female participants (Table 1). There were no significant differences between coaches in terms of participants’ diagnoses, age at diagnosis, treatment intensity, and time since treatment. The sample associated with each coach also did not differ significantly with regard to pre-intervention intelligence, SES, and age at enrollment on this study. However, effect sizes associated with SES and age at diagnosis were medium. Participants matched with Coach 1 and Coach 2 demonstrated similar adherence in terms of the overall number of sessions completed (p = .38; d = .32), number of weekly sessions completed (p = .98; d = .00), and overall duration of training (p = .90; d = .04; Figure 1).

Table 1.

Demographic Data

Coach 1 Coach 2 p Cramer’s V/φ d
Gender (% male) 73 36 .04* .37
Diagnosis (n) .28 .20
 ALL 9 11
 BT 6 3 .28a .20
Treatment Intensity (n)
 Chemotherapy only 9 8
 CSI with or without chemotherapy 4 2
 CRT with or without chemotherapy 2 1
 Chemotherapy plus BMT/TBI 0 3
Socioeconomic status (BSMSS; X̄ ± SD) 36.70 ± 17.00 43.96 ± 13.24 .21 .47
Abbreviated intelligence (SS; X̄ ± SD) 107.27 ± 17.82 108.14 ± 14.57 .82 .05
Age at enrollment (years; X̄ ± SD) 12.82 ± 2.09 12.08 ± 2.72 .41 .30
Age at diagnosis (years; X̄ ± SD) 5.67 ±3.15 4.23 ± 2.81 .20 .48
Time since treatment (years; X̄ ± SD) 4.95 ± 2.85 5.74 ± 3.39 .49 .25
Open in a new tab

Note. d = Cohen’s d; ALL = acute lymphoblastic leukemia; BT = brain tumor; CSI = craniospinal irradiation; CRT = conformal radiation therapy; BMT/TBI = bone marrow transplant/total body irradiation; BSMSS = Barratt Simplified Measure of Social Status, where scores range from 8 to 66 with higher scores indicating higher socioeconomic status; SS = standard score (M = 100; SD = 15).

a

Treatment intensity was dichotomized to patients treated with cranial radiation therapy and those whose treatment did not involve radiation to the brain for chi-square test of independence.

*

p < .05.

Figure 1.

Figure 1

Open in a new tab

Adherence data including total and weekly sessions completed by participants and the overall duration of training across coaches (Coach 1 and Coach 2).

Caregivers’ ratings of their satisfaction with the Cogmed training program revealed that coaches were equivalent in terms of caregivers’ perceptions of the utility of instructions provided, weekly phone calls made and weekly progress reports sent home by coaches (Table 2). This was also the case with regard to caregivers’ perceptions that participants completed sessions independently. Most notably, caregivers’ ratings of coaches were equivalent with regard to whether changes in children were apparent or noticed by caregivers and others. Caregivers’ perceptions that their child’s grades improved and their child benefited from Cogmed training was also equivalent across coaches. Finally, coaches were equivalent in terms of whether caregivers would recommend Cogmed to other parents. Equivalence between coaches was not apparent on any other aspects of caregiver satisfaction.

Table 2.

Caregiver and Participant Satisfaction

Rater/Item Coach 1 (X̄ ±SD) Coach 2 (X̄ ±SD) Δ/ε p1 p2 Relevance
Caregiver
 Cogmed computer program functioned 4.13±0.99 4.50±0.65 1 .03 <.01* Indeterminate
 Instructions were helpful and easy 4.67±0.48 4.79±0.42 1 <.01* <.01* Equivalence
 Phone calls addressed difficulty 4.80±0.41 4.79±0.42 1 <.01* <.01* Equivalence
 Phone calls were useful 4.87±0.35 4.79±0.42 1 <.01* <.01* Equivalence
 Sending weekly progress is a good idea 4.40±0.82 4.71±0.46 1 <.01* <.01* Equivalence
 Scheduling daily session was easy 4.07±0.88 3.36±1.27 1 <.01* .24 Indeterminate
 Child was agreeable to sessions 3.87±0.91 3.57±1.01 1 <.01* .03 Indeterminate
 Child enjoyed training program 3.80±0.86 3.14±0.86 1 <.01* .15 Indeterminate
 Child enjoyed as much as other video games 3.00±0.75 2.62±1.12 1 <.01* .05 Indeterminate
 Child was not easily bored 3.60±0.98 3.21±0.80 1 <.01* .04 Indeterminate
 Child was not frustrated 3.20±1.01 2.86±1.09 1 <.01* .05 Indeterminate
 Child completed sessions independently 4.53±0.51 4.50±0.51 1 <.01* <.01* Equivalence
 Child looked forward to racing game 3.93±0.99 3.86±1.09 1 <.01* .01 Indeterminate
 Able to upload information to the internet 4.07±0.82 4.69±0.63 1 .10 <.01* Indeterminate
 Child was motivated by gift card 4.33±0.90 4.14±1.02 1 <.01* .02 Indeterminate
 Noticed a change in child 3.63±0.61 3.43±1.02 1 <.01* <.01* Equivalence
 Others noticed a change in child 3.20±0.41 3.36±0.63 1 <.01* <.01* Equivalence
 Child’s grades improved 3.07±0.45 3.08±0.49 1 <.01* <.01* Equivalence
 Child benefitted directly 3.87±0.51 3.71±0.72 1 <.01* <.01* Equivalence
 Would recommend this study to other parents 4.33±0.61 4.29±0.61 1 <.01* <.01* Equivalence
Participant
 Understood rules of the games 4.53±0.64 4.57±0.64 1 <.01* <.01* Equivalence
 Easy to complete daily sessions 3.47±0.83 3.36±0.92 1 <.01* <.01* Equivalence
 Rarely complained when completing sessions 3.33±1.17 3.14±0.77 1 <.01* .02 Indeterminate
 Enjoyed games 3.07±1.03 3.21±1.25 1 .03 <.01* Indeterminate
 Enjoyed as much as other video games 2.20±0.94 2.21±1.18 1 <.01* <.01* Equivalence
 Sessions kept my attention 3.27±0.79 3.64±0.84 1 .03 <.01* Indeterminate
 Completed sessions independently 4.80±0.41 4.57±0.75 1 <.01* <.01* Equivalent
 Looked forward to racing game 3.20±1.01 3.64±1.55 1 .13 <.01* Indeterminate
 Motivated by gift card 3.79±0.97 3.50±1.16 1 <.01* .05 Indeterminate
 Games helped with better performance at school 3.14±0.86 3.64±0.63 1 .05 <.01* Indeterminate
 Other children would like being in this study 3.00±1.10 2.71±1.13 1 <.01* .05 Indeterminate
Open in a new tab

Note. Caregiver ratings of 5 = strongly agree, 4 = agree, 3 = neutral, 2 = disagree, 1 = strongly disagree.

*

p < .01.

Participants’ ratings of their satisfaction with the Cogmed training program demonstrated equivalence between coaches with regard to participants’ understanding of the rules of Cogmed tasks and ability to complete these tasks daily (Table 2). This was also the case in terms of participants’ relative enjoyment of Cogmed tasks and their perception that they completed tasks independently. Other aspects of participants’ satisfaction were not found to be equivalent between coaches.

In terms of cognitive outcomes, coaches were found to be equivalent across the majority of measures (Table 3). Equivalent outcomes were found on tasks of WM (WISC-IV Integrated WMI, Digit Span Backward and Letter-number Sequencing, BRIEF MI), attention (CPT-II Omissions, Conners’ 3-P Executive Function) and processing speed (CPT-II Hit RT). This was also the case with regard to academic fluency (WJ-III Ach Reading Fluency, Math Fluency). However, there were a number of measures for which equivalence between coaches was not found.

Table 3.

Cognitive Outcomes

Measure/Subtest Coach 1 (X̄ ±SD) Coach 2 (X̄ ±SD) Δ/ε p1 p2 Relevance
WISC-IV Integrated Change Scores
 Spatial Span Forward (ScS) 3.93 ± 3.51 2.86 ± 4.31 3 <.01* .10 Indeterminate
 Spatial Span Backward (ScS) 3.13 ± 3.62 3.14 ± 2.93 3 .01 .01 Indeterminate
 Working Memory Index (SS) 9.40 ± 9.85 9.57 ± 8.82 15 <.01* <.01* Equivalence
 Digit Span Forward (ScS) 1.00 ± 4.08 1.07 ± 2.16 3 .01 <.01* Indeterminate
 Digit Span Backward (ScS) 2.40 ± 2.52 2.14 ± 2.62 3 <.01* <.01* Equivalence
 Letter-number Sequencing (ScS) 1.26 ± 2.54 1.71 ± 1.68 3 <.01* <.01* Equivalence
CPT-II Change Scores
 Omissions (T) −1.51 ± 8.15 0.70 ± 5.37 10 <.01* <.01* Equivalence
 Hit Response Time (T) −1.61 ± 9.31 −2.55 ± 6.98 10 <.01* <.01* Equivalence
Conners’ 3-P Change Scores
 Inattention (T) −6.73 ± 11.96 −7.64 ± 9.47 10 <.01* .02 Indeterminate
 Executive Function (T) −8.26 ± 8.72 −5.85 ± 7.58 10 <.01* <.01* Equivalence
BRIEF Change Scores
 Working Memory (T) −1.33 ± 9.44 −5.85 ± 4.65 10 <.01* .03 Indeterminate
 Metacognition Index (T) −3.20 ± 6.89 −5.00 ± 4.29 10 <.01* <.01* Equivalence
WJ-III Ach Change Scores
 Reading Fluency (SS) 2.00 ± 5.50 1.92 ± 7.63 15 <.01* <.01* Equivalence
 Math Fluency (SS) .27 ± 4.25 1.64 ± 5.14 15 <.01* <.01* Equivalence
Open in a new tab

Note. All X̄s reflect change scores from baseline to immediate post-treatment; WISC-IV = Wechsler Intelligence Scale for Children, Fourth Edition; ScS = scaled score (M = 10; SD = 3); CPT-II = Conners’ Continuous Performance Test II; T = t-score (M = 50; SD = 10); Conners’ 3-P = Conners’ Third Edition, Parent; BRIEF = Behavior Rating Inventory of Executive Function; WJ-III Ach = Woodcock-Johnson Tests of Achievement, Third Edition; SS = standard score (M = 100; SD = 15).

*

p < .01.

To further investigate the potential impact of gender matching with coaches, posthoc analyses comparing gender-matched dyads (i.e., male coach with male participant, female coach with female participant) to nongender-matched dyads (i.e., male coach with female participant, female coach with male participant) in terms of adherence, satisfaction, and cognitive outcomes were conducted through use of t-tests. Analyses demonstrated that there were no differences in terms of adherence and cognitive outcomes between gender-matched and nongender-matched dyads. With regard to caregiver satisfaction, ability to upload information to the internet each week was significantly different between groups with a large effect size (p = .01, d = 2.60). Additionally, participants differed significantly based on gender matching in terms of their perceptions that Cogmed was as enjoyable as other video games (p = .10). Again, the associated effect size was large (d = 2.57).

Discussion

This study demonstrated that program adherence was similar and the majority of cognitive outcomes were equivalent across coaches, but that caregiver and participant satisfaction were less consistent between coaches. However, there was the potential for significant differences in demographic variables between the participant groups with whom the coaches worked.

Demographic and Clinical Variables

Both coaches worked with participants who were similar in terms of the majority of demographic and clinical variables associated with cognitive outcomes (Castellino et al., 2014; Mulhern & Butler, 2004). Participants were reasonably balanced across coaches with regard to child-based risk factors for the development of cognitive late effects including estimated IQ prior to intervention. Risk factors associated with type of CNS malignancy (i.e., BT versus ALL) and treatment (i.e., treatment intensity, time since treatment) were also well-balanced across coaches. Though participants were not similar across coaches in terms of gender, female sex has been variably associated with increased risk for cognitive late effects particularly among patients with ALL (Butler & Mulhern, 2005; Castellino et al., 2014). Additionally, adherence and cognitive outcomes were similar whether participants were gender matched or not; however, aspects of satisfaction were influenced by gender matching such that caregiver and participant satisfaction were increased with gender matching. There may also be potential differences in SES and age at diagnosis based upon effect sizes. Given the association between these child-based risk factors and the development of cognitive late effects, it is possible that between group differences in these variables influenced satisfaction and cognitive outcomes. However, it warrants mention that these variables were distributed such that one coach may have worked with a higher SES group diagnosed at a younger age; thus, neither coach was at an absolute disadvantage in terms of these variables.

Adherence

Participants working with respective coaches evinced similar duration of training as well as total number of sessions and number of weekly sessions completed indicating comparable adherence across coaches. Roughly 88% of participants paired with Coach 1 and about 87% of participants matched with Coach 2 completed at least 20 sessions. This compliance with treatment is consistent with the adherence rates found for the overall samples in previous studies investigating computer-based cognitive intervention (Cox et al., 2015; Hardy et al., 2013).

Satisfaction

Importantly, caregiver’s satisfaction with Cogmed was equivalent across coaches in terms of their perceptions of the overall benefit or pragmatic utility of the program (e.g., items such as “noticed a change in child,” “others noticed a change in child,” “grades improved”). Caregivers’ ratings also suggested that they found the phone-based coaching component of the intervention and weekly provision of progress equally useful across coaches. In contrast, caregivers’ satisfaction was not equivalent across coaches in terms of their child’s apparent enjoyment of and motivation to engage in or complete the Cogmed intervention. Caregiver satisfaction also differed based on gender matching with regard to ability to upload information to the internet suggesting, for example, that caregivers may be more comfortable engaging in trouble shooting with someone of the same gender. While these findings may have also been influenced by group-based differences in clinical variables, it is possible that differences between coaches or caregivers played a role. For example, one coach may have devoted greater time to trouble-shooting scheduling concerns with families. Alternatively, caregivers may have differed in terms of their time-management skills or the autonomy they allowed participants in scheduling their own sessions.

Participants’ satisfaction with the Cogmed intervention was equivalent across coaches with regard to general logistics of the program (e.g., items such as “easy to complete daily sessions,” “completed sessions independently,” “understood rules”). Consistent with caregiver ratings, participants’ enjoyment as well as interest in and motivation to complete the intervention was not equivalent across coaches. Further, participants’ enjoyment of the program was influenced by gender matching possibly indicating that the program was more enjoyable when working with a coach of the same gender. Unlike caregiver ratings, however, participants’ ratings were not equivalent in terms of their perceptions that the intervention resulted in better performance at school. Again, these findings may have been related to between group differences found with regard to demographic and clinical variables, but the impact of coach-specific variables should be considered as well. It is possible that one coach may have devoted more time to the discussion of the nature of improvements in cognitive abilities in the naturalistic setting (i.e., school). These differences could also be associated with difficulty among participants in recognizing improvements in cognitive abilities in the naturalistic environment despite explicit discussions with coaches.

Cognitive Outcomes

Equivalent improvements in performance between coaches were evident across all relevant domains of functioning including attention, WM, processing speed and academic fluency; however, equivalence in performance within these domains was variable. Though improvements on a measure of sustained attention were equivalent across coaches, improvements on a measure of attention span were not. Additionally, improvements in WM were not consistently equivalent across coaches such that performance on tasks of verbal WM was equivalent whereas performance on a task of visuospatial WM was not. This is also the case in terms of caregiver ratings of improvements in attention and WM in the naturalistic setting. Again, in addition to demographic and clinical variables, the role of coach-specific variables may be influencing these findings. Additionally, it is possible that differences in participants’ use of cognitive strategies when completing Cogmed sessions impacted individual performance.

Limitations and Future Directions

There are limitations associated with this study. The sample size was small, which impacted power; however, estimates of effect size and inferiority or equivalency statistics were used in part to address this concern. The potential differences in demographic and clinical variables across participant groups make interpretation of some findings difficult. Given the possible role of individual coaching factors (e.g., number and duration of contacts, time spent trouble shooting versus implementing cognitive behavioral strategies during these contacts), another limitation of this study is that these factors were not examined in a systematic fashion. Coaches were employed within the same institution allowing for frequent contact with each other and were supervised by the same licensed psychologist, which could have resulted in more consistent approaches to coaching than may occur in other settings. The influence of the use of monetary incentives upon adherence is unclear. This was implemented in the current study given its use in previous studies examining the feasibility and acceptability of Cogmed (Hardy et al., 2013). Monetary incentives have been used in multiple studies to increase adherence to treatment regimens (Finney, Lemanek, Brophy, & Cataldo, 1990; Robison et al., 1992; Smith et al., 2006), and a number of individually tailored incentives used in the current study were nonmonetary. Nevertheless, adherence could change based upon monetary incentives. This becomes more salient when considering that it may not be feasible for all families implementing Cogmed to provide monetary incentives.

Overall, this study showed that the coaching component of Cogmed can be standardized such that similar adherence among pediatric cancer survivors can be achieved across coaches. Similar satisfaction with regard to pragmatic utility among caregivers and logistical ease among participants was apparent between coaches, which is suggestive of the potential effectiveness of this program. Similar improvements in cognitive performance were evident within broad cognitive domains between coaches. However, numerous variables can impact caregiver as well as participant satisfaction and cognitive outcomes, which can make controlled investigation of disseminability challenging. Nonetheless, this is an important step in establishing the external validity of this and similar computer-based interventions. Future studies should use well-matched participant groups and focus upon investigation of coaching factors that may increase adherence to, satisfaction with and the efficacy of Cogmed. Studies more closely examining coaching factors such as the frequency and duration of contacts and activities coaches engage in during these contacts could assist in optimizing treatment outcomes when using computerized cognitive training programs that include coaching. Inclusion of a method for monitoring coaching fidelity within the Cogmed program may also be beneficial. It may also be useful to evaluate adherence, satisfaction and efficacy across coaches when implemented in remote or distant locations, which would reduce the opportunity for communication across coaches and thereby provide a more robust indication of external validity. Similarly, implementing this program without monetary incentives would assist with the establishment of disseminability.

Acknowledgments

This work was supported in part by the National Cancer Institute at the National Institutes of Health (St. Jude Cancer Center Support [CORE] Grant [P30 CA21765]; the American Cancer Society (RSGPB CPPB-119423 to HC); and the American Lebanese Syrian Associated Charities (ALSAC)

The authors thank the patients and their families who volunteered their time to participate in this study. Cogmed software was provided by Pearson, Inc. for research purposes. Pearson did not play a role in design or conduct of this study; analysis or interpretation of the data; preparation, review or approval of the manuscript.

References

  1. Butler RW, Copeland DR, Fairclough DL, Mulhern RK, Katz ER, Noll RB, … Sahler OJ. A multicenter, randomized clinical trial of a cognitive remediation program for childhood survivors of a pediatric malignancy. Journal of Consulting and Clinical Psychology. 2008;76(3):367–378. doi: 10.1037/0022-006X.76.3.367. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Butler RW, Copeland DR. Attentional processes and their remediation in children treated for cancer: A literature review and the development of a therapeutic approach. Journal of the International Neuropsychological Society. 2002;8(1):115–124. [PubMed] [Google Scholar]
  3. Butler RW, Mulhern RK. Neurocognitive interventions for children and adolescents surviving cancer. Journal of Pediatric Psychology. 2005;30(1):65–78. doi: 10.1093/jpepsy/jsi017. [DOI] [PubMed] [Google Scholar]
  4. Castellino SM, Ullrich NJ, Whelen MJ, Lange BJ. Developing interventions for cancer-related cognitive dysfunction in childhood cancer survivors. Journal of the National Cancer Institute. 2014;106(8):1–16. doi: 10.1093/jnci/dju186. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Cohen J. Statistical power analysis of the behavioral sciences. 2. Hillsdale, NJ: Erlbaum; 1988. [Google Scholar]
  6. Conklin HM, Helton S, Ashford J, Mulhern RK, Reddick WE, Brown R, … Khan RB. Predicting methylphenidate response in long-term survivors of childhood cancer: A randomized, double-blind, placebo-controlled, crossover trial. Journal of Pediatric Psychology. 2010a;35(2):144–155. doi: 10.1093/jpepsy/jsp044. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Conklin HM, Khan RB, Reddick WE, Helton S, Brown R, Howard SC, … Mulhern RK. Acute neurocognitive response to methylphenidate among survivors of childhood cancer: A randomized, double-blind, cross-over trial. Journal of Pediatric Psychology. 2007;32(9):1127–1139. doi: 10.1093/jpepsy/jsm045. [DOI] [PubMed] [Google Scholar]
  8. Conklin HM, Ogg RJ, Ashford JM, Scoggins MA, Zou P, Clark KN, … Zhang H. Computerized cognitive training for amelioration of cognitive late effects among childhood cancer survivors: A randomized controlled trial. Journal of Clinical Oncology. 2015 doi: 10.1200/JCO.2015.61.6672. Advance Online Publication. doi:JCO.2015.61.6672. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Conklin HM, Reddick WE, Ashford J, Ogg S, Howard SC, Morris EB, … Khan RB. Long-term efficacy of methylphenidate in enhancing attention regulation, social skills, and academic abilities of childhood cancer survivors. Journal of Clinical Oncology. 2010b;28(29):4465–4472. doi: 10.1200/JCO.2010.28.4026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Conners CK. Conners’ Continuous Performance Test II [Assessment instrument] San Antonio, TX: Pearson; 2004. [Google Scholar]
  11. Conners CK. Conners 3rd Edition [Assessment instrument] Toronto, Ontario: Multi-Health Systems; 2008. [Google Scholar]
  12. Cox LE, Ashford JM, Clark KN, Martin-Elbahesh K, Hardy KK, Merchant TE, … Conklin HM. Feasibility and acceptability of a remotely administered computerized intervention to address cognitive late effects among childhood cancer survivors. Neuro-Oncology Practice. 2015 doi: 10.1093/nop/npu036. Advance Online Publication. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Finney JW, Lemanek KL, Brophy CJ, Cataldo MF. Pediatric appointment keeping: Improving adherence in a primary care allergy clinic. Journal of Pediatric Psychology. 1990;15:571–579. doi: 10.1093/jpepsy/15.4.571. [DOI] [PubMed] [Google Scholar]
  14. Fisher EB, Boothroyd RI, Coufal MM, Baumann LC, Mbanya JC, Rotherman-Borus MJ, … Tanasugarn C. Peer support for self-management of diabetes improved outcomes in international settings. Health Affairs. 2012;31(1):130–139. doi: 10.1377/hlthaff.2011.0914. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Fisher EB, Strunk RC, Highstein GR, Kelley-Sykes R, Tarr KL, Trinkaus K, Musick J. A randomized controlled evaluation of the effect of community health workers on hospitalization for asthma: The asthma coach. Archives of Pediatrics and Adolescent Medicine. 2009;163(3):225–232. doi: 10.1001/archpediatrics.2008.577. [DOI] [PubMed] [Google Scholar]
  16. Garbutt JM, Yan Y, Highstein G, Strunk RC. A cluster-randomized trial shows telephone peer coaching for parents reduces children’s asthma morbidity. Journal of Allergy and Clinical Immunology. 2014;135(5):1163–1170. doi: 10.1016/j.jaci.2014.09.033. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Gioia GA, Isquith PK, Guy SC, Kenworthy L. Behavior Rating Inventory of Executive Function [Assessment instrument] Odessa, FL: Psychological Assessment Resources; 2000. [Google Scholar]
  18. Hardy KK, Willard VW, Allen TM, Bonner MJ. Working memory training in survivors of pediatric cancer: A randomized pilot study. Psychooncology. 2013;22(8):1856–1865. doi: 10.1002/pon.3222. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Hicks CL, von Baeyer CL, McGrath PJ. Online psychological treatment for pediatric recurrent pain: A randomized evaluation. Journal of Pediatric Psychology. 2006;31:724–736. doi: 10.1093/jpepsy/jsj065. [DOI] [PubMed] [Google Scholar]
  20. Holmes J, Gathercole SE, Dunning DL. Adaptive training leads to sustained enhancement of poor working memory in children. Developmental Science. 2009;12:F9–F15. doi: 10.1111/j.1467-7687.2009.00848.x. [DOI] [PubMed] [Google Scholar]
  21. Kaplan E, Delis D, Fein D, Maerlender A, Morris R, Kramer J. Wechsler Intelligence Scale for Children, Fourth Edition Integrated [Assessment instrument] San Antonio, TX: Pearson; 2004. [Google Scholar]
  22. Kesler SR, Lacayo NJ, Jo B. A pilot study of an online cognitive rehabilitation program for executive function skills in children with cancer-related brain injury. Brain Injury. 2011;25(1):101–112. doi: 10.3109/02699052.2010.536194. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Klingberg T, Fernell E, Olesen PJ, Johnson M, Gustafsson P, Dahlstrom K, … Westerberg H. Computerized training of working memory in children with ADHD—A randomized, controlled trial. Journal of the American Academy of Child & Adolescent Psychiatry. 2005;44(2):177–186. doi: 10.1097/00004583-200502000-00010. [DOI] [PubMed] [Google Scholar]
  24. Krishna S, Francisco BD, Balas EA, Konig P, Graff GR, Madsen RW. Internet-enabled interactive multimedia asthma education program: A randomized trial. Pediatrics. 2003;111:503–510. doi: 10.1542/peds.111.3.503. [DOI] [PubMed] [Google Scholar]
  25. La Greca AM, Mackey ER. Adherence to pediatric treatment regimens. In: Roberts MC, Steele RG, editors. Handbook of pediatric psychology. 4. New York, NY: The Guilford Press; 2009. pp. 227–237. [Google Scholar]
  26. Mulhern RK, Butler RW. Neurocognitive sequelae of childhood cancers and their treatment. Pediatric Rehabilitation. 2004;7(1):1–14. doi: 10.1080/13638490310001655528. [DOI] [PubMed] [Google Scholar]
  27. Nguyen B, Shrewsbury VA, O’Connor J, Steinbeck KS, Hill AJ, Shah S, … Baur LA. Two-year outcomes of an adjunctive telephone coaching and electronic intervention for adolescent weight-loss maintenance: The Loozit randomized controlled trial. International Journal of Obesity. 2013;37(3):468–472. doi: 10.1038/ijo.2012.74. [DOI] [PubMed] [Google Scholar]
  28. Palermo TM, Wilson AC. eHealth applications in pediatric psychology. In: Roberts MC, Steele RG, editors. Handbook of pediatric psychology. 4. New York, NY: The Guilford Press; 2009. pp. 227–237. [Google Scholar]
  29. Patel SK, Katz ER, Richardson R, Rimmer M, Kilian S. Cognitive and problem solving training in children with cancer: A pilot project. Journal of Pediatric Hematology and Oncology. 2009;31(9):670–677. doi: 10.1097/MPH.0b013e3181b25a1d. [DOI] [PubMed] [Google Scholar]
  30. Puccio JA, Belzer M, Olson J, Martinez M, Salata C, Tucker D, Tanaka D. The use of cell phone reminder calls for assisting HIV-infected adolescents and young adults to adhere to highly active antiretroviral therapy: A pilot study. AIDS Patient Care and STDs. 2006;20:438–444. doi: 10.1089/apc.2006.20.438. [DOI] [PubMed] [Google Scholar]
  31. Robison JI, Rogers MA, Carlson JJ, Mavis BE, Stachnik T, Stoffelmayr B, … Van Huss WD. Effects of a 6-month incentive-based exercise program on adherence and work capacity. Medicine and Science in Sports Exercise. 1992;24(1):85–93. [PubMed] [Google Scholar]
  32. Schuirmann DJ. A comparison of the Two One-Sided Tests procedure and the Power Approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Biopharmaceutics. 1987;15(6):657–680. doi: 10.1007/BF01068419. [DOI] [PubMed] [Google Scholar]
  33. Smith SR, Jaffe DM, Highstein G, Fisher EB, Trinkaus KM, Strunk RC. Asthma coaching in the pediatric emergency department. Academic Emergency Medicine. 2006;13:835–839. doi: 10.1197/j.aem.2006.03.565. [DOI] [PubMed] [Google Scholar]
  34. Smith SR, Jaffe DM, Fisher EB, Highstein GR, Strunk RC. Improving follow-up for children with asthma after an acute emergency department visit. Journal of Pediatrics. 2004;145(6):772–777. doi: 10.1016/j.jpeds.2004.08.029. [DOI] [PubMed] [Google Scholar]
  35. Wade SL, Carey J, Wolfe CR. An online family intervention to reduce parental distress following pediatric brain injury. Journal of Consulting and Clinical Psychology. 2006;74:445–454. doi: 10.1037/0022-006X.74.3.445. [DOI] [PubMed] [Google Scholar]
  36. Walker E, Nowacki AS. Understanding Equivalence and Noninferiority Testing. Journal of General Internal Medicine. 2011;26(2):192–196. doi: 10.1007/s11606-010-1513-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Wechsler D. Wechsler Abbreviated Scale of Intelligence [Assessment instrument] San Antonio, TX: Harcourt Assessment; 1999. [Google Scholar]
  38. Westerberg H, Jacobaeus H, Hirvikoski T, Clevberger P, Ostensson ML, Bartfai A, Klingberg T. Computerized working memory training after stroke—A pilot study. Brain Injury. 2007;21(1):21–29. doi: 10.1080/02699050601148726. [DOI] [PubMed] [Google Scholar]
  39. Wise M, Gustafson DH, Sorkness CA, Molfenter T, Staresinic A, Meis T, … Walker NP. Internet telehealth for pediatric asthma case management: Integrating computerized and case manager features for tailoring a web-based asthma education program. Health Promotion and Practice. 2007;8(3):282–291. doi: 10.1177/1524839906289983. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Woodcock RW, McGrew KS, Mather N. Woodcock-Johnson Tests of Achievement, Third Edition [Assessment instrument] Itasca, IL: Riverside; 2001. [Google Scholar]

RESOURCES