Abstract
Objective.
Traumatic brain injury (TBI) is a leading cause of acquired disability in children, who are at risk of significant impairment in executive function (EF). Virtual reality technology provides a novel strategy to offer rich and immersive training content that is both appealing to children and of potential value in improving their daily functioning. The present study aimed to evaluate the feasibility and safety of implementing an innovative VR-based interactive cognitive training (VICT) system for EF rehabilitation designed to meet the developmental and clinical needs of children with TBI.
Methods.
A parallel-group random-block randomized controlled trial was conducted among 26 children 7–17 years with TBI, who completed baseline, post-intervention, and 2-month follow-up visits. Feasibility was assessed for recruiting children, measuring outcomes, and implementing the intervention. VR satisfaction was assessed via five-point Likert scales. Safety outcomes included simulator sickness (0–4) and physical exertion (6–20). Preliminary efficacy was assessed by NIH Toolbox Cognitive Battery tasks.
Results.
Findings supported the feasibility of recruitment, outcome assessment, and delivery of the intervention. The intervention group reported adequate VR satisfaction in terms of pleasure (M=3.25, SD=0.50) and motivation (M=2.75, SD=0.96), as well as low levels of physical exertion (M=6.25, SD=0.50) and simulator sickness (M=0.16, SD=0.19). Preliminary evidence supported potential efficacy of the intervention, particularly for moderate and severe TBIs.
Conclusions.
The present study found high feasibility, safety, and preliminary efficacy of the VICT system. Further research is required to fully examine the intervention’s efficacy as a possible rehabilitation tool for children and adolescents with TBI.
Keywords: traumatic brain injury, virtual reality, executive function, children, rehabilitation
Traumatic brain injury (TBI) is one of the leading causes of mortality and acquired disability among US children (Centers for Disease Control and Prevention, 2015; Faul et al., 2010; Shen et al., 2021). Defined as a disruption in the normal function of a child’s brain that can be caused by a bump, blow, or jolt to the head, or a penetrating head injury, pediatric TBIs can result in significant impairment in cognitive functions, such as executive functions (EFs), especially in moderate to severe injuries (Faul et al., 2010; Leblanc et al., 2005; Scheibel & Levin, 1997; Scott & Schoenberg, 2011). EFs include a set of core cognitive capacities for self-regulation, creativity, and flexibility (Diamond, 2013; Diamond & Lee, 2011). Deficits in EF were shown to have profound implications for the children’s daily executive behaviors (Diamond, 2013) and quality-of-life (Câmara-Costa et al., 2020), as reflected in increased attention problems (Le Fur et al., 2020), poorer academic performance (Ilie et al., 2020), and poorer psychosocial adjustment (Keenan et al., 2018; Kennedy et al., 2017).
However, evidence-based EF training programs specifically designed for childhood TBI are unavailable (Cicerone et al., 2019; Niemeier et al., 2015; Shen, Johnson, et al., 2020). As early as 2015, the CDC reported to Congress that post-TBI cognitive rehabilitation was the No. 1 unmet health care need among children with TBI and called for more innovation in this area (Centers for Disease Control and Prevention, 2015). Although a combination of computerized and non-computerized cognitive games has been effective in improving healthy children’s EFs, adherence and generalizability remain two key obstacles that hamper wide implementation of such interventions in children with TBI (Carruthers et al., 2014; Diamond & Lee, 2011). For example, existing EF training typically uses paper-and-pencil tasks or flat-screen computer tasks, both of which are unappealing to children and adolescents who are increasingly accustomed to interactive video games that are much more engaging (Greenfield, 2014). This may negatively affect children’s long-term rehabilitation outcomes by decreasing their willingness to participate in planned training activities, especially for follow-up sessions after discharge. Furthermore, partially due to the lack of variety and richness, the skills acquired through existing EF interventions often show limited transfer to untrained life activities (Diamond & Lee, 2011).
Virtual reality (VR) offers an exciting alternative EF rehabilitation strategy for at least two reasons. First, VR technology has the capability to offer a multitude of activities for training children with TBI in core EFs within a versatile virtual environment (Diamond, 2013), and training of this type has been theoretically considered effective in promoting structural and functional recovery of the human brain (Grealy & Heffernan, 2001). Second, unlike traditional paper- or flat screen computer-based EF training, VR-based EF training – a cutting-edge technological platform in the realm of video gaming – has the potential to facilitate long-term rehabilitation outcomes by improving intervention adherence (Hsu et al., 2016). Based on these advantages, research using VR for both cognitive and physical rehabilitation among adult patients with TBI has been increasing in recent years (Hsu et al., 2016; Shin & Kim, 2015). However, a recent systematic review found limited research on VR applications in childhood TBI rehabilitation, especially in the realm of cognitive rehabilitation (Shen, Johnson, et al., 2020).
To address this gap, we developed a novel VR-based interactive cognitive training (VICT) program specifically designed to meet the developmental and clinical needs of children with TBI. The design, rationale, and methods are detailed in a previous publication (Shen, Xiang, et al., 2020). Briefly, a series of VR games were designed to follow a general story line in which participants are asked to complete various EF training tasks to rescue an animated creature named the ‘Lubdub’ from a guarded castle. The overall goal of the ongoing larger project was to examine the safety and feasibility (Phase II, ORBIT model) and efficacy (Phase III, ORBIT model) of the VICT system in improving both trained and untrained EF skills following pediatric TBI. This Phase II project has been completed and this paper presents findings regarding the safety (e.g., tolerance and acceptability), feasibility of the research paradigm (e.g., recruitment, measurement, intervention metrics) as well as preliminary efficacy data to identify changes in clinical outcomes worthy of examining in a future efficacy trial of the VR program.
Method
Transparency and Openness
This report includes information regarding determination of sample size, data exclusions (for attrition and data error), and study design, procedures, and measures. CONSORT reporting standards were followed for this pilot clinical trial, with the flow diagram reported in Figure 2 and the CONSORT checklist available as supplementary materials. De-identified data, analysis code, and research materials are available upon request by emailing the corresponding author. Data were analyzed using SAS 9.4 and IBM SPSS Statistics 26. This study’s design and hypotheses were registered at clinicaltrials.gov (NCT03611062).
Study Design
A parallel-group random-block randomized control trial (RCT) design was adopted as the research paradigm to examine the protocol’s feasibility for a future efficacy trial. Participants were randomized to either the intervention group (VICT training) or the control group (a comparable VR game without cognitive training).
Study Setting
Participants were recruited from a Level 1 Pediatric Trauma Center in the Midwester U.S. The study protocol was approved by the institutional review board at the participating institution (IRB18-00472) Potential study participants were recruited from both the inpatient rehabilitation unit and outpatient clinics, as well as through trauma registries provided to the study team monthly during the recruitment period. All study procedures, including the intervention and assessments, were completed within a hospital setting (patient rooms for inpatient participants and exam rooms designated for clinical research for outpatient participants).
Participants
Sample Size and Characteristics.
The study recruited 26 children aged 7–17 with TBI. Because this was a pilot feasibility study, no a-priori power analysis was conducted for sample size estimation; the sample size was determined primarily based on available patient volume at the recruitment site.
Eligibility Criteria.
The following inclusion criteria were used for eligibility screening: 1) history of hospital admission for TBI (ICD-9 codes 803, 850, 851, or 854) and between 7 to 17 years old (inclusive) when admitted; 2) sustained a complicated mild to severe TBI, defined as a lowest post-resuscitation Glasgow Coma Scale (GCS) score = 13–15 combined with trauma-related abnormalities on neuroimaging or a depressed skull fracture (complicated mild TBI), GCS score = 9–12 (moderate TBI), or GCS score = 3–8 (severe TBI); 3) fluent in English; and 4) current Agitated Behavior Scale (ABS) score <28. To minimize the possibility of confounding by comorbidities and cognition, the following exclusion criteria were applied: 1) severe comorbidities secondary to TBI or premorbid neurological disorder or neurodevelopmental issues prior to injury that would have confounded study outcomes and administration of the study protocol (e.g., visual field disturbances, frequent nausea/vomiting, not being medically stable yet, intellectual disability, severe autism); and 2) medical restrictions that precluded patients from using electronic gaming devices (e.g., due to post-injury seizure activity).
Consent.
Once identified as eligible to participate, research staff approached legal guardians and eligible children either in hospital (if inpatient) or via IRB-approved mail, email, or phone (if outpatient) to introduce the research project. Staff ensured children are awake, alert, and able to sign the assent form (if ≥ 9 years) and demonstrated understanding of the study before being approached. Staff proceeded with consenting only if the patient and patient family were interested in participating in this research after initial study introduction. During consenting, the researchers ensured that the patients and family understood that participation was voluntary and would not affect any treatment she/he may receive in the current or future visits. The legal guardians of participants in this study gave consent and the participate gave assent if age >=9 years.
Randomization.
Following informed consent from parents and assent from children (if older than 9 years old) at the baseline assessment, all participants were randomly assigned to one of the following two groups: intervention group (VICT training) or control group (a comparable VR gaming environment without EF training). The random allocation sequence was generated using a SAS-based randomization program with 1:1 allocation ratio and random block sizes of 2 and 4. No stratification was used in randomization. Randomization was implemented through the “randomization module” on REDCap. Study staff who conducted the outcome assessment after intervention delivery were masked to group assignment. However, complete masking of assessors to group assignment was not feasible as participants may have inadvertently referred to playing the VR games when answering the safety/feasibility questions.
Interventions
Detailed description and pictorial illustrations of both the hardware and software for both intervention and control groups have been published elsewhere (Shen, Xiang, et al., 2020). Notably, the VR equipment was sanitized with germicidal wipes after each use and both groups were provided the same hardware for delivering the VR experience. The protocols implemented for the two groups during the feasibility trial are described below.
Content and Protocol for the Intervention Group.
After randomization, children assigned to the intervention group played an animated VR game developed by the research team called “Rescue the Lubdub.” The game consists of three challenging and child-friendly tasks designed to train three EFs: inhibitory control (defeating castle guards), working memory (unlocking castle gates), and cognitive flexibility (rescuing the Lubdub from the castle). The child completes a one-on-one tutorial with trained research staff on how to play each training task. After receiving visual/audio feedback to learn the tasks, children were asked to go through at least one training session (consisting of 30 trials per training task for approximately 20–30 minutes in total) but were allowed to play more sessions if they had the opportunity and chose to do so.
Content and Protocol for the Control Group.
Children in the control group were provided with the same hardware setup and comparable virtual environment but played a non-cognitively taxing training VR game that required no specific EF skill training to play, e.g., casting different types of spells (bees, bouncy balls, sparkler spells) to objects in a virtual playground. The control group was asked to play their VR game for a similar period as the intervention group (approximately 20–30 minutes). Like the intervention group, this group was also permitted to play for longer periods if they had the opportunity and chose to do so.
Measures
Demographic/Clinical Variables.
Data were obtained from the participant’s electronic medical records, and included participant age, sex, race, ethnicity, severity of TBI, GCS at the time of injury, and Rancho Los Amigos Scale score at admission. The participant’s experience of playing video games (1–5 scale, higher score indicating more experience) was obtained via self-report.
Feasibility Outcomes.
A Protocol Implementation Sheet consisting of the following three sections were used to evaluate the feasibility of the research paradigm: (a) Recruitment Feasibility: average number of children screened/enrolled per month, proportion of eligible children who enroll, and treatment-specific retention rates; (b) Intervention Feasibility: proportion of children who adhered to assigned treatment (defined as not requesting to switch to another group after assignment) and children’s time spent in VR; (c) Measurement Feasibility: Because one important goal of the study was to evaluate the feasibility of a clinical trial protocol for future evaluation of the efficacy of the VICT program, the study assessed the “administrational efficacy” of the efficacy outcome measures in terms of the number children who completed them and the time required for children to complete them (see “Preliminary Efficacy Measures” for details).
Furthermore, a Patient VseR Experience Survey collected both quantitative VR satisfaction ratings and open-ended qualitative feedback regarding children’s experience after playing the VR games. VR satisfaction was assessed using a 6-item survey consisting of questions about the overall VR experience, such as levels of perceived fun, enjoyment, and motivation to complete the VR games. An example item is: “How did you like the virtual reality games you just played?” Ratings were provided using a 5-point Likert scale, and scores were averaged across items, with higher scores indicating higher levels of satisfaction. The survey was conducted twice: immediately following the first VR training session, and upon completion of training. The second administration was conducted to minimize potential bias that may have contributed to the initial rating due to the novelty of VR games. Qualitative VR experience was probed by asking a single-item open-ended interview question about the perceived benefits and challenges involved in using the VR program. Responses were recorded verbatim by the research staff.
Safety Outcomes.
Trained research staff documented any adverse event (AE) or serious adverse event (SAE) during study participation. AE was defined as any unfavorable and unintended sign or symptom temporally associated with the intervention, regardless of whether it was considered related to the intervention. SAE was defined as any untoward medical occurrence that resulted in patient death or life-threatening illness/injury during the study participation, whether or not it was causally related to the intervention. Any AE/SAE was to be reported to the IRB by the study PI in real-time and study protocols were to be re-evaluated by both the study team and IRB after each event. To assess potential side effects of playing the VR games, children provided ratings of simulator sickness using a 15-item Simulator Sickness Questionnaire or SSQ (Kennedy et al., 1993). The SSQ is a widely used questionnaire consisting of 15 items related to potential side effects common to individuals after playing in virtual environments, such as “general discomfort,” “headache,” and “sweating.” The questionnaire asks about symptoms being currently experienced as rated on a 0–3 scale. An example item is: “How much is headache affecting you right now?” The total score was the average rating, with higher scores indicating higher levels of simulator sickness. Finally, children rated their physical exertion after playing the VR games using the Borg Rating of Perceived Exertion Scale (Williams, 2017). This scale consists of a single question asking children to rate their subjective feeling of exertion, without considering the kinds of physical demands that might lead to this feeling. The scale ranges from 6 (no exertion at all) to 20 (maximal exertion).
Efficacy Outcomes.
Preliminary efficacy of the VR program for improving children’s EF skills was measured NIH Toolbox Cognitive Battery. Specifically, an NIH Toolbox Composite Score was computed by taking the average of age-corrected standard scores (M=100, SD=15) from the following three NIH Toolbox tasks related to EF: Flanker Inhibitory Control and Attention Test, List Sorting Working Memory Test, and Dimensional Change Card Sort Test. Higher scores indicated better EF performance.
Additionally, the following efficacy measures were included: Conners’ Continuous Performance Test 3rd Edition (children), PedsQL Core Scale (parents/legal guardians), Child Behavior Checklist (CBCL; parents/legal guardians), and Behavior Rating Inventory of Executive Function 2 (BRIEF2; parents/legal guardians). CPT3 and CBCL scores provided measures of children’s attentional problems and could be controlled as confounding variables in future efficacy trials. BRIEF2 and PedsQL served as measures of daily EF functions and distal health outcomes, respectively. Note that these additional measures were intended to assist the study team in gauging the feasibility of measurement (see details in Feasibility Outcomes), not efficacy evaluation, as is consistent with the overall objectives of Phase 1 of this project.
Procedures
Baseline Visit.
All participants completed baseline measures prior to randomization, including the NIH Toolbox Cognitive Battery tasks and Conners’ Continuous Performance Test 3rd Edition (children). All participants were provided a tutorial to familiarize them with the VR environment and instruct them how to operate and navigate through the VR games. A member of the research team recorded feasibility data on the proportion of planned baseline measures completed and the duration of each assessment. Neither the assessor nor the child was aware of group assignment at the baseline visit.
Post-Intervention Visit.
Immediately after playing the VR games, children in both the intervention and control groups completed the quantitative measures of VR satisfaction, qualitative assessment of the VR experience, and ratings of simulation sickness and physical exertion. Examiners masked to children’s baseline performance and intervention assignment also re-administered the NIH Toolbox Cognition Battery tests and the Conners CPT 3. The proportion of planned post-intervention measures completed and duration of each assessment were additionally documented.
Follow-Up Visit.
All children completed a follow-up assessment approximately 2 months after the post-intervention visit. Examiners masked to group assignment and baseline/post-intervention performance re-administered the NIH Toolbox battery and the Conners CPT 3, the children completed the children version of the PedsQL Generic Core Scales, and their parents or legal guardians completed the BRIEF2, CBCL-APS, and parent version of the PedsQL. The proportion of planned follow-up measures completed and duration of each assessment were again documented.
Note that in order to reduce potential practice effects on computerized tasks such as the NIH Toolbox Cognitive Battery, the study team scheduled the three visits (baseline, post-intervention, and follow-up assessments) on different days.
Data Analysis
To evaluate protocol feasibility in recruitment, measurement, and intervention delivery, the following descriptive statistics were calculated: average number of children screened and enrolled per month, proportion of eligible children enrolled, treatment-specific retention rates, proportion of children who adhered to their assigned condition, proportion of planned measures completed, and average time to complete each measure. To evaluate intervention safety, the total number of AEs/SAEs, mean simulator sickness scores, and mean Borg exertion scores were computed, and the intervention and control groups were compared on these variables. The VR experience of the participants was analyzed using both quantitative and qualitative methods. Descriptive statistics including mean and standard deviations were computed for the quantitative measures. Qualitative feedback from patients on the perceived benefits and challenges of the VR program were recorded verbatim and submitted to the VR development team for improvement in future iterations. Based on the exploratory nature of the feasibility study and its small sample size, inferential statistical analyses were performed only on the demographic/clinical variables for the purpose of verifying the randomization procedure. Finally, exploratory analysis was conducted on the NIH Toolbox composite score at baseline, post-intervention, and follow-up visits between intervention and control groups to provide preliminary efficacy data to guide future efficacy trials. As indicated in Figure 2, data for some outcomes were missing due to attrition. There was also a data error in the variable “time since injury” for one participant, resulting in a smaller sample size in analysis of that variable.
Results
Sample Characteristics
Demographic and clinical characteristics of the study participants are presented in Table 1. A CONSORT flow diagram is shown in Figure 1. The comparable size of the intervention and control groups supported the utility of the blocked randomization scheme for this small sample study. Participants in the two groups were also comparable in pre-intervention baseline characteristics, including age, sex, race, ethnicity, TBI severity, video gaming experience, and total VR “dosage” as measured by the number of minutes each participant spent in either VICT training or the VR control game.
Table 1.
Characteristics of the Study Participants (N=26)
| Variable Name | Total (n=26) N (%) |
Intervention (n=14) N (%) |
Control (n=12) N (%) |
p |
|---|---|---|---|---|
| Age * | 12.96 (3.27) | 14.00 (3.09) | 11.75 (3.17) | 0.08 |
| Sex | ||||
| Males | 16 (61.64%) | 8 (57.14%) | 8 (66.67%) | 0.70 |
| Females | 10 (38.46%) | 6 (42.86%) | 4 (33.33%) | |
| Race: | ||||
| White | 20 (76.92%) | 11 (78.57%) | 9 (75.00%) | 0.81 |
| Black | 5 (19.23%) | 3 (21.43%) | 2 (16.67%) | |
| Other | 1 (3.85%) | 0 (0.00%) | 1 (8.33%) | |
| Ethnicity: | ||||
| Non-Hispanic | 25 (96.15%) | 14 (100.00%) | 11 (91.67%) | 0.46 |
| Unknown | 1 (3.85%) | 0 (0.00%) | 1 (8.33%) | |
| Traumatic Brain Injury Severity | ||||
| Complicated Mild | 9 (34.62%) | 4 (28.57%) | 5 (41.67%) | 1.00 |
| Moderate | 3 (11.54%) | 2 (14.29%) | 1 (8.33%) | |
| Severe | 13 (50.00%) | 7 (50.00%) | 6 (50.00%) | |
| Missing | 1 (3.85%) | 1 (7.14%) | 0 (0.00%) | |
| Glasgow Coma Scale * | 9.04 (4.72) | 8.62 (4.59) | 9.50 (5.02) | 0.65 |
| Video Gaming Experience | ||||
| Never | 2 (7.69%) | 2 (14.29%) | 0 (0.00%) | 0.48 |
| A little bit | 3 (11.54%) | 1 (7.14%) | 2 (16.67%) | |
| Sometimes | 7 (26.92%) | 4 (28.57%) | 3 (25.00%) | |
| Often | 7 (26.92%) | 5 (35.71%) | 2 (16.67%) | |
| Always | 6 (23.08%) | 2 (14.29%) | 4 (33.33%) | |
| Missing | 1 (3.85%) | 0 (0.00%) | 1 (4.00%) | |
| Total number of minutes spent in VR * | 31.43 (15.87) | 34.17 (14.05) | 28.45 (17.83) | 0.40 |
Mean (Standard deviation) for continuous variables
Figure 1.
CONSORT extension for Pilot and Feasibility Trials Flow Diagram
Protocol Feasibility
The study protocol demonstrated high feasibility in recruitment, measurement, and intervention. Specifically, as shown in Table 2, we screened an average of 6.2 children per month (SD=4.26) and enrolled an average of 1.4 children (SD=1.19) per month. 100% of participants assigned to the intervention group and 100% of participants assigned to the control group remained in their assigned group upon completion of the intervention session. There were three children in the intervention group who were lost at either post-intervention or 2-month follow-up visits due to patient request (n=1) or the pandemic interruptions (n=2). In the control group, there was one child who did not attend the 2-month follow-up visit due to scheduling issues and another child who only completed part of the study measures at post-intervention visit due to technical issues of the VR system at the time of the visit. Overall, more than 80% of children completed all study measures. As shown in Table 3, participants in both groups showed high completion rates (>90%) and reasonable completion time for all measures.
Table 2.
Recruitment Feasibility
| Variable | N (%) |
|---|---|
| Average number of children screened per month* | 6.2 (4.26) |
| Average number of children enrolled per month* | 1.4 (1.19) |
| Proportion of eligible children who enroll | 26 out of 88 (29.55%) |
| Children in the intervention group who completed assigned training | 14 out of 14 (100%) |
| Children in the control group who completed assigned training | 12 out of 12 (100%) |
| Children who completed assigned training after randomization overall | 26 out of 26 (100%) |
| Children in intervention group who completed all study visits | 11 out of 14 (78.57%) |
| Children in control group who completed all study visits | 10 out of 12 (83.33%) |
| Children who completed all study visits overall | 21 out of 26 (80.77%) |
Mean (Standard deviation) for continuous variables.
Table 3.
Measurement Feasibility: Completion Rates and Duration (N=25)
| Measures | Timepoint | Total |
Intervention |
Control |
|||
|---|---|---|---|---|---|---|---|
| Completion Rate (%) | Duration (minutes) | Completion Rate (%) | Duration (minutes) | Completion Rate (%) | Duration (minutes) | ||
| Perceived Exertion | 1st Session | 96 | 1.18 | 93 | 1.08 | 100 | 1.30 |
| All Sessions | 100 | 1.14 | 100 | 1.17 | 100 | 1.10 | |
| Simulator Sickness | 1st Session | 96 | 1.75 | 93 | 1.64 | 100 | 1.89 |
| All Sessions | 100 | 1.59 | 100 | 1.33 | 100 | 1.90 | |
| VR Experience | 1st Session | 96 | 1.73 | 93 | 1.55 | 100 | 1.91 |
| All Sessions | 100 | 1.81 | 100 | 2.00 | 100 | 1.60 | |
| NIH Toolbox | Baseline | 100 | 16.30 | 100 | 16.50 | 100 | 16.09 |
| Post | 100 | 18.05 | 100 | 18.18 | 100 | 17.90 | |
| Follow-up | 100 | 18.65 | 100 | 18.11 | 100 | 19.09 | |
| Conners CPT3 | Baseline | 100 | 15.44 | 100 | 15.57 | 100 | 15.27 |
| Post | 100 | 15.09 | 100 | 15.08 | 100 | 15.09 | |
| Follow-up | 100 | 14.90 | 100 | 14.60 | 100 | 15.18 | |
| BRIEF2 | Follow-up | 100 | 7.84 | 100 | 6.22 | 100 | 9.30 |
| PedsQL Parent | Follow-up | 100 | 3.94 | 100 | 2.25 | 100 | 5.30 |
| PedsQL Child | Follow-up | 100 | 3.80 | 100 | 2.44 | 100 | 4.91 |
| CBCL | Follow-up | 100 | 16.18 | 100 | 16.57 | 100 | 15.90 |
Both the quantitative and qualitative assessments indicated high levels of patient satisfaction after using the VICT program. As shown in the upper section of Table 4, participants in both the intervention and control group reported positive experience playing the VR games, as assessed by reports of game quality and of having fun, liking, and being motivated to play. These high satisfaction ratings were reported after both the first session and repeated sessions of the VR games. The two experimental groups reported largely comparable experiences. The only significant difference was the control group’s marginally higher rating on the item “fun playing the VICT games” (p < 0.05).
Table 4.
Quantitative VR Experience and Safety of VICT Program
| 1st Session |
All Sessions |
|||||
|---|---|---|---|---|---|---|
| Intervention | Control | p | Intervention | Control | p | |
| Quantitative VR Experience (1–5) | ||||||
| Quality of the VR games | 2.86 (1.17) | 3.33 (1.50) | 0.37 | 2.25 (0.96) | 2.00 (0.00) | 0.64 |
| Fun playing the VR games | 3.64 (1.22) | 4.58 (0.67) | 0.03 | 3.25 (0.50) | 4.00 (1.41) | 0.36 |
| Like the VR games | 3.57 (1.28) | 4.33 (0.65) | 0.07 | 3.75 (0.96) | 4.00 (0.00) | 0.64 |
| Want to play again in the future | 3.71 (1.27) | 4.25 (0.87) | 0.24 | 3.25 (2.06) | 2.50 (0.71) | 0.66 |
| Want similar VR games in future visits | 3.79 (1.25) | 3.92 (1.51) | 0.81 | 3.75 (1.50) | 5.00 (0.00) | 0.19 |
| More motivated to attend follow up sessions | 3.71 (1.27) | 4.17 (1.19) | 0.36 | 2.75 (0.96) | 4.50 (0.71) | 0.09 |
| Safety | ||||||
| Perceived Exertion (6–20) | 8.61 (3.33) | 7.29 (1.54) | 0.20 | 6.25 (0.50) | 11.50 (2.12) | 0.17 |
| Simulator Sickness (0–4) | 0.15 (0.29) | 0.03 (0.04) | 0.17 | 0.16 (0.19) | 0.03 (0.04) | 0.43 |
Review of qualitative feedback to the open-ended question on the benefits and challenges of using the VICT system revealed that about 50% of the sample (7 in the intervention group and 5 in the control group) had nothing to add to their quantitative ratings. The remaining participants provided mostly positive comments on the VICT system, emphasizing the variety of gaming content, their high level of pleasure in playing the games, and the perception that the games helped to improve their cognitive abilities (e.g., memory). Participants also provided constructive feedback on ways to improve the VR games for future iterations. Examples of such feedback included the recommendation to add more games and more challenging levels to existing games, fix occasional visual/technical glitches that occurred during the games, and improve user-game interaction mechanisms to allow for more precise control of game elements.
Intervention Safety
Overall, this pilot study revealed that the VICT program demonstrated adequate safety for children with TBI. Because of their full immersion in a virtual environment, some children were expected to experience minor eye fatigue with electronic screens in general and a slight simulation-related motion sickness particular to VR gaming. As anticipated, these symptoms were low in frequency, temporary, and short-lived. Specifically, only two AEs were documented over the course of the study; one child reported blurry vision and eye strain (control group) and another reported headache after playing the VR games (intervention group). Both participants were withdrawn from participation after reporting these events, and further follow-up of the participants and their electronic medical records indicated that their symptoms resolved shortly after study withdrawal. Both events were reported to the local IRB in a timely fashion. As shown in the lower section of Table 4, participants from both groups who remained in the study reported low levels of simulation-related sickness on a scale of 0–3, as well as low levels of perceived physical exertion, even after repeated exposure to the games, with no significant differences found on any of these variables between the groups.
Preliminary Efficacy
Although the primary objectives of the current study were evaluation of feasibility and safety of the VR program, an exploratory analysis of children’s EF performance on NIH Toolbox tasks was conducted. Table 5 summarized the detailed descriptive statistics and effect sizes by injury severity (moderate-severe TBIs and mild complicated TBIs). Results indicated that despite the fact that both intervention and control groups started at similar levels, the VR program showed promising trends of improving the EF performance in NIH Toolbox tasks, but only among children with moderate to severe TBIs especially at follow-up assessment (Hedges’ g = 0.42, 95% CI = [−5.51, 6.00]), not children with mild complicated TBIs (Hedges’ g = −0.59, 95% CI = [−8.68, 7.50]).
Table 5.
Preliminary Efficacy of VICT Program Assessed by NIH Toolbox Composite Scores
| TBI Severity | Visit | Condition | Mean | SD | p | Hedges’ g | 95% CI of g |
|---|---|---|---|---|---|---|---|
| Mild Complicated | Baseline | Intervention | 99.00 | 10.15 | 0.85 | −0.11 | −6.58, 6.35 |
| Control | 100.27 | 9.71 | |||||
|
| |||||||
| Post | Intervention | 85.67 | 11.41 | 0.05 | −1.59 | −8.25, 5.08 | |
| Control | 103.20 | 8.58 | |||||
|
| |||||||
| Follow-up | Intervention | 91.11 | 12.17 | 0.39 | −0.59 | −8.68, 7.50 | |
| Control | 99.00 | 11.42 | |||||
|
| |||||||
| Moderate-Severe | Baseline | Intervention | 83.52 | 10.23 | 0.72 | 0.18 | −5.60, 5.95 |
| Control | 81.31 | 13.58 | |||||
|
| |||||||
| Post | Intervention | 87.24 | 6.48 | 0.91 | 0.06 | −5.96, 6.07 | |
| Control | 86.52 | 14.89 | |||||
|
| |||||||
| Follow-up | Intervention | 96.46 | 8.87 | 0.41 | 0.42 | −5.15, 6.00 | |
| Control | 91.57 | 13.09 | |||||
Note. Hedge’s g was calculated as an unbiased measure of effect sizes with higher values indicating larger group differences using the following guidelines: small (Hedges’ g: 0.15~0.4), medium (Hedges’ g: 0.4~0.75), large (Hedges’ g: >0.75)
Discussion
Findings of the present study support the feasibility and safety of implementing the VICT system with children with TBI for post-injury rehabilitation of EF. High levels of feasibility in recruitment, measurement, and gaming experience were found in both the intervention and control groups. Except for two participants who were withdrawn because of symptoms of either eye strain or headache, participants reported high levels of acceptability with the VICT system and low levels of AE/SAEs and side effects after using the intervention or control VR games.
Both recruitment and measurement proved highly feasible. One concern regarding recruitment was the contrast between the large number of children per month identified as eligible relative to those enrolled into the study. The study’s success in finding eligible participants is credited to the substantial effort invested in identifying candidates from a large pool of data sources, including admission data in the hospital’s electronic medical record (EMR), analysis of monthly trauma registry data, and active communication with both inpatient and outpatient units within the hospital, such as neurosurgery, physical medicine and rehabilitation (PM&R), and psychology. Despite the low rate of enrollment relative to children screened as eligible, recruitment of 1–2 participants per month was in line with our initial expectation given the stringent inclusion/exclusion criteria (e.g., age, injury severity) and low volume of pediatric TBI cases relative to the overall pediatric population. Possible reasons for the low rate of enrollment among eligible patients included limited availability of working caregivers (e.g., conflicts with work schedules), conflicts with schedules for children during the school year or in summer (e.g., family vacation plans), and families living too far away from the study site. In support of the study protocol’s feasibility for future efficacy trials, the proportion of children who adhered to assigned condition was at or close to 100% for both the intervention and control groups. Moreover, the completion rate for most measures was 100%, with rates exceeding 85% for all measures.
With respect to VR safety and satisfaction, none of our participants experienced severe side effects from playing the VR games. Interestingly, participants found the control game more “fun” to play than the EF training game. This finding is not surprising though, considering the cognitive effort required of the intervention participants to practice their skills in inhibitory control, working memory, and cognitive flexibility, compared to the non-taxing activities involved in playing the control game. This explanation was echoed by safety data indicating that children in the intervention group reported slightly higher levels of perceived physical exertion and more simulator-related sickness symptoms than those in the control group, despite similarity in the physical characteristics of the two VR environments. However, in reports made following completion of VR training sessions, the control group reported non-significantly higher exertion than the intervention group. This trend may have been reflected a lack of adequate variety in the control game after repeated exposure, compared to the more challenging tasks presented to the intervention group. As this group difference were not statistically significant, further research is warranted for such an interpretation to be validated and more research is needed to identify the specific VR characteristics that may appeal to pediatric patients in the rehabilitation settings.
Finally, this pilot study also provided preliminary evidence regarding the potential efficacy of the VR program on improving children’s EF skills after a TBI. The descriptive data did point to a potential positive effect of the VR intervention for children with moderate to severe TBIs. This suggested that future research of formal efficacy evaluation of the VR program might be most cost-efficient to focus on pediatric populations with more severe TBIs. Additionally, the descriptively larger group differences between intervention and control conditions at the follow-up visit, in comparison to the post-intervention visit, seems to indicate a possible delayed but longer-term interventional effect, although such effects might have been confounded with the natural recovery processes of children with TBI. Therefore, more intensive longitudinal research, such as those using ecological momentary assessment, may be needed to help better understand the underlying changes that had led up to such delayed effects between post-intervention and follow-up visits. It is also possible that children in the intervention group may have experienced more “cognitive fatigue” after the VR training compared to the more relaxed VR game played by the control group, which may have negatively affected the performance of children in the intervention group at the post-intervention assessment. Future research incorporating measures of both physical exertion (as done in this study) and cognitive fatigue is needed to examine this possibility.
Several study limitations require emphasis. First, the sample was relatively small. Although the distributions may be representative of the population characteristics of pediatric TBI patients treated at the recruitment site in terms of age, sex, race, ethnicity, and injury severity, restricted variability in these factors precludes meaningful examination of their associations with the feasibility and safety of the study protocol. Second, related to the small sample size, all study activities took place in the hospital setting, which helped to increase adherence to completion, but possibly limited total enrollment especially for those families who live further away and do not need to return to hospital for follow-up appointments within the study timeline. Future research should explore interventional modes that can be implemented at home settings for improved accessibility and potentially larger enrollment rate. Third, due to the small sample size of this pilot study, we were unable to statistically test the relationships between experience, age, and time since injury with our feasibility and efficacy outcomes. Outcomes may be further moderated by other factors not measured in the present study, such as discrepancy of game quality between the VR and participants’ previous games. Future research with more statistically power and refined measures of each of these variables is needed for shed light on potential moderators of outcome. Fourth, we only examined children’s perceptions of satisfaction and not the perceptions of caregivers or medical providers. Future research applying our study protocol would likely benefit from the inclusion of multi-informant perspectives. Finally, caution is needed in interpretation of the preliminary efficacy findings as none of these group comparisons reached statistical significance. This is perhaps at least partially due to the small size of this pilot study sample that should be addressed in future larger-scale efficacy trials.
Conclusion
The present study examined the feasibility, safety, and preliminary efficacy of a VR-based cognitive training system for rehabilitation of EFs following pediatric TBI. The findings document the feasibility of our recruitment methodology, measurement, and intervention, acceptable safety metrics for the VR system, and promising efficacy data among children with moderate to severe TBI. Future research is needed to formally assess the efficacy of the VR intervention for post-TBI executive dysfunction and to examine potential associations of socio-demographic and medical factors with interventional effects.
Supplementary Material
Impact.
Childhood traumatic brain injury (TBI) poses significant impairment in children’s executive functions (EFs), yet interventions specifically designed for children’s EF rehabilitation post-TBI and rigorous clinical trials to establish the safety and efficacy of such interventions remain lacking.
Per the ORBIT model, this study served as the Phase II (a-b) of randomized clinical trials to conduct preliminary testing on the feasibility and safety of a novel virtual reality (VR)-based training program for EF rehabilitation for childhood TBI.
Knowledge from this research serves as the empirical basis for the next phase of the research program to evaluate the efficacy of the VR program in pediatric TBI cognitive rehabilitation.
Acknowledgements:
Thanks to Kimberly Lever, Megan Armstrong, and Deborah Grayson for assistance with participant recruitment, data collection, and data management. Research reported in this publication was supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development of the National Institutes of Health under award number K99/R00HD093814. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Footnotes
Disclosures: Trial registration: ClinicalTrials.gov Identifier NCT03611062.
Authors declare no conflict of interest.
References
- Câmara-Costa H, Opatowski M, Francillette L, Toure H, Brugel D, Laurent-Vannier A, Meyer P, Watier L, Dellatolas G, & Chevignard M (2020). Self-and parent-reported Quality of Life 7 years after severe childhood traumatic brain injury in the Traumatisme Grave de l’Enfant cohort: Associations with objective and subjective factors and outcomes. Quality of life research, 29(2), 515–528. [DOI] [PubMed] [Google Scholar]
- Carruthers K, Zampieri C, & Damiano D (2014). Relating motor and cognitive interventions in animals and humans. Translational Neuroscience, 5(4), 227–238. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Centers for Disease Control and Prevention. (2015). Report to congress on traumatic brain injury in the United States: epidemiology and rehabilitation. National Center for Injury Prevention and Control, 1–72. [Google Scholar]
- Cicerone KD, Goldin Y, Ganci K, Rosenbaum A, Wethe JV, Langenbahn DM, Malec JF, Bergquist TF, Kingsley K, & Nagele D (2019). Evidence-based cognitive rehabilitation: systematic review of the literature from 2009 through 2014. Archives of physical medicine and rehabilitation, 100(8), 1515–1533. [DOI] [PubMed] [Google Scholar]
- Diamond A (2013). Executive functions. Annual review of psychology, 64, 135–168. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Diamond A, & Lee K (2011). Interventions shown to aid executive function development in children 4 to 12 years old. Science, 333(6045), 959–964. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Grealy MA, & Heffernan D (2001). The rehabilitation of brain injured children: the case for including physical exercise and virtual reality. Pediatric rehabilitation, 4(2), 41–49. [PubMed] [Google Scholar]
- Greenfield PM (2014). Mind and media: The effects of television, video games, and computers Psychology Press. [Google Scholar]
- Hsu K-S, Jiang J-F, Wei H-Y, & Lee T-H (2016). Application of the environmental sensation learning vehicle simulation platform in virtual reality. Eurasia Journal of Mathematics, Science and Technology Education, 12(5), 1477–1485. [Google Scholar]
- Ilie G, Trenholm M, Boak A, Mann RE, Adlaf EM, Asbridge M, Hamilton H, Rehm J, Rutledge R, & Cusiman MD (2020). Adolescent traumatic brain injuries: Onset, mechanism and links with current academic performance and physical injuries. PLoS one, 15(3), e0229489. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Keenan HT, Clark AE, Holubkov R, Cox CS, & Ewing-Cobbs L (2018). Psychosocial and executive function recovery trajectories one year after pediatric traumatic brain injury: the influence of age and injury severity. Journal of Neurotrauma, 35(2), 286–296. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kennedy E, Heron J, & Munafò M (2017). Substance use, criminal behaviour and psychiatric symptoms following childhood traumatic brain injury: findings from the ALSPAC cohort. European child & adolescent psychiatry, 26(10), 1197–1206. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kennedy RS, Lane NE, Berbaum KS, & Lilienthal MG (1993). Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. The international journal of aviation psychology, 3(3), 203–220. [Google Scholar]
- Le Fur C, Câmara-Costa H, Francillette L, Opatowski M, Toure H, Brugel D, Laurent-Vannier A, Meyer P, Watier L, & Dellatolas G (2020). Executive functions and attention 7 years after severe childhood traumatic brain injury: Results of the Traumatisme Grave de l’Enfant (TGE) cohort. Annals of physical and rehabilitation medicine, 63(4), 270–279. [DOI] [PubMed] [Google Scholar]
- Leblanc N, Chen S, Swank PR, Ewing-Cobbs L, Barnes M, Dennis M, Max J, Levin H, & Schachar R (2005). Response inhibition after traumatic brain injury (TBI) in children: Impairment and recovery. Developmental neuropsychology, 28(3), 829–848. [DOI] [PubMed] [Google Scholar]
- Niemeier JP, Grafton LM, & Chilakamarri T (2015). Treating persons with traumatic brain injury: history and updates. North Carolina medical journal, 76(2), 105–110. [DOI] [PubMed] [Google Scholar]
- Scheibel RS, & Levin HS (1997). Frontal lobe dysfunction following closed head injury in children: Findings from neuropsychology and brain imaging [Google Scholar]
- Scott JG, & Schoenberg MR (2011). Frontal lobe/executive functioning. In The little black book of neuropsychology (pp. 219–248). Springer. [Google Scholar]
- Shen J, Johnson S, Chen C, & Xiang H (2020). Virtual reality for pediatric traumatic brain injury rehabilitation: a systematic review. American Journal of Lifestyle Medicine, 14(1), 6–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Shen J, Shi J, Cooper J, Chen C, Taylor HG, & Xiang H (2021). A population-based study of the incidence, medical care, and medical expenditures for pediatric traumatic brain injury. Journal of surgical research, 268, 87–96. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Shen J, Xiang H, Luna J, Grishchenko A, Patterson J, Strouse RV, Roland M, Lundine JP, Koterba CH, & Lever K (2020). Designing a Virtual Reality-based Executive Function Rehabilitation System for Children with Traumatic Brain Injuries. JMIR serious games [DOI] [PMC free article] [PubMed] [Google Scholar]
- Shin H, & Kim K (2015). Virtual reality for cognitive rehabilitation after brain injury: a systematic review. Journal of physical therapy science, 27(9), 2999–3002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Williams N (2017). The Borg rating of perceived exertion (RPE) scale. Occupational Medicine, 67(5), 404–405. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.