Validation of a Virtual Reality Balance Module for Use in Clinical Concussion Assessment and Management

Abstract

Objective

To determine the criterion and content validity of a virtual reality (VR) balance module for use in clinical practice.

Design

Retrospective, VR balance module completed by participants during concussion baseline or assessment testing session.

Setting

A Pennsylvania State University research laboratory

Participants

A total of 60 control and 28 concussed students and athletes from the Pennsylvania State University

Interventions

None

Main Outcome Measures

This study examined: (1) the relationship between VR composite balance scores (final, stationary, yaw, pitch, and roll) and area of the center-of-pressure (eyes open and closed) scores and (2) group differences (normal volunteers and concussed student-athletes) on VR composite balance scores.

Results

With the exception of the stationary composite score, all other VR balance composite scores were significantly correlated with the center of pressure (COP) data obtained from a force platform. Significant correlations for the eyes open conditions ranged from r= −.273 to −.704 and from r= −.353 to −.876 for the eyes closed condition. When examining group differences on the VR balance composite modules, the concussed group did significantly (p<.01) worse on all measures compared to the control group.

Conclusions

The VR balance module met or exceeded the criterion and content validity standard set by current balance tools and may be appropriate for use in a clinical concussion setting.

Introduction

The cornerstone of clinical concussion management is based on a combination of neuropsychological testing, balance assessment, and symptom evaluation^1,2 and this “gold standard” has remained largely unchanged. While there are a variety of tools used in clinical balance assessments, the Balance Error Scoring System (BESS) and Sensory Organization Test (SOT) are two of the most common postural control assessments. Both of these tools are sensitive to concussion deficits^3–6. Typically, these tools indicate that postural stability reaches its worst 24 hours after concussion, with balance restored to pre-injury measures around 3–5 days post-injury^4,5,7,8. This short recovery window has raised concerns whether the currently used clinical balance tools are capable of picking up the residual deficits often associated with concussion.

While clinical management has remained consistent, continuing technological advancements have allowed for the creation of virtual reality (VR) paradigms. VR is designed to be an interactive environment generated by a computer that mimics the real world. An important component of the VR setting is that the participants inside the environment feel a large sense of presence and the illusion of forward self-motion. In unpublished data, participants undergoing the VR balance module used in this study report a sense of presence of 9.2 ± 1.4, with 0 being no sense of presence and 10 being high sense of presence. This high sense of presence found in the VR environment creates an immersive and more life-like setting, which is an added benefit to VR testing. Compared to more traditional and clinically used concussion assessment tools, other benefit of VR are the 3D nature of the tests and transferability to real-life situations⁹.

Along with its ability to assess depth perception, several studies have found that decision-making processes and visual search behavior are different in a 2D vs. 3D environment^10–12. Along with creating a more realistic testing environment, these additional visual qualities have made VR modules susceptible to balance deficits seen after concussion up to 30-days post-injury^9,13–15.

While VR balance assessments are used in research settings, VR technology has yet to become a common part of clinical assessment. In order to transition from a research to clinical setting, VR balance assessments must first be shown as a legitimate postural assessment tool. Therefore, the primary purpose of this study was to validate a VR balance assessment module for use in clinical settings. We hypothesized that the VR system would be significantly correlated with area of center of pressure data obtained from force platform. It should be noted that the center of pressure (COP) area is considered as a major clinical index of postural stability¹⁶. Additionally, we hypothesized that concussed participants would have significantly worse balance measures compared to normal controls.

Methodology

Part I: Criterion Validation

Participants were recruited from the Pennsylvania State University. For this study, both normal controls (N=60) and concussed (N=21) participants were included and received both standard balance assessments (force plate) and VR balance assessments. Participants from both groups were excluded if they had any known neurological disorder, any lower body injury affecting balance, or a diagnosis of an attention deficit disorder. Each concussed participant was diagnosed by a certified athletic trainer or physicians and tested within 7–10 days after his/her injury. This time window to chosen to reduce the likelihood of the participation experiencing nausea from the VR system and to attempt to identify more subtle balance deficits seen further out from injury. The Pennsylvania State University Institutional Review Board approved the study prior to the initiation of testing. All participants, regardless of group, completed a signed consent form before testing began.

Prior to VR balance assessments, each participant’s balance was measured using an Advanced Mechanical Technology, Inc. (AMTI) force plate. A CAS software collected and processed the center of pressure (COP) data. The data were collected with a sample frequency of 90 Hz. The signals were amplified via a 6 channel AMTI model SGA6-4 amplifier. A maximum gain of 4000 was used, with a low-pass filter of 10.5 Hz. The subject completed two, one-minute conditions on the force platform. Under the first condition, the participant’s eyes remained open, while the participant closed their eyes for the second force platform condition. For both conditions, the participant was told to remain as still as possible throughout the minute-long duration. The area of the center of pressure (COP) was calculated from the time series of the center of pressure displacement in the anterior-posterior (A-P) and medial-lateral (ML) directions. The area of the COP for the eyes open and eyes closed conditions were used as outcome measures. The center of pressure at each instantaneous time defined by the sample rate reflecting the degree of postural motion was calculated as:

• COPx = (−My+Fz)/Fz

  COPy= (−Mz+Fy)/Fz

There is high correlation between several of the standard force platform measures (Slobounov & Newell, 1994), so the area of COP was used as a traditional measure to characterize the postural stability. Specifically, there is an inverse relationship between the area of COP and stability index (see Slobounov et al., 2008).

The VR system used in this study includes several technological components. A VisMini^™ by Vizbox Ultra Portable Passive Stereo 3D projection system allowed for flicker-free stereo on the projection surface, which reduces the chances of inducing more symptoms in recently concussed individuals with photophobia. The software was provided by HeadRehab, LLC (www.HeadRehab.com). InterSense's patented inertial-ultrasonic hybrid tracking technology (IS-900 PCT tracker system) offered real-time tracking of position and orientation in yaw, pitch, and roll directions. The sensor was located on the subject’s head to interact with the visual field motion induced by VR moving room paradigm (see Slobounov et al., 2011). Lastly, a Cinefold portable 83" × 144" projection screen was used to display the image. This technology was used to create a “moving room” experiment. For specifics about the virtual room utilized in this experiment, please see Slobounov et al⁹. Figure 1 show the virtual room utilized in this VR module.

Figure 1.

Virtual reality tilting room as seen during the VR balance module.

Participants in the VR groups were placed in the virtual room and a balance testing protocol was initiated. The field sequential stereo images viewed in the VR environment were separated into right and left eye images using liquid crystal shutter glasses worn by participants. Before trials began, participants were secured in a harness to prevent injury in case of loss of balance. Each participant stood in the tandem stance (one foot in front of the other, hands on hips) and was asked to remain as still as possible as the virtual room swayed in varying directions for 30 seconds. Figures 2 and 3 show participants place in the virtual room during testing.

Figure 2.

Virtual Reality set-up.

Figure 3.

Virtual reality Romberg test set-up.

During the first trial, the room remained still (stationary score). During the subsequent nine trails, the room rotated in one of three directions (yaw, pitch, and roll). The stationary, yaw, pitch, and roll scores, as well as an overall (final) balance score, were used as outcome measures in this study.

Part II: Content Validation

In addition to the data used in Part I of this study, 34 control and 7 concussed participants completed only the VR balance module. This brings the total number of participants for Part II to 94 controls and 28 concussed participants. The additional participants included in this part of the study completed only the VR balance model and did not complete any standard balance assessments (force platform). The same inclusion/exclusion criteria, methodology, equipment used in Part I were used in Part II, with the only difference coming from the exclusion of force platform testing.

Data Analysis

All outcome measures were analyzed using IBM SPSS Version 19.0¹⁷. Although the VR system produces data about head deviation (in cms), the only VR balance outcome measures being used in this study are VR balance outcome scores are based off of a 0–10 scale, with 10 being the best possible performance. Since the purpose of this study is to validate the VR module for clinical use, the authors felt that the outcome scores are the most clinically applicable measure of balance. For the criterion validation portion, all participants in Part I of this study had their data analyzed together. The COP area data for both the eyes opened and eyes closed conditions were compared against the VR balance outcome measures (final, stationary, yaw, pitch, and rolls scores) using correlation/regression analysis to determine significance. For the content validation portion, VR balance outcome measures (final, stationary, yaw, pitch, and rolls scores) were compared between the concussed and control using multiple, repeated measures of analysis of variance (one-way ANOVAs) to determine if significance. Correlations between group and VR balance outcome measures were also analyzed.

Results

Part I: Criterion Validation

COP (open and closed) was significantly (p<.01) and negatively correlated with final, yaw, pitch, and roll VR measures. For both COP conditions, stationary balance scores were not significant. Significant correlations for the eyes open conditions ranges from r= −.273 to −.704. Roll (r= −.704) and final (r= −.656) had the highest correlations under the eyes open condition. For the eyes closed conditions, correlations ranges from r= −.353 to −.876. Similar to the eyes open portion, roll (r= −.876) and final (r= −.778) showed the highest correlations. All correlation coefficients and indications of significance can be seen in Table 1.

Table 1.

Pearson’s correlation coefficients for the COP and VR data.

	COP Eyes Open	COP Eyes Closed
Final	−.656**	−.778**
Stationary	−.034	−.003
Yaw	−.390**	−.353**
Pitch	−.273*	−.364**
Roll	−.704**	−.876**

^*indicates significance at the p<.05 level and

^**indicates significance at the p<.01 level.

Part II: Content Validation

Group (concussed vs. control) was significantly (p<.01) and negatively correlated with all VR balance outcome measures. Stationary scores (r= −.246) showed the lowest correlation to the groups, while roll (r=−.641) displayed the highest correlations.

When examining between group differences, ANOVAs revealed that the concussed group did significantly (p<.01) worse on all VR balance measures compared to the control group. Group means, standard deviations, and significance levels are reported in Table.

Discussion

In this two-part study, a VR balance paradigm was evaluated for its criterion and content validity, in hopes that this new technology can be incorporated as part of clinical concussion assessment and management. Part I showed that, in a group of concussed and control participants, VR final, yaw, pitch, and roll scores were significantly (p<.01) and negatively correlated with the area of the COP in both eyes open and closed condition. Since higher VR scores and lower COP scores indicate better balance, the negative correlation is appropriate.

In a validation of the clinically used BESS, Riemann et al.⁶ found that tester errors score significantly correlated (r= .308 to .788) with force-plate sway measures. In a review of the BESS literature, Bell et al.¹⁸ described these results as “moderate to high” criterion-related validity. VR outcome measures ranged from r= −.273 to −.704 in the eyes open force platform condition and r= −.353 to −.876 in the eyes closed force platform condition. These results are similar to those found for the BESS and suggest that the VR balance paradigm has acceptable levels of criterion validity for use in clinical practice. Although stationary, yaw, pitch, and roll measures were analyzed in this study, the VR final balance score is designed to incorporate all of these components and is recommended as the most important clinical evaluation component. The final score correlations were r= −.656 (eyes open) and r= −.778 (eyes closed). These were the second highest correlations for the VR measures. Since the Riemann et al. study established validity with correlations between r= −.273 to −.704, the VR final score correlations with COP data meet or exceed the established clinical validity markers.

Once the VR balance paradigm was determined to meet the clinically acceptable levels of validity for postural stability tools, Part II of the study was initiated to determine the content validity of the paradigm. Group was significantly (p<.01) and negatively correlated with all VR balance outcome measures. Correlations ranged from r= −.246 to −.641, with the roll condition having the strongest correlation to group. Post hoc analysis was run to determine if significant mean differences existed between groups. Under all balance outcome measures, the concussed group performed significantly (p<.01) worse than the control group.

Several studies have shown that BESS^4–6 and SOT^3–5 are valid clinical tools for concussion assessment and management because they are capable of detecting balance deficits commonly seen after concussion. Based on these criteria, the results of this study indicate that the VR balance module is a valid clinical tool due to its ability to expose subtle postural stability deficits. Under all of the balance outcome measures, with the largest differences seen under the roll condition, the concussed group performed significantly worse compared to control subjects. These results suggest that VR balance is an acceptable tool for use in a clinical concussion assessment and management setting.

While the majority of this study compares the findings of a VR balance paradigm to the BESS and SOT, it is important to note a difference between the paradigms is the exclusion of any eyes closed conditions during the VR paradigm. While there is great merit to teasing out the part the visual system plays in balance, the VR paradigm necessitates that the participant has his/her eyes open throughout all trials. This is because the VR affects balance by inducing optic flow⁹. The immersive environment of VR would also be sacrificed by closing the eyes. Additionally, athletic participation requires using and integrating visual information, so the authors feel keeping the eyes open for the duration of balance testing is the most clinically applicable scenario.

Studies investigating both the BESS and SOT have found that postural stability deficits resolve around 3–5 days post-concussive injury^4,5. Due to the 3-D and immersive nature of the VR paradigm, concussed participants were tested between 7–10 days post-injury in order to reduce the chances of inducing nausea and motion sickness. Therefore, the VR balance paradigm is capable of detecting balance deficits with a longer interval (approximately two times longer) between concussive injury and testing. This suggests that the VR balance module may be more sensitive to the residual and subtle deficits of concussion.

Limitations

Although this study uses a novel balance assessment tool in the clinical assessment of sports related concussions, it is not without its limitations. For the concussed group, there were no baseline balance data available. Therefore, it is unknown how much of the group differences were due to concussive effects or poor balance in general. While this study has shown the VR balance module to valid, the single testing session did not allow for reliability data. As reliability of outcome measures over multiple testing sessions is critically important for clinical diagnostic tools, further research should investigate the reliability of the VR balance paradigm. An obvious limitation to this study is that the VR paradigm is indirectly compared to the BESS and SOT, instead of including these balance measures directly in our study. This is because the data used in the manuscript was obtained retroactively and BESS and SOT data were not available for these participants. Since both the BESS and SOT were validated using force platform data and this study directly compares the VR data to force platform data, we feel that the VR balance module is a valid form of balance assessment that can be used clinically. Lastly, the cost of a VR system, especially compared to a traditional BESS test, is substantial. However, as new technology continues to make prices more affordable and future research may highlight the additional benefits of using VR technology, it is our hope that VR paradigms can find a more prominent place in concussion research and clinical care.

Conclusion

According to Mayers and Redick¹⁹, in order for a tool to be clinically useful it must have sound psychometric properties and proven validity. In order for VR modules to be accepted into clinical practice, they must first be shown as valid tools for detecting balance deficits. Based on the criteria set by currently used clinical balance assessments, these results indicates that the VR balance model validly measures postural stability and that the nature of the test is capable of detecting balance deficits post-concussive injury. Although additional research needs to be done to examine other important psychometric properties, this study supports the use of a VR balance module in clinical concussion assessment and management.

Table 2.

Concussed (top) and control (bottom) group mean, standard deviation, and significance level for all VR balance outcome measures.

	Mean	Standard Deviation	P-Value
Final	7.61 8.72	1.62 0.93	<.001
Stationary	8.69 9.23	1.73 0.42	.006
Yaw	7.83 9.01	1.72 0.50	<.001
Pitch	7.96 8.81	1.72 0.67	<.001
Roll	6.07 8.40	1.93 0.85	<.001

Clinical Relevance.

VR balance module is a valid tool for concussion assessment in clinical settings. This novel type of balance assessment may be more sensitive to concussion diagnoses, especially later (7–10 days) in the recovery phase than current clinical balance tools.