Differential sensitivity between a virtual reality (VR) balance module and clinically used concussion balance modalities

Abstract

Objective

Balance assessments are part of the recommended clinical concussion evaluation, along with computerized neuropsychological testing and self-reported symptoms checklists. New technology has allowed for the creation of virtual reality (VR) balance assessments to be used in concussion care, but there is little information on the sensitivity and specificity of these evaluations. The purpose of this study is to establish the sensitivity and specificity of a VR balance module for detecting lingering balance deficits clinical concussion care.

Design

Retrospective, case-control study

Setting

Institutional research laboratory

Participants

Normal controls (n=94) and concussed participants (n=27)

Interventions

All participants completed a VR balance assessment paradigm. Concussed participants were diagnosed by a Certified Athletic Trainer or physician (with 48 hours post-injury) and tested in the lab between 7-10 days post-injury. ROC curves were performed in order to establish the VR module’s sensitivity and specificity for detecting lingering balance deficits.

Main Outcome Measures

Final balance score

Results

For the VR balance module, a cutoff score of 8.25 was established to maximize sensitivity at 85.7% and specificity at 87.8%.

Conclusions

The VR balance module has high sensitivity and specificity for detecting sub-acute balance deficits after concussive injury.

Introduction

Currently, the clinical “gold standard” in the evaluation of patients recovering from concussive injury is the clinical evaluation, which is supported by a battery of tests including computerized neuropsychological evaluations, clinical balance assessments, and patient reported symptom checklists.^3,14 The Balance Error Scoring System (BESS) is a widely used clinical postural control test that evaluates an athlete’s ability to maintain balance with eyes closed and hands on hips in a two-footed, one-footed, and tandem position on solid and foam surfaces.¹² More dynamic than the BESS, the Sensory Organization Test (SOT) is a commonly used tool that consists of six conditions, which provides insight into an athlete’s ability to process and integrate sensory and visual information.⁴

Continuing improvements in technology and affordability has opened up the door to incorporating virtual reality (VR) testing into clinical concussion care. Compared to more traditional tests, the benefits of the VR environment includes the 3-D nature of the tests, the ability to assess depth perception, increases in the subject’s sense of presence within the virtual environment, and the transferability to real-life situations.¹⁹ Several clinical studies have found that VR assessments are sensitive to concussive deficits^19-21, risk of falling in the elderly subjects⁷, and balance deficits in stroke and/or cerebral palsy patients.^16,17

Balance deficits have been found after concussion, with deficits typically resolving 3-5 days post-injury.^11,15 It has been hypothesized that balance dysfunction following concussion is due to the brain having difficulties integrating vestibular, visual, and somatosensory information.⁶ VR technology has been well-documented to induce egomotion, or actual motion in response to optic flow, and vection, or illusionary thoughts of self-motion due to the moving environment. As vision is a critical component of postural control, VR paradigms that utilize egomotion and vection may be able to provoke and identify balance deficits following concussion.

While the benefits of VR environments are fairly universal, there are a large number of VR based platforms. The VR platform used in this study is designed to imitate a health-care provider’s office, to create a plausible environment for the participant to be completing post-injury testing. While this VR technology has been around for several years and offers several benefits, the inconsistency between the various platforms is currently a limitation.

In order for VR technology to become part of clinical concussion assessment and management, VR paradigms must be shown to adequately distinguish concussed patients from healthy controls. Therefore, the primary purpose of this study is to examine the specificity and sensitivity of a VR based balance assessment (VR balance module) to detect sub-acute balance deficits.

Methods

Data were retrospectively gathered on 94 normal controls and 27 concussed participants. Within 48 hours, a certified athletic trainer and team physician diagnosed concussions based on the results of a clinical evaluation, symptom checklist, neuropsychological testing, and clinical balance assessment. Concussed participants were tested on the VR balance module between 7-10 days post-injury. All participants, regardless of group, were excluded if there were any known neurologic disorder, lower extremity injury affecting balance, or ADD/ADHD. This study followed the ethical guidelines put in place by the Pennsylvania State University, whose Institutional Review Board approved this protocol prior to testing. All participants signed an informed consent form before testing began.

A VisMini by Vizbox Ultra Portable Passive Stereo (Saint Joseph, Illinois) 3D projection system, which makes use of Infitec stereo (Mainz, Germany), allowed flicker-free stereo. InterSense’s (Billerica, Massachuesetts) patented inertial-ultrasonic hybrid tracking technology (IS-900 PCT tracker system) offered real-time tracking of position and orientation in Yaw, Pitch, and Roll directions. The sensor was located on the subject’s head to interact with the visual field motion induced by VR moving room paradigm (see Slobounov et al. 2011). A 83” × 144” projection screen was used to display the VR animations. The software was developed and provided by HeadRehab, LLC (Chicago, Illinois).

Before testing began, each participant was given liquid crystal shutter glasses to separate the field sequential stereo images into right and left eye images and secured in a harness to prevent injury in case of loss of balance. Each participant stood in the Romberg position (one foot directly in front of the other, hands on hips) and was asked to remain as still as possible as the virtual room he/she was viewing swayed in one of three directions for 30 seconds (Figure 1).

Figure 1.

Example of the VR set-up used during data collection. Participant is harnessed and viewing the VR environment (health-care office) during the stationary balance condition.

During the first trial, the virtual room remained completely still. During the subsequent nine trials, the room rotated exclusively in one of three planes: yaw (rotation about the vertical (z) axis between 10-30 degrees at 0.2Hz), pitch (rotation about the interaural (x) axis between 10-30 degrees at 0.2Hz), or roll (rotation about the y-axis between 10-30 degrees at 0.2Hz). The final balance score, a composite score generated from the combination of all ten balance trials, was used as an outcome measure in this study. The final balance score is automatically generated by the VR software used in this study and is determined by the amount of head deviation (in square centimeters) of each participant during each trial. Each of the ten trials contributes equally to the final balance score, which is an averaged score that ranges from 0 (worst) to 10 (best).

All outcome measures were analyzed using IBM SPSS Version 19.0 (Armonk, New York). A receiver operating characteristic (ROC) curve was run for the final balance score to determine which cutoff point maximized the sensitivity and specificity of the VR balance module. A priori alpha level was set at 0.05.

Results

Participants from both groups were all college-aged (18-24 years), Division-I varsity athletes participating in football, ice hockey, or soccer (football) or club rugby. The control group (non-concussed athletes at time of testing) completed the VR balance module one time during their athletic career. All of the athletes in the concussed group were tested 7-10 days after their concussion and were cleared to begin the return-to-play protocol by their team physician at the time of testing. In order to begin the return-to-play protocol, all concussed athletes must have been asymptomatic and passed clinical neuropsychological and balance testing prior to completing the VR balance module.

Due to the non-normal distribution (positively skewed), data were transformed using the natural log for the statistical analysis. Independent samples t-test were run between the two groups. After statistical analysis, data were retransformed in their original metric and are presented in this metric throughout. There were no differences between the control and concussed group (p=0.067; control: mean=8.58, 95% CI: 8.17-8.99; concussed: mean=7.87, 95% CI: 7.62-8.13).

For the VR balance module, a cutoff score of 8.25 was determined to maximize sensitivity and specificity. At this score, the VR balance module was found to have a specificity of 85.7% and a specificity of 87.8%. The AUC was .862 (95% CI; .767-.958). A table detailing sensitivity and specificity at different cut off scores (Table), as well as the ROC curve for the data (Figure 2), is shown below.

Table.

Cutoff scores and the given sensitivity and specificity for the VR balance module.

Cutoff Score	Sensitivity	Specificity
9.25	96.3	11.6
9.00	88.9	33.7
8.75	85.7	62.1
8.50	85.7	78.9
8.25	85.7	87.8
8.00	74.1	90.5
7.75	37.0	94.7
7.50	25.9	98.9
7.25	14.8	98.9

Figure 2.

ROC curve established for the VR balance module. The blue line represents the trade off between sensitivity and specificity at given cut-off values. The further the blue line is to the left and above the green line (45-degree diagonal through the ROC space), the better the diagnostic value of the test.

The positive predictive value of the VR balance module was 65.7%, while the negative predictive value was 97.7%. The likelihood ratio was given as 18.28 and odds ratio was listed at 0.24 (95% CI: 0.11-0.52).

Discussion

The primary purpose of this study was to determine the sensitivity and specificity of a VR balance module in detecting lingering balance deficits in order to determine if the paradigm meets the current standard for use in clinical care. This was achieved by having concussed and control participants complete a VR balance module designed for use in concussion assessment and management. A ROC curve was run to establish cutoff scores and determine the sensitivity and specificity of the VR balance module. For the VR balance module, the AUC was found to be .862 (95% CI; .767-.958), where a perfect diagnostic test would have an AUC of 1.¹³ A cutoff score of 8.25 was determined to maximize the combined sensitivity and specificity of the VR balance module to detecting sub-acute balance deficits (85.7% sensitivity and 87.8% specificity).

The BESS and the SOT are two commonly used postural assessment tools in concussion assessment and management. In a study by Furman et al.⁵ the overall BESS score was found to have an AUC of 0.74 (95% CI, 0.53-0.94) when differentiating between healthy participants and concussed individuals tested approximately 8 days post-injury. Furman et al. established a cutoff score of 21 for the BESS, maximizing the sensitivity at 60% and specificity at 82%. ROC curves were run for each of individual BESS conditions as well, with the most sensitive conditions being the tandem stance on a foam surface (AUC, 0.80; 95% CI, 0.66-0.95; P < .01). While this study shows the BESS is capable of detecting concussive deficits 8 days post-injury, other studies indicate that BESS scores return to baseline 3-5 days after concussive injury.^11,15 Although, to our knowledge, no other studies completed ROC curves on the BESS, other studies have evaluated the BESS for sensitivity and specificity. A 2005 study by McCrea et al. found that sensitivity and specificity of the BESS was maximized at the time of injury (34% and 91% respectively).¹⁰

Two studies by Broglio et al.^1,2 have examined the sensitivity and specificity of the SOT. In a study of 129 participants (63 concussed, tested within 24hr of injury) using reliable change (RC) scores, the SOT was found to maximize a combined sensitivity and specificity at a RC of 1.38 at the 75% CI.¹ At this cutoff, overall sensitivity was 57% and specificity was 80%. An earlier study by Broglio et al.,² using the same population as the aforementioned study, looked at the overall sensitivity of the SOT. However, instead of using RC scores, Broglio et al. used changes of more than one standard deviation from baseline scores as a clinically meaningful finding. When using these criteria, the SOT had a sensitivity of 61.9%.

Compared to the BESS and the SOT, the VR balance paradigm has better overall sensitivity and specificity. Particularly, when looking at the sensitivity of the overall balance assessment instead of individual components, the VR paradigm was capable of discriminating 85.7% of concussed participants compared to 60% in BESS and 61.9% of SOT. Although the VR balance paradigm does not represent a perfect clinical tool, it exceeds the current standard of sensitivity and specificity set by the BESS and SOT. This is not to suggest that the BESS and SOT are poor tools or should be replaced in clinical care. Instead, the authors are suggesting that the VR balance module may be more sensitive to on-going balance deficits and that the VR paradigm meets the sensitive and specificity standards needed to be implemented in clinical care.

Along with sensitivity and specificity, it is important to consider other psychometric properties of diagnostic tools before they can be included in the clinical concussion battery. The VR balance module used in this study has previously been shown as a valid postural stability assessment tool.²² In two studies using D-1 college football players, it was shown that there are no differences between VR final balance scores over three separate testing sessions as well as scores before and following a full practice.¹⁸ More formal reliability statistics, such as intraclass correlation coefficients, have yet to be established for this technology. This current lack of more formal reliability statistics makes it difficult to compare the serial nature of VR balance testing to other modalities such as the BESS and SOT.

When making comparisons between the VR battery and other testing paradigms, it is important to highlight the timelines after injury used in these studies. Concussion symptoms and deficits change fairly rapidly after injury, so it important to consider how the amount of time after injury may affect outcome variables. The participants completing the VR testing battery were tested between 7-10 days post-injury. The only other study using a similar timeline was the Furman et al.⁵ study, which tested participants approximately 8 days post-injury. For all other studies included in this discussion, participants were tested within 72hrs post-injury. Most studies show that balance deficits resolve between 3-5 days post-injury^11,15 and neuropsychological deficits resolve within 7 days post-injury.^8,9 Therefore, the fact that VR technology showed similar or better levels of sensitivity and specificity than other tools at 7-10 days post-concussive injury may indicate that VR paradigms are capable of detecting residual deficits of concussion missed by other clinical tools. Therefore, when returning athletes to play, clinicians should be aware that athletes might be continuing to experience lingering balance deficits not readily detectable by common clinical tools.

While this study indicates that VR programs have the potential to be useful clinical tools in concussion diagnosis, VR paradigms do not come without their limitations. The hardware and software needed to run a VR system is costly. With the creation of 3-D televisions, the cost has been greatly reduced from previous systems. However, the initial cost of setting up a VR environment will be a considerable investment. Another limitation of VR environments is their mobility. Generally, VR systems are stationary and take a great deal of effort and expertise to relocate. Portable display screens and 3-D head mounting display systems significantly increase the ease of movement and can even allow for sideline evaluations, but these devices are again costly. Lastly, VR systems are very technically advanced. Typically, they require a great deal of expertise for installation, which may require outsourcing for initial setup. While these are serious limitations to using VR systems, the potential benefits, in terms of diagnosis and rehabilitation, should not be ignored.

Conclusion

There is no perfect clinical tool for concussion assessment and management and VR testing is no exception to that. Current tools, such as the BESS and SOT, represent solid assessment modules that have clinically stood the test of time. However, researchers need to continue to push for better tools in order to protect concussed individuals from long-term damage due to misdiagnosis or returning to play too early. Advancing technology has opened the door for VR technology to become part of clinical concussion injury testing. While future research will be needed to continuously evaluate the appropriateness of VR technology in clinical settings, this study provides support for the implementation of a VR balance module into clinical concussion care. The high sensitivity and specificity of the VR balance module, which exceeds the minimum standards set by current clinical tools, indicate the appropriateness of VR as a testing tool and may provide a new and improved way to assess individuals after a suspected concussive injury.

Clinical Relevance.

The VR balance has a high sub-acute sensitivity and specificity as a stand-alone balance assessment tool and may detect on-going balance deficits not readily detectable by the BESS or SOT. VR balance modules may be a beneficial addition to the current clinical concussion diagnostic battery.