TEST-RETEST RELIABILITY OF THE LIMITS OF STABILITY TEST PERFORMED BY YOUNG ADULTS USING NEUROCOM® VSR SPORT

Abstract

Background

A reliable measure of dynamic postural control is needed for inclusion in the sports-related concussion assessment battery. Currently, there is not a clinical gold standard. The Limits of Stability (LOS) test has potential to be a useful tool to collect objective data on important dynamic postural stability variables. Psychometric properties of the LOS test with healthy young adults are yet to be established.

Hypothesis/Purpose

The purpose of this study was to examine the intra-session and test-retest reliability for the LOS on the NeuroCom® VSR Sport when performed by young adults.

Study Design

Reliability study

Methods

Twenty-seven healthy university students completed four trials of the LOS in each of two testing sessions one week apart. Relative reliability was measured within each session with an intraclass correlation coefficient (ICC[3,k]) for Session 1 and Session 2, respectively, on each of the five dependent variables (movement velocity [MVL], directional control [DCL], maximum excursion [MXE], endpoint excursion [EPE], and reaction time [RT]) provided by the Neurocom. Test-retest reliability was assessed using a repeated-measures analysis of variance along with an ICC (3,k) for relative reliability. An ICC value of 0.90 or higher was defined as having a high reliability, moderate reliability for ICC values between 0.80-0.89, and below 0.80 as questionable.

Results

The reliability within each session for LOS composite scores for MVL, DCL, and MXE was moderate to high (ICC[3,k]=0.89-0.95). These same three variables also had high levels of test-retest reliability (ICC[3,k]=0.95-0.96). EPE and RT had moderate reliability over time (ICC[3,k]=0.88) but differences for within session reliability.

Conclusions

LOS provides a reliable measure of dynamic postural control for young adults. Two trials are recommended at baseline with the first being an adaptation trial to ensure accuracy of findings. Care needs to be taken when interpreting EPE and DCL scores on post-injury tests due to a learning effect for those variables.

Level of Evidence

2c

INTRODUCTION

Postural control is used as part of an assessment battery for diagnosing sports-related concussions (SRC) and making return-to-play decisions for college athletes.^1,2 The most commonly used tools assess postural steadiness, such as the ability to stand as still as possible under different conditions.^2,3 Tests such as the Balance Error Scoring System (BESS) are clinically accessible, easy, and inexpensive to perform.^4,5 However, reliability scores of the BESS have been found to be less than optimal and this test used in isolation may be insufficient for testing the postural control skills needed by athletes.^6-6 Athletic participation requires complex dynamic postural control and rapid response to changing field conditions putting demands on visual and perceptual motor abilities.¹¹ Tests of postural steadiness are found to return to baseline performance levels within three to seven days following SRCs;^3,12 whereas deficits have been found in measures of dynamic postural control for a longer period of time, even after clearance for return to sport participation.^9,13 It appears that tests of postural steadiness are not sensitive enough to identify subtle deficits^8,14,15 and to follow changes throughout the concussion recovery time period.^14,15 Further, performance on static and dynamic measures have been found to vary independently of each other suggesting that performance on these balance tests are not related.^15,16 In summary, normal performance on a test of postural steadiness may not indicate that a person is prepared to accurately respond to dynamic demands on the playing field.

Currently, no gold standard exists for assessment of dynamic postural control for the young adult population, especially during on-field examination where sophisticated equipment may be unavailable or cumbersome. Gait studies have provided evidence of impaired dynamic postural control processes following SRCs, but the specialized equipment required for those studies limits clinical applicability.^9,14,17 A need exists for a clinically accessible test of postural control that produces consistent, objective information about dynamic postural control variables that can also identify prolonged impairments. The Limits of Stability Test (LOS) performed on the NeuroCom® VSR Sport allows quantifiable variables of dynamic stability to be obtained in a clinically accessible manner due to its portability. Test-retest reliability and the practice effect need to be established as first steps in determining the usefulness of the LOS as part of a SRC management program for college athletes. Limits of stability testing has been studied using a variety of testing techniques resulting in differing reliability scores in healthy adolescents and young adults.^15,18-20 To be clinically useful, an assessment must be able to establish a baseline within a few trials due to time constraints when performing pre-season testing with college athletes. Test-retest reliability of all five of the LOS variables was found to be moderate to high when performed on the NeuroCom® VSR Sport by healthy adolescents.¹⁵ However, this testing protocol had subjects perform only two trials so it is unknown if a true baseline was reached.¹⁵

Establishment of a true baseline (prior to injury) is important for tests included in a pre-season and post-injury SRC management program to determine if change in performance can be attributed to injury rather than to test inconsistency. Further, test psychometrics need to be established for the population of intended use. Therefore, the purpose of this study was to examine the intra-session and test-retest reliability for the LOS on the NeuroCom® VSR Sport when performed by young adults.

METHODS

Participants

Twenty-eight healthy university students (7 men, 21 female; age: 24.4 + 1.6 years; height: 170.2 + 9.7 cm) volunteered to participate and completed informed consent approved by the college Institutional Review Board. Participants were recruited by an oral introduction and informed consent documents were left for interested individuals to complete. Participants had to be between 18 and 30 years old, not currently engaged in college sponsored athletics and verbally attest to having no injury, illness or condition that impaired balance or ability to see the computer screen. All participants completed the first testing session, but one was unable to schedule follow-up testing within the seven to ten day time period; therefore 27 participants completed the study. Sample size was estimated at 24 participants with a type I error of a 2-sided test set at 0.05, 80% statistical power, and an effect size of 0.25 for the test-retest analysis.

Measures

The LOS test required participants to stand in a designated foot position, about hip width apart, on a fixed force plate. The computer screen, placed at eye level in front of the participant, depicted a center box with eight target boxes equally spaced in an elliptical arrangement around the center at the individual's computer-generated limit of stability based on height. Participants were instructed to shift their weight so that the projection of their COP on the computer screen, indicated by an icon, was in the center box. All trials started with holding steady in the center (Figure 1). Participants were instructed, “When you see the circle jump to the target and hear the tone, shift your weight to move the icon as fast and accurately as you can to the target and hold steady until you hear a second tone.” Participants were free to use whatever movement strategy they chose as long as they did not lift or move their feet. All trials began with shifting to the target in the 12:00 (forward) position and moved sequentially in a clockwise direction.

Figure 1.

Limits of Stability Test. The test begins in the center square. The first target is in the forward position and the test proceeds in a clockwise direction.

Performance on five programmed variables set by the NeuroCom® VSR Sport was recorded for each trial in each of the eight directions: reaction time (RT), movement velocity (MVL), directional control (DCL), maximum excursion (MXE) and endpoint excursion (EPE) (Table 1). Reaction time is used as a measure of cognitive efficiency.^21,22 For the LOS, the reaction time is recorded as the time from the cue to the time the COP sway exceeds the random range indicating volitional movement has begun.²³ Movement velocity is the average speed of the center of gravity (COG) shift toward the target measured in degrees per second. Directional control is the amount of movement in the intended direction minus the amount of off-axis movement given as a percentage. Endpoint excursion is the distance traveled by the COG on the primary attempt to reach the target expressed as a percentage of the LOS. Endpoint excursion is considered to be a measure of an individual's confidence in approaching their LOS.²³ Maximum excursion is the furthest distance traveled by the COG in a given trial. Composite scores were generated for each of the five variables by averaging the performance across all eight targets within a trial. Composite scores were found to demonstrate better reliability than individual target scores when the LOS was performed by high school students¹⁵and aligns most closely with the current sample, according to the available literature.

Table 1.

Definitions of measurements from the NeuroCom® VSR Sport.

Measure	Definition
Reaction Time	Time from the cue to the time the COP sway exceeds the random range indicating volitional movement has begun
Movement Velocity	Average speed of the COG shift toward the target measured in degrees per second
Endpoint Excursion	Distance traveled by the COG on the primary attempt to reach the target expressed as a percentage of the LOS
Maximum Excursion	Furthest distance traveled by the COG in a given trial
Directional Control	Amount of movement in the intended direction minus the amount of off-axis movement given as a percentage

COP: Center of Pressure; COG: Center of Gravity; LOS: Limits of Stability

Table 2.

Test-Retest Reliability Measures, presented as means +/- SD's.

	Session 1				Session 2				ICC (95% CI)	SEM
	Trial 1	Trial 2	Trial 3	Trial 4	Trial 5	Trial 6	Trial 7	Trial 8
Reaction Time*	0.64 ± 0.10	0.61 ± 0.10	0.61 ± 0.10	0.70 ± 0.18	0.66 ± 0.15	0.67 ± 0.15	0.67 ± 0.16	0.67 ± 0.18	0.88 (0.80, 0.94)	0.04
Movement Velocity**	6.1 ± 1.5	6.7 ± 1.9	6.7 ± 1.9	6.5 ± 2.3	6.4 ± 1.9	6.7 ± 1.8	6.7 ± 1.9	6.6 ± 2.1	0.96 (0.93, 0.98)	0.35
Endpoint Excursion†	85 ± 7	89 ± 5	88 ± 6	90 ± 5	90 ± 7	91 ± 5	91 ± 4	90 ± 7	0.88 (0.80, 0.94)	1.61
Maximum Excursion†	96 ± 5	95 ± 5	95 ± 4	95 ± 5	95 ± 4	96 ± 4	95 ± 4	95 ± 5	0.95 (0.91, 0.97)	0.64
Directional Control‡	80 ± 6	82 ± 7	82 ± 7	84 ± 6	83 ± 5	84 ± 4	85 ± 4	85 ± 6	0.95 (0.91, 0.97)	1.17

^*measured in seconds

^**measured in degrees per second

^†measured as percent of projected 100% LOS

^‡measured as percent of movement in the right direction compared to off target trajectory

Procedures

Participants attended two data collection sessions 7.8 + 0.8 days apart. On both occasions they performed the LOS four times in a row with a two-minute break between trials to prevent fatigue. Although the number of trials may lead to overestimation of reliability,¹⁵ four trials were necessary in each session to examine practice effects.²⁴ Demographic information was collected before the first trial on testing day one. Testing was performed on the NeuroCom® VSR Sport (Natus Medical Incorporated, Pleasanton, CA; Figure 2) with bare feet, and all data were sampled at rate of 100Hz. Participants were given a one-minute warm-up to practice shifting their COP in order to move the cursor icon to the target positions depicted on the computer screen. The force plate was calibrated to manufacturer specifications before each testing session. Both testing sessions took place in the same quiet room and participants were questioned to assure no illness or injury occurred between testing sessions that could impact performance.

Figure 2.

NeuroCom® VSR Sport.

Statistical analysis

Descriptive statistics were used to find means and standard deviations. Normality of data was assessed using the Kolmogorov-Smirnov test and the assumption was met (p > 0.05) for all dependent variables (reaction time, movement velocity, endpoint excursion, directional control, and maximum excursion). The possible presence of systematic error, such as a learning effect, was evaluated using a one-way repeated measure analysis of variance (RM ANOVA) for each of the five different dependent variables with trial as the independent variable (trials 1-4). If the assumption of sphericity was violated, a Greenhouse-Geisser adjustment was implemented. To measure relative reliability within each session, an intraclass correlation coefficient (ICC), specifically an ICC [3,k] model was used.²⁵ The intra-session reliability was calculated separately for Session 1 and Session 2.

To compare differences between sessions, the four trials within each session were averaged together. Then the test-retest reliability was determined using a RM ANOVA for each of the five dependent variables stated above. A 2-way fixed effects model was used to determine the relative reliability (ICC [3,k]). An ICC value of 0.90 or higher was defined as having a high reliability, moderate reliability for ICC values between 0.80-0.89, and below 0.80 as questionable.²⁶ Ninety-five percent confidence intervals (CIs) and standard error of measure (SEM) were determined for each of the ICC values. The SEM, a measure of absolute reliability, was calculated using $\text{SEM}=SD\sqrt{\left(1-IC{C}_{3,k}\right)}$. For all statistical analysis, the Statistical Package for the Social Sciences (IBM SPSS, Inc., 22.0, Chicago, IL) was used with an alpha level of 0.05.

RESULTS

Intra-Session Reliability

More variability was seen in Session 1 compared to Session 2. Differences were seen in reaction time (F_1.92,48=3.52, p = 0.04) for Session 1, specifically between Trial 2 (0.60 ± 0.10) and Trial 4 (0.68 ± 0.17) (P = 0.02) with Trial 4 having a slower time. Endpoint excursion trials were different (F_3,75=4.67, p<.01) for Session 1, only between Trial 1 (86.±7.) and Trial 4 (90.±5.) (p = 0.03). There were also differences between trials (F_2.08,51.94=10.54, p < 0.01) for directional control, specifically between Trial 1 (79.±6.) and Trial 4 (84.±6.) (p < 0.01) and between Trial 2 (81.±7.) and Trial 4 (P < 0.01). However, there were no differences between trials for movement velocity (F_1.66,41.50=1.71, p = 0.20) or maximum excursion (F_3,78=.89, p = 0.45) suggesting no learning effects for these two variables in Session 1. In Session 2, there were no learning effects as indicated by the lack of statistically significant differences (reaction time: F_2.16,56.28=0.13, p = 0.91; movement velocity: F_3,78=0.92, p = 0.44; endpoint excursion: F_3,78=0.89 p = 0.45; and maximum excursion: F_2.25,56.19=1.38, p = 0.26) except for directional control (F_3,78=3.94, p = 0.01) which were different between Trial 5 (83.±6.) and Trial 7 (85.±4.).

Reliability was high for movement velocity in both Session 1 (ICC [3,k]=0.92, 95% CI=0.85-0.96) and Session 2 (ICC [3,k]=0.95, 95% CI=0.92-0.98). This was also true for directional control in both Session 1 (ICC [3,k]=0.92, 95% CI = 0.87-0.96) and Session 2 (ICC [3,k]=0.93, 95% CI = 0.88-0.97). For maximum excursion, the reliability was high in Session 2 (ICC [3,k]=0.94, 95% CI = 0.89-0.97) but moderate for Session 1 (ICC [3,k]=0.89, 95% CI=0.79-0.94). Reliability was questionable in Session 1 for both reaction time (ICC [3,k]=0.62, 95% CI=0.31-0.81) and endpoint excursion (ICC [3,k]=0.77, 95% CI=0.59-0.89). Both of these measures, reaction time (ICC [3,k]=0.88, 95% CI=0.78-0.94) and endpoint excursion (ICC [3,k]=0.87, 95% CI=0.76-0.93) had moderate reliability values in Session 2.

Test-Retest Reliability

There were no significant differences between session averages for reaction time (t₂₅=-1.64, p = 0.11), movement velocity (t₂₅=-0.303, p = 0.765), or maximum excursion (t₂₅=-0.47, p = 0.64). However, there were significant differences between session averages for endpoint excursion (t₂₅=-3.28, p < 0.01) and directional control (t₂₅=-3.12, p < 0.01). The second session average was significantly higher (EPE: 90.29 ± 4.83 and DCL: 83.82 ± 4.56) than the first session (EPE: 87.90 ± 4.51 and DCL: 81.64 ± 5.78). Three of the five variables demonstrated high levels of reliability over time, between Session 1 and Session 2 (movement velocity: ICC [3,k]=0.96, 95% CI=0.93-0.98, SEM=0.35; maximum excursion: ICC [3,k]=0.95, 95% CI=0.91-0.97, SEM=0.64; and directional control: ICC [3,k]=0.95, 95% CI=0.91-0.97, SEM=1.17 as seen in Table 1. Reaction time (ICC [3,k]=0.88, 95% CI=0.80-0.94, SEM=0.04) and endpoint excursion ICC [3,k]=0.88, 95% CI=0.80-0.94, SEM=1.61) had moderate reliability over time.

DISCUSSION

Tests for perceived stability limits have been used as a measure of dynamic postural stability.^15,18,19 Comparison of reported reliability and learning effects for LOS testing is challenged due to variations in testing protocols, testing equipment and metrics reported. Unidirectional leaning tests from a stable platform have been found to demonstrate larger COP excursions with less variability than circling tests or those with a moving platform.^18,19 Thomsen et al. hypothesize that circling is more difficult than leaning in the four cardinal directions (forward, backward, side to side) because circling requires control of movement in two directions at once.¹⁹ LOS testing performed with the NeuroCom® protocol requires the subject to shift center of gravity in eight directions: at a 45 degree diagonal toward each corner as well as in the four cardinal directions. The four corner leans require control in two planes at once, increasing the difficulty of the task and this may potentially provide more challenge than the four-way leaning test.

Reliability

Good test-retest reliability (ICC [3,k]=0.88-0.96), and little practice effect, were found for five variables both within and between testing sessions in this study when the LOS was performed by young adults using the portable NeuroCom® VSR Sport, a stable platform system. Our findings are higher than the reliability reported by Pickerill and Harter (ICC [2,k]=0.69-0.88) when they tested LOS with young adults.¹⁸ Differences in methodology may contribute to the difference in findings for the three variables they reported (DCL, MVL, EPE). Pickerill and Harter¹⁶ had subjects perform the LOS assessments on two different pieces of equipment during both of their testing sessions. The NeuroCom® Smart Balance Master® utilized a short, stable force plate and visual surround whereas the Biodex® DLOS test had subjects standing on an unstable platform.¹⁸ Different postural control demands are created under the two different conditions potentially causing interference in test performance and reducing reliability for both testing protocols. It is also possible that four trials in each of the testing sessions may have led to overestimation of reliability. However, the current findings are similar to others^15,18 who used fewer trials when testing with the NeuroCom® VSR Sport.

Alsalaheen et al.¹⁵ had adolescent subjects perform one practice trial and one testing trial in each of two sessions, one week apart. They reported test-retest reliability as ICC[2,1]=0.81-0.96 for RT, MXE, MVL and EPE.¹⁵ They also had subjects perform assessments other than LOS during the testing sessions, but in their study the other assessments examined postural steadiness.¹⁵ It seems the steadiness tests did not interfere with consistency of LOS performance since their reliability scores are quite good. Both the Pickerill and Alsalaheen studies reported the lowest reliability (0.69 and 0.73, respectively) and the widest confidence interval for DCL.^15,18 This conflicts with the present findings in which DCL demonstrated excellent reliability (0.95) and a narrow confidence interval (95% CI=0.91-0.97) indicating good test precision. The subjects in this study were allowed to move however they naturally would, so long as they kept feet down in place on the force plate. In the other studies subjects were constrained to keep their arms crossed on their chest,^19,20 palms on their thighs,¹⁵ and/or their body rigid without flexing knees or hips.¹⁸ Constraining individual movement strategy may impact consistency of performance and therefore lower reliability findings for directional control. There was a learning effect for DCL when scores were examined for differences within and between testing sessions. Learning effects were not addressed in the previous studies, but could contribute to the lower DCL reliability scores reported by other authors.^15,18

Learning effects

Four trials in each testing session were necessary to examine practice effect in this study. Three variables, RT, MVL, and MXE, reached baseline by the second trial. Two variables, EPE and DCL, demonstrated a small practice effect beyond the second trial. Improvement in EPE in the first testing session occurred primarily between Trials 1 and 2 with no significant difference seen between consecutive trials in either testing session after the second trial. However, when scores for all four trials in each testing session were averaged, scores were slightly higher in the second testing session for both EPE and DCL. Directional control scores persisted with small improvements in consecutive within session trials indicating that a clear baseline was never established. Healthy young adults continue to show small improvements in EPE and DCL, so any decrement in post-testing performance may be considered an abnormal response. Directional control could be clinically important in the prevention of further injuries, for example, in soccer, when two players go up to head the ball at the same time, inaccuracy in directional control, of even a few degrees, could cause a player to connect with another player's head instead of with the ball. Therefore, careful consideration must be taken by clinicians in interpretation of post-test scores for directional control.

Clinicians and researchers provide acclimation time and practice trials to mitigate the impact of learning in an attempt to obtain true ability scores when testing. The number of trials suggested when performing tests of stability limits varies from two to eight depending on testing methods.^15,18,19 Thomsen et al. suggest performing eight trials when using the four-way leaning test, since they found a learning effect up to the eighth trial.¹⁹ The high number of trials reduces clinical usefulness due to time factors, especially since the only metric obtained is excursion distance. Both Alsalaheen et al. and Pickerill and Harter provided three to five minutes of acclimation and had subjects perform two trials of the LOS using the NeuroCom® protocol.^15,18 The current findings support the use of two trials, using the first as practice, unless a firm baseline is required for directional control. One-minute acclimation time appears to be sufficient for young adults according to the good reliability estimates found in this work, or better than studies providing longer times.

Limitations and Future Directions

Although this study is an important step in determining if the LOS test is reliable, the authors concede that only healthy volunteers were used as the purpose was to determine reliability for baseline concussion testing. Future studies are needed to determine the reliability across a sport season and to determine baseline testing modifiers, such as lower extremity injury, fatigue, or testing session, that could alter results and cause invalid scores. Determining the sensitivity and specificity to identify concussion injury and to monitor recovery is a vital next step in determining the clinical usefulness of LOS testing using the NeuroCom® VSR Sport. Future studies should aim to measure these metrics as well as others such as odds-likelihood ratios. Finally, the authors caution against broad generalizations in the direct application of the data as they were collected from a small, convenience sample of college aged individuals.

CONCLUSIONS

Composite scores from all five variables obtained with LOS testing performed by young adults on the NeuroCom® VSR Sport show moderate to high test-retest reliability. Two trials are recommended to establish a baseline for MXE, RT and MVL. Caution is advised in interpreting EPE and DCL results since a practice effect was found for those variables. The LOS on the NeuroCom® VSR Sport is a reliable test of dynamic postural stability for young adults which may offer more challenge than tests of steadiness.