Assessment of wheelchair driving performance in a virtual reality-based simulator

Abstract

Objective

To develop a virtual reality (VR)-based simulator that can assist clinicians in performing standardized wheelchair driving assessments.

Design

A completely within-subjects repeated measures design.

Methods

Participants drove their wheelchairs along a virtual driving circuit modeled after the Power Mobility Road Test (PMRT) and in a hallway with decreasing width. The virtual simulator was displayed on computer screen and VR screens and participants interacted with it using a set of instrumented rollers and a wheelchair joystick. Driving performances of participants were estimated and compared using quantitative metrics from the simulator. Qualitative ratings from two experienced clinicians were used to estimate intra- and inter-rater reliability.

Results

Ten regular wheelchair users (seven men, three women; mean age ± SD, 39.5 ± 15.39 years) participated. The virtual PMRT scores from the two clinicians show high inter-rater reliability (78–90%) and high intra-rater reliability (71–90%) for all test conditions. More research is required to explore user preferences and effectiveness of the two control methods (rollers and mathematical model) and the display screens.

Conclusions

The virtual driving simulator seems to be a promising tool for wheelchair driving assessment that clinicians can use to supplement their real-world evaluations.

Background

Virtual environments (VEs), in the form of computer games, are traditionally used for entertainment purposes. With their increase in popularity and better affordability, gaming consoles and other consumer electronics are becoming more commonplace additions to a typical living room. Significant advancements in graphics and gaming technologies have made VEs an ideal platform for training, collaboration, and information exchange. Moreover, since the skills learned in the VEs seem to transfer well into real-world activities,^1,2 researchers are increasingly using VEs for therapy and rehabilitation. Researchers have developed and tested simulators for the purposes of wheelchair driving training.^3–6 Virtual driving performances in these simulators show high correlations with performance on real-world driving tasks.^3,5,7 Most of the driving simulators developed by researchers so far are designed as training environments to expose inexperienced wheelchair drivers to unfamiliar driving scenarios. However, no validated wheelchair driving assessment tools exist in the VE. There are also no recommended virtual driving performance metrics that have shown clinical relevance in accessing real-world driving. A long-term goal of our research is to build a valid and reliable driving performance assessment tool in a VE. This paper reports the development and preliminary evaluation of performance measures using a virtual reality (VR)-based simulator (VRSIM) developed for wheelchair driving. VRSIM can also be used as a training tool that can assist clinicians in exposing their clients to the various features of power-wheelchair driving and to various driving scenarios before assessing them in a real wheelchair.

Our research group has developed and evaluated simulators specifically for people with traumatic brain injury,^8,9 multiple sclerosis,¹⁰ and cerebral palsy.¹¹ VRSIM is designed as a customizable platform that could be adapted to the unique needs of these and other populations such as people with spinal cord injury who may be candidates for power wheelchair mobility. VRSIM provides capabilities for clinicians to choose displays and control methods that individuals can use to interact with the VE. Immersive virtual reality environments (IVRE) that are used with car driving simulators have more than 160° of view frustum and provide significant immersion and a better sense of presence in the virtual world. However, these displays are expensive, need significant physical space to setup, and require computers with powerful graphical processors. Computer display screens and computer-based virtual environments (CVEs) are a cost-effective alternative. The CVEs provide a limited field of view (FOV) but they can be rendered on conventional desktop or laptop computers. Few studies have compared the relative merits and task performances of users while using display systems¹² with varying sizes. In this research study, we therefore aimed to compare the driving performances of wheelchair users using CVE and IVRE and gather feedback from their subjective preferences. Our first hypothesis was that subjects would show better driving performance in the IVRE than in the CVE.

In addition to the simulation graphics of the VE, user interaction with elements in the VE helps to improve their sense of presence. Users can interact with VRSIM using a custom-designed roller system or with a customized position sensing wheelchair joystick similar to a standard gaming joystick. Driving on the rollers gives users vibro-tactile feedback equivalent to driving on plain ground. We expect that this feedback added to the visual stimulus from the VEs will increase the realism in users’ virtual interactions. The customized joystick on the other hand uses a mathematical model to simulate virtual wheelchair's kinematics and provides limited additional feedback. Our second hypothesis was that, compared to the driving trials using a customized joystick and mathematical model, trials using a stationary roller system would show better driving performance metrics.

Finally, we will determine inter- and intra-rater reliability of a virtual version of a real-world Power Mobility Road Test (PMRT)¹³ outcome measurement tool, which we will simulate in the VE.

Methods

Subject recruitment

The protocol for this research study was approved by the Institutional Review Boards of the Veteran Affairs Pittsburgh Healthcare System and the University of Pittsburgh. Subjects were recruited by posting flyers at local rehabilitation clinics and hospitals. Subjects between the ages of 18 and 80 years, who used a power wheelchair (with standard proportional joystick) or an attendant-propelled manual wheelchair for all or part of their mobility, and those who had basic cognitive, visual, and motor skills to interact with an interface were invited to participate in this study. The subjects were screened based on having sufficient short-term memory to recall that interaction with the joystick produces results on the computer screen, ability to perceive the moving simulated wheelchair on the display screens, and ability to tap or hit the joystick (exert approximately 2N of force on the joystick, which is the typical amount of force it requires for operation¹⁴ and which will result in movement of the simulated wheelchair on the computer screen).

Experimental setup

Participants were seated in their own power wheelchairs at all times during this protocol. They were asked to park their wheelchairs on a 33′ × 33′ × 6′ roller platform (Fig. 1). Two sets of dual rollers were instrumented in the roller platform such that each of them interfaced with one drive wheel of the wheelchair. Four security straps of the Q'Straint 4 Point Securement System¹⁵ were mounted on corners of the roller platform to tie down the wheelchair to the rollers. Incremental encoders mounted in the rollers read the wheelchair wheel rotations. Analog voltages from a customized movement sensing joystick, similar to the participant's wheelchair joystick, were read into a computer through a National Instruments Data Acquisition card 6024E. The simulation was projected on a single generic 22′ widescreen LCD monitor that gave a 90° FOV and on three 6′ × 8′ back projected screens that gave more than 160° FOV into the VE from the user's perspective. Both the screens showed graphics at the resolution of 1024 × 768 (Fig. 2).

Figure 1.

Platform with rollers and tie down straps.

Figure 2.

First person viewpoint on the three VR screens.

Virtual power mobility road test

Clinicians rely on their clinical judgments during their real-world assessments of their client's wheelchair driving capabilities. However, few subjective driving assessment tools have shown good intra- and inter-rater reliability and validity.^13,16–18 The PMRT¹³ has shown good reliability in assessing real-world wheelchair driving performance of clients who may be candidates a power wheelchair. Massengale et al.¹³ found that clients who had average total PMRT scores ≥95% showed better scores on motor co-ordination tasks, had better near and far visual acuity and FOV, were less likely to bump into obstacles, and completed the real-world PMRT in less time thus indicating, in general, a better wheelchair driving performance.

The real-world PMRT consists of 12 structured tasks with static obstacles and 4 unstructured tasks with moving/dynamic obstacles. The virtual power mobility road test (VPMRT) modeled in the simulation was based on a real-world testing scenario of the rehabilitation clinic where the real-world PMRT was developed and validated by Massengale et al.¹³ The wheelchair driver was expected to maneuver through a course without hitting the static and dynamic obstacles. The tasks subjects were expected to complete were defined by sequentially displayed preset milestones marked with arrow pointers. To guide subjects, the arrow pointers were placed to indicate the ideally expected trajectory for that specific driving task. Computer-generated audio instructions were played when required by the VPMRT task. Subjects were instructed to complete the VPMRT driving course as quickly and accurately as possible without hitting walls or furniture.

Similar to the real-world PMRT,¹³ the VPMRT was scored using a 4-point scale. The possible scores were: 1 (unable to complete task), 2 (completes task hesitantly, requires several tries, requires speed restriction, bumps objects lightly without causing harm), 3 (bumps objects and people in a way that could cause harm to driver or other persons or objects), and 4 (completely independent in completing task, with optimal performance and able to perform task smoothly and in one attempt). Composite scores were derived for the structured and unstructured tasks and a total score was derived from each clinician's ratings on all tasks. After completing the 16 tasks of the VPMRT, subjects were asked to drive along a long hallway. The width of the hallway decreased progressively (in steps of 1.524, 1.372, 1.067, 0.914 m) in order to simulate an increase in cognitive load while driving. The hallway with decreasing widths is not part of the real-world PMRT validated my Massengale et al.¹³ This task was added to the driving course in order to test whether subjects could navigate hallway widths recommended by the Americans with Disabilities Act.¹⁹ The anticipation was that the hallways would help isolate certain risky driving behaviors such as impulsive driving and other driving deficits that distinguish drivers with experience from new or borderline safe wheelchair drivers. Scores from the decreasing hallways section were not used to compute the cumulative VPMRT scores.

The ‘actor’ in the VE was a person sitting in a power wheelchair (seat width = 0.671 m, seat length (depth) = 0.701 m) that was controlled by the user's inputs through the joystick or rollers. The user saw the VE from a ‘first person’ point of view.²⁰ The VE also showed animation of a virtual joystick that mimicked the user's physical inputs (tilt of the joystick post). To give a sense of wheelchair boundaries, a red wire frame box was placed around the complete footprint of the virtual wheelchair. This box also aided in detection of collisions of the chair with other VE components. A short beep sounded to indicate collision with obstacles. After a collision, the virtual chair was slightly bounced back to facilitate maneuvering of the chair away from the obstacle. Subjects had a limited about of time to move the virtual wheelchair away from the obstacle depending on their speed before impact. If the user did not move the chair soon enough after the collision and needed the clinician's instructions/assistance to do so, the program terminated the driving trial. This feature allowed users to anticipate and avoid accidents/collisions and react promptly to move the chair away from the collision site.

Research protocol

Subjects performed driving trials for four test conditions: combinations of the two display screens (personal computer (PC) and VR screens) and two driving modes (Rollers ON and OFF) (see Table 1). During the ‘Rollers ON’ mode, subjects used their own joystick to drive their wheelchair on the rollers which also moved the virtual wheelchair. Encoder readings from each wheel were used by the simulation program to determine the wheelchair's instantaneous linear and rotational speeds. During the ‘Rollers OFF’ mode, subjects used the customized joystick, and the simulation program applied a mathematical model to estimate the virtual wheelchair's linear and rotational speeds. Subjects performed one to two practice trials to familiarize themselves with the experimental setup and driving in the VE. A balanced randomization scheme was used to set the sequence of the four test conditions. Up to three repetitions were performed for each of the test conditions, resulting in a maximum of 12 driving trials per subject. Subjects were allowed to take rest breaks for a few minutes if they felt any fatigue. Subjects self-selected their acceleration of the virtual wheelchair during the practice trials, and this value was kept unchanged during the rest of the experiment. Two clinicians experienced in real-world wheelchair driving evaluations independently assessed the driving performance of participants during every virtual driving trial. Clinician 1, an occupational therapist, had more than 9 years of experience and Clinician 2, a physiatrist, had more than 6 years of experience in power mobility evaluations. After finishing the research protocol, subjects were asked informal questions about their subjective feedback on the VEs. Subjects were specifically asked to describe their preference for PC and VR screens and the ‘Rollers ON and OFF’ modes. Subjects were asked whether the virtual driving was comparable to real-world driving and for their general comments and suggestions for improvement in VRSIM.

Table 1.

Four experiment test conditions

Test condition	Driving mode	Display	Inputs to VE
1	Rollers OFF	PC (CVE)	Customized joystick + Math Model
2	Rollers OFF	VR (IVRE)	Customized joystick + Math Model
3	Rollers ON	PC (CVE)	Encoders on Rollers
4	Rollers ON	VR (IVRE)	Encoders on Rollers

Data preprocessing and statistical analyses

The VPMRT software predetermined an ideally expected trajectory for the virtual wheelchair along the complete driving course. The ideal trajectory for any section of the driving circuit was defined as the trajectory equidistant from the objects (walls or furniture) lining the path. For example, for a 6-ft wide hallway, the ideal trajectory is the line that is 3 ft from the walls lining the hallway. For cases where the driving task involved a turn, ideal trajectory was defined as a circular arc between the midpoints of the preceding and following hallways. A root mean squared deviation (RMSD) of the virtual chair from the ideal trajectory was recorded for all task components. Also, during every driving trial, the simulation program recorded joystick voltages and encoder inputs, actual and model predicted wheelchair speeds, virtual wheelchair position and orientation coordinates, and collisions with static and moving obstacles.

Raw data from the driving trials were processed to derive certain performance metrics for driving performance. These metrics are derived from their equivalents in computer access research²¹ and have been used in past research to evaluate wheelchair^8,9,11,22 and car driving²³ in VEs. Trial completion time was the time taken to complete all components of the driving circuit in the virtual simulation. Reaction time was defined as the time it took for the subject to move the virtual wheelchair at least 0.01 m when the driving trial began. The length of the actual path taken by the virtual wheelchair and the number of collisions were also recorded.

SPSS (version 18.0)²⁴ and MATLAB (version 7.11)²⁵ were used for all analyses. Significance level was set at 0.05 a priori. The data were evaluated for normality and other assumptions for the statistical analyses. Any driving trials with outliers in outcome measures were removed prior to running the statistical comparisons. Logarithmic transformation was used to normalize variables that still had skewed distribution. A 2 × 2 (two displays and two driving modes) completely within subjects repeated measures Analysis of variance (ANOVA) was performed to compare wheelchair driving performance in CVE and IVRE using the above mentioned driving performance metrics as outcome measures. Post hoc ANOVAs were performed when the main effects were significant.

A user's performance on a task of driving a wheelchair along a predefined path is affected by the limited information processing and control capacity of humans.²⁶ Owing to random noise in the human action-perception system or in the environment, errors start compounding while a task is being performed. If the task is to drive a wheelchair along a predefined path, then due to accumulation of noise, the wheelchair will start deviating from the desired or ideal path. The higher the current speed of the wheelchair, the faster errors will compound in the wheelchair's trajectory, the faster the lateral deviation will increase and the sooner the wheelchair will reach the lateral edge of the path. The wheelchair driver takes a certain amount of time to process the information about deviation in a trajectory. On narrow roads, where the possibility of committing error is high, a driver may drive more slowly to allow enough time for correcting the trajectory if a deviation was to occur and vice versa. The Steering Law attempts to model this change in the efficiency of task completion per unit increase in task difficulty.²⁷ The efficiency in completing a task such as navigating a wheelchair along a marked path in a VE²⁷ is directly related to the difficulty of the task.

The Steering Law models a relationship between task completion time and difficulty of that task²⁸ and is given by T_C = a + b*ID_C. T_C is the task completion time and a and b are constants in the linear regression equation. ID_C is the index of difficulty of the task and is obtained by integrating the inverse of path width along the trajectory. For a linear hallway of fixed width (W) and length (L), the ID_C integral becomes L/W. If a wheelchair of width D is navigating along this hallway, effectively a width of W − D is available to the user before an error is committed by impacting the hallway walls. Hence the task completion time equation becomes

Equation 1: Steering Law equations for a hallway of length L and width W.

Researchers have modeled this relationship in multiple contexts such as performing a two-dimensional computer access task,^29–31 three-dimensional navigation tasks in a VE^27,32–34 and in real-world driving tasks.³⁵ The law provides a standardized framework that can be used to compare different input devices or different virtual-world setting. The index of performance or throughput (calculated as 1/b, units: bits/second) values can be used for this comparison. The lower the slope (b) of completion time versus ID plot, the higher the throughput (index of performance) will be and the higher the task completion efficiency will be. Steering Law relationships can assist in designing activities in virtual and the real world (office spaces, hallways) such that they have an index of difficulty that does not seriously affect the user's mobility performance.

The index of difficulty values was computed using Equation 1, and linear regression equations were derived for ‘task completion times’ versus ‘index of difficulty plots’. For the exploratory analysis, data from all driving trials for every subject were analyzed to evaluate the minimum hallway width subjects could traverse comfortably. If there was a collision along the decreasing hallway, that hallway width was noted. Secondly, an exploratory analysis of VPMRT composite scores from all subjects was performed to determine whether the CVE or IVRE-based assessment could help to identify driving deficits in borderline safe drivers. The researchers used their clinical observations to identify subjects who may be borderline safe wheelchair drivers in the real world.

Composite scores from the VPMRT tasks were used to establish intra- and inter-rater reliability of the VPMRT for the VE. To evaluate test–retest or intra-rater reliability of the two clinicians, the assessment scores on the repeated driving trials were compared. Since adequate training time was allowed for the participants, they were not expected to perform significantly different in trial repetitions. The intra-rater reliability is measured here using Intra Class Correlation (ICC)³⁶ as the consistency in the clinician ratings of two driving trials rather than an exact agreement between them. Analyses for the two display screen were performed separately. For determining inter-rater reliability, analyses were performed with the three compounded scores. A two-way random effects model was used since both participants and clinicians were considered to be random samples. The ICC³⁶ values for PC and VR screens were computed separately. Bland Altman plots³⁷ for the VPMRT total scores from the two clinicians for every driving mode were drawn to give a visual representation of agreement between the clinicians.

Results

Eleven participants (seven men) with average age of 39.5 ± 15.9 years who regularly used their power wheelchairs for all or part of their daily mobility were recruited for this research protocol. Four participants had spinal cord injury, three had cerebral palsy, one had muscular dystrophy, one had spinal muscular atrophy, and two had other disabilities. Participants used a computer for on average 32.4 ± 8.2 hours per week at home and at the office. Clinicians terminated the driving session for one subject who could not complete any of the virtual driving trials because of fatigue and dizziness. Another subject partially completed the required number of driving trials but could not return to complete the remainder of the protocol due to scheduling conflicts.

Quantitative driving performance measures

Before running any statistical test, data were verified to meet its statistical assumptions and outliers were removed. All driving trials from subject VR07 had extreme outliers in the dependent variables and hence they were not included in the statistical comparisons for quantitative measures. VR07's data were used for all other analyses. The repeated measures ANOVA indicated that there was a significant main effect of ‘drive mode’ (P < 0.001, η² = 0.511) and ‘display type’ (P < 0.001, η² = 0.677) for trial completion time. Participants took about 111.97 seconds more to complete driving trials when using the VR screen than when using the PC screen. Also, irrespective of the display screens, trial completion times were 54.82 seconds higher when participants used Rollers ON as the input than when they used the customized joystick and mathematical model (Rollers OFF).

The main effect of ‘reaction time’ was significant (P < 0.006, η² = 0.286). There were no differences in reaction times across PC and VR screen driving trials. While driving with the rollers ON, reaction time was about 0.69 seconds higher than when not using the rollers (Table 2). The path length covered by the virtual chair was significantly different only across the two drive modes (P < 0.001, η² = 0.609) and not between the two screens. When the rollers were OFF, participants covered an extra 5.08 m compared to when rollers were ON. The main effect of RMSD was significant only for the type of display screens (P < 0.035, η² = 0.195) and not for the drive mode. Irrespective of the driving mode, RMSD values for the VR screens were 0.11 m higher than those on PC screen. All but two participants had at least one driving trial in which they hit a wall or moving obstacle and could not self-correct before the program self-terminated the trial. Nine of 11 such trials were on the PC screen.

Table 2.

Repeated measures ANOVA for the two driving modes and two display screens

Mode	Display	Trial time (seconds)	Reaction time (seconds)	Path length (m)	RMSD (m)
Rollers OFF	PC (CVE)	231.7 ± 49.13*	0.966 ± 0.67*	137.69 ± 6.02*	0.415 ± 0.15*
Rollers OFF	VR (IVRE)	344.96 ± 93.58*	1.61 ± 0.97	139.03 ± 6.99	0.51 ± 0.37*
Rollers ON	PC (CVE)	287.81 ± 72.96*	2.09 ± 1.48*	132.38 ± 3.16*	0.333 ± 0.08
Rollers ON	VR (IVRE)	398.49 ± 162.72*	1.87 ± 1.21	134.18 ± 7.0	0.456 ± 0.39

*Indicates a pair with a statistically significant difference.

While driving with the rollers ON, reaction time was about 0.69 seconds higher than when not using the rollers.

Steering law validation

The index of difficulty (ID) values for the four hallways, the slope and the R² values for the Trial time versus ID regression curves for the four conditions, and throughput values for the four driving conditions are shown in Table 3. The regression curves for all driving conditions gave a very good linear regression fit (R² statistic 0.82–0.99). This gives reasonable confidence that the Steering Law can be applied to design driving tasks and to predict the virtual driving behavior in this VE. Regression analysis (see Table 3) shows that the throughput values during the ‘Rollers OFF’ driving mode were about twice the values during ‘Rollers ON’ mode. Also the throughput values were higher for the PC screen than for the VR screens. This indicates that tasks performed on a PC screen with Rollers OFF should yield the best performance.

Table 3.

Slope of the curve (b) and R² values for the ‘trial time versus index of difficulty’ plots and throughput for the four test conditions

Mode	Display	Slope (b)	Throughput (bits/seconds)	R2 value
Rollers OFF	PC (CVE)	0.08	12.5	0.84
Rollers OFF	VR (IVRE)	0.12	8.33	0.83
Rollers ON	PC (CVE)	0.15	6.67	0.84
Rollers ON	VR (IVRE)	0.24	4.17	0.998

Reliability of VPMRT scores

The VPMRT total scores for the two repeated trials assessed by clinicians were highly correlated and statistically significant. Clinician 1 showed intra-rater reliability ICC values of 0.896 (95% confidence interval: 0.745, 0.96; P < 0.001) for PC screen and 0.887 (95% confidence interval: 0.731, 0.955; P < 0.001) for VR screens. Clinician 2 showed intra-rater reliability ICC values of 0.711 (95% confidence interval: 0.378, 0.881; P < 0.001) for PC screen and 0.758 (95% confidence interval: 0.473, 0.899; P < 0.001) for VR screens. The VPMRT composite scores show a moderate-to-high degree of agreement between the two clinicians. The inter-rater ICC values for driving trials from the PC screen (CVE) were 0.901 (95% confidence interval: 0.8, 0.9; P <0.001) and for the VR screens were 0.78 (95% confidence interval: 0.7, 0.9; P <0.001).

Fig. 3 shows Bland Altman plot for the Total composite VPMRT scores from Clinician 1 and Clinician 2. The mean ± standard deviations of difference in the ratings of the two clinicians were computed. For the PC screen, for both ‘Rollers OFF’ (0.0001 ± −0.011) and ‘Rollers ON’ (0.0007 ± −0.011) the mean difference in scores from the two clinicians and variability of these differences were very small. For PC screen all differences were within 2 standard deviations from the mean difference. Compared to the PC screen, the differences in clinician ratings on the VR screen were significantly larger for both ‘Rollers OFF’ (−0.013 ± −0.041) and ‘Rollers ON’ (−0.012 ± −0.03) driving modes. There were cases when the differences were more than two standard deviations away from the mean difference in ratings.

Figure 3.

Bland-Altman plot showing agreement between the total VPMRT scores from Clinician 1 and Clinician 2 for the four driving conditions. The horizontal lines represent overall mean and two standard deviations of differences in ratings from the two clinicians.

Exploratory analysis to evaluate driving deficits

Except for one participant who did not have wall collisions in the decreasing hallway section, participants had a median of 3 (range 1–7) of 12 driving trials in which they had a collision in the decreasing hallway walls. Since subject VR07 had a collision with the walls of hallway 2 (width = 1.067 m or 3.5 ft) the minimum hallway width this subject could navigate was 1.37 m (4.5 ft). All others had at least one collision with the walls of hallway 1 (width 0.91 m or 3 ft). Hence, the minimum width most could safely navigate safely was 1.067 m (3.5 ft). When collisions during decreasing hallway tasks were computed for the two driving modes and screens, the PC screen with Rollers OFF driving mode showed the least number of collisions (5) while the VR screens with Rollers ON mode showed the most number of collisions (10).

Table 4 shows the average values of driving performance indicators. C1TOTAL and C2TOTAL are the composite VPMRT scores given by Clinician 1 and Clinician 2. The ‘total’ scores derived from every trial are averaged in order to generate one representative value for per subject. Similarly, averages for other outcome measures were computed. Clinician 1 rated driving trials of subjects VR07 and VR09 with VPMRT scores less than 0.9 while Clinician 2 rated them with scores less than 0.93. Among all participants, these two subjects showed the highest amount of RMSD and longest path length. Their median boundary collisions were highest among all subjects. The trial time and reaction time for these two subjects were significantly above the group average of rest of the group.

Table 4.

Average VPMRT scores and driving performance indicators for all subjects

Subject ID	C1TOTAL	C2TOTAL	Trial time	Reaction time	RMSD	Path length	Collisions
VR01	0.996	0.996	208.835	0.747	0.269	137.549	2
VR02	0.997	0.992	499.634	2.212	0.300	135.554	1
VR03	0.992	0.991	334.089	1.111	0.361	132.348	2
VR04	0.984	0.983	260.125	1.110	0.377	134.469	4
VR05	Did not complete the experiment
VR06	0.986	0.986	274.254	1.246	0.510	138.819	6
VR07	0.786	0.901	1020.712	2.729	1.310	176.415	61
VR08	0.982	0.988	319.885	1.561	0.374	127.170	3
VR09	0.899	0.929	330.746	2.533	0.805	147.404	10
VR10	0.997	0.999	241.554	1.468	0.306	134.247	0
VR11	0.982	0.978	419.414	3.116	0.386	135.674	8

When asked for their subjective preferences, eight subjects preferred the PC screen (CVE) over the VR screen (IVRE) while one did not have a preference. Those who preferred the PC screen over the VR screen suggested that the driving in IVRE was more realistic but it made them feel dizzy. They reported that the three screens presented a lot of information which sometimes overwhelmed them and made them feel tired. Most of the subjects felt there was no significant difference between the two driving modes. Subjects were able to adapt to the change in the virtual wheelchair's dynamics when the mathematical model was used (‘Rollers OFF’ mode). All subjects agreed that the virtual simulation was a good first step toward a driving training tool and they would recommend and use it if such a tool were commercially available. Subjects suggested that the future versions of the simulation should include a wider range of and more challenging tasks including navigating outside home, through traffic, and in tight spaces (for example, public transportation).

Discussion

Comparisons of driving performance scores across the two screens and two driving modes clearly showed significant differences in the driving modes. The driving trials on the VR screen took longer to complete than the driving trials on PC screen yet had no significant difference in the length of path covered. Subjects also had higher RMSD on VR screens than when on the PC screen. A 4-ft wide hallway on the VR screen, for example, might have been perceived as bigger with respect to the chair than it appeared on the PC screen. In other words, larger FOV on VR screens likely lead to greater visually induced self-motion.³⁸ Compared to the single PC screen (FOV = 90°) the three VR screens together (FOV = >160°) gave a much wider FOV and displayed a larger part of the VE. It could be that the VR screens, due to their larger FOV than the PC screens, increased the cognitive processing load on the subjects. This increased cognitive load likely lead to slower driving but with more driving errors when using the VR screens in order to compensate for increased information-processing requirements from the extra-visual inputs. The higher FOV from the VR screens was of some advantage for the subject's response to wall collisions, as they were able to avoid and correct their paths away from potential obstacles. Subjects got stuck, or they were unable to remove the wheelchair away from the site of collision, more often on the PC screen than on the VR screen. Similar results were found by Tan et al.³⁹ In the beginning of the research session subjects were allowed to practice virtual driving until they felt comfortable. In future studies, we plan to have a more structured training regimen which may help in reducing errors while driving.

When the rollers were not used, the customized joystick was mounted on the subject's wheelchair. We expect the mounting to have minimal impact on the virtual driving since most of the subjects felt comfortable with driving with the newly positioned joystick. However, this may be an issue with subjects who have substantial seating and positioning requirements and who use customized joysticks. The future versions of the virtual driving simulator will accept inputs directly from the user's wheelchair joystick. When rollers were not used, subjects showed lower reaction times compared to when rollers were used. Subjects were comfortable driving faster without the rollers and using the mathematical model but they showed a slight increase in the length of path travelled compared to when the rollers were used. This indicates that subjects took a path with more turns when not using rollers. The mathematical model used to simulate wheelchair dynamics tends to make the wheelchair slightly more sensitive to turns. Half of all participants noted this difference in turn sensitivity when Rollers were OFF compared to when rollers were ON, but they were able to adapt to the turning rate after one practice trial. The differences in virtual driving with and without the rollers were not significantly perceptible for the other half of study participants.

The Steering Law evaluations showed that the driving performances in the decreasing hallways could be modeled using the Steering Law equation. Of the four driving modes, subjects showed the best index of performance values for Rollers OFF + PC screen combination and the worst values for the Rollers ON + VR screen combination. Similar trends were seen in the number of wall collisions in the decreasing hallway section. However, more research is currently being conducted to optimize the driving modes for virtual driving and evaluate effects on potential/current wheelchair users. The PC screen provides a low cost and portable display for implementing the virtual driving experience in a home environment. Moreover, some users who are prone to fatigue and cybersickness from the immersive VR screens report being more comfortable with the PC screens over immersive VR displays.^40–42 The mathematical model, although not perfect, closely simulates wheelchair driving and enables users to interact with the VE using only a standard joystick. This software could be implemented and customized by an experienced clinician through a web interface to which the potential users could log in from their homes or remote clinics.

In this pilot research study, we compared the four roller and screen combinations based on a sample of regular wheelchair users. We need more data to evaluate whether the differences in driving performances of users were actually because of differences in the simulator driving modes or because the evaluation tools and metrics were less sensitive to the variables of interest in those simulator modes (combination of screen and rollers). For example, while using a certain simulator mode, if users show higher number of driving errors, it could also mean that the simulator mode may be more sensitive to driving errors. Having a simulator mode with higher sensitivity may be important to detect errors that may be critical to wheelchair driving. Hence, future studies will compare driving performance and errors in virtual and real worlds.

The intra-rater reliabilities of the two raters were moderate to high for driving tasks. The correlations were significant for trials repeated across two driving modes and for the trial repetitions performed for every driving condition. This indicates that overall the clinicians were fairly consistent in their assessments on driving tasks. Clinician 1 showed higher intra-rater reliability than Clinician 2. This was expected since Clinician 1 had more direct and ‘hands on’ experience of evaluating wheelchair driving in a real-world rehabilitation clinic. Although experienced with assessing real-world wheelchair driving performance in a rehabilitation clinic, Clinician 2 would primarily focus on medical aspects of client assessment. Another interesting observation was that the ICC values from inter-rater reliability analysis by clinicians for the tasks on the VR screen were lower than the than those on the PC screen. One possible reason could be a bias or preference of the clinicians to one or other screen. One clinician expressed of feeling ‘slight dizziness’ from continuously observing participants drive on the VR screens for a long time. Virtual reality displays are known to cause simulator sickness (similar to motion sickness) after prolonged use without a break.^1,43 In future studies a larger number of more experienced and less experienced clinicians will be recruited for the virtual driving evaluations. The consensus building protocol would be implemented so that clinicians with all levels of experience can interpret and evaluate the VPMRT scale in a more uniform fashion. Besides, this will also aid in minimizing any bias in evaluations because of the clinician's prior experience with virtual simulators.

This study was a first step toward exploring the clinical relevance of the quantitative measures used for wheelchair assessments. While validating the real-world PMRT, Massengale et al.¹³ found that wheelchair drivers with ≥95% score were safe drivers. Drivers who get a high score on the PMRT showed better motor coordination, higher near and far visual acuity, lower collisions with obstacles, and had a higher alertness to details in environment. Out of the 10 subjects who completed this research protocol, two subjects (VR07 and VR09) received significantly low cumulative/total PMRT scores (<95%) from both clinicians. Their poor driving performance was also indicated through quantitative performance metrics (high trial time, high reaction time, high RMSD, high path length, and more collisions). These two participants could be classified as borderline ‘good/safe’ drivers as far as virtual driving tasks are concerned. Although both of these participants were regular power chair users, clinicians observed these drivers to have a few instances of impulsive driving in real-world hallways especially when they arrived to the research center. Since impulsive driving in the real world may be one of the main reasons for wheelchair accidents, it will be useful to introduce driving tasks and evaluation metrics that can specifically quantify impulsive driving and assist in safe driving practices. Recent research on using joystick excursion or movement of the joystick post as an evaluation metric is promising in detecting impulsive driving behaviors.⁴⁴ This pilot study indicates the threshold of real-world PMRT score of 95% and above also applies to the VPMRT to help identify ‘good/safe’ drivers. In future studies, collecting data from a larger cohort of wheelchair drivers with different driving capabilities will help us identify and validate certain safe score ranges for the VPMRT and other driving performance metrics like average driving speed and RMSD. Furthermore, VPMRT could be administered multiple times to track the participant's progress and identify specific deficits in their driving. Other subjective feedback received from the subjects was useful in deciding the future plan of action with the virtual simulation software.

There were some limitations to this study. The mathematical model that was implemented was designed only for front wheel drive chairs. This limited us to recruit only those participants who regularly used front wheel drive wheelchairs. The future versions of the virtual simulators will be evaluated with people who use different types (front wheel drive, mid-wheel drive, and rear wheel drive) of wheelchairs from different manufacturers. The mathematical model itself was slightly more sensitive to turns which caused the virtual chair to over steer. A future version of this model used in will employ tuning parameters to control the linear and rotational accelerations. Also, for use in a future clinical application, a library of mathematical models of commonly prescribed front wheel, mid-wheel, and rear wheel drive wheelchairs will be created. The mathematical model can be customized by experienced clinicians as required by clinical needs of a potential user.

Small sample size was another limitation of this study: the within subjects repeated measures design improved our ability to detect statistical significance. All participants recruited for this study were regular power chair users. Because users had significant experience driving the wheelchair in the real world, the driving deficits identified in this study should be compared with driving in real-world tasks. Comparison of virtual and real-world driving performances of wheelchair users will be examined in the second phase of this study. The participants in this study were not specifically evaluated for their motor coordination and visual acuity/FOV. The screening relied on self-report from the participant. In future studies, tests such as the Motor-Free Visual Perception Test⁴⁵ could be used to test overall visual perceptual skills.

Conclusion

The VPMRT was implemented in a virtual driving simulator. VPMRT scores from the two clinicians show good-to-very-good inter-rater reliability and moderate-to-high intra-rater reliability. There were significant differences in the driving performance of participants when they were driving the virtual chair in the two driving modes and using the two display screens. Participants showed the best driving performance when using the combination of the computer screen with ‘Rollers OFF’ (using the mathematical model) while their worst driving performance was seen with the combination of VR screen and ‘Rollers ON’. Using the total VPMRT score, two participants were classified as borderline safe drivers and this was consistent with the driving performance metrics derived from their trajectory data. The Steering Law shows good validity in modeling a participant's driving performance when driving along a virtual hallway with decreasing widths. Overall, this VE seems to be a promising platform for future work with virtual driving assessments.