A Standardized Obstacle Course for Assessment of Visual Function in Ultra Low Vision and Artificial Vision

Abstract

We describe an indoor, portable, standardized course that can be used to evaluate obstacle avoidance in persons who have ultralow vision. Six sighted controls and 36 completely blind but otherwise healthy adult male (n=29) and female (n=13) subjects (age range 19-85 years), were enrolled in one of three studies involving testing of the BrainPort sensory substitution device. Subjects were asked to navigate the course prior to, and after, BrainPort training. They completed a total of 837 course runs in two different locations. Means and standard deviations were calculated across control types, courses, lights, and visits. We used a linear mixed effects model to compare different categories in the PPWS (percent preferred walking speed) and error percent data to show that the course iterations were properly designed. The course is relatively inexpensive, simple to administer, and has been shown to be a feasible way to test mobility function. Data analysis demonstrates that for the outcome of percent error as well as for percentage preferred walking speed, that each of the three courses is different, and that within each level, each of the three iterations are equal. This allows for randomization of the courses during administration.

Abbreviations: preferred walking speed (PWS) course speed (CS) percentage preferred walking speed (PPWS)

Download video file ^{(25.9MB, mp4)}

Introduction

Low vision rehabilitation assessments must determine whether intervention results in improvement in function. Performance metrics typically involve computer based reading or functional assessments^1-9 as well as quality of life questionnaires^10-15. Being able to also assess the low vision patient's ability to navigate around obstacles might also provide clues to functional improvements¹⁸ particularly in the case of artificial vision devices. Geruschat et al. recently published navigation outcomes with a retinal implant chip, highlighting the need for a standard metric in this area¹⁷. Currently there are no widely accepted, objective, validated, and comprehensive standards for determining capacity for obstacle avoidance.

Development of a functional test that would correlate to navigation performance for persons with low vision or "ultra low vision" as produced by artificial vision would be desirable, but has remained an elusive goal. The burgeoning field of artificial vision devices such as retinal implant chips^18-24 or sensory substitution devices such as the BrainPort ²⁵ and The vOICe²⁶, necessitates a test of obstacle avoidance that might correlate to increased navigational abilities conferred by these devices. Such an assessment would not only allow subjects to understand their own limitations as they traverse their surroundings, but might provide a means for measuring improvement with orientation and mobility training or between iterations of vision enhancement prototype devices. Ideally, there might be some ability to assess an individual's risk for fall accidents²⁷.

Our goal was to create an obstacle course that would be useful for evaluation of navigation ability in patients using artificial vision devices and transferrable to the field of low vision in general. A review of the published literature on obstacle courses and visual impairment was undertaken using the PubMed database. There have been numerous attempts at creating standardized obstacle courses^{16,17,28-31,34}. Most of these are not portable in the sense that it would be difficult to exactly reproduce the setting, particularly for outdoor courses. Maguire et al. describe obstacle course which is used to show mobility performance in patients with Leber's Congenital Amaurosis. This course has the benefit of being portable and small, but it is not clear whether different iterations have been made available to prevent memorization effects, nor are there any provisions for obstacles which are not on the floor, texture changes, or stepovers. Leat provides an incisive description of potential pitfalls in designing a course and puts forth a description of an outdoor course which unfortunately would not be able to be reproduced exactly in an alternative location³⁰. Velikay-Parel et al. described a mobility test for use with retinal implant chips. This design had the benefit of being portable and simple to execute. While this course could be reproduced at an alternative site, no specific details on course construction are provided. Moreover, and more concerning was that they showed the learning effect reached asymptotic levels due to course familiarity, therefore being able to prevent course memorization altogether might eliminate the concern for loss of learning effect over time¹⁸. None of the courses described so far have been widely adopted by the low-vision or rehabilitation communities.

The authors subsequently consulted with a team of six low vision occupational therapists and orientation and mobility specialists from the Western Pennsylvania School for Blind Children (Pittsburgh, PA) and the Blind and Vision Services Rehabilitation of Pittsburgh (Homestead, PA) regarding proposed course design. Desirable attributes of a functional obstacle course identified included: Portability for easy assembly/disassembly and storage, flexibility to test under both dim and bright lighting conditions, and to mirror "real life" situations by including obstacles that represent objects in a patient's home environment that are sturdy enough to withstand repeated collision while being ductile in order to prevent patient injury. In addition it was deemed necessary to have several types of environments designed in such as way so that when administered in a randomized order prevents course memorization. In addition, the course should demonstrate reproducible results in multiple settings, have strong inter and intra rater reliability and be an objective measure of spatial awareness.

The culmination of this effort was development of an obstacle course which could reasonably be expected to be reproduced in a standard institutional hallway. The course is designed to test different skill sets, all important for navigation. Each level of the course attempts to focus several particular types of obstacles encountered in everyday navigation activities. The first course evaluates the ability to navigate through relatively high contrast targets that are all placed on the floor, but requires a large number of turns. The second course evaluates the ability to navigate through obstacles that are both high and low contrast, floor texture changes, and objects suspended in air. The final course evaluates the ability to navigate Styrofoam obstacles that are low contrast, surface glare changes on the floor, the addition of nonStyrofoam obstacles (fabric), floor tile color changes, obstacles that must be stepped over, and obstacles that are not on the floor. The courses are labeled 1, 2, and 3 for ease of labeling, but this designation should not be construed as increasing in level of difficulty. Within each level, there are three versions of the course, which can be randomized to prevent course memorization.

Protocol

1. Course Construction

1. Install course floor. Course dimensions are 40 ft long by 7 ft wide consisting of 280 1 ft² portable floor tiles (beige event floor tiles). Place with black trim around perimeter only (Figure 1).

2. Paint the adjacent walls to match the floor tiles creating a somewhat monochromatic environment. Colors we used with greyscale values are provided (Table 5). If the specific colors are not available, we recommend taking a tile to a hardware store for color matching.

3. Install lighting according to the lighting template (Figure 1). Connect lights to dimmer switch.

4. Paint obstacles according to painting instructions (Figure 2).

2. Prepare Testing Area

1. Adjust lighting to desired condition and check with light meter at the beginning, middle, and end of the hallway containing the course.

2. Make sure that the video camera is set to record and that placement of the camera is appropriate to capture the subjects as they walk through the course. We recommend a ceiling mount or alternatively the camera can be hand held.

3. Record Preferred Walking Speed PWS

1. Position subject in center of walkway (course column "D"). Note: toes should be behind border of walkway. Read instructions to the subject (Figure 3).

2. Begin stopwatch once foot crosses black border and onto pathway. Stop time once foot crosses black border at other end of walkway. Record time as PWS1. Turn subject around and repeat procedure in opposite direction. Record time as PWS2. Average PWS1 and PWS2 and record as final PWS.

4. Obstacle Course Navigation

1. From the randomization scheme, set up the first course (Figure 4). Floor tiles should be used as the grid upon which the obstacles are mapped. Refer to the provided diagram for correct mapping of the obstacles. It is helpful to number the tiles along the vertical and horizontal axis with an indelible marker to permit easy placement of the obstacles. It is also helpful to label the obstacles according to the provided diagrams in an inconspicuous location.

2. Guide subject to start of 40 ft walkway. Read instructions to the subject (Figure 3). Subject should be positioned in center of walkway (column "D") with toes behind border. Begin stopwatch once foot crosses black border and onto pathway. Stop time once foot crosses black border at other end of walkway. Record this time as Course Speed (CS).

3. Record when obstacles are hit, grading the severity of the hit on a 3 point scale. The course run should be videotaped for later confirmation by an independent observer.

5. Obstacle Identification

1. Upon completion of course navigation task, turn subject around to face the course and position at center of course (column D). Note: make sure that any obstacles that require repositioning in order to view correct color from end of course is rotated. Read instructions to subject (Figure 3). At this time the first object identification task will be administered. Ask the subject to turn around and tell the Research Assistant the total number of objects they can discern within 30 sec. This number should be recorded.

2. Tell the subject to walk back through the course and point to each obstacle they can see. It does not matter if they collide with the obstacle. The number of obstacles they can see is recorded. It is helpful to record which obstacles they are able to detect. This is not timed.

Representative Results

Subjects

Six sighted blindfolded, sighted, and 36 completely blind but otherwise healthy adult (age range 19-85 years), male (n=29) and female (n=13) subjects were enrolled in one of three studies involving testing of the BrainPort sensory substitution device (Wicab, Madison WI). All studies were approved by the University of Pittsburgh IRB and all subjects signed an approved informed consent document. All studies were a within subjects, repeated measures design such that each subject acted as their own control. Sighted subjects were blindfolded to simulate a newly blind condition for all testing procedures. Visual acuity of light perception or worse for those with blindness was confirmed with BaLM light perception test and the FrACTSnellen score of < 2/5,000 and an eye exam prior to enrollment. All subjects completed the entire obstacle course at baseline and then again after a 15-20 hr structured training protocol with the BrainPort device. This protocol is designed to confer basic proficiency with the device and includes approximately 2 hr of ambulation/mobility training within the office environment (locating doors, windows, chairs, etc.). The primary outcome for course navigation is measured by percentage preferred walking speed (PPWS), which is a gold standard for mobility research. This is calculated by dividing CS by PWS (see instructions). Our secondary outcome is percent error, defined as the percentage of possible collision with obstacles on the course.

The first 16 subjects were sent through all 9 course iterations in both bright and dim lighting situations for a total of 18 runs through the obstacle course per subject. Means and standard deviations were calculated across control types, courses, lights, and visits. To adjust for random effects among repeated measurements in each nested cluster within a subject, the linear mixed effects model was used to compare different categories in the PPWS and percent error data. The nested clustering was in the order of subject identification number, visit (pretraining and post training), light (dim and bright), and course level (1, 2, and 3). A preliminary analysis of the first 16 subjects showed that there were no statistically significant differences between the individual versions of the course within each level of difficulty. Therefore, in order to minimize subject burden, the remaining subjects were randomized to one version of course 1, 2, and 3 in dim light and another version of course 1, 2, and 3 in bright light. This reduced the time to complete the course from 3 hr to just less than 1 hr. Both the order of the courses and lighting conditions were randomized to prevent the potentially negative effects of waning concentration and/or fatigue.

Data for all subjects is presented in Table 1 for PPWS and Table 2 for Percent Error. The data is arranged as follows in each table: All (pre and post training data combined), Pre training (no BrainPort) and Post training (with BrainPort), respectively. Note for pretraining values, subjects are without vision, which tended to result in larger standard errors for this condition. All reported p-values are two-sided and statistical analyses were done using Stata/IC12.1. We found that for the outcomes of PPWS and Percent Error, the three course levels (1, 2, and 3) were not equal. We also found that the levels of the nine sub courses were not equal. Our results showed that the three subcourse iterations (a, b, and c) for level 1 were equal , as were three subcourse iterations (a, b, and c) for level 3 for all conditions. However for level 2, the sub courses were shown to be equal when using the BrainPort, but not at baseline (no BrainPort /pretraining condition), which affected results for the combined condition.

Figure 6 is a histogram showing our results for PPWS, which demonstrates that subjects using the BrainPort walked more slowly than without it (PPWS 1.90 for No BrainPort Condition vs. 3.92% for the BrainPort Condition, p=0.001) Review of the video cameras was particularly helpful in explaining this result. When the blind subjects walk through the course at baseline, they walk at their normal pace but hit anything in their path as they have no means for detecting obstacles. However, subjects using the device engaged in visual scanning, a behavior which was absent without the BrainPort, and reflects in an increase in PPWS values (see video).

Figure 7 shows our baseline versus BrainPort condition results for the outcome of percent possible errors. Using the BrainPort, subjects had a trend towards a reduced number of collisions with obstacles compared to the no BrainPort condition. A deficiency of current artificial vision devices is lack of depth perception, so although they might detect an obstacle, it is quite difficult to estimate its distance. This is due to limited resolution capabilities of the BrainPort and use of a single camera system. In order to provide additional insight into the obstacle avoidance capabilities of the BrainPort, two visual identification tasks are conducted during the performance trial. The first takes place at the completion of the course when the subject is asked to turn around and tell the examiner the number of objects in the total course that he/she can discern. We found that the resolution of the BrainPort was not sufficient to perform this task, but it remains to be tested in a low vision cohort. The second identification task involves an untimed walk through one version of each level of the course and asking the subject point to obstacles that they can detect. This visual identification tasks is conducted separately from the timed course navigation tasks so as not to influence walking speed³⁴. In addition, for the obstacle detection task collisions are not recorded. The "no BrainPort " or pretraining condition was not tested as none of our blind subjects would have been able to complete this task. Table 4 shows our results for the obstacle detection task using the BrainPort in dim light and bright light. We were able to further analyze this by the color of the obstacle detected. This is important for artificial vision, which is heavily dependent on contrast. Overall, we found that subjects were able to detect the presence of any obstacle about 48% of the time whether the obstacle was high or low contrast. Generally, high contrast obstacles were detected more easily than low contrast obstacles irrespective of lighting condition (56.25% versus 40%, respectively). Obstacle detection did not vary significantly between lighting conditions, likely due to the presence of luminance averaging software on the BrainPort device.

Table 1. Representative summary of results which compares percent preferred walking speed at baseline to post - BrainPort training values. The Kruskall-Wallis test was used to compare baseline values (no BrainPort condition, or pretraining) to those obtained after one week of BrainPort training (BrainPort condition, or post training). Click here to view larger image.

Table 2. Representative summary of results which compares percent possible errors at baseline to post BrainPort training values. The Kruskall-Wallis test was used to compare baseline values (no BrainPort condition, or pretraining) to those obtained after one week of BrainPort training (BrainPort condition, or post training). Click here to view larger image.

Table 3. Detailed description of obstacles used for the course.Click here to view larger image.

Table 4. Percentage of light and dark objects identified in both dim and bright lighting during the obstacle detection task.Click here to view larger image.

Table 5. Detail of the materials required for obstacle course construction. Click here to view larger image.

Figure 1. Flooring and lighting set-up template.Click here to view larger image.

Figure 2. Obstacle painting instructions.Click here to view larger image.

Figure 3. Instructions for staff when administering course.Click here to view larger image.

Figure 4. Illustration of each of the 9 course iterations grouped by course level including a description of number of turns and path width. Vertical path refers to number of open tiles to be traversed in the forward direction, Horizontal path refers to number of open tiles to be traversed in the right or left direction. Turn refers to when the subject must change orientation or direction to avoid an obstacle. Click here to view larger image.

Figure 5. Illustration of the idealized path trajectory through each course.Click here to view larger image.

Figure 6. Percent Preferred walking speed at Baseline and Post Training. Click here to view larger image.

Figure 7. Percentage of Possible errors made at Baseline and Post Training.Click here to view larger image.

Discussion

We describe an indoor, portable, easily reproducible, and relatively inexpensive course that can be used to evaluate obstacle avoidance in persons who are blind or have low vision. Most current obstacle course designs and tests (i.e. TUGS) are difficult to compare across sites and observers, or are permanent instillations which cannot readily be performed at alternate locations^16,17,30. Our goal was to create a course that could be standardized for use at different locations and with different observers, and which would provide some predictive ability as to whether an intervention (i.e. artificial vision device or mobility training) had any effect.

We constructed a portable obstacle course measuring 40 ft long by 7 ft wide consisting of 280 1 ft² portable floor tiles. The floor tiles flanking the perimeter of the course are also beige, but do have a 1 in darker border at the outside edge, which serves to delineate the border of the course. The adjacent walls are painted to match the floor tiles creating a somewhat monochromatic environment. This serves to reduce ambient contrast and render the obstacles more prominent. A total of 16 obstacles representing objects encountered in the day to day environment such as chairs, desks, trash cans, etc. were identified by the orientation and mobility consultants. We recreated the objects in representative block shapes out of Styrofoam, (see Table 3 for exact specifications), with 10 obstacles being used for any given course iteration. These obstacles are located either on the floor or are hung from the ceiling at a height of 63 in from ground level, as the average height of American females is 63.8 in vs. average American male height of 69.3 in ³³. Styrofoam obstacles were manufactured according to custom specifications. The sides of the obstacles are painted darker or lighter than the ambient color to vary contrast. Other obstacles include a dark pile of fabric, changes in floor color and changes in floor texture, the latter created by placing a carpeted mat on the obstacle course. These were added at the suggestion of the occupational therapists, who noted that fall accidents often occur when glare or other texture changes are misconstrued as an obstacle. Ambient illumination is controlled and measured with a light meter. The total cost for all course related materials including all outcome measurement is approximately $5,200 USD.

Obstacles are arranged in three prespecified levels, with 3 subcourses or iterations for each level. Each course level contains the same set of obstacles arranged in 1 of 3 configurations. Course levels are determined by number of turns and path width, as well as type and placement of obstacles. Each course is color coded and mapped onto a grid (floor tiles) for rapid and easy reproducibility (Figure 4). Each of the 3 course permutations within each level of difficulty is designed with a similar, if not identical, number of path widths and turns between obstacles (Figure 5). Courses can be run in both photopic (light) and mesopic (dim) illumination. All runs through the course are videotaped. Each course takes approximately 0.5-5 min to navigate, depending on baseline navigation skills, preferred walking speed, and course level. For timed assessments, subjects are instructed to find their way through the obstacle course as quickly as possible using normal gait while avoiding objects to the best of their ability.

The primary outcome for course navigation is measured by looking at percentage preferred walking speed (PPWS). PPWS is widely used in balance and gait research and is an ideal measure because it offers the advantage of allowing subjects to act as their own controls, thus normalizing results for physical factors such as height and weight as well as for sex and age³². Using this metric has the added advantage of negating any effect of previous mobility training between subjects.

While use of percent PPWS is a good primary outcome measure to determine a difference between baseline and post intervention performance (i.e. low vision rehabilitation or artificial vision device use), it is only one of several assessments that we used. As the subject is walking the course, the number of errors or "collisions" is also recorded. Errors are quantified on a 3 point scale as first described by Marron and Bailey³¹. Errors were scored as 1 point if the subject made contact with an obstacle but was able to correct in ≤5 sec, 2 points if the subject took 5-15 sec to correct errors, and 3 points if the subject took >15 sec to self-correct or required the assistance of one of the research assistants to correct the error³¹. We also have two object identification tasks, both of which are untimed. The first requires the subject to view the course just completed and count the number of obstacles they can detect. The second requires the subjects to navigate the course and point to objects they think are in their path.

We found PPWS to be an appropriate primary outcome measure to determine a difference between baseline and post intervention performance. For our study, this metric reliably demonstrated that subjects slowed down significantly when using the BrainPort, a finding which was confirmed by the fact that subjects scanned their environments (see video). We are currently collecting data on whether PPWS scores can improve after prolonged use of the BrainPort with additional orientation and mobility training. Percent Error data consistently suggested trends for improved performance across every course level. A large gap in function for camera based artificial vision devices is lack of depth information. It is likely that Percent Error outcomes would improve if artificial vision devices had the ability to enable this precept. Indeed, we have conducted pilot studies comparing several vibrotactile canes to the BrainPort as well as studies with multimodal input (BrainPort plus vibrotactile canes) using this obstacle course (data not shown). Preliminary results suggest that use of vibrotactile systems, which can convey depth cues, improve both PPWS and Percent Error Performance. The two tertiary outcomes of obstacle detection can be used to provide depth to the navigation analysis. For example, although Percent Error scores did not improve appreciably, subjects were able to detect if an obstacle was present about half the time, when presumably this would be none of the time for a blind person without an assistive device.

Comment should be made regarding making comparisons between each level of the course. As mentioned in the introduction, each "level" possesses its own unique conditions designed test a specific combination of navigation skills. Therefore, it is important not to conclude that there is a progressive increase in difficulty between levels 1, 2, and 3. For example, there are fewer obstacles to hit in level 3, but more floor and texture changes. We do account for these factors in calculating percent possible error calculations. For any given course, we only count the actual obstacles a subject can hit, but not floor texture changes. For texture or color changes located on the floor, behavioral changes (i.e. hesitations, etc.) are recorded, and are reflected in the PPWS calculation. In the obstacle detection tasks, floor texture and glare "obstacles" are included in the calculation. The specific details for recording are included in the instruction document.

Further studies need to be undertaken to verify if the course iterations are identical within each level for patients with low vision. Several features of the perceptions enabled by the BrainPort may not transfer to patients with remaining sight. For example, when using the BrainPort, lighter high contrast objects are easier to detect than those with low contrast. The device does have an invert function, which can make darker objects stand out against a lighter background. Moreover, because of luminance averaging software, the lighting condition (dim versus bright light) did not make a statistically significant difference in performance with the BrainPort, but we would expect ambient illumination would generally affect performance for persons suffering from diseases such as glaucoma or macular degeneration.

We feel that our course possesses several attributes which make it attractive for both research and clinical purposes compared to existing obstacle avoidance platforms. Most importantly, we found the course to be reproducible. We have two instillations, and there was no difference in performance between sites. Furthermore, the setup is easy to arrange and administer, with average test time taking less than 90 min. The fact that there are a total of 36 possible course permutations makes memorization unlikely to occur even during repeated testing, providing randomization schemes are used. Having both dim and bright light conditions allows for examination of whether ambient illumination is having a negative impact on mobility. Several outcomes measures are possible, including PPWS, percent error, two untimed visual identification tasks, and the ability to analyze according to both the color and the type of obstacles.

Disadvantages of our course include the need to have a hallway that is 40 ft in length that can be painted the same color as the floor tiles, and a storage closet to house the obstacles. It is also helpful if once can permanently install the floor tiles and keep the lights affixed to the ceiling. Once installed, these are both unobtrusive, but depending on the décor of the facility could be noticeable.

In conclusion, we describe a portable, standardized obstacle course tool that has been used to assess some mobility functions for use with artificial vision devices and states of ultra-low vision. The course is relatively inexpensive, simple to administer, and has been shown to be reliable and reproducible. Future work should investigate its usefulness in low vision populations.

Disclosures

The authors have nothing to disclose.