Collision judgment when using an augmented-vision head-mounted display device

Abstract

Purpose

We have developed a device to provide an expanded visual field to patients with tunnel vision by superimposing minified edge images of the wide scene, in which objects appear closer to the heading direction than they really are. We conducted experiments in a virtual environment to determine if users would overestimate collision risks.

Methods

Given simulated scenes of walking or standing with intention to walk towards a given direction (intended walking) in a shopping mall corridor, participants (12 normally sighted and 7 with tunnel vision) reported whether they would collide with obstacles appearing at different offsets from variable walking paths (or intended directions), with and without the device. The collision envelope (CE), a personal space based on perceived collision judgments, and judgment uncertainty (variability of response) were measured. When the device was used, combinations of two image scales (5× minified and 1:1) and two image types (grayscale or edge images) were tested.

Results

Image type did not significantly alter collision judgment (p>0.7). Compared to the without-device baseline, minification did not significantly change the CE of normally sighted subjects for simulated walking (p=0.12), but increased CE by 30% for intended walking (p<0.001). Their uncertainty was not affected by minification (p>0.25). For the patients, neither CE nor uncertainty was affected by minification (p>0.13) in both walking conditions. Baseline CE and uncertainty were greater for patients than normally-sighted subjects in simulated walking (p=0.03), but the two groups were not significantly different in all other conditions.

Conclusion

Users did not substantially overestimate collision risk, as the 5× minified images had only limited impact on collision judgments either during walking or before starting to walk.

1. INTRODUCTION

Diseases such as retinitis pigmentosa, choroideremia, and glaucoma may cause severe peripheral visual field (VF) restriction (known as tunnel vision), which greatly impairs mobility.^1–4 Presenting a minified view of a scene may help the patients cope with the mobility problems arising from a restricted VF. Minifying devices proposed for this purpose include handheld divergent lenses,⁵ reversed telescopes,^{6, 7} amorphic lenses (reversed telescopes that minify only horizontally) in constant use⁸ or in bioptic configuration,⁹ and a video remapper (head mounted display that presents minified images captured by a video camera).¹⁰ However, when the VF is expanded with these visual aids, the ability of central vision to discern fine details (e.g. reading small letters) is compromised.

Peli and colleagues^{11, 12} have developed an augmented-vision head-mounted display (HMD) system based on a principle of spatial multiplexing, whereby minified edge images of the ambient scene are superimposed over the wearer's natural view seen through an optical see-through display. Because edge pixels in the display occupy only a very small portion of the field of view, the edge images minimally occlude the wearer's see-through view. Thus, the image quality of the see-through view is almost as good as the unobstructed view, yet the device can provide additional information about the peripheral scene to the wearer. Although the peripheral information provided by the minified edge images is coarse, we have shown that it can reduce search time and increase directness of gaze path of patients when they look for objects outside their VFs.¹³ Figure 1 shows such an augmented view provided by the HMD device. It is a photograph captured directly through the augmented-vision spectacle lens. The superimposed edge image is about 5× minified. It allows a patient with a small VF, e.g. 15°, to look forward at the door and still be able to notice the presence of a person on the left and the trash bin on the right. The 16°×12° white rectangle is the boundary of the see-through display and represents an 80°×60° field in the real scene.

Figure 1.

A photograph captured through an augmented-vision HMD spectacle lens. From the superimposed edge image (16°×12°), a patient with tunnel vision looking ahead can detect the woman and the trash bin that otherwise would not be seen. Note the presented visual directions of these potential obstacles may be much closer to current gaze point (marked by a white circle) than they really are.

As the augmented-vision HMD device is see-through, it does not obstruct the view of the real scene. Thus, collision judgments can be made by first detecting an obstacle in the minified edge view and then looking at the real object through that transparent display. However, it is preferable for the judgment to be based on the minified edge view alone. If patients can make effective judgments from the minified view only, without the need to move their eyes or heads towards real objects, collision-avoidance responses would be faster and the device may have wider acceptance. When using the minified edge view, perceived directions of objects may appear much closer to gaze points than they really are. A major concern about using this device was that this change in presented visual direction could be detrimental to collision judgment. If patients overestimate the risk of collision with objects seen in the minified edge view, it could provoke many unnecessary collision-avoidance maneuvers, and may discourage use of the device. To address this concern, we conducted a collision judgment study in a safe and controlled virtual environment. The major question, whether minified edge images had a substantial impact on collision judgment, was examined in normally-sighted subjects and tunnel-vision patients by measuring the perceived likelihood of collision.

2. METHODS

Normally-sighted subjects and patients with tunnel vision performed a collision judgment task while standing before a large virtual environment display. Judgments were made with a natural view of the scene (baseline) or with the augmented-vision device. The augmented-vision view was either 1:1 scaled or minified in either grey scale or edge image type. The presented scene either simulated a walk through a shopping mall corridor or the intention to walk, in which case, the scene was stationary. In all cases, the judgment was whether the participant would collide with an adult-human size obstacle.

2.1 Virtual obstacle course

We have created a virtual shopping mall corridor using the World Toolkit (Sense8, Mill Valley, CA).^{14, 15} A 172×127cm rear projection screen presented a photo-realistic representation of a shopping mall at a resolution of 1280×960 pixels. Participants stood at 77cm from the screen, thus it subtended a visual angle of 96°×79°. The virtual environment comprised an infinitely-long and 10.4-meter wide corridor with tile-textured floor, and sidewalls that were textured with pictures of storefronts in a real shopping mall. Objects such as benches and vending machines were also added to the corridor. Scenes presented on the projection screen simulated a view seen by a participant walking at speed of 1.5m/s (or about to walk) in preset directions. The task of participants was to report whether they would collide with stationary obstacles that appeared in the virtual corridor.

The virtual obstacle course was a preset zigzag path through the corridor, over which the participants had no control. Each trial consisted of one straight segment of the path (Figure 2). During each trial of the simulated-walk experiment, the motion scene was presented for 2 seconds for participants to establish a sense of heading direction, and then a stationary human-sized obstacle (0.7m wide×0.7m deep×2m tall) appeared 5m ahead, and disappeared 1 second later when the participant was 3.5m from it. During each trial of the intended-walk experiment, the obstacle appeared 3.5m away for 3 seconds, and then disappeared.

Figure 2.

Illustration of the experimental layout for one trial. Participants “walked” or “intended to walk” a zig-zag path in a virtual shopping mall corridor. Each trial consisted of one straight segment of the path. Obstacles appeared at variable offsets from the walking path. Participants reported whether they would make contact with the obstacles if they continued walking in the same direction (simulated-walk experiment), or were going to walk towards a marked direction (intended-walk experiment).

The obstacles were placed at different path offsets (Path offset was defined as the distance from the closest boundary of the obstacle to the walking path. An obstacle that obstructs the walking path has a negative offset, as the closest boundary is on the opposite side of the walking path from its center.). For each participant, 44 tested path offsets were evenly distributed from −20cm to +120cm on each side (left and right, total 88 trials). Participants reported verbally whether they would have made any contact with the obstacle if they had continued on the same trajectory (path). In the simulated-walk experiment, participants could interpret the heading based on motion cues, and in the intended-walk experiment, the direction that they were about to walk towards was indicated by a thin, green, vertical line. The question to participants was whether they believed that they would make any physical contact with the obstacle if they continued walking in the same direction (simulated-walk experiment) or walked towards the given direction (intended-walk experiment). The participants made forced choices even if they were uncertain about their decisions.

2.2 Test Conditions

We were interested in how participants would make collision judgments when looking at minified images of stationary obstacles. The ability to judge potential collisions was first tested in a simulated walking scenario, and then in a simulated intended-walk scenario. We anticipated that participants could use motion information in the simulated-walk condition. However, motion information is not available before a person starts walking. If the device users hesitate to start because of an inability to estimate collision risk, they will not have any opportunity to use motion information, even if it would be useful. In the real world people know their intended walking direction, so in the intended-walk experiment we provided an indication of the intended direction by placing a vertical line on the screen.

For the simulated-walk experiment, the apparent walking speed was 1.5 m/s while the participants actually stood still (to limit the impact of fatigue). Each participant was tested with and without the augmented-vision device. When the device was used, combinations of two image types (gray-scale or edge image) and two image scales (5× minified or 1:1) were tested. The 1:1 scale was achieved by using a camera lens with the same field of view as the display. As we did not find any effect of image type (see Results section), we only used the edge image type with the two scales in the subsequent intended-walk experiment. In the minified image conditions, the participants were able to view the equivalent of an 80°×60° field of view covering most of the projection screen. In 1:1 scale conditions, the participants had a 16°×12° field of view of the screen, so they often needed to move their heads to scan the screen to spot obstacles. In this condition, it usually took longer for the participants to find obstacles due to restricted field of view. If the obstacle was not detected, the trial was repeated until the obstacle was detected. The order of conditions was randomized across participants.

Since the augmented-vision device presents images to only one eye, participants were tested monocularly. Each participant used the same eye in all conditions. When wearing the device, see-through views of both eyes were blocked and participants could only see the images in the display with one eye (Figure 3). In the without-device conditions (baseline), the eye that would not be fitted with display in the with-device condition was patched. We use the term “display/viewing side” to refer to those obstacles presented to the same side of the heading direction as the open eye (baseline) and the eye fit with display. We use the term “camera/patched side” to refer to objects on the side of the always-covered eye. For participants who needed correction, we placed closely-matched, stick-on Fresnel lenses on the plano display lens.

Figure 3.

(a) A participant wearing the augmented-vision device. (b) The see-through view of both eyes was blocked when using the device and the camera was exposed through the cut out. (c) A typical view in the simulated-walk experiment when using the device in 1:1 grayscale image mode. Note the limited field of view (16°×12°). (d) View in the intended-walk experiment when using the device in its minified edge image mode. Note the vertical line indicating the intended direction of heading. The field of view of the device was 16°×12°.

2.3 Participants

Twelve normally sighted subjects participated in the simulated-walk experiment (age: 22 to 58 yr; visual acuity of tested eyes: 20/15 to 20/26; 6 males). Twelve normally sighted subjects participated in the intended-walk experiment (age: 24 to 62 yr; visual acuity of tested eyes: 20/15 to 20/26; 8 males). Eight subjects participated in both experiments. Six subjects used the left eye in the simulated-walk experiment, and 5 subjects used the left eye in the intended-walk experiment. Fifteen of the 16 normally sighted subjects had very limited experience with the augmented-vision device, or no prior experience at all. They were given less than 10 minutes to be familiar with the device use prior to the study.

Seven male patients with tunnel vision (due to retinitis pigmentosa or choroideremia) participated in both experiments (age: 40 to 67 yr; visual acuity of tested eyes: 20/26 to 20/66; maximum horizontal width of the VF of viewing eye: 6 to 16 deg.). Five patients used their left eye. No patient had residual peripheral islands (Goldmann V4e). Most patients had a VF that was smaller than the display, so they had to scan to detect the obstacle in all viewing conditions. Six of the 7 patients had prior experience with the augmented vision device as participants in our previous studies. ^{13, 16} The patient without prior experience spent about 30 minutes becoming familiar with the device before testing.

The research followed the tenets of the declaration of Helsinki. Participants signed an informed consent form approved by the Schepens IRB.

2.4 Data analysis

For each participant, the yes/no responses were assigned numeric values (yes=1, no=0), and these values at different path offsets were fitted with a cumulative Gaussian function separately for the right and left obstacles (Figure 4). We defined the mean of the corresponding Gaussian functions as the perceived safe passing distance (PSPD) for left or right obstacles, and the standard deviations of the Gaussian functions were defined as the judgment uncertainty.¹⁷ Collision envelope size (CE) was calculated by summing the PSPD for both sides. The CE reflects the size of a space within which the likelihood that an obstacle was judged to cause collisions is at least 50% (not actual collision chance). We have also analyzed the same response data using a safer criterion for PSPD, 20% point of the cumulative Gaussian function, and the outcomes were not substantially different from those using the 50% point.

Figure 4.

Illustration of the calculation of perceived safe passing distance (PSPD) and collision envelope (CE) based on participants' collision judgment responses. Cumulative Gaussian functions were fitted to the response data separately for the right and left obstacles. CE is the sum of PSPD for both sides.

Repeated measures ANOVAs and paired t-tests were primarily used in statistical analyses for normally-sighted subjects. As the number of tunnel-vision patients was small, we used Wilcoxon nonparametric test for patients. When the two groups were compared, we used the two-sample Kolmogorov-Smirnov non-parametric test. Effects with significance, p < 0.05, were considered to be statistically significant. SPSS 11.5 (SPSS, Chicago, IL) was used to conduct the statistical analyses.

3. RESULTS

3.1 Normally sighted subjects

To test the main hypothesis, that use of the minified view would substantially increase perceived collision risk, we compared CE without the device (baseline) to the CE with the device. When the augmented-vision device was used in the simulated-walk experiment, neither image scale (minified or 1:1; repeated measure ANOVA, F_1,10=0.02, p=0.89) nor image type (gray-scale or edge; F_1,10=0.16, p=0.70) had an effect on the CE size. Therefore, CE data for edge and gray-scale image types were combined in the subsequent analyses, and are plotted beside the baseline CE (without the device) in Figure 5a. Compared with the baseline CE (74cm), CE with the device was not significantly different either for minified (paired T test, T₁₁=1.7, p=0.12) or 1:1 scaled (T₁₁=1.4, p=0.18) images, although it seemed to slightly increase by 12% to 83cm, and 14% to 84cm respectively. In the intended-walk experiment, however, as shown in Figure 5b, the device significantly increased CE by 30% for minified scale (111 cm; T₁₁=6.9, p<0.001) and by 11% for 1:1 scaled images (T₁₁=2.9, p=0.01) compared to baseline CE (86cm). These increases are small compared to the 500% minification factor, and the small increases would be expected to result in actions that were a bit more conservative.

Figure 5.

The CE of normally-sighted subjects without the augmented-vision device (baseline) and with the device in 5× minified and natural scales. (a) Simulated-walk experiment (n=12): Image type had no effect, so data for edge and grayscale images were combined here. CE with the device was not significantly different from that without it. (b) Intended-walk experiment (n=12): CE when using the device in minified edge-image mode significantly increased by 30%, and in 1:1 scaled edge-image mode increased by 11% compared to baseline. Error bars are standard error of the mean.

To test the hypothesis that use of the minified view would substantially degrade the ability to make collision-risk judgments, we compared judgment uncertainty without the device to that with the device. In the simulated-walk experiment, image type (gray scale or edge) did not affect judgment uncertainty (F_1,10=0.02, p=0.88), so the data for these conditions was combined. Since judgment uncertainty was not significantly different between sides in either the simulated-walk (F_1,10=3.8, p=0.08) or the intended-walk (F_1,10=0.12, p=0.74) experiment, the judgment uncertainty was averaged for both sides. Figure 6 shows the judgment uncertainty in the two experiments. Overall, the judgment uncertainty was significantly smaller in the intended-walk experiment than in the simulated-walk experiment (F_1,22=41.6, p<0.001), because heading direction was provided in the intended-walk condition.¹⁷ Compared with the baseline condition, only when using the device in 1:1 scaled image mode in the simulated-walk experiment did the judgment uncertainty significantly increase by 4.8 cm (T₁₁=2.8, p=0.02). The device in minified mode in the simulated-walk condition (T₁₁=1.2, p=0.25) and the device in either scale (T₁₁<=0.3, p>0.75) in the intended-walk condition did not significantly change the judgment uncertainty.

Figure 6.

Judgment uncertainty of normally sighted subjects with and without the augmented-vision device in the (a) simulated-walk (n=12) and (b) intended-walk experiments (n=12). A lower the value indicates a more consistent collision judgment. The only significant change from baseline was the 4.8 cm uncertainty increase when using the device in the natural scale mode in the simulated-walk experiment. Overall, the judgment uncertainty was significantly lower in the intended-walk experiment than in the simulated-walk experiment. Error bars are standard error of mean.

3.2 Patients with tunnel vision

Like the normally-sighted subjects, use of the minified view did not substantially affect collision judgments for CE or judgment uncertainty of the patients with tunnel vision. The CE and judgment uncertainty of the seven patients are plotted in Figure 7 and Figure 8, respectively. CE based on minified edge images was not significantly different from their baseline (Wilcoxon signed rank, z₆=0.34, p=0.74) for both simulated-walk and intended-walk experiments. Like normally sighted-subjects, patients had larger judgment uncertainty in the simulated-walk than in intended-walk experiment (Repeated measures ANOVA, F_1,6=42.7, p=0.001). Device use did not significantly affect the uncertainty compared to baseline in either the simulated-walk (z₆<=1.2, p>=0.24) or the intended-walk (z₆<=1.5, p>=0.13) experiments.

Figure 7.

CE of the tunnel-vision patients (n=7) without using the augmented-vision device (baseline) and with the device in minified or natural scales. CE based on minified edge images was not significantly different from the baseline in either the simulated-walk or the intended-walk experiment. Error bars are standard error of mean.

Figure 8.

Judgment uncertainty of the tunnel-vision patients (n=7) with and without the augmented-vision device in the simulated-walk and intended-walk experiments. The uncertainty was significantly larger in the simulated-walk than in the intended-walk experiment. Error bars are standard error of mean.

Two-sample Kolmogorov-Smirnov tests were conducted to compare normally-sighted subjects and patients for each device-use condition. Patients had a significantly larger baseline CE (105cm) than normally sighted subjects (CE: 74cm, p=0.03) and a larger baseline judgment uncertainty (12cm) than normally sighted subjects (7cm, p=0.03) in the simulated-walk experiment. However, for all the other conditions (simulated-walk with device and intended-walk) the two groups were not significantly different in either CE or uncertainty (p>0.11).

3.3 PSPD and walking conditions

For all the participants in this study, there was an interesting difference in the PSPD between the two walking conditions. As shown in Figure 9, the PSPD was asymmetric in the simulated-walk experiment, i.e. it was significantly smaller on the display/viewing side (F_1,17=14.9, p=0.001). However, in the intended-walk experiment there was no significant difference between viewing and patched sides (F_1,17=0.03, p=0.87). In either walking condition, there was no difference in PSPD between left and right eye (F_1,17<=0.2, p>=0.66).

Figure 9.

Perceived safe passing distance (PSPD) of all subjects in the (a) simulated-walk (n=19) and (b) intended-walk experiments (n=19). PSPD was asymmetric in the simulated-walk experiment but not in the intended-walk experiment. PSPD was not different between left and right eyes used in either experiment. Error bars are standard error of mean.

4. DISCUSSION

This study investigated the impact of the augmented-vision HMD device on collision judgments. Our main finding is that viewing 5× minified images did not substantially change collision judgment compared to baseline for both normally-sighted subjects and tunnel-vision patients. The larger baseline CE (105cm) of the patients compared to the normally-sighted subjects (74cm) might be because they are usually more cautious about the risk of collisions than are people with normal sight. Objects within the range of their long cane swing, which is usually wider than body width, are high-risk obstacles for them. Except for this difference, the two groups were not significantly different when using the device. As 5× minification did not significantly affect the CE or judgment uncertainty of the patients (i.e. collision judgment was similar to that with direct viewing) and the image type (grayscale or edge) did not have any effect, it suggests that minified edge images can provide users with information that is as useful for collision judgment as natural color images. Therefore, the augmented-vision concept implemented in this device is promising in this regard.

It is not clear from this study why the participants' collision judgment abilities were only minimally affected by 5× minification. The absence of an effect of minification in the simulated-walk condition may be explained by Regan and Kaushal's direction discrimination model.¹⁸ They showed that the obstacle offset can be estimated from the ratio between the translational velocity of the obstacle and its expansion rate (e.g. very small ratios correspond to likely collisions). Because the translational velocity and expansion rate of obstacles decrease proportionally with minification, the ratio between them does not change. Therefore, collision judgments with the device could be as good as without the device (provided that the obstacles are large enough for the estimation of translation and expansion in the minified view).

In the intended-walk experiment, no motion information was available. We think that the participants normalized the scale of the scene and then determined obstacle offset. Correctly associating body with the environment (where image size of the same object is different depending on distance) is a capability humans develop as they grow. ¹⁹ When asked how they made judgments in minification conditions, all participants reported that they imagined themselves scaled down in the “small” corridor. As the clues to scale down were limited, it seemed that normally sighted subjects erred towards safer judgments, increasing their CE by 30% (Figure 5). We believe that the collision judgment in the intended-walk condition could be reliable if people use a simple skill – using the human-sized obstacle as a gauge to estimate how far away it was from the intended direction. One of the participants reported that he used this strategy, and his CE with 5× minification increased only 2% from baseline (128cm to 131 cm). Of course, this skill will work well only for objects with known size. If an object is not recognized or its size is not known, it can not be reliably used as a gauge.

In our explanations proposed above, the visual information needed for collision judgment can be derived from central vision. This is probably the primary reason why the CE of the patients was not greatly different from that of the normally-sighted subjects (they only differed in baseline CE, which might be related to patients' cautiousness, as mentioned above), and why the judgments of normally-sighted subjects did not change much whether they had full monocular field of view or restricted field of view of 16°×12°.

The see-through view was blocked in our experiment, while real users of the augmented-vision HMD device will have two superimposed views (the minified edge image view and the see-through natural view). This might affect their ability to judge collisions to some degree. However, we do not expect the impact to be large, as the two views are so distinct that they may be easily separated perceptually. In a previous study, ²⁰ even without depth difference, participants could easily ignore one view while they attended to another different view, where edge images were superimposed on color images and presented on screen. When motion cues are available, even if a user might misinterpret objects in the edge image to belong to the see-through view, potential collisions can be detected correctly, as we have shown that image scale does not affect judgment (Figure 5a and Figure 7). However, users must differentiate the minified view and see-through view when no motion information is available. Further studies are needed to investigate collision judgment when both views are seen simultaneously.

It is interesting to note that PSPD was asymmetric in the simulated-walk experiment, but appeared symmetric in the intended-walk experiment (Figure 9). The asymmetry in PSPD demonstrates that our testing paradigm was capable of measuring small changes in collision judgment. This also provides evidence that participants used different information in the two experiments. The asymmetry in the simulated-walk experiment did not seem to relate to the offset of the camera (mounted on the opposite side of the display), because a similar asymmetry existed in the baseline condition, where the eye viewed directly. A possible explanation for the asymmetry is illustrated in Figure 10. For two obstacles at the same path offset, the obstacle on the viewing eye side is further away from the egocenter than the obstacle on the occluded eye side. If the distance between the egocenter and the viewing eye is taken into account by the visual system (e.g. body self-awareness), the participant might require a larger safety margin to the occluded eye side and reduce the extra safety margin for the viewing eye side accordingly. Thus, obstacles with symmetric path offset for the egocenter appear to be at different distances from the viewing eye (i.e. smaller on the viewing eye side and larger on the occluded eye side). According to this explanation, the difference between PSPDs for both sides should be equal to the inter-pupillary distance. The actual difference in the baseline condition was found to be 5.4cm, which is not significantly different from the typical inter-pupillary distance of 6cm (T₁₈=0.2, p=0.83). In the intended-walk experiment the heading direction was clearly indicated by a vertical line. The participants only needed to judge how far obstacles were from that direction. So they did not need to make compensation for monocular viewing. Thus, the PSPD judgment should be expected to be symmetric, as was found. This hypothesis needs to be confirmed by further investigation.

Figure 10.

Explanation of the asymmetric PSPD in the simulated-walk experiment. The solid-line rectangles represent two obstacles at equal path offset for the viewing eye (dot-dash line). Since the obstacle on the occluded eye side is actually closer to the egocenter (shown as a dashed line), the participant might require a larger safety margin to this side. Similarly, the safety margin on the viewing eye side may be reduced, as the obstacle on this side is further from the egocenter. Thus, obstacles with a symmetric path offset for the egocenter (illustrated by the dotted-line rectangles) appear asymmetric from the viewing eye.

A concern for the augmented-vision HMD device stemmed from the mounting the camera on the opposite side of the head from the monocular display. This camera mounting was dictated by cosmetics and to balance the weight. The parallax resulting from the camera placement might affect collision judgments. As the parallax was not simulated in our study, the 2D presentation of the virtual world appeared the same to the camera and the viewing eye despite the offset. Thus, the PSPD asymmetry was present in both with and without device conditions. In the real world, the perspectives from the camera and viewing eye are slightly different depending on distance. The effect of such a parallax on collision judgment should be investigated in future studies. However, we do not expect any effect of the 10cm parallax would be large.

In conclusion, using minified edge images, as provided by the augmented-vision HMD device, did not have a substantial impact on collision judgment. Thus, the device has the potential to provide mobility assistance to people with severely-restricted peripheral vision.