DEVELOPMENT AND EVALUATION OF VISION MULTIPLEXING DEVICES FOR VISION IMPAIRMENTS

Abstract

We have proposed a novel concept of vision multiplexing for the visually impaired – superimposing contour images over their natural view of a scene or over original images presented on a TV screen. We have implemented the concept using an optical see-through head mounted display to provide either visual field expansion to patients with peripheral field loss or vision enhancement to patients with central vision loss. We have also implemented the concept in a video player with dynamic magnification for patients with central vision loss. Results of initial testing in the lab and on the street are promising.

1. Background

With normal vision we enjoy the benefits of a wide field of view, primarily used for navigation, orientation, and obstacle avoidance, together with high-resolution capabilities that enable discrimination of fine details. Visual impairments due to disease or injury usually affect only one of these aspects, either restricting the wide peripheral visual field (VF) in conditions such as retinitis pigmentosa (RP) and glaucoma, or damaging the high-resolution fovea in conditions such as age-related macular degeneration (AMD).

When peripheral VF loss is severe (leaving useful VFs of less than 20° in diameter – tunnel vision), a patient’s mobility can be affected due to reduced ability to spot obstacles and difficulties in navigation. When patients experience central vision loss, the ability to discriminate fine detail is affected due to an increase in contrast threshold for high-frequency patterns, which causes difficulties with reading regular-sized print and recognizing faces and watching TV.

Traditional optical devices for the visually impaired restore, at least partially, the lost visual ability, but at a high cost for the remaining functionality. For example, magnification improves resolution but inherently limits the field of view. Similarly, minifying devices increase the field of view, but cause a loss of resolution in addition to spatial distortions, and restriction of free-scanning eye movements. Our novel design approach, called “vision multiplexing”, attempts to avoid or reduce these limitations by combining both the wide field-of-view and the high-resolution capabilities in devices in ways that permit these functionalities to be both separable and useful¹. Augmented vision systems lend themselves to vision multiplexing. In this paper we report on the development and preliminary evaluations of two augmented vision systems that implement spatial multiplexing via superposition¹ (See Fig. 1).

Fig. 1.

Simulation of the appearance of spatial vision multiplexing. (a) A street-crossing scene as it might appear to a patient with tunnel vision using the augmented vision system that provides an expanded VF. The woman’s head is seen in full resolution through the display as well as in the superimposed minified contour image. The contour (edge-image) provides a wide field of view enabling detection of the pedestrian (on the left) crossing in the other direction (a potential collision that would not be visible without the display). (b) Illustration of the effect of enhancement by superimposing 1:1 scaled edge images of the scene over the natural view, an application that may be beneficial for patients with central vision loss.

The first system incorporates a head-mounted display (HMD) and has applications for patients either with tunnel vision or central vision loss. Contour images derived from a miniature video camera are presented in an optical see-through HMD. For patients with tunnel vision, a camera with a wider field of view than that of the HMD is used to provide an expanded field of view. The minified edge images enable the patients to see and detect potential obstacles and other objects that, without the minification, would fall outside of their residual VF (Fig. 1a). Since the edges occupy only a small portion of the display, the users can see through the transparent display. Therefore, once an object is detected in the HMD, it can be examined through the transparent display and be seen in full resolution and color. Thus a street sign may be detected in the minified contour display and then read through it. For patients with central vision loss, a camera with the same field of view as the HMD can be used to provide high contrast contour images that are precisely aligned with the see-through view (Fig 1b.). The addition of the contours to the natural vision can result in increased local contrast of features and an increase in the spatial bandwidth of images. The presentation of the contour images on a transparent see-through display enables patients with tunnel vision to maintain natural, high resolution central vision (while benefiting from the expanded field of view), and patients with central vision loss to maintain intact peripheral vision for search and navigation (while benefitting from the central vision enhancement). The development and testing of the HMD field expander for patients with tunnel vision is reported in section 2, and the testing of the on-axis HMD for central vision enhancement is described in section 3.

The above application for central vision enhancement does not include magnification; however, the same vision multiplexing concept may also help patients with central vision loss when magnification is used. A superimposed, 1:1 scale contour image could be used to restore the reduced field-of-view that occurs when magnification is applied. The implementation of this second spatial multiplexing system in a video player that provides dynamic magnification of display is described in section 4.

2. HMD field expander

2.1. HMD development

Early prototypes of the augmented vision HMD field expander were evaluated by 2 RP patients². The VF expansion and vision multiplexing potential of the system were demonstrated and the patients, both with severely restricted VFs, felt that the augmented-vision concept could be useful for navigation, obstacle avoidance, and hazard prevention. They did not find the small-field display to be restrictive, provided the design was open and allowed clearance around the display³. They also preferred a relatively high level of minification and minimal user control (automated system). A number of designs and experimental iterations then served to refine the carrier lens and the frame to ensure that the device will fit securely and comfortably on people with varied facial dimensions. Compared to the early prototypes, the current generation of HMD (Fig. 2) developed with MicroOptical Eng., Inc. (Westwood, MA), represents a substantial improvement in cosmetics and ergonomics (reduced weight and better fit). In addition it provides a vertically expanded exit pupil that serves to compensate for the up-and-down movement of the display during walking. The current field of view of the display is 16°(H) by 12°(V).

Fig. 2.

The current generation of augmented vision HMD field expander for patients with tunnel vision. The wide-angle image captured by the video camera is processed by the controller to provide edge contour images of the scene. The edge images are displayed on the see-through display providing an expanded view. Once an object is detected from the minified edge images, it can be examined with full resolution and color through the transparent display.

A number of off-the-shelf video cameras were tested. We chose the Supercircuits PC182XS camera equipped with Sony’s Ex-View CCD chipset (80° field of view), and re-assembled it in a customized enclosure. This camera is currently used for both the day and night time application of the system to provide 5.0× field expansions. In early prototypes, edge detection at the rate of 30 frames/sec was performed by a dedicated FPGA-based system developed for us by DigiVision (San Diego, CA). This system was powered by a 3lb SLA battery, and was used in the visual search study described in the following section. In the latest system (Fig. 2), the edge detection was implemented on the FPGA (100K gates) of the control box of the display system. The algorithm used a 3 line and 3 pixel difference filter with adjustable threshold. The deck-of-cards sized controller box includes the edge detection functionality, and also drives the camera and display. This controller weighs only 175g including a 2-hour battery. The weight of the spectacle-mounted components is approximately 110g.

Another important improvement of the augmented vision HMD field expander was the inclusion of a center mark in the display. Pilot trials with the device showed that patients with severe tunnel vision might have difficulty locating a real target, even though they could see a contour image of the target in the HMD. We found that with residual VFs much smaller than the display, patients could not determine where within the display the target was located, and therefore they had difficulty in registering the minified view to the real-world view. A center mark was added to the display as a registration aid to help users locate targets in the real world. If the camera and display are spatially aligned using a simple calibration procedure, then when a target contour is noted in the display, moving the head to align the center mark with the target contour image brings the real target into the see-through view.

2.2. Laboratory and outdoor evaluations

A series of studies were conducted to test the value of the HMD field expander for a variety of tasks and conditions. Laboratory studies allowed us to assess prototypes, identify limitations, acquire early users’ feedback, and thereafter improve design and configuration. These studies were interleaved with successive prototype development.

2.2.1. Visual search

Within a controlled environment, visual search tasks resemble some of the visual demands of daily life (e.g., navigation, scanning the environment, and finding objects of interest). Kuyk et al.^{4, 5} reported that scanning ability in a visual search task was one of the main predictors of mobility in visually impaired adults. We conducted a search study similar to Kuyk’s. Subjects with severe tunnel vision searched for targets with and without the augmented vision HMD field expander, and with an auditory cue indicating target direction⁶. The search targets were displayed on a gray blank background in random positions outside of subjects’ residual VFs. Therefore, the targets could not be detected with natural vision when looking straight ahead at a fixation point at the center of the screen (Fig. 3).

Figure 3.

The visual search task. Targets were presented outside subjects’ visual fields. The minified target contour (edge) images seen in the HMD field expander provided cues for both the direction and eccentricity of targets. Auditory direction cues were provided by buzzers around the screen.

Targets were composed of a random low-contrast letter surrounded by a black frame (triangle, square, or circle selected randomly), 3° or 5° in size. Only the target frame could be detected and recognized in the minified contour view. Subjects had to look through the display to view the targets foveally to identify the letter. Two studies were conducted using different search area widths (90°(H) and 66°(H), respectively). Subjects were allowed to move their eyes and heads freely during the search. Head and eye positions were recorded and used to compute the gaze positions, directness of search path, the angular speed of gaze, and to determine the search time to find the target. The directness score for a whole search path was computed as the weighted average of cos(θ), where θ was the angle between the current sample gaze shift and the target direction, weighted by the step length. A perfect path would have a directness of 1.0, regardless of gaze speed. Normally-sighted subjects performing the same task had directness scores of about 0.95.

For the larger search area in the first study, all three subjects found the targets significantly faster using either auditory cues or contour (edge-image) cues than without a cue (Fig. 4a). For the smaller search area in the second study, the contour cues significantly improved the directness score (Fig. 4b), but the search time performances with the HMD field expander were mixed. Six out of the 9 subjects with VFs >10° found targets more quickly with the device, but the 3 with smaller VFs (<10°) were slower. Gaze speed was quite slow when using the HMD field expander (about half the speed without the device), which we believe was associated with lack of device familiarity (subjects had only about an hour of training before the study). Regression analyses suggest that, if the gaze speed increases, the minimum VF required to benefit from the device would be less than the 10° we found⁶. Since the directness of the search improved for all subjects, any increase in gaze speed would then result in reduced search times, even for patients with smaller VFs. In fact the analyses suggest that, in relative terms, patients with smaller VFs would benefit more, but of course their absolute performance would remain lower than those of patients with wider residual VFs.

Fig. 4.

(a) Visual search time of the three subjects in the larger area search. Auditory cues and contour (edge-image) cues from the HMD field expander significantly reduced search time for all subjects. (b) Mean directness of the 9 subjects in the smaller area search. In both studies, directness with either auditory or contour cues was better than without cues.

In this laboratory-based visual search study, targets were presented on a simple blank gray background, which is not representative of the real world. When the contours of targets were within the subjects’ VFs, they could be easily detected from the minified edge-image view. However, this might not be the case if targets were embedded among many distracters. The usefulness of the augmented vision HMD field expander in the real world needs further investigation.

2.2.2. Collision detection

Once a person with tunnel vision detects an object, a judgment regarding potential collision is needed, so that timely avoidance maneuvers can be initiated. When a patient is wearing the augmented vision HMD field expander, the collision risk could be judged from the see-through view. However, it might be preferable if the risk of collision could be judged directly from the minified edge images, as that would speed up the response. The minified edge images may impede the user’s ability to make such judgments accurately and confidently. One concern is that with minification patients might feel that they are going to collide with everything seen in the display. This could cause too many unnecessary collision-avoidance maneuvers. To evaluate the ability to make collision judgments with the minified edge images, we conducted a study in a virtual environment⁷.

Ten normally sighted subjects stood 77cm from a wide (94°(H) ×79°(V)) rear-projection screen that displayed a photo-realistic video representation of a shopping-mall corridor. The movie scene was updated as if a subject was walking at 1.5m/s down a preset path. Each trial consisted of moving down one straight segment of the path and a stationary human-sized obstacle appeared at 5m and stayed for one second. The obstacles were placed at varying distances from the trajectory of the path segment (path offset). For each subject, 44 path offsets were distributed from −20cm to 120cm on each side of the path. Subjects reported verbally whether they would have any contact with the obstacle if they continued on the same trajectory without an avoidance maneuver. The subjects were required to make a choice. Response values at different path offsets were fit to a Gaussian cumulative density function. The mean of the Gaussian representing the perceived safe passing distance (PSPD), was calculated for left and right sides separately, while the standard deviation represents the decision certainty. Each subject performed the task with and without the augmented vision HMD field expander. The see-through views of both eyes were blocked so that subjects could only see the images in the display with one eye. In the without-device condition, the eye that would not be fit with the display in the with-device condition was patched.

As shown in Fig. 5, the minification from the HMD field expander caused a small, but significant increase in the PSPD (p=0.004) for the camera side. The effect was not significant for the display side (p=0.890). The small increase in PSPD (18%) was much smaller than the anticipated effect of the 5× minification factor of the device tested. The difference between the two sides is likely to be a result of the parallax due to the position difference between the display and the camera. There appeared to be a trend for the HMD field expander to reduce judgment certainty, but that effect was not statistically significant (repeated measures ANOVA, F_1,9=3.6; p=0.089). It is encouraging that despite the very small and low resolution images seen in the HMD, subjects’ judgments of potential collisions did not change much compared to the natural viewing condition. This suggests that with training users should be able to use the minified images directly for obstacle avoidance in addition to obstacle detection.

Fig. 5.

An augmented vision HMD device increased the PSPD only on the camera side, but had no effect on the display side. A PSPD asymmetry appeared in both with- and without-device conditions.

2.2.3. Night vision system evaluation

Tunnel vision is often accompanied by night blindness (e.g. in patients with RP), which causes severe difficulties with outdoor nighttime mobility. In fact, many patients with night blindness modify their lifestyle to avoid outdoor mobility after sunset. Two generations of the prototype HMD field expander for nighttime application were evaluated by patients with RP and night blindness. The first study⁸ evaluated an early prototype (LV-3), in which the edge-image mode had not been implemented, and therefore the device displayed gray-scale images. Visual function measurements (acuity, letter contrast sensitivity and VFs) and indoor mobility assessments (high-density obstacle course) were conducted without a device, with the LV-3, and with a commercially-available comparison device (Multi-Vision), which is an opaque HMD showing gray-scale images in 1:1 scale. The measurements were conducted at light levels representative of well-lit (~16 lux) and poorly-lit streets (~2 lux). As expected from the minification, the LV-3 substantially expanded patients’ VFs by 18.4° (or 287%) at 2 lux. Patients preferred its better comfort, fit and lighter weight in comparison to the Multi-Vision device (Fig. 6). However, walking speed was slower and mobility errors were greater with the LV-3 than without a device. Insufficient light sensitivity of the prototype camera limited LV-3 performance, subjects had only brief training in how to use the device, this early prototype did not include a center mark on the device display, and the subjects found it difficult to make use of the gray-scale images superimposed over the see-through view.

Fig. 6.

Median ratings for various aspects of night vision device performance based on experiences during the indoor and outdoor assessments (1= Very Poor; 2= Poor; 3= Fair; 4=Good; 5= Very Good). The first prototype of the augmented vision HMD field expander (LV-3) was highly rated (score of = 4) for comfort and weight. The comparison opaque HMD night vision device (M-V) was highly rated for ease of use and image quality. Error bars represent inter-quartile range.

In response to the feedback received and device limitations noted in the first study, the prototype underwent further development and an improved version (NV-3) was evaluated in a small-scale, extended wear pilot study. The improved prototype incorporated a full implementation of the minified edge-image mode, a software-adjustable center (registration) mark, and the Supercircuits PC182XS camera with better sensitivity at low light levels. Four RP patients participated in the second study, each taking the device home for a minimum of 2 weeks. All patients demonstrated VF expansion with the device when tested at low light levels in the clinic environment. Questionnaire responses and informal observations of mobility during training walks with the device on a specially designed outdoor course⁹ confirmed that patients were able to see the minified edge images superimposed over their natural view, and that with practice they could identify objects from the image and to a limited extent could use the image for orientation and to guide mobility.

These preliminary findings provide evidence that patients with night blindness can use the augmented vision HMD field expander within a real world environment, and derive benefit from the VF expansion provided by the minified edge-images. However, the patients had very poor night vision and therefore, the see-through feature of the HMD did not help much in badly-lit (very dark) areas. We also noted that, even with the better camera, the edge images were degraded at low light levels while the gray-scale images were still useable. We therefore conclude that a better design will automatically switch from edge images at moderate light levels to grey scale images only at very low light levels. Such level could be adjusted individually for each user.

3. On-axis HMD for image enhancement

Our lab has developed and evaluated numerous approaches to image enhancement for patients with central vision loss. We have demonstrated that contrast enhancement applying spatial filtering approaches increases face recognition performance^{10, 11}. We have also demonstrated that superposition of double polarity edges¹² on the original TV image is preferred by visually impaired patients¹³. To enhance the visibility of real world objects for patients with central vision loss, we have proposed and developed an on-axis optical see-through HMD system¹⁴, in which the computed high contrast edges are superimposed on the natural see-through view. Unlike in the augmented vision HMD field expander for patients with tunnel vision, accurate registration is critical in this image enhancement application. A miniature video camera with the same field of view (15°) as the display was aligned with the viewer’s eye in the system, and 1:1-size contour images derived from it were precisely superimposed over the patients’ natural view to create an appearance illustrated in Fig. 1b. To obtain such accurate registration and avoid parallax, the display and the camera shared the same optical path.

The registration error of the on-axis HMD system was evaluated. It showed consistent registration across a wide depth range (Fig. 7.). The registration difference between targets at 3′ and 7′ was only 0.07°. We found the variability in the position of the HMD relative to the eye to be the major cause of registration error. For example, 2mm deviation of eye position relative to the HMD caused 0.06° registration difference. The on-axis alignment of the HMD itself can be very accurate.

Fig. 7.

An illustration of consistent registration performance across a wide range of distances for the on-axis HMD system. Five identical white bar targets were placed at 3, 4, 5, 6, and 7 feet from the HMD system. The registration was purposely shifted down to provide a simultaneously clear view of the real objects and their virtual edge images. This picture was captured with a digital camera aimed through the HMD. Five identical gauges depicted next to the white bars show that the registration was consistent from 3′ to 7′. Note, however, that the edges are dimmer than the white target in this implementation.

Although the registration error of the on-axis HMD system is low, it is not yet ready for evaluations by patients with central vision loss. The contour images are not of sufficient quality as the on-axis optics reduces the light reaching camera, and the miniature camera used in the registration evaluation has limited light sensitivity.

4. Dynamic magnification with vision multiplexing

Patients with central vision loss may face a problem of reduced field-of-view when using magnifying devices designed to help with their primary visual impairment, the loss of resolution. When images on TV screens or computer monitors are magnified, large portions of the scene are eliminated from the display. To improve access to TV for patients with central vision loss, we proposed and implemented an approach of dynamic control of magnified display combined with spatial vision multiplexing^{1, 15}.

Electronic zoom is easy to implement. However, if the image is enlarged around the center of the display (Fig. 8a), the enlarged portion may not include the most relevant part of the scene (Fig. 8b). We proposed that better results would be attained if we selected a point in each video frame that should be centered on the screen when the image is magnified. This point should be selected where the most relevant details are available (e.g. the player with the ball in Fig. 8c). We are using eye movement recordings from normally sighted observers watching videos to determine this point-of-regard for each frame to provide the dynamic magnification aspect of the device.

Fig. 8.

(a) A full frame of a video. (b) A portion of the image enlarged around the center of the frame. (c) The player holding the ball may be the most relevant portion of the scene. (d) Image of another frame is enlarged around the ball player with the edge image of the full frame superimposed.

We recorded the eye movements of 20 normally-sighted subjects as each watched 6 movie clips, totaling 37.5 minutes. More than half of the time the distribution of subject gaze points fell within an area statistic that was less than 12% of the movie scene area¹⁶. That study confirmed that, in general, people with normal sight looked at the same area of the screen most of the time (although we did find small age and gender group differences). Thus it should be possible to obtain a point-of-regard and apply it to the device.

We have developed a video player that can play videos using the point-of-regard data as the center for the magnification. This dynamic magnification may be combined with unmagnified (1:1 scale) contour images in a picture-on-picture fashion to provide a complete outline view that maintains the context of the original scene (Fig. 8d). Based on the outline view, patients may choose when to lower the magnification level to achieve a larger field of view if needed. We are planning evaluations of this device by patients with central vision loss.

5. Discussion

The HMD field expander appears in initial laboratory-based evaluations to enable more efficient visual search, which with more training may provide increased speed of search for tunnel vision patients even with very narrow VFs. The ability to properly judge collisions using the minified edge image carries the promise that users will be able to respond directly to that image initiating early avoidance maneuvers. Both functions should increase the safety and comfort of patients’ mobility.

Active eye-movement scanning is an important strategy by which patients can compensate for their VF loss. It is much more natural, easier, and more effective to move the eyes than to move the head. Despite restricted VFs, eye movements of patients with tunnel vision are not largely confined. In walking, only the horizontal eye position dispersion was found to be smaller than that of normally sighted people¹⁷. Saccade sizes of patients with tunnel vision are similar to those of normally sighted people, and they make many saccades into areas not seen when the eye movement is initiated.

Therefore, a visual aid for tunnel vision should not restrict eye movements. Optical see-through HMD systems with an open design have advantages over opaque HMD systems for daytime use, as patients can move their eyes freely when wearing them. However, because patients with night blindness can not benefit from the see-through view on poorly-lit streets conditions⁸, an opaque HMD presenting a gray-scale image may have advantages over an optical see-through HMD, provided the opaque HMD system has a sufficiently wide field of view to minimally limit eye movements (30°). The multiplexing concept can be implemented in opaque HMD systems. If VF expansion multiplexing is desired, the minified edge images of the wide field can be superimposed on gray-scale intensified night vision images shown in the display, creating a video-augmented view in place of the optical-augmented view of our basic system.

The field restriction inherent to magnification can create artificial tunnel vision when patients with central vision loss use magnifying devices. The same approach used in the HMD field expander can supplement the dynamic magnification of a display, helping the user with reduced resolution keep track of the overall picture while benefiting from the magnification effect.

In noting the well-known interplay of central and peripheral visual functions in integrating vision, the concept of vision multiplexing was developed as a useful guide that has led to a number of new vision rehabilitation approaches and devices proposed here and in other areas. The development and evaluations of the spatial vision multiplexing devices described in this paper provide evidence from laboratory and real-world settings of the promise of the approach.