Recording and automated analysis of naturalistic bioptic driving

Abstract

Purpose

People with moderate central vision loss are legally permitted to drive with a bioptic telescope in 39 US states and the Netherlands, but the safety of bioptic driving remains highly controversial. There is no scientific evidence about bioptic use and its impact on safety. We propose searching for evidence by recording naturalistic driving activities in patients' cars.

Methods

In a pilot study we used an analogue video system to record two bioptic drivers' daily driving activities for 10 and 5 days, respectively. In this technical report, we also describe our novel digital system that collects vehicle maneuver information and enables recording over more extended periods, and discuss our approach to analyzing the vast amount of data.

Results

Our observations of telescope use by the pilot subjects were quite different from their reports in a previous survey. One subject used the telescope only 7 times in nearly 6 hours of driving. For the other subject, the average interval between telescope use was about 2 minutes, and cell phone use in one trip extended the interval to almost 5 minutes. We demonstrate that computerized analysis of lengthy recordings based on video, GPS, acceleration, and black box data can be used to select informative segments for efficient off-line review of naturalistic driving behaviors.

Conclusions

The inconsistency between self reports and objective data as well as infrequent telescope use underscores the importance of recording bioptic driving behaviors in naturalistic conditions over extended periods. We argue that the new recording system is important for understanding bioptic use behaviors and bioptic driving safety.

Introduction

Millions of people worldwide struggle with reduced visual acuity due to retinal disorders, such as age-related macular degeneration, optic atrophy, and diabetic retinopathy. Because of failure to meet the vision requirements for licensing, many people lose their driving privileges. This has a major impact in many developed countries where driving is a major means of transportation. Thirty nine US states have implemented regulations that allow people with moderate visual acuity loss to drive with spectacle-mounted bioptic telescopes.¹ Recently, bioptic driving has become a legal option in The Netherlands.²

Bioptic driving is allowed in some jurisdictions, but not in others, mainly because its safety remains highly controversial. Advocates believe proper bioptic use enables users to drive safely, ^3-5 while opponents argue that telescope use may cause field restriction, and therefore failure to see traffic. Opponents also claim that bioptic drivers use telescopes purely as a means for licensing but do not use them while driving.^{6, 7} Studies that have compared accident rates of bioptic drivers and matched control drivers resulted in inconsistent conclusions – some found that the accident rate of bioptic drivers was less than or same as the general public^{8, 9} while others found bioptic drivers were involved in more accidents.^{10, 11} These studies do not provide any information about how or why the accidents happened. Specifically, it is not known whether the accidents of bioptic drivers occurred when looking through the telescope (which might cause a failure to see traffic in the periphery), or occurred because of failures to look through the telescope (which might result in poor perception of important traffic situations).

Since bioptic telescopes are designed to be used infrequently, conventional on-road driving assessments conducted by driving professionals may be inappropriate or too brief. In addition, existing knowledge and training guidelines are based on self-reports from bioptic drivers and opinions from clinicians, but there is insufficient evidence to support any bioptic usage pattern as optimal or even beneficial. We believe that comprehensive observations of actual bioptic telescope use during daily driving are necessary. Using a video surveillance system, we indeed found that recording naturalistic bioptic driving is possible and may be valuable, as reported here.

Monitoring naturalistic driving has become a trend in driving research, where capture of infrequent events such as accidents¹² and running red lights ¹³ is desirable. It is considered an attractive methodology, as it can avoid the often over-simplified laboratory setting, as well as limitations of the short time sampling of on-road driving assessments.¹⁴ In this paper we first report a pilot experiment demonstrating the feasibility and value of recording daily bioptic driving activities using an in-car video surveillance system. We then present a new recording and analysis system that we developed for naturalistic bioptic driving studies, which was inspired by the findings from the pilot experiment. Driving studies using this new system are on going. Here we only describe the system's functionalities and demonstrate its feasibility.

Naturalistic recording of two bioptic drivers

1. Methods

In a pilot study we installed an analog video recording system in two bioptic drivers' own vehicles for 10 and 5 days, respectively, during which time they performed daily driving as usual, mostly in their local areas for tasks such as driving to work and picking up children from school. A video multiplexer combined videos of the traffic view, the driver's head, and the car interior, and they were recorded on 10-hour video tapes by a VCR. Running on batteries, the system started recording when the vehicle engine started. Video tapes were collected after the system's batteries were depleted. In total, we collected 351 and 290 minutes of video recordings from the two subjects, respectively. We manually reviewed the videos and interpreted telescope use events, which were indicated by quick downward head tilts that aligned the eye with the telescope. To interpret what objects were viewed through the telescope, we visually estimated how far the driver's head was turned from the usual head position, and determined what objects in that direction might be points of interest. Only when there was just one possible point of interest did we mark the object. Therefore, with this method we were able to interpret only 39% of the points of interest viewed with the bioptic telescope.

Our research followed the tenets of the Declaration of Helsinki and was approved by the institutional review board at the Schepens Eye Research Institute.

2. Results

The telescope use events of the two subjects were intermittent and brief, consistent with professionals' recommendations. Such recommendations were based on the premise that the user should look through the telescope for the minimum time necessary to discern the details of interest, and thereby minimize the risk of not seeing traffic due to the restricted field-of-view of the telescope. Subject 1(VA:20/60, 20/25 with telescope) was a bioptic driver with 3 years of experience. She used her telescope only 7 times during the 351 minutes of 55 drives, a little more than once per hour on average – too few occurrences to be analyzed. Subject 2 (VA: 20/50, 20/25 with telescope) was a bioptic driver with 17 years of experience. He used his telescope 204 times during the 290 minutes of 16 drives (once every 1:25 min on average). Among his 204 instances of bioptic use, we were able to confidently estimate the telescope-viewed targets for 80 times. Of these 47% were for vehicles ahead, 13% for road signs, and 40% for examining intersections and upcoming curved road segments.

We observed for subject 2 that cell phone use during driving reduced his telescope use (Figure 1). Among the 16 trips that he drove within 5 days, the average interval between telescope uses was between 1 and 3 min. During one trip to work in which he used his cell phone for 91% of the driving time, the average interval between telescope uses was 4:56 min. This finding is consistent with many reports that mobile phone use has a distracting effect on driving.^{15, 16} Although his telescope use frequency suggested a distracting effect of cell phone use, we also observed that while he was using his cell phone he was able to spot two pedestrians on his side of the road. He then double checked through the telescope, and shifted to the left to keep a safe distance from them (Figure 2).

Figure 1.

Histogram of average telescope use interval (driving time divided by the number of telescope uses in each trip). When the driver used his cell phone, the interval was more than doubled.

Figure 2.

Subject 2 noted pedestrians while he was using the cell phone. (a) Driving on the right side of the road. (b) Shifting to the left to avoid the pedestrians. Note the change in lane marker position.

Both subjects participated in a survey on bioptic use habits¹⁷ about two years before the recording. When asked what percentage of driving time they used their telescopes, they reported 2% and 5%. On the contrary, our recorded data show 0.03% and 1.2%, respectively. These are factors of about 60 and 4 times difference. The previous survey also indicated that reading road signs is the most difficult task without a bioptic, and therefore all participants reported using the telescope to read road signs.¹⁷ Specifically, subject 2 reported reading road signs was somewhat difficult and reading street name signs was extremely difficult. In our study, he did use his telescope frequently for spotting road signs on unfamiliar highways when he was driving home to Vermont after our system was installed in New Hampshire (of the 13% of bioptic uses for road signs, all were for road signs in unfamiliar areas). However, as might be expected, we did not find that he looked at road signs through his telescope during subsequent drives in his local area.

3. Discussion

To our knowledge, this is the first time that bioptic telescope usage has been monitored over extended periods in naturalistic conditions. Previously, data on bioptic use was usually collected by questionnaire survey¹⁷ or observations during on-road driving test situations.^{18, 19} Our findings regarding bioptic driving behaviors from daily driving activities, as well as the large difference between our objective data and subjects' self reports, highlight the value of recording in naturalistic conditions. Yet, the limited data from the two drivers are far from being sufficient to suggest any generalized conclusion regarding bioptic driving behaviors or safety. For instance, it is not possible to interpret the impact of cell phone use, given the fact that subject 2 reduced telescope usage but responded properly to pedestrians on the road while he was on his cell phone. In order to understand what factors contribute to differences in bioptic usage patterns among bioptic drivers, and whether these differences are associated with driving safety, substantially more data recording is necessary.

Using current digital video technologies, recording is not difficult, but analyzing the large amount of recorded data can be challenging. The 10-hour video collected in this experiment took us more than double that time to just review and collect a minimal level of data. For studies involving more subjects, longer recordings, and more detailed analyses, the total time may easily grow to thousands of hours. Therefore, it is important to develop recording systems that allow at least some level of automated processing and analyses. In the next part of this paper we introduce such a system that we developed.

Digital recording system and automated analyses

Inspired by the findings from the pilot experiment, we developed a digital recording system that can be installed in participants' own vehicles and be uninstalled without leaving a mark. In addition to conventional recordings of video and other data (including GPS coordinates, black box and acceleration measurements), the system also includes an infrared head tracking unit that can be used for automatic detection of bioptic telescope use and for the determination of the object of regard during such use.

1. Recording system

As shown in Figure 3, the recording system includes four windshield-mounted micro video cameras. Two side-by-side cameras mounted in front of the rear-view mirror combine to provide an 88°×33° forward view of traffic and road conditions, which matches the wide horizontal and restricted vertical view the driver has through the windshield (about 90°×30°)²⁰. One camera mounted on the passenger side allows observation of drivers' behaviors. This camera has a customized exposure control to deal with the dramatically varied outdoor illumination. With conventional cameras the exposure control is based on the overall scene luminance, therefore, when outside is very bright and the driver's head is shaded, the view of the driver's eye and telescope is often too dark to be seen. This customized camera has 16 selectable areas for exposure control, and we set it to adjust exposure only based on the area where the telescope is expected (as indicated by the dashed rectangle in Figure 3). Thus, under any light level outside the vehicle, the driver's eye and the telescope are always recorded clearly. This enables us to easily observe the driver's bioptic use behaviors, for instance verifying if the eye is aligned with telescope when the head tilts down. An infrared camera is mounted on the driver side for head movement tracking. That camera is supplemented by an active infrared light source and a narrow band pass infrared filter to minimize the impact of ambient light change on head tracking. A digital video recorder in the trunk records the four-channels of videos in synchrony. It also records GPS coordinates at 0.5Hz, XYZ acceleration data at 30Hz, vehicle speed, and multiple ON/OFF signals, including turn signals and braking.

Figure 3.

The digital in-car camera system is comprised of two side-by-side scene cameras for the road view and two for the driver's head, all mounted on the windshield. A digital video recorder (DVR) is installed in the trunk of the car. The system also records GPS, XYZ acceleration, and vehicle black box data. The head view camera has customized exposure control based on the luminance within the dashed rectangle, which ensures that the eye and the telescope are always clear, even though other, unimportant, areas may be saturated (as shown).

2. Computerized analyses

Our ultimate goals are to assess the safety of bioptic driving, distinguish good and poor practices, and eventually develop training guidelines that will make it possible for more bioptic users to drive safely. While accident data are considered a gold standard for safety, accidents are very rare and not well documented. For example, the high profile 100-car study captured 69 crashes (mostly minor physical contact and only 12 reported to the police) with 43,000-hour and 2-million-mile recording from 241 drivers over one year¹². This amounts to an average of 0.29 accidents per driver per year. It is clearly impractical to conduct driving assessment studies based solely on such low-occurrence data/incidents. We believe that off-line assessment of general driving performance by driving instructors based on naturalistic driving can reliably identify drivers' skills, and possibly predict safety. However, recording of naturalistic driving usually generates too much data to be easily reviewed. It is necessary to manually review only important segments rather than the full recordings, most of which contains nothing of interest. We therefore developed a set of techniques and tools to automatically and objectively identify informative and important segments based on automated analysis of geographical locations, vehicle maneuvers and driver behaviors. In this section we describe the system and its intended use with illustrative data.

(A) GPS

As most accidents occur at intersections, observing behavior when driving through intersections is important for assessment of driving skills.^{21, 22} We are able to identify these segments using the GPS recording and geographic information database. We have acquired a database of the geographic coordinates of all of the 654,491 intersections and interchanges in the New England area (Massachusetts, Vermont, New Hampshire, Maine, Connecticut, and Rhode Island). The database also includes road class information (highway, major road, local street, etc.) that allow us to select intersections by road class for different analyses (Figure 4). Since our recorded GPS data are saved along with a time tag, the corresponding video segments can be extracted and reviewed.

Figure 4.

A computer program has been developed to identify, from GPS recordings, segments where the vehicle crosses intersections. The class of roads or intersections can be specified. For example, we can select only intersections with limited access roads (pins on the left figure), major road intersections (circle markers on the middle figure), or minor streets (square markers on the right figure).

Taking turns generally requires careful visual examination of the environment, which may be difficult for some drivers with low vision. Driving through curves was reported as one of the major problems by some bioptic drivers, especially when driving at high speed.²³ Therefore, it may be of interest to evaluate how bioptic drivers behave and perform when driving through these segments. By analyzing the geometric shapes of recorded driving routes, turns and curved road segments can be identified by our program (Figure 5).

Figure 5.

Automatic detection of turns and curved road segments in driving routes (Driving direction is from top of the figure). R: right turn; L: left turn; C: curved road.

(B) Acceleration

Maintaining lane position is an important driving skill which may be adversely affected by bioptic use. On the other hand, steering changes may also occur in response to certain traffic situations (e.g. swerving to avoid pedestrians, animals, or objects on the road, see Figure 2), which require awareness of surrounding situations, correct judgment, sufficient vision and appropriate operation of the vehicle. Segments with steering changes can be detected based on lateral acceleration changes. For instance, a weaving maneuver can be located by detecting a single sine wave in the lateral acceleration signal (Figure 6). It is possible then to determine if the cause of swerving is due to the use of bioptic or possibly the lack of bioptic use.

Figure 6.

Detection of weaving based on lateral acceleration fluctuations. These figures show an example where the car veered to the left and then repositioned. This event is featured by a single sine wave in the lateral acceleration signal (left figure). Note the shifting horizontal position of the lane marker at the bottom of the video images (right figure). The times of these video frames are indicated on the acceleration plot on the left.

Similarly, sudden stops may indicate (near) crashes or responses to urgent situations, and provide important information about the driver's skill.²⁴ These segments can be easily detected by looking for forward deceleration that exceeds a certain threshold.

(C) Bioptic telescope tracking

While looking through the telescope can be an infrequent event, it is of the utmost interest for bioptic driving studies. We have developed a tracking technique that allows us to detect telescope use, as indicated by a quick downward head movement. Furthermore, by tracking three infrared retro-reflective markers pasted on the bioptic spectacle frame, we are able to estimate the telescope aiming point registered on scene images. Only one-time calibration is required when the recording system is installed. The accuracy is not affected by the driver's body movements because the system actually tracks the telescope rather than the driver. See our published paper²⁵ for details about this technique. The tracking accuracy of about 1° over a wide depth range and a wide field of view allows experimenters to visually interpret what objects are viewed through the telescope based on scene videos superimposed with tracked gaze points (Figure 7).

Figure 7.

An example of estimation of gaze point through the telescope from naturalistic driving recordings. The top two images are captured by the two side-by-side scene cameras. Head tracking is based on the lower-right infrared image. Estimated gaze point, a red star surrounded by a white box here, is registered on the scene image. In the depicted instant, the driver is looking at a car driving across the intersection through the telescope. The bottom center inset image shows typical head position when the telescope was not used.

(D) Combination of information

Segments in driving routes to be evaluated can be selected based on each of the aforementioned criteria separately. We can also combine different criteria to select specific segments for different purposes. As shown in Figure 8, a 5200m driving trip was purposely recorded to include various segments. The driving route was first automatically labeled based on 8 types of information, and then three types of segments were identified by combining turn signals, turning maneuver and intersection information using Boolean operations. Examples of such combined selections here are turning without signaling (TNS), signaled lane changing (CL), and driving through major intersections without turning (TI). Behaviors and performance at particularly selected segments can be correlated with bioptic use or lack thereof.

Figure 8.

Identifying specific driving segments based on combinations of labeling. (a) An example of recorded data and analysis results of a 5200m driving trip. Marked points are GPS samples. Using Boolean operations, specific segments can be identified, e.g. turning without signaling (TNS), signaled lane change (CL), and driving through major intersections (TI). (b) The satellite picture of a major intersection on the trip. TNS, CL and TI segments, also underlined in (a), all occurred at this location. The inset map indicates the driving route and the location of the intersection.

General Discussion

The two subjects in our pilot experiment had very different bioptic usage patterns. Subject 1 used the telescope very infrequently, apparently supporting bioptic driving opponents' claim that the telescope is barely used in actual driving. However, it is possible that she did not need to use her telescope because she was driving in familiar local neighborhood. Subject 2 commented that he liked to be sure of what was happening even though he did not have to use the telescope. This might be a good bioptic use habit, as long as critical situations in the periphery are not missed. Despite the very different bioptic use behaviors, we did not observe in this pilot study that either rare or frequent telescope use caused any problems for the two bioptic drivers, who both had relatively good VA. While habit and personality can play important roles in behaviors, we believe that visual function should also be an important factor when a wider range of vision conditions (e.g. VA from 20/20 to 20/200) is considered. We anticipate that other bioptic drivers with poorer VA may present different behaviors and effect of bioptic telescope. Obviously no general conclusions about bioptic driving skills or safety may be drawn from the limited data of just two drivers. Our pilot data and system development work illustrate the feasibility of such long term recording and its potential value. While it is not clear whether the bioptic telescopes contributed to the driving safety of the two participants, our pilot recording of naturalistic bioptic driving did uncover some interesting behaviors. Some of the behaviors, for example, using telescope only 7 times within 10 days, could not be revealed by any previously used methods other than naturalistic recording.

Our experience of reviewing the 10 hour recording strongly suggests that naturalistic driving studies require advanced recording systems and analysis methods. We have shown here that with current video, GPS and computer technologies it is possible to build a recording system and facilitate evaluation by automatically annotating the vast amount of recorded data. This system will be used in bioptic driving studies that involve many more bioptic drivers over much longer driving periods. Similar techniques will be useful for other driving studies.