Anti-Blur Feedback for Visually Impaired Users of Smartphone Cameras

Abstract

A wide range of smartphone applications are emerging that employ image processing and computer vision algorithms to interpret the contents of images acquired by the phone’s built-in camera, including applications that read product barcodes and recognize a variety of documents and other objects. However, almost all of these applications are designed for normally sighted users; a major barrier for visually impaired users (who might benefit greatly from such applications) is the difficulty of taking good-quality images. To overcome this barrier, this paper focuses on reducing the incidence of motion blur, caused by camera shake and other movements, which is a common cause of poor-quality, unusable images. We propose a simple technique for detecting camera shake, using the smartphone’s built-in accelerometer (i.e. tilt sensor) to alert the user in real-time to any shake, providing feedback that enables him/her to hold the camera more steadily. A preliminary experiment with a blind iPhone user demonstrates the feasibility of the approach.

1. INTRODUCTION

As smartphones have grown in power and popularity, a huge range of applications have emerged for these devices, including those that employ image processing and computer vision algorithms to interpret the contents of images acquired by the phone’s built-in camera. Examples of such applications include the iPhone Red Laser app for reading product barcodes, Google Goggles for identifying a variety of objects, the kReader Mobile for reading printed documents at close range and a paper currency identifier [6].

Most of these applications are designed for normally sighted users, who use the visual feedback provided by the camera viewfinder to find and center the target object of interest. In some cases there is a significant delay between the time an image is captured and when the results become available (especially if the image has to be sent to a remote server before it can be analyzed by a computer or human assistant), and so it is important for the user to make sure that the image is of sufficient quality before attempting to recognize it [1]. The lack of immediate feedback about image quality makes such applications difficult to use for persons with visual impairments, who often must take repeated images of the same target to obtain reliable recognition results.

A major barrier to capturing good-quality images is motion blur, caused by camera shake and other movements, which commonly occur when holding and panning a camera. Such movements are especially problematic in low-illumination conditions such as indoor scenes, when the camera exposure is long and the magnitude of the resulting blur is therefore larger. Unfortunately, motion blur can cause serious image degradation that can impair the recognition of image information (see Fig. 1 for an example using ABBYY FineReader Online OCR).

Figure 1.

Motion blur example. Top: the same sign is shown in a clear image on the left and in a motion blurred image on the right. Bottom shows text zoomed in from above images: on left, text is clear enough for correct OCR result; on right, blurring causes incorrect OCR result (“MIOHANICAL”).

A variety of techniques have been developed to prevent motion blur, and even reverse its effects [4], but such techniques require either special hardware or extensive computations, which are too costly and slow to be practical for smartphone platforms. As an alternative, we propose a simple technique for detecting motion blur that harnesses the smartphone’s built-in accelerometer (i.e. tilt sensor) to signal the onset of any sudden movements. Audio or tactile feedback alerts users in real-time to such movements, helping them to hold the camera more steadily.

While the idea of using the accelerometer to help the user stabilize the camera is not new (e.g. the Night Camera and Moon Lighter iPhone apps provide this functionality, see http://www.creativeapplications.net/iphone/10-creative-ways-to-use-the-accelerometer-iphone/), we are specifically investigating the utility of this approach for visually impaired users. A preliminary experiment with a blind volunteer subject demonstrates the feasibility of our approach for reducing motion blur.

2. APPROACH

The accelerometer (i.e. tilt sensor) included in modern smartphones measures the x,y and z components of the phone’s acceleration [2]. When the phone is at rest or moving at a uniform velocity, the accelerometer vector points in the direction of gravity and has fixed magnitude. However, when the phone is moved at a varying velocity (e.g. unsteadily), the accelerometer vector changes over time. Thus, we define a simple measure of unsteadiness calculated by applying a high-pass filter to the time sequence of acceleration vectors (which reflects how much the acceleration changes from one moment to the next), and then taking the norm of the resulting vector. This “shake metric” equals 0 if the acceleration is perfectly constant, and increases the more that the acceleration changes over time.

While the accelerometer does not directly measure the amount of motion blur, we note that under typical conditions (i.e. handheld use of the camera) the shake metric is empirically correlated with the amount of motion blur (see next section for details). A more direct approach would be to estimate the amount of motion blur from the pixel information contained in each image, but this type of approach is computationally very intensive [5] and may therefore be problematic for smartphone applications that process up to a few or more image frames per second.

3. EXPERIMENTAL RESULTS

We programmed an iPhone 3GS to acquire video frames and save them to flash memory at a rate of 2 Hz, along with the corresponding accelerometer values. A blind volunteer subject was instructed to slowly sweep the camera across a large bulletin board several times, in such a way as to minimize the incidence of sudden movements. A total of 360 image frames (at 480 × 320 resolution) were acquired in the experiment and saved to a desktop PC, and each frame was manually categorized (by visual inspection) as being sharp, slightly blurred or strongly blurred.

We conducted a simple analysis to test the hypothesis that higher values of the shake metric are associated with a higher probability that subsequent image frames are blurred. Specifically, a “shake flag” is activated whenever at least one of the current frame and two most recent frames have shake metrics above an experimentally determined threshold. Similarly, a “blur flag” is activated at each moment in which at least one of the current frame and the two subsequent frames is categorized as being slightly or strongly blurred. (The inclusion of recent and subsequent frames allows for time lags between movement onset and accelerometer response.)

If activation of the shake flag (the criterion for inferring whether blur is present) is regarded as a predictor of activation of the blur flag (the true state of the blur), then the reliability of the shake flag on our data set can be evaluated in terms of the true positive and false positive rates, which are 69% and 19%, respectively. These rates demonstrate that the shake flag predicts the incidence of blur fairly well with a small probability of false alarms. (Even with some incorrect predictions, the shake flag should improve the quality of images acquired by the user on average.) In the future we will experiment with calculating the shake flag on the smartphone in real time so that audio/tactile feedback is issued to the user whenever the flag is activated.

4. CONCLUSION

We have described a simple technique for using the smartphone’s built-in accelerometer to detect the onset of motions likely to cause motion blur. An experiment with images acquired by a visually impaired user demonstrates that accelerometer disturbances are well correlated with the incidence of motion blur.

Future work will focus on issuing feedback based on the accelerometer to help the user minimize motion blur. It may be useful to integrate such feedback into a number of camera-enabled smartphone applications; some users may wish to dispense with the feedback altogether after sufficient training with it. Finally, we will explore the possibility of augmenting the anti-motion blur functionality with additional accelerometer feedback to help the user hold the camera close to the horizontal; such feedback (see [3] for an example) may be useful for applications in which targets of interest (e.g. printed text in signs) are known to be oriented horizontally.