A Mobile Phone Application Enabling Visually Impaired Users to Find and Read Product Barcodes

Abstract

While there are many barcode readers available for identifying products in a supermarket or at home on mobile phones (e.g., Red Laser iPhone app), such readers are inaccessible to blind or visually impaired persons because of their reliance on visual feedback from the user to center the barcode in the camera's field of view. We describe a mobile phone application that guides a visually impaired user to the barcode on a package in real-time using the phone's built-in video camera. Once the barcode is located by the system, the user is prompted with audio signals to bring the camera closer to the barcode until it can be resolved by the camera, which is then decoded and the corresponding product information read aloud using text-to-speech. Experiments with a blind volunteer demonstrate proof of concept of our system, which allowed the volunteer to locate barcodes which were then translated to product information that was announced to the user. We successfully tested a series of common products, as well as user-generated barcodes labeling household items that may not come with barcodes.

1 Introduction

The 1D barcode is a ubiquitous system for labeling packaged goods with codes that uniquely identify product information. This type of barcode (which includes the UPC and EAN symbologies, widely used in North America and Europe, respectively) was designed to be read by a laser scanner, which is a standard tool in supermarkets and shops, but there is increasing interest in being able to read barcodes using the most ubiquitous portable device: the mobile phone.

Applications such as the iPhone Red Laser, Nokia's Point and Find and Realeyes3D have recently been released to provide this functionality, but all such products force the user to locate and home in on the barcode using visual feedback from the camera viewfinder. Without such feedback, it is nearly impossible to guide the camera close enough to the barcode to view it with sufficient resolution. As a result, these products are only accessible to people with sufficiently good vision.

Dedicated devices such as the i.d. mate OMNI talking barcode reader (from Envision America), which uses a laser to rapidly scan a product and read any barcode within its field of view, have been designed expressly for blind and visually impaired users, but such products are expensive (e.g. over $1000 US) and require the use of a dedicated piece of hardware. One possible alternative to such a dedicated device is a system such as Trinetra [3], which uses a small barcode reader that plugs into a standard mobile phone; however, the barcode reader is still an additional device that makes the system less convenient to use.

A computer vision algorithm enabling a blind person to locate and read barcodes can be implemented on a camera phone, perhaps in addition to other functionality providing access to printed information (such as the KNFB reader, which is a commercial OCR system that runs on a Nokia camera phone). Such a system would be much more convenient than conventional laser scanner technology that requires the user to carry a dedicated device, such as the i.d. mate.

The alternative approach that we describe in this paper is to use a portable computer vision-based system to find a barcode, direct the user with audio feedback to this barcode until it is well-centered in the camera frame, and then decode the barcode. This system is implemented as a software application on an off-the-shelf mobile phone, building on a previous desktop computer version of the system implemented by the authors [5,4]. Once the barcode is decoded it is matched to an on-device database to identify the product (no data connection is required), and the corresponding product information is presented using text-to-speech functionality.

2 Finding and Reading Barcodes with Computer Vision

In this work we specialize to UPC-A barcodes, which are widely used in North America to label commercial products, but it will be straightforward to extend our system to other common symbologies such as EAN (common in Europe).

The computer vision algorithm we have developed has two stages, the first for finding barcodes in cluttered images, the second for reading them when they are viewed at close range, as presented in the system overview in Fig. 1. We note that past work on computer vision algorithms for reading barcodes on mobile phones [2,6] has deemphasized the important issue of finding the barcode, instead assuming that the user can center the barcode in the viewfinder.

Fig. 1. System overview.

Our application starts in a low-resolution mode for the first part of our algorithm, which is the barcode detection part, so as to run at a reasonable speed. It first determines if there is a piece of a barcode in each video frame it receives. This is accomplished by searching lines of pixels along the 0^°, 45^°, 90^° and 135^° orientations, and seeing if there are multiple lines that are approximately perpendicular to the swept lines. Areas with such lines are recorded as candidate areas for further investigation.

The algorithm then focuses on the biggest such verified candidate area if there are multiple such candidates. If this area is too small, or too close to the edge of the image (implying that there may be a yet unseen portion of a barcode), then the user is notified by directional feedback (which can be chosen to be a different tone or voice command) until the candidate area is large enough and relatively well positioned in the frame. We then use a higher resolution version of the barcode area to detect and localize the edges more accurately. If enough edges are found, we use a Bayesian statistical approach, [4], to find the barcode digits that are most likely, based on the edge locations. Multiple slices are used if needed, until we have sufficient confidence about the digit estimates.

Then the detected barcode is matched against the product databases, first the customizable user database and then the full database. If the information for this product is found, it is read out using text-to-speech. If we cannot find an entry, then we match the first six digits of the barcode against a manufacturers' database. If the manufacturer is found, then this information is read out loud; if this information is not available either, the user is notified of the lack of information regarding this product.

3 System Operation and User Interface

The preceding barcode location and decoding algorithm was developed and tested on a desktop computer and then ported to the Nokia N97 camera phone running Symbian C++. The algorithm was implemented to run as quickly as possible on the mobile phone, so that up to a few images could be processed per second. To achieve operation as close to real-time as possible, we had the system run in a video capture mode (in which several images are automatically captured every second).

The system's user interface guides the user through two stages for each product, first finding and then reading a barcode. The user holds the camera approximately 10-15 cm above the surface of the product, slowly scanning the likely locations of the barcode (see Fig. 2); the system is silent until it finds sufficient evidence for a barcode in its field of view. When a barcode is detected, the system attempts to guide the user to center the barcode in the field of view, using four distinct signals (tones or verbal instructions) to indicate that movement is required to the left, right, up or down. If the barcode is centered but too far to be clearly resolved and read, another tone directs the user to bring the camera closer to the barcode. Once the barcode is sufficiently close, the system continues capturing images of the barcode until a barcode estimate can be determined with likelihood exceeding a threshold, and indicating these attempts with another tone.

Fig.2. Blind volunteer subject (with face covered for anonymity) using our system.

Finally, when a barcode is read with sufficient confidence, a “successful read” signal is issued, the barcode is looked up in a freely available UPC database [1], and the results of this look-up are read aloud using text-to-speech functionality (built into the mobile phone). We note that the UPC database contains over one million entries (and is augmented with any user-generated barcode entries, if any) but is small enough (under 70 MB total) to be downloaded to the mobile phone's flash memory. Thus, the system software runs locally on the mobile phone without the need for any data plan to access a database on a remote server, which is an important advantage if the user is in a place without mobile reception.

4 Experimental Results and User Testing

We explained the purpose and operation of the system to a completely blind volunteer subject, and advised him on how best to hold the mobile phone (Fig. 2). Particular emphasis was placed on the importance of moving the camera slowly to avoid motion blur and to ensuring the camera lens was not covered (e.g. by the user's fingers). In addition, we discussed the importance of using prior knowledge of where barcodes are likely to be found, and at what orientation, on different packages to narrow the search. For instance, barcodes tend to be aligned with the edges of a package, and the camera is better able to resolve a barcode when it is aligned to the barcode (or at a 90^° angle); on a cylindrical can, the barcode is on the circumference of the cylinder near the flat top or bottom.

We first conducted a brief training session on several barcoded packages, in which we gave advice to the subject as he was using the system; some of the packages in this session that he was able to read included user-generated barcodes, i.e. barcodes that we printed out to label personal items that did not already originally have their own barcodes.

Next, a simple experiment tested the subject's ability to find and read barcodes for ten items using the system without our help. These ten packages were commonplace grocery items in the form of boxes, cans and bottles (separate from the items used in the training session). One item was a bottle of Ibuprofen on which we pasted the correct product barcode, since the bottle lacked its own. The subject was able to locate barcodes on nine of the ten packages but was unable to locate the barcode on a plastic jar, because of false positive detections due to the presence of densely grooved lines on the top of the jar (which resemble barcode lines). Of the nine barcodes that were successfully located, seven were read correctly and one was located but not read; the other was read with two incorrect digits (out of the entire 12-digit barcode digit sequence), but enough digits were correct to determine the product manufacturer (indicated by the first six digits in the sequence).

While our experiments establish proof of concept of our system, there are currently two main problems that need to be solved to make this a truly useful product. The biggest problem is that the Nokia N97 phone we used has a slow and erratic camera autofocus mechanism, which results in some of the images being out of focus, thereby degrading the system's performance. Also, the phone CPU is much less powerful than a standard desktop computer, which makes the system much slower to analyze images than the desktop version [4], and limits the system's ability to quickly rule out false positives. However, both these problems should be ameliorated in newer-generation devices such as the iPhone or the Google Nexus One, which we plan to experiment with in the future.

5 Conclusion

We described a new mobile phone system that enables blind and visually impaired users to find and read barcodes with a mobile phone. The system is implemented as a software application that can be downloaded on an off-the-shelf phone, and it is the first such system we are aware of that is accessible to blind users that doesn't require the use of any additional hardware. Preliminary experiments with a blind subject demonstrate the system's feasibility.

Future improvements to this system include porting it to more popular and powerful mobile platforms, such as iPhone and Android, improving the speed and accuracy of the computer vision algorithms for finding and detecting barcodes, extensive testing with blind and low vision subjects, and extending the system to handle a greater variety of barcode symbologies (such as EAN).