A Low-Vision Navigation Platform for Economies in Transition Countries

Abstract

An ability to move freely, when wanted, is an essential activity for healthy living. Visually impaired and completely blinded persons encounter many disadvantages in their day-to-day activities, including performing work-related tasks. They are at risk of mobility losses, illness, debility, social isolation, and premature mortality. A novel wearable device and computing platform called VIS⁴ION is reducing the disadvantage gaps and raising living standards for the visually challenged. It provides personal mobility navigational services that serves as a customizable, human-in-the-loop, sensing-to-feedback platform to deliver functional assistance. The platform is configured as a wearable that provides on-board microcomputers, human-machine interfaces, and sensory augmentation. Mobile edge computing enhances functionality as more services are unleashed with the computational gains. The meta-level goal is to support spatial cognition, personal freedom, and activities, and to promoting health and wellbeing. VIS⁴ION can be conceptualized as the dovetailing of two thrusts: an on-person navigational and computing device and a multimodal functional aid providing microservices through the cloud. The device has on-board wireless capabilities connected through Wi-Fi or 4/5G. The cloud-based microservices reduce hardware and power requirements while allowing existing and new services to be enhanced and added such as loading new map and real-time communication via haptic or audio signals. This technology can be made available and affordable in the economies of transition countries.

I. Introduction

The unemployment rate of visually impaired persons is estimated to be between 60-80% globally among the base population of 39 million blind and 246 million individuals with low vision worldwide, according to the World Health Organization (WHO) [1]. This reality is undoubtedly tied to the harsh mobility losses that compromise activities of daily living and impact quality of life, as well as health and wellbeing. Studies have highlighted the significant association of visual impairment with anxiety, depression, decreased daily physical activity, and increased risk of imbalance, fear of falling, being hit by or colliding with objects and long-bone fractures [2,3]. Visual impairment restricts or prevents access to three-dimensional information about the world that we are living in and the objects in it, leading to poor spatial cognition and an inability to navigate successfully, whether that be living quarters, public transportation systems, places of employment, grocery stores or shopping and entertainment areas. Our hypothesis is that wearables enable and enhance spatial cognition in visually impaired populations, augmenting personal freedom and agency as well as answering the desire to connect with other human beings, and promote health and wellbeing. Once deployed, this platform will serve as a foundation that can be built upon. We will seek to enhance this technology with next-generation mapping and localization software to further support spatial cognition. Thus, there is a significant need to provide consistent and reliable access to the information needed for mobility and orientation instruction during personal navigation to increase personal mobility.

In economies in transition countries, the main barriers to employment are low education status. A significant impediment to higher education is the lack of inclusive spaces for visually impaired (VI) students, including blind-unfriendly physical structures and poor campus-based support services [4]. Lower levels of education make this group of demographics disenfranchised, financially impoverished, and overall sicker with adverse health outcomes. This work focuses on improving VI students’ mobility and quality of life.

II. Related Works

Long/white cane and guide dog continue to be the most used primary tools for mobility, but not everyone has access to these solutions. While many secondary or expanded primary accessibility devices and applications have been developed, few have been adopted by the blind/low vision community. Despite a dismal history of acceptance, advances in technology continue to bring waves of new accessibility solutions, for example, the BAT ‘K’ Sonar-Cane, UltraCane, GuideCane, and Nurion cane. These solutions are an extension to white cane walking stick by adding laser or ultrasound sensor to detect obstructions in an immediate vicinity, no latest edge computing and other types of sensors. The low acceptance of past efforts is that designs were based upon technological availability, rather than on user needs and experience. There are only a few studies that explore VI behavior. These studies suggest that VI people travel less in general and are 8-times more likely to be driven to work and other places. Most blind and low-vision students spend time after school alone without social interaction, as opposed to sighted students that fraternize with friends. University executives do not address blind-unfriendly physical environments on campus because removing architectural barriers is both challenging and expensive [4]. As a result, the daily challenges blind students encounter is overwhelming. Unfriendly environments can discourage these students from active classroom participation because such environments seriously hamper their ability to join in lecture-based interactions and social gatherings [5].

III. VIS⁴ION

Visually Impaired Smart Service System for Spatial Intelligence and Onboard Navigation (VIS⁴ION) is a novel wearable platform. VIS⁴ION can be conceptualized as the dovetailing of two thrusts; an on-person navigational and computing device and a multimodal functional aid providing microservices through the cloud. It provides a personal mobility solution that serves as a customizable, real-time update of services, human-aware and in-command, sensing-to-feedback platform to deliver functional assistance to a targeted group. The system offers advanced technology for area mapping and navigational localization to support spatial cognition in the VI. The technology enables real-time access to distance and direction information about the 3D position of spatial landmarks and barriers – important to personal movement, affording an understanding of spatial layout and scale within peri-personal space. This system therefore enables end-users to receive tailored navigation prompts to safely wayfind in complex urban environments, helping with both travel and orientation needs. For example, the end user may hear: “turn left in 200 feet at the next intersection” or “head left in a 2 o’clock direction to reach your final destination” or even “veer right to your 4 o’clock to avoid a potential hazard.” Given this newfound capacity, the platform mitigates fall and immobility risk and associated adverse health outcomes such as anxiety, mental frustration, immobility, obesity, cardiovascular disease, diabetes, etc.

In an era of automation, digital route mapping, applied AI and edge computing, there is a pressing need to move away from the cane used in conjunction with a limited-use assortment of gadgets and apps that may not work well in combination to improve the lives of VI. A paradigm shift is required as we move away from the assistive technology of the past and on to have comprehensive tools making safe mobility a reality for the VI. Quality of life can be tremendously improved by providing wearable devices and platforms that are fully integrated for independence living.

A. System Arachitecture and Related Components

The VIS⁴ION platform provides real-time situational and obstacle awareness in three-dimensional (3D) space of a VI’s immediate environment, allowing individuals to travel with more confidence and safety. The travel backpack has sensors and feedback mechanisms, namely a bone conduction headset and a haptic interface or touch communication interface. This device can detect high-body/head, mid-body and low-body hazards to inform the end user for personal navigation. It remedies many of the cane’s shortcomings, and further augments the ability of VI persons to both maintain balance and to localize and identify nearby objects in their environment [6], while providing networking connectivity to the Internet based on Wi-Fi and 4/5G.

The backpack consists of five main components: (1) several distinct distance, ranging and image sensors. These sensors extract pertinent information about obstacles and the environment in real-time, (2) an embedded micro-computer system with Wi-Fi and 4G/5G, (3) a haptic interface via waist strap that communicates the spatial information computed from the sensory data to the end-user in real-time via an intuitive, torso-based, ergonomic, and personalized vibrotactile scheme, (4) a headset that has both binaural bone conduction speakers and a noise-cancelling microphone for oral communication [6], (5) GPS module. The system leverages stereo cameras as its sensing foundation, through the use of advanced computer vision algorithms deployed on an Nvidia board. This enables continuous ‘mapping’ of a dynamically changing environment [6].

B. Operational and Functional Features

The two forms of human-computer interaction for VI are audible messages and vibrotactile feedback. The system continuously monitors and inform its user of environmental features. The audible messages are delivered through bone conduction. The waist belt strap is used for the vibrotactile feedback, whereby the scene that has been mapped is broken into a grid of segments and displayed to the users in a pixelated form factor, in Figure 2.

Figure 2.

A simulated view (Top) of a scene decomposed into capture fields that spatiotopically correspond to actuators in the haptic interface; (Bottom) color-coded depiction demonstrating the scene decomposed into a segmented grid for belt-based vibratory warnings of various threat level based on proximity and spatial position (red denotes ‘high threat’, amber to yellow depicts ‘medium threat’ and green shows ‘low threat’).

For each given scene, the live camera feed is broken down frame by frame and then each frame is decomposed into capture fields that correspond to the density of the tactors on the haptic interface in a spatiotopic representation, i.e. an egocentric view of peri-personal space is calculated for the individual user. It is intuitive and body-centered so the midline of the scene is aligned with the midline of the user’s body. For example, in Figure 2, obstacles on the user’s immediate left are notified through vibrating actuators on the left aspect of the belt. Hazards that are shorter vibrate fewer (lower) actuators in one column of actuators on the belt, and obstacles that are closer to the user are communicated through a higher frequency of vibration, giving a sense of tactile looming or ‘approach.’ This haptic interface is robust without adding significant cognitive load to the end user, minimizing the processing needed to react, plan, adapt and respond.

C. Hardware Requirements

An Nvidia TX2 is used to perform computer vision tasks and provides real-time navigation feedback. It is capable of processing 720p video at ~5-10 fps. Video-based real-time automated navigation support is computationally and power intensive. In other autonomous applications, e.g., the PAL Robotics TIAGo mobile manipulator or Anybotics ANYmal robot (carrying ~ 700Wh), batteries can weigh 3Kg and provide a mere hour of function. This is obviously not an option for our human wearable navigation. Our current wearable runs off a laptop-battery with approximately 66Wh at 0.5kg yielding 2-3 hours of function.

TX2 has a built-in Bluetooth 4.1, 802.11a/b/g/n/ac at 867Mbps, mPCIe connected with a cellular modem such as the Huawei ME909s-120, supporting LTE/UMTS Bands. GPS support is provided by ublox’s EVK-M8 to get outdoor location, allowing edge computing applications to send real-time data including location and update existing or new APPs from the VIS⁴ION microservices.

D. Microservices

Software and digital ‘twin’ updates and services are available on the platform. The current and proposed system is innovative in the following respects:

1. Map-Net is next-gen 3D semantic mapping, based on CASENet [7], by combining a highly novel semantic segmentation neural net together with an image-based 3D reconstruction method (i.e., Structure-from-Motion; SfM) to digitize spatial structures with semantic information. This information can be uploaded to the TX2 for on-demand navigation without relying on a Wi-Fi or cellular network. Map-Net does not need expensive 3D Lidar sensors. It creates a semantically rich and geometrically accurate digital twin of an environment using only regular images. It is a low-cost solution for transitional countries.

2. Loc-Net is the query-image-based user localization network, providing mapping information to users through another neural net localizing the positions and orientations of users via query images. Loc-Net is based purely on images and can operate either fully on-board, or partly using cloud computing, if wireless networks are available.

3. Emergency response platform includes SOS call to designated contacts and real-time assistants with chatbots.

4. Wearable haptic device directly connects end users’ spatial position with navigation information. This reduces the cognitive load of using the navigation system.

5. Audit logs are uploaded for data analytic purposes to enhance the platform in an iterative capacity.

IV. DISCUSSION

There is a pressing need to expand VIS⁴ION function by refining localization features and steering its development toward a defined social problem, where behavioral research influences iterative design. Fundamental research is required to fill the following gaps: (1) limited ability to interpret real-world scenes through on-board computer vision algorithms, (2) experimentally-based understanding of users’ needs and behavioral patterns in complex settings such as Thailand, a transitional country, (3) understand the implementation methods required to deploy and sustain such methods at scale effectively, and (4) lack of hypotheses-driven studies with ecological validity.

V. Conclusion and Future Works

We will use next-generation mapping and localization technology embedded in a novel wearable travel aid to improve the mobility and health of individuals in these populations. Our multi-disciplinary and global team will use a user-centered approach that leverages input from study participants for improved function and usability, ultimately benefitting visually impaired persons worldwide.

Figure 1.

Proposed five component wearables connecting to microservice on the Internet via Wi-Fi, 4G, 5G network