Embedded solution to detect and classify head level objects using stereo vision for visually impaired people with audio feedback

Abstract

This work presents an embedded solution for detecting and classifying head-level objects using stereo vision to assist blind individuals. A custom dataset was created, featuring five classes of head-level objects, selected based on a survey of visually impaired users. Object detection and classification were achieved using deep-neural networks such as YoloV5. The system computes the relative range and orientation of detected head-level objects and provides audio feedback to alert the user about nearby objects. Four types of tests were conducted: a dataset-based test, achieving a mAP@0.95 of 0.89 for head-level objects classification; a quantitative assessment of range and orientation, with an average error of 0.028 m ± 0.004 and 2.05°±0.09, respectively; a field test conducted over a week at different times and lighting conditions, yielding a precision/recall of 98.21%/93.75% for head-level object classification; and user tests with Head-level identification accuracy of 91% and obstacle-avoidance/local-navigation where users reported an average of 88.75% for low or middle risk.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-025-01529-7.

Introduction

Persons with disabilities (PwD) encounter access barriers in various common environments, including medical care institutions, banks, malls, grocery stores, educational institutions, and restaurants. According to the World Report on Disability¹, 80% of the world’s one billion PwD live in Low and Middle-Income Countries (LMICs). Only 3–15% of PwD who need assistive devices and technologies have access to them². Beyond mere access, it is crucial that these devices and technologies meet user requirements and preferences, as well as support bodily functioning. This is increasingly important as considerations such as diversity among blind individuals and access to rehabilitation become more prominent today.

Safe navigation is an important aspect for users’ autonomy, especially for VIP. In LMICs, outdoors and indoors environments are challenging environments where pedestrian walks are full of dangerous objects. It is important to merge these needs with user requirements and preferences by actively involving users in the design process, thereby increasing the likelihood of technology adoption.

Nowadays, many electronic devices and systems have been developed, as described in related works section, to assist blind individuals in navigating unfamiliar environments and enhancing their autonomy. These devices range from automated canes with tactile or sound feedback to advanced systems utilizing computer vision techniques and artificial intelligence to describe surroundings to the user. While commercial initiatives like MyEye 2.0 ³ offer a compelling array of functionalities, access to such technologies remains challenging for users in LMICs.

This work proposes an embedded solution that considers user requirements and preferences to detect and classify head-level objects near the user’s position (those identified by the deep neural model presented in this work, and those not identified by our prototype). Figure 1 illustrates the platform setup concept, where the primary sensor, a stereo camera, is positioned at the user’s forehead level. The main computer device and power supply are carried in a backpack. The main computer processes the incoming images, builds a local map of the environment, and outputs audio feedback, which is delivered through a pair of headphones extending from the backpack.

Fig. 1.

System setup for the embedded solution proposed.

This work is organized as follows: the next section describes the related works; then, the proposed solution is presented, which includes the system setup, the dataset built, and the CNN (convolutional neural networks) architectures used; afterwards, the results obtained are presented; and finally, our conclusions.

Related works

Visual substitution systems, particularly those designed for Electronic Travel Aids (ETA) must meet several key requirements: real time response, working indoors/outdoors, navigation and planning functionalities, detection of a wide range of objects: at medium and long ranges, static and dynamic, and at low and head-level. Nowadays, computer vision systems can be deployed to satisfy these requirements at relatively low cost. With these criteria in mind, relevant works were analyzed and summarized in Table 1. The columns in Table 1 evaluate each system based on indoor/outdoor functionality, night/day operation, short/medium/large range operation, dynamic obstacle detection, the sensors used, the computation platform, object recognition capabilities (Unevenness, Low Obst., High Obst., Holes), and the data processing algorithm employed.

Table 1.

Related works comparison.

Refs.	Ind.	Out.	Night	Day	Short R.	Med. R.	Large R.	Static Ob.	Dyn. Ob.	Sensors	CPU Plat.	Obj. Rec.	Nav.	CI Alg.
4	Ok	No	No	Ok	OK	Ok	No	Ok	Ok	RGBD	Google Tango	LO	Ok	Semantic Mapping
5	Ok	No	No	Ok	Ok	Ok	No	Ok	Ok	RGBD	Embedded	LO	Ok	Cloud Point Proc.
6	Ok	No	No	OK	Ok	Ok	No	Ok	No	RGBD	Laptop	HO LO	Ok	Prog. Hough
7	Ok	OK	No	Ok	Ok	Ok	Ok	Ok	Ok	LRF, Monocular	Laptop	HO LO	Ok	Fuzzy Logic
8	Ok	OK	No	Ok	Ok	Ok	No	Ok	Ok	Monocular	Smart phone	LO	Ok	BOW, SVM
9	OK	OK	No	OK	Ok	No	No	Ok	No	Monocular	MCU	No	No	Roving Net.
10	Ok	OK	No	Ok	Ok	Ok	No	Ok	Ok	Ultrasonic, Monocular	Smart phone	HO LO	Ok	SVM
11	Ok	No	No	Ok	Ok	Ok	No	Ok	Ok	Dynamic Vision Sensor	Embedded	HO LO	No	Stereo Img. Proc.
12	Ok	OK	No	Ok	Ok	Ok	No	Ok	Ok	Stereo pair, IMU, GNSS	Laptop	HO LO	Ok	SpikeNet
13	Ok	Ok	No	Ok	Ok	Ok	No	Ok	No	Google glass, smart cane	Embedded	LO	No	YoloV3
14	Ok	No	No	Ok	Ok	No	No	Ok	No	Camera, sonar, button	Embedded	LO	No	MobileNet V2
15	Ok	Ok	No	Ok	Ok	Ok	No	Ok	No	Stereo camera	Embedded	LO	No	MobileNet V1
16	Ok	Ok	No	Ok	Ok	No	No	Ok	No	Camera	Embedded	LO	Ok	MobileNet
17	Ok	Ok	No	Ok	Ok	Ok	No	Ok	No	Camera	Embedded	LO	No	E-BiSeNet
18	Ok	Ok	No	Ok	Ok	Ok	No	Ok	No	Stereo camera	Embedded	LO	No	Yolo V4
19	Ok	Ok	No	Ok	Ok	No	No	Ok	No	Camera, light, sonar, humidity	Embedded	LO	No	No

In^4–6 RGBD sensors are used, they work indoors, at daytime and up to medium range (5 m)^4,5 can deal with dynamic objects which ensures a safe path for VIP. These solutions can detect obstacles nearby and they have navigation functionalities^4,5 run in portable and easy-to-use devices. In⁴ semantic mapping methods are implemented to create user-friendly digital maps, and in⁶ high obstacles are detected as walls in indoor narrow hallways. These works show relevant academic advances to increase a safe navigation for VIP. However, these proposals were not tested with enough VIP, and they not considered a user-centered design of the prototype.

Lowering the cost while maintaining the high environmental description is a key challenge to be addressed on this project. Then, monocular vision plays an important role. In^{7–10,14,16,17}, and¹⁹ authors use monocular vision to satisfy most features of the ETA systems. These solutions work indoors and outdoors, but at daytime. ⁸ and ¹⁰ work up to medium range, and⁷ thanks to its dual operational model, it can work at long range (up to 10m). In⁷, high obstacles are detected in outdoors scenarios such as poles and traffic signs. In ⁸ and ¹⁰ use smartphones for data processing, which is a more portable device for VIP. However, in general these works present a technological prototype without enough tests with VIP. In⁷, a monocular camera and Lidar are used together, which rises the implementation costs and the prototype is not affordable. In^8,9, and¹⁰, monocular camera is used to detect proximity of dynamic objects but the accuracy of the relative pose is poor, even when the ultrasonic sensors are used. In^14,17, and¹⁹, the proposed system focuses on low-cost sensor, but the accuracy of detection is inconclusive. And, in¹⁶, the CNN approach provides results without relative pose information, only recognition in quite ideal environments.

Currently, stereo vision systems are more affordable and compact, thus they can be used in ETAs. In ¹¹ and ¹², authors use stereo sensor to build in 3D the surroundings for VIP. These solutions can work indoors and outdoors, but at daytime, and up to medium range. In^10–12, high obstacles are also detected in indoors and outdoors scenarios. In¹² thanks to the fusion of different sensors such as Inertial Measurement Units (IMUs) and Global Navigation Satellite Systems (GNSS) devices, the system can give navigation clues of the environment. Other potential risks for VIP are potholes, in¹⁵ proposes a solution using stereo camera to estimate the relative location of the potholes previously detected using a MobileNet neural network. In¹⁸, the stereo camera is used to compute the relative location of common low level obstacles, and the user feedback is performed using vibrating motors. In addition, as observed in Table 1, road unevenness or holes in the road are not common features in these works. Even though, road unevenness or holes are very common in LMICs. However, in ¹¹ and ¹², despite the fact that these works use stereo cameras, the amount of information which is sent to the user is a issue. And, in ¹⁵ and ¹⁸, the prototypes are well conceived since they satisfy a urgent need, but the accuracy results are inconclusive.

Finally, given the amount of data captured and processed in this kind of aid systems which range from global positioning data, RGB images, RGBD images to point clouds, sophisticated processing methods must be used. Table 1 shows that the following computer intelligent algorithms are used: semantic mapping⁴, Fuzzy logic⁷, bag-of-words⁸, CNN^9,12–18, machine learning^8,10 and classic processing methods^5,6, and¹¹. It can be observed that CNN provide a robust solution specially in tough environments where there are changes in illumination, point of view, occlusion, and sudden movements of the camera. The integration of highly optimized algorithms with off-the-shelf electronic components—such as cameras, processors, and audio interfaces—paired with 3D-printed elements, significantly reduces maintenance and production costs. This design approach not only facilitates easy access to spare parts and local repairs but also results in a system that is 5 to 15 times more affordable than comparable products on the market. Additionally, the cost is expected to decrease further with industrialization and bulk procurement of components, making this solution increasingly accessible for widespread adoption. As noted before, it is very important to consider the user requirements for the final appeal of the prototype, as well as, a report of accuracy levels using user tests with satisfaction surveys measures and having a system which is visually attractive, portable and discrete.

Methods—proposed solution

This work proposes an embedded travel aid designed to detect and classify nearby obstacles using stereo vision. Based on the analysis of the related works along with a context study carried out by our research team in the city of Cali, Colombia²⁰ and a systematic review of UCD standards applied in assistive technology development²¹, several important requirements were identified: the system must be low-cost to ensure accessibility in LMICs, discreet to avoid attracting unwanted attention, comfortable to wear, and aesthetically appealing. From a technical standpoint, the system must incorporate high-performance single-board computers, compact sensors with rich information data flow, a primary sensor with a wide field of view and operational range, and A. I. algorithms with high generalization capabilities. Additionally, the technology must be user-friendly and provide appropriate feedback for VIP. These criteria guided the development of an embedded solution that detects and classifies head-level objects using stereo vision. This project has the approval of the ethics committee of the Universidad del Valle No. 008–022, and the approval of the ethics committee of the Instituto para Niños Ciegos y Sordos (INCS) No. CEI-2022-03.

Hardware overview

As shown in Fig. 2, the main input data is generated by the ZED mini stereo camera via a USB connection. This data is processed by the ROS application running on the Jetson Nano embedded system. The ROS application pre-processes the RGB and depth images, issuing early warnings using the image depth, detecting and classifying head-level objects, building a local environment map and issuing the corresponding audio feedback. The audio feedback is delivered to the user through Bluetooth-connected headphones.

Fig. 2.

Hardware setup to detect low-level objects.

The head level object detection prototype can be configured and monitored using an HDMI monitor, keyboard, and mouse as shown in the bottom of Fig. 2. However, these devices (indicated by the blue dotted rectangle) are not required when the GlobalViewAid hardware is in active use.

Dataset

The head-level obstacles dataset was built using the OpenImages dataset²², the Detection Lamp Cabinet dataset²³ and images captured by the authors at the Universidad del Valle campus. Based on advisory advising sessions held at the INCS (Instituto para Niños Ciegos y Sordos)²⁴ with volunteer VIP, the following categories were identified as most relevant: ceiling lamps, sconce, tree branches, and chandeliers. An example of these objects is shown in Fig. 3a.

Fig. 3.

(a) Head-level objects balanced dataset. (b) Orientation preferences for VIP.

Initially, the dataset consisted of 1326 images, with ceiling lamps and sconce being overrepresented. To balance the dataset, additional images of chandeliers and tree branches were collected, along with performing data augmentation techniques applied to increase the variety of the dataset. These transformations included scaling (from 0.5 to 1.5), rotating (± 60°), translation on X axis (up to 30%), translation on Y axis (up to 30%), horizontal flip, brightness scaling (from 0.5 to 1.5), and gaussian noise (up to σ = 3.0). Figure 3b displays the final histogram of the dataset, which now consists of 3,257 images, providing a more balanced representation of head-level obstacles. Table 2 shows the labels distribution between classes for traiing and validation.

Table 2.

Dataset description.

Class	Train	Validation
Ceiling lamp	638	109
Sconce	816	140
Branches	714	184
Chandelier	553	103
Total	2721	536

Software development

Selecting the right neural network architecture suited to the application is critical to meeting user requirements effectively. Deep Convolutional Neural Networks (DCNN) have demonstrated high efficiency in object detection and classification²⁵. These types of neural networks can be divided in three categories: sliding window based, region proposal based and end-to-end based methods. Standard convolutional neural networks (CNN) use the sliding window approach. Region proposal methods are implemented in two well-known architectures such as R-CNN, Fast R-CNN and Faster R-CNN. These architectures are very robust but have slow learning times, and runtime delays as described in²⁶. In contrast, end-to-end detection solutions such as YoloV5 ²⁷ are faster than their counterparts, this kind of neural models can be optimized for different levels or precision, and they have proved efficiency in embedded solutions as NVIDIA Jetson Nano²⁶. Therefore, in this work we implemented two versions of YoloV5 (small and medium models) to handle head-level object detection for VIP.

Software development for the head-level object detection system followed the Rational Unified Process (RUP) methodology²⁸, which includes deliverable products such as: functional and nonfunctional requirements, concept diagrams, real use cases, sequence diagrams, and class diagrams. Due space limitations, only the functional requirements and concept diagrams of both prototypes are presented in this paper.

Audio feedback

Audio feedback for VIPs should be presented in a simple and easily understandable manner. To figure out how to present the range and orientation information about closer obstacles, a test session was performed in the INCS with volunteer VIP (20 persons). This test session considered the experimental setup shown in Fig. 3c. Then, for each participant the following procedure was performed:

1. The VIPs were instructed with movement commands such as: walking 1–2 m ahead, moving two steps in different directions (forward, left, diagonal-left, right, and diagonal-right), and moving two steps based on clock-hour directions (12:00, 3:00, 9:00, 2:00, and 10:00).

2. The VIP starts moving and initiating video recording.

3. At the end of completing the movement command the video recording concludes.

All the results were analyzed, and considering the literature review of related works section, the range and orientation information to be issued by audio to users must be transformed to steps and orientation instructions depicted in Fig. 3c correspondingly. The relationship between the steps and range to the object in meters is transformed using the anthropometric user data²⁹ and Eq. (1).

1

Where, ρ_Obj is the range to the detected object, and user_Step is the anthropometric measure in meters of one step given the user’s height. Figure 3c shows the orientation preferences for the VIP surveyed. Then, the following equivalences apply: Right, up 50°; Diagonal Right, between 50° and 75°; Front, between 75° and 105°; Diagonal Left, between 105° and 130°; and Left, larger than 130°.

Head-level objects detection

Considering the advising sessions performed in the INCS with volunteer VIP, and the literature review of the related works section, the head-level object detection and classification system has the following functional requirements: Data perception is composed by the stereo image and IMU data; the system will perform data preprocessing to enhance the image acquisition; it will use the IMU data to discard frames that observes other than looking ahead; the system will be able of computing the relative distance between users and the nearest object using the disparity map; the system will be able of detect and classify head-level objects of interest i.e. those specified in the dataset section; the system will be able to detect other obstacles not classified by the neural network model; and the system will be combine the inference result and the disparity map to generate audio alerts to users.

Figure 4a shows the complete developed system for the head-level object detection considering the above functional requirements. There are two stages, one offline for training and validations of the deep-neural model, and another for online detecting and classifying head-level objects. The offline stage was developed in the Jupyter framework and executed in the computational cluster of the University of Valle to train the YoloV5 small and medium deep-neural models.

Fig. 4.

(a) Software developed for the head-level object detection and classification system. (b) Node graph for the ROS2 data acquisition module. (c) Node graph for the ROS1 planner module.

Once the deep neural model is trained, it was optimized using the NVIDIA TensorRT SDK³⁰ in order to be used in the Jetson Nano embedded system. The head-level object detection and classification system was developed using ROS (Robotic Operating System)³¹ since it facilitates the system reproduction, system parametrization, and it offers many high level debugging tools. To start with, the input data from the ZED mini stereo camera and its IMU are captured using the image acquisition thread, which is a third-party application provided by the manufacturer. Using this thread, the intrinsic and extrinsic calibration parameters are also loaded into the solution from the device itself.

Then, the preprocessing thread enhances the input images through a histogram equalization and detecting blur images which could be further discarded. Figure 4b shows the graph node diagram proper for ROS where this is implemented. These tasks are implemented in the nodes and topics highlighted in blue. The other nodes and topics are third-party or provided by ROS. In addition, this thread uses the IMU data to discard images when the VIP is looking upwards or downwards. This is done because this solution is focused on detecting head-level objects only. At this point, the following calibration procedure is performed at the first time the user is using the prototype:

1. The user put on the device.

2. Using audio communication, the user is noticed that she/he has 5 s to look ahead. The regressive count is also issued by audio.

3. Then, the user is noticed that in 5 s she/he should not move at all. The regressive count is also issued by audio.

4. Afterwards, the system is getting the IMU data over 4 s. The regressive count is also issued by audio.

5. The IMU data captured (≅ 40 readings) are averaged, and this average will define the user’s head coordinate frame, which is saved to further use.

Then, all readings of the Mini ZED stereo camera are transformed to this coordinate frame, obtaining a suitable local representation of the environment. After preprocessing, the images are feed to two threads: neural network inference and disparity map. The former executes the neural network model previously loaded, and it gives as a result the classification of head-level objects, the bounding box and its probability value. The disparity map thread is implemented using third-party software called Isaac Image Pipeline³². The result of this thread is the disparity image which will be used to compute the object distance and orientation with respect to the user. This is performed in the disparity to 3D node, which also has as input the neural network inference result. This inference result is performed by the perception_node node in Fig. 4c, which generates a local cost map weighted by the relative distance between the main sensor (stereo cameras) and the obstacles in the surroundings.

The disparity to 3D thread uses the range and orientation transformation described in the audio feedback section. In case of having a head-level object which is not classified by the neural network model, the object detector thread segments those objects using the 3D point cloud computed using the disparity map. Then, the range of orientation of this head-level obstacle is also computed considering the transformation described in the audio feedback section. Finally, the audio feedback text template is described as follows:

There is a/an + OBJECT_CLASS + at + STEPS + ORIENTATION.

Where, OBJECT_CLASS is the type of object detected and classified; STEPS is the number of steps given by Eq. (1), and ORIENTATION is one of the possible orientations depicted in Fig. 3c. This generated string is sent to the Google Text to Speech API³³ to finally issue the audio feedback to users using a Bluetooth headphones. The translation between a 3D environment representation and the audio feedback is performed by the gelbert_local_planner node as shown in Fig. 4c. This node deals with the Google Text to Speech API and the Bluetooth connected headphones.

Results

In this section, different results are reported: first, training and validation process for the head level object detection and classification; second, quantitative results on the estimation of range and orientation of nearby objects; third, field tests test results under varying lighting conditions, and times of day; and fourth, user tests including head-level object detection, obstacle avoidance and local navigation.

Training and validation process for the head level object detection and classification

The validation-training results for head-level object classification are shown in Fig. 5 considering the YoloV5 small and medium models. These processes were performed using the dataset described in the corresponding section, with training conducted on the PSI (Perception and Intelligent Systems) research group’s cluster at Universidad del Valle.The cluster is equipped with 130 CPU cores, 376GB of RAM, 12 TB of storage, 23,808 CUDA cores, and 176GB of GPU memory. The training process followed these key steps: first, the YoloV5 backbone was frozen over 1000 epochs, during which 70% of the dataset was used for training. Afterward, all the hidden layers were unfrozen using a small initial learning rate of 0.00334.

Fig. 5.

Head-level object classification. (a) Validation results of YoloV5 small. (b) Validation results of YoloV5 medium.

As a result, the total mAP@0.5 reached 0.89 for the YoloV5 small model and 0.882 for YoloV5 medium model. The small model was selected for optimization using TensorRT and to be executed by the Jetson Nano embedded system.

Estimation of range and orientation of objects in the surrounding environment

Measuring the range and orientation accuracy of the low and head level object classification solution proposed in this work is crucial as the correct perception of the environment depends on these measurements. This is done in two cases: first, when the distance to objects is transformed into user steps; and second, when the orientation is transformed as depicted in Fig. 3c.

To measure the accuracy of range and orientation, the following experiment was conducted: first, the head level object classification solution was placed at the origin of a coordinate system (see Fig. 6). Then, head level objects were placed at three different orientations and distances. At each location, the system classified the objects and computed their range (in meters) and orientation (in degrees), taking at least 30 measurements per configuration. These measurements were averaged, and the standard deviation was computed.

Fig. 6.

Configuration setup to perform the range and orientation quantitative tests.

The results of this experiment are presented in Table 3. The table includes three orientations: objects placed in front of the user (90°), and at the left and right limits of the camera’s field of view. Additionally, three different distances were tested for each orientation. The classification system achieved an average range error of 0.028 m ± 0.004, and an average orientation error of 2.05° ± 0.09.

Table 3.

Results of the range and orientation quantitative tests.

Ground truth	Error	Ground truth	Error
0.82 m	0.01 m ± 0.003	54.74°	1.27° ± 0.04
1.22 m	0.04 m ± 0.001	54.74°	0.3 ± 0.04
1.63 m	0.05 m ± 0.02	54.74°	2.3° ± 0.21
1.0 m	0.01 m ± 0.002	90°	1.98° ± 0.05
1.5 m	0.014 m ± 0.001	90°	2.73° ± 0.05
2.0 m	0.09 m ± 0.001	90°	3.6° ± 0.05
0.82 m	0.003 m ± 0.001	125.25°	0.33° ± 0.03
1.22 m	0.016 m ± 0.002	125.25°	2.01° ± 0.06
1.63 m	0.02 m ± 0.001	125.25°	3.9° ± 0.25

These results demonstrate that the head-level objects classification solution can model the surrounding environment to provide the users with accurate information about it. The high level of accuracy is attributed to the prior filtering of the disparity image using the median filter, ensuring robust environmental modeling for the user.

Head level object detection and classification under different illumination conditions, and times of the day

The third set of tests conducted in this work involved field tests to evaluate the performance of the head-level object classification system in various real-world lighting conditions. The experiment was designed as follows:

1. Five scaled-up images of each class of head level objects were printed.

2. A 15-meter-long hallway in the University of Valle’s E53 building was chosen as the test site.

3. The printed images were distributed throughout the hallway.

4. The hallway was traversed using the head level objects classification solution at different times of the day (morning, midday, afternoon, and night) over the course of a week.

5. The classification results were stored locally on the Jetson Nano for further analysis.

The aim of this test was to assess the system’s performance before any VIP can utilize it, ensuring the solution’s detection capabilities. in challenging environments. The chosen hallway presented significant challenges as it was only partially indoors—one side featured a combination of high and low walls, while the other side consisted of a low wall and a large 10-meter open window, leading to considerable variations in illumination.

The results, consolidated in Fig. 7, demonstrate the system’s robustness. Despite changes in lighting conditions, the precision-recall graphs show high performance. On average, the head-level object classification solution achieved a precision of 98.21% and a recall of 93.75%. These results confirm that the proposed solution is effective in detecting head-level objects, even in a dynamic and challenging environment.

Fig. 7.

Field tests. Head-level object classification.

User tests

Building on the previous results, the GlobalViewAid prototype was tested with VIPs under three specific tasks:

1. Head level objects identification: At the very beginning, users received a detailed explanation about this test. Then, users were guided to different locations where head level objects such as chandeliers, tree branches, ceiling lamps, and wall lamps were placed (as shown in Fig. 8a). At each location, users were asked various questions to assess their experience. Following the test, participants completed a survey, with results discussed below.

2. Obstacle avoidance: At the very beginning, users received a detailed explanation about this test. Then, this test involved new head-level obstacles (see Fig. 8b). The GlobalViewAid prototype recognized the l obstacles and provided the users with audio feedback about the type, distance (in steps) and orientation of each obstacle. Users then used this information to avoid the obstacles. The test results and accompanying survey feedback are further detailed below.

3. Local navigation: At the very beginning, users received a detailed explanation about this test. Then, in this task, users navigated through an environment with a different configuration of head level obstacles (as shown in Fig. 8c). The prototype provided audio guidance, suggesting paths to avoid obstacles. Results of the task and survey responses are presented later in this section.

Fig. 8.

User tests scenarios. (a) Head level objects identification. (b) Obstacle avoidance. (c) Local navigation.

Figure 8 shows three examples of the head-level obstacle distribution. Each route is 3 m wide and 9 m long. The blue arrows indicate the recommended trajectory for the VIPs. All obstacles are hanging from the ceiling, and a 1-meter diameter circle (indicated by the purple circle) is marked on the ground below each obstacle, defining a warning region used to assess the risk of collision during the tests.

The user tests were performed considering the following global procedure: first, the users arrived at the Diagnostic and Therapeutic Support Coordinating Office in the University Hospital, Cali, Colombia; second, the users were asked about demographic data and the participant consent is explained in detail; third, the three tasks described above are performed previous explanation to the users is done about what to expect and do for each task; finally, users are given a snack.

The users demographic information include the following: age range between 22 and 50 years old; 50/50% of participation of women and men; 40% of users had secondary education, 35% of users had professional education and 25% of users were technicians; 50% of users had visual disability before the 5 years old; 60% of users had digital skills above the basic level, and 40% of users had basic digital skills (considering the European Union level to build the Digital Competence Indicator); and 65% of users were single and 35% of them were married.

Head level objects identification test

To perform this test, a set of instructions were given to the VIP in the following sequence:

1. Starting position: The VIP stands at the starting position with assistance from a companion.

2. Introduction: At this point, the test’s goal and procedure are explained The VIP is informed that they will be moving to specific positions, where the GlobalViewAid prototype will describe nearby head-level objects via audio feedback. They will continue this process at each position along the path.

3. System Activation: The supporting engineer starts the GlobalViewAid software in head-level object identification mode.

4. Initial Object Approach: With help from their companion, the VIP moves toward the first head-level object.

5. Audio Feedback: The VIP listens to the audio feedback and verifies whether they could hear the head-level object description. They can also touch the object to confirm this information.

6. Repetition: Steps 4 and 5 are repeated until the end of the path. Meanwhile, a digital backup of the identification process, performed by the GlobalViewAid prototype, is stored in the ROSbag log file.

7. Completion of Route: Upon reaching the end, the VIP is instructed to stop and assisted in returning to the starting position. The support engineer then stops the data recording.

8. Survey: Back at the starting position, the VIP completes a survey regarding the test experience.

9. Obstacle Rearrangement: The obstacles are rearranged before the next test to prevent participants from memorizing their locations.

Table 4 displays the head-level object identification results for all participants. The current GlobalViewAid prototype uses the YoloV5 Small deep-neural network model, which achieved a mean Average Precision (mAP) score of 0.89 in validation tests. Table 3 indicates an average success rate of 91% for the system. Notably, in cases where GlobalViewAid is unable to classify an object, it can detect that there is an obstacle and notify the user accordingly.

Table 4.

Head level object identification during the test.

Object	Percentage of success
Wall lamp	100%
Tree branch	100%
Chandelier	60%
Ceiling lamp A	100%
Ceiling lamp B	100%
Average	91%

At the end of the test, users were asked two key questions:

1. Were you able to fully understand the audio instructions?

2. How confident are you in the device’s ability to correctly identify objects for you?

Figure 9a and b present the participants’ responses. Figure 9a shows that the audio instructions were generally understood, since 85% of participants understood the audio instructions, while Fig. 9b reveals that around 60% of participants reported feeling “Very confident” or “Confident” in the device’s object identification ability. Based on user feedback, participants indicated they would have benefited from a learning phase to become more comfortable with the device before testing.

Fig. 9.

(a) Answers to question No. 1. (b) Answers to question No. 1.

In this test, even though a detailed explanation was done, it was obvious the lack of training in using the proposed device. The chandelier class had a relative low success since the used one was up to scale.

Obstacle avoidance test

To conduct this test, a series of instructions were provided to the VIP in the following sequence:

1. Starting Position: The VIP stands at the starting position with assistance from their companion.

2. Introduction: At this point the goal and procedure of the test are explained. The VIP is informed that he/she will start walking and the GlobalViewAid prototype will use audio feedback to indicate where obstacles are located, which he/she should then avoid.

3. System Activation: The support engineer initiates the GlobalViewAid software in obstacle avoidance mode.

4. Navigation and feedback: The VIP begins walking, listening carefully to the audio feedback instructions to help avoid obstacles nearby. Throughout the test, the VIP accompanied by an Orientation & Mobility Specialist.

5. Completion of Route: At the end, the VIP will be requested to stop and return to the starting position. Meanwhile, a digital record of the GlobalViewAid’s inference process is stored in the ROSbag log file and the support engineer stops data acquisition.

6. Survey: Once back at the start position, the VIP completes a survey regarding the test experience.

7. Obstacle Rearrangement: The obstacles are moved to a new configuration for the next participant to avoid memorization of obstacle locations.

In this test, a 0.7 m diameter circle was drawn around each obstacle on the floor to measure collision risk. This risk was classified as high, medium, or low. High risk indicates the VIP entered the circle; medium risk means the VIP stepped on the circle’s boundary; and low risk indicates the VIP passed near the circle boundary without touching it.

Figure 10 presents the collision risk levels for different obstacles. In case of tree branches, 70% of participants reported a low risk level (Fig. 10a). In case of ceiling lamp, 100% of participants reported a low risk level (Fig. 10b). In the event of having shelves, 70% of participants shown a low-level risk (Fig. 10c). The low-level risk for wall lamp was the 90% of participants (Fig. 10d). The low-level risk of chandelier was 75% of participants (Fig. 10e). On average, 90.5% of the tests showed a low or middle collision risk with only 9.47% resulting in a high-risk encounter. Some objects, like shelves, wall lamps and chandeliers, had high risk incidences of 25%, 5% and 15%, respectively as shown in Fig. 10f.

Fig. 10.

Collision warning level for (a) Branch tree. (b) Ceiling lamp. (c) Shelve. (d) Wall lamp. (e) Chandelier. (f) Average. Survey results. (g) Question (1) (h) Question (2) (i) Question 3.

It is worth noting that, although shelves were not recognized by the YoloV5 small model, the prototype could still detect their presence. Observing Fig. 10, it is notable that most of the head level objects have low or middle collision risk when the GlobalViewAid prototype was used.

When the obstacle avoidance test was completed, participants answered three questions:

1. Were you able to fully understand the audio instructions?

2. How was the speed of the audio instruction?

3. How confident are you to avoid more obstacles if you would repeat the test?

The survey responses are shown in Fig. 10g – i. For question 1, 90% of users reported fully understanding the audio instructions.The 10% who struggled mentioned they were not accustomed to rapid audio instructions and would have benefited from an initial warm-up to adjust to the device.

In Fig. 10h, feedback on audio speed shows that most users, accustomed to Talkback’s fast audio instructions were comfortable with the pace. For Question 3, regarding confidence in obstacle avoidance, 70% of users reported feeling “confident” or “very confident” in using the device. Comments indicated that familiarity with the device would enhance confidence, particularly with orientation cues. Most users preferred simple directional instructions like “front”, “diagonal left” or “diagonal right” over clockwise based instructions, though they noted that some practice would improve their comfort with these instructions.

In this test, many users suggest different audio feedback configurations which we are working on, even they suggest using beeps and 2D sounds which the new GlobalViewAid prototype already implemented in the new version. Given audio feedback used, it constraints the speed of walking of VIP, which is a feature to deal with in the new version of GlobalViewAid.

Local navigation test

In the Local Navigation Test, the GlobalViewAid prototype detects any head-level obstacles, and computes a safe path for the user to follow providing audio instructions to guide them around the obstacles, Similar to the Obstacle Avoidance Test, a 0.7-meter diameter circle was drawn on the floor around each obstacle to assess collision risk as either low, medium, or high.

Figure 11 shows the results of the collision risk for each obstacle type. In case of shelves (Fig. 11a), 80% of participants reported a low-level risk. The low-level risk for chandelier was 75% of participants (Fig. 11b). In case of wall lamp No. 1 (Fig. 11c), 80% of participants reported a low-level risk. The low-level risk for tree branch (Fig. 11d) was 65% of participants. And, in case of wall lamp No. 2 (Fig. 11e) 70% of participants reported a low-level risk. The data shows that on average 87% of users experienced a low or middle collision risk as shown in Fig. 11f. It can be noted in Fig. 11 that most of the head-level objects represent low or middle risk in the moment of suggesting safe paths by the GlobalViewAid prototype tested. Users generally found this navigation mode preferable compared to other modes, as noted in their feedback.

Fig. 11.

Collision warning level: (a) Shelve. (b) Chandelier. (c) Wall lamp No. (1) (d) Tree branch. (e) Wall lamp No. (2) (f) Average. Survey results: (g) Question (1) (h) Question (2) (i) Question 3.

Upon completing the Local Navigation Test, users were asked the following questions:

1. Were you able to fully understand the audio instructions?

2. How was the speed of the audio instruction?

3. How confident are you in avoiding more obstacles if you repeated the test?

Survey results are shown in Fig. 11g – i. For question 1, all users reported understanding the audio instructions. In response to Question 2, only 5% found the audio instructions too fast. For Question 3, 90% of users reported feeling confident or very confident in using the device.

In this test, users got more comfortable with the movement directions suggested by the GlobalViewAid prototype, since they are shorter, the VIP can walk quickly. However, users need to get use to diagonal orientations, since the instructions that include them tend to confuse users. Or, another option is include a training session to explain better these type of instructions.

Final remarks of user tests

A recurring question throughout the tests was, “Were you able to fully understand the audio instructions?”. Responses showed increasing positive feedback across tests, with 85%, 90% and 100%, of users reporting comprehension in each successive test, indicating that users became more comfortable with the device’s audio feedback over time.

Another common question was “How confident are you in avoiding obstacles if you repeated the test?”. User confidence grew across tests, with responses of confident or very confident rising from 60 to 90%. User comments suggested that an initial learning phase could further improve comfort and familiarity with the GlobalViewAid prototype. However, users demonstrated a quick increase in confidence even within the scope of the tests.

In the Obstacle Avoidance and Local Navigation Tests, users were asked about the audio instruction speed. Responses indicating the instructions were “too fast” dropped from 25 to 15%, suggesting that users adapted well to the device’s feedback pace as they became more familiar with its operation.

The user tests were performed between 8:00 AM and 5:00 PM, this range of hours include a good variation in illumination conditions. In terms, of real-time constraints, the more important constraint is the time spend in playing the audio feedback for obstacle avoidance. Which limits the walking speed of VIP, it is of course a feature to deal with in the next version of GlobalViewAid. Since, the GlobalViewAid depends on the stereo vision system, it is not recommended to be used at night or in low illumination conditions.

Conclusions

This work introduces a low-cost, stereo vision-based assistive device designed to enhance the mobility of visually impaired individuals by detecting and classifying head-level objects in real time. Using YoloV5 deep neural networks, the device provides intuitive audio alerts that empower users to navigate complex environments independently and safely. This technology addresses a significant need, as approximately one billion people live with disabilities globally, with 80% residing in low- and middle-income countries (LMICs). Currently, only 5–15% of these individuals have access to assistive devices, leaving the majority reliant on human assistance for navigation. By offering an accessible, affordable solution adapted to LMIC conditions, our device has the potential to reduce this access gap and improve autonomy and quality of life for millions.

The GlobalViewAid prototype was developed following the RUP methodology for software engineering. The key functional requirements of the head-level object classification solution included utilizing stereo vision and IMU sensor data, enhancing captured images, classifying nearby head-level objects, suggesting safe routes to avoid obstacles and delivering audio feedback to inform users of events in their surroundings.

The GlobalViewAid prototype uses a stereo camera to capture both visual range data. Pre-processing methods such as image filtering, blurry image detection, and histogram equalization were employed to enhance the input images. Object identification was achieved using YoloV5, optimized using tools such as TensorFlow lite and TensorRT for improved performance on embedded systems. Audio feedback was provided through concise, dynamic sentences, relaying object range and orientation, or the suggested safe routein a user-friendly format—converting range and orientation from the geometric world, i.e. meters into steps and angles into orientation instructions (as shown in Fig. 3c) based on the test results with VIP.

After training and validating the final neural models achieved a mAP@0.5 of 0.89 for head level object classification. To validate the solution, two additional tests were conducted prior to user testing:

1. Quantitative assessment of range and orientation estimation accuracy.

2. Field tests conducted under various lighting conditions and times of day.

The head-level object classification solution demonstrated an average range error of 0.028 m ± 0.004, and an average orientation error of 2.05°±0.09 providing a reliable representation of the environment. Field tests further revealed a precision of 98.21% and recall of 93.75%, confirming the solution’s efficiency in real-world conditions. The system is compact, portable, and affordable, making it a valuable option for improving the navigability of VIPs.

User tests were conducted with 20 VI participants at the Hospital of Universidad del Valle, Cali, Colombia, covering a range of ages, experience, and digital skills. Three tests were performed: head-level object identification, obstacle avoidance and local navigation. As a result, in head-level object identification, the GlobalViewAid system achieved a success rate of 91%. In the obstacle avoidance test, 90.5% of the user tests resulted in a low or middle risk with users entering the defined “danger zone” just 9.47% of the time. Certain Head level objects, shelves, wall lamps and chandeliers, have high risk measures of 25%, 5% and 15% respectively. For local navigation, 87% of the trials were classified as low or middle risk. It can be noted in Fig. 11 that most of the head level objects represent low or middle risk in the moment of suggesting safe paths by the GlobalViewAid prototype tested. Considering the user comments, they felt that this mode of operation of the electronic device was liked more in comparison with the others.

Notably, users became progressively more comfortable with the audio instructions: 85%, 90%, and 100% to the question “Were you able to fully understand the audio instructions?”); the users get more confident as the tests passed (increasing percentages of 60%, 70% 90% to the question “How confident are you to avoid more obstacles if you would repeat the test?”); and, users progressively better perceive the audio feedback regardless its speed (decreasing percentage of 25–15% to the question “How was the speed of the audio instruction?”).

Therefore, by offering an affordable, adaptable, and effective mobility aid, this solution has the potential to significantly reduce the existing AT access gap, enhancing autonomy and quality of life for millions of visually impaired individuals.

Future work: Based on these findings, future enhancements to the GlobalViewAid prototype include additional audio feedback formats, extending object recognition to cover low-level obstacles, utilizing the computed local map to detect dynamic hazards, and developing a comprehensive global navigation system with obstacle avoidance features.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Author contributions

Muñoz Kevin, main software developer.Chavarria Mario, project manager, review and correct the manuscript.Ortiz Luisa, perform user testsSutter Silvan, project manager, review and correct the manuscript.Schönenberger Klaus, project director.Bacca-Cortes Bladimir, project leader in Colombia, and wrote the main manuscript text.

Data availability

Survey results about the context study: https://correounivalleeduco-my.sharepoint.com/:f:/g/personal/bladimir_bacca_correounivalle_edu_co/ErRkJnnTI-FDoZlncxAdYYkBFDkdBFI6F-9W7t4vS3rSew? e=5JtKSgUser test results surveys: https://correounivalleeduco-my.sharepoint.com/:f:/g/personal/bladimir_bacca_correounivalle_edu_co/EqjEP13PYSBFkDESUnRJZeMBtDkch5qhPW4MhK5Y1nil9A? e=yp3v2nSoftware of the embedded system developed: https://correounivalleeduco-my.sharepoint.com/:f:/g/personal/bladimir_bacca_correounivalle_edu_co/EpBLnRaIB0NJnBMn0SMOkT8BtoHaz1JcWNViwfWDqLkBEg? e=B6ZXMT.

Declarations

Competing interests

The authors declare no competing interests.