Variability in Reactions to Instructional Guidance during Smartphone-Based Assisted Navigation of Blind Users

Abstract

‘Turn slightly to the left’ the navigational system announces, with the aim of directing a blind user to merge into a corridor. Yet, due to long reaction time, the user turns too late and proceeds into the wrong hallway. Observations of such user behavior in real-world navigation settings motivate us to study the manner in which blind users react to the instructional feedback of a turn-by-turn guidance system. We found little previous work analyzing the extent of the variability among blind users in reaction to different instructional guidance during assisted navigation. To gain insight into how navigational interfaces can be better designed to accommodate the information needs of different users, we conduct a data-driven analysis of reaction variability as defined by motion and timing measures. Based on continuously tracked user motion during real-world navigation with a deployed system, we find significant variability between users in their reaction characteristics. Specifically, the statistical analysis reveals significant variability during the crucial elements of the navigation (e.g., turning and encountering obstacles). With the end-user experience in mind, we identify the need to not only adjust interface timing and content to each user’s personal walking pace, but also their individual navigation skill and style. The design implications of our study inform the development of assistive systems which consider such user-specific behavior to ensure successful navigation.

1. INTRODUCTION

Designing the guidance feedback of assistive indoor navigation systems requires carefully considering the specific needs of the end-user [6, 56]. While technologies for navigation assistance have been previously developed by researchers [2, 11, 16, 17, 34, 69], current turn-by-turn guidance systems for people with visual impairments often do not consider individual differences between users. However, such differences can potentially impact the navigation experience and the ultimate success of the navigation task [46]. For instance, consider a scenario where the system may ask the blind user to ‘turn slightly to the left’ to merge into a different corridor. However, due to longer reaction time, the current user turns too late and proceeds into the wrong hallway. In this example, personal user motion characteristics (e.g., reaction time, task performance, over-turning, etc.) can impact the success of the navigation task. In an unfamiliar environment, the recovery from a navigation error such as this one may be lengthy [2]. Therefore, a lack of a model for personal differences in reaction characteristics can result in navigational errors, collision, or user confusion.

In this work, we focus on analyzing personal characteristics in real-world user motion data with respect to different instructional context. To gain insight into how effective navigational interfaces can be designed, our work studies two interconnected but less explored components of the interaction between a blind user and a navigation system, (1) data-driven analysis of user motion behavior, and (2) the role of inter-user reaction variability.

We propose a data-driven analysis framework of user motion (i.e., reactions) following interface guidance cues. In-depth, quantitative analysis of blind user reaction characteristics is often lacking in related research studies [17, 32, 56]. Studies in assistive indoor navigation systems often evaluate system performance in terms of system localization error, overall route statistics (e.g., total completion time, total incidents, etc.), and subjective user reports [11, 16, 18, 20, 34, 38, 41, 47, 60, 69]. Moreover, studies are mostly conducted in lab or in simplified environments. In contrast, our work employs continuous tracking of real-world users to extract and summarize a set of data-driven reaction measures. The proposed framework provides quantitative trends which can be automatically extracted from sensor data, hence useful when analyzing behavior in larger-scale settings and facilitating a rigorous comparison among future studies. Our findings regarding user behavior, challenges, and strategies are leveraged to infer several novel system design implications, as discussed next.

We use the analysis framework to argue that the navigation interface should support each user’s unique navigation style to ensure pleasant navigation. Specifically, we uncover the significant impact of inter-user variability on the end-user experience. The practical issue of accommodating the information needs of diverse users has been discussed in previous research only briefly and qualitatively [17, 22]. Hence, we emphasize it throughout the quantitative analysis. We find inter-user variability to play an essential role in determining the quality of the interaction experience and ultimately the success of the navigation. We explain the variability by discussing differences in navigation techniques and personal mobility style among users. By examining summarized motion measures within each user, across multiple users, and across varying instructional context, we tie novel insights regarding user behavior patterns with implications for system design. The implications can be used to enable a more useful collaboration between the system and varying types of users.

To better understand the role of reaction variability of blind users in assisted navigation settings, we studied sensor data collected in a real-world deployment of a smartphone and Bluetooth beacon-based navigation solution over a multi-building area. The analysis of motion and timing reaction measures in a total of 1,553 interaction events is used to uncover different system-user interaction patterns which can impact the navigation success. To our knowledge, this is the first study to employ automatic extraction of sensor readings to analyze, on a time scale of seconds, blind user path following in a real-world large-scale indoor navigation study.

1.1. Key Findings

Our study revealed three main findings. First, we find that timing of instructional content, in particular when approaching turns, should be made adaptive. Through extensive analysis on measures of reaction time and motion, we demonstrate how different users can end up several meters apart (i.e., reaction time and type vary significantly) during a task which is critical to the navigation task. Hence, successful turning becomes challenging in certain environmental context. To avoid missing turns due to late or early turning (the most common cause of navigation error in the data), the interface timing should accommodate users traveling at different speeds and reacting to instructions differently. Yet, existing navigation interfaces do not consider tailoring timing of instructions to different users.

Second, we find that our defined reaction measures allow for automatic identification of a variety of fine-grained user-specific navigation strategies, including cane techniques, cautious navigation, or challenging navigation tasks, as discussed in Section 5.3. This finding is encouraging, as it enables real-time systems that can adjust the content of the guidance (e.g., level of context and verbosity) across diverse users and user states.

Third, beyond analysis of task-dependent variability among individual users, we cluster user motion trends to identify broader interaction patterns and system design implications. We find that users can be categorized into roughly four broad interaction trends (Section 5.5), based on how they interpret and utilize different types of instructions, e.g., slowing down to a near stop during turns cautiously with careful attention to instructions or maintain speed with minimal slow down.

In addition, we present several additional novel findings about user behavior during specific navigation tasks, such as large variability in slight turns and reaction during large turns. The findings highlight information needs and opportunities in designing better assistive navigation systems for people with visual impairments.

2. RELATED RESEARCH

User behavior analysis and modeling (e.g., task performance time, reaction time, motion analysis) has been long-studied in interactive and ubiquitous technologies [23, 24, 57, 59, 62, 70]. Inspired by behavior and reaction time models in other application domains, such as driving-based applications with sighted people [19, 30] and interactive computer systems [12, 27, 35, 39, 50, 58], we analyze the motion of blind users in reaction to navigational guidance in real-world settings. Below, we compare our work to relevant studies analyzing the mobility and navigation behavior of blind people.

2.1. Behavior Analysis in Mobility of Blind People

Researchers have employed a variety of measures to analyze the impact of different training strategies and assistive technologies on the independent mobility of blind people [37]. Such studies are relevant to our work, as we aim to better understand how to meet the needs of a blind user under assisted speech-based navigation. Soong et al. [55] and Black et al. [8] studied walking efficiency and error score of visually impaired people with and without O&M training. Most commonly, studies emphasize route completion time or number of navigation errors, with only anecdotal or subjective feedback regarding behavior along the route.

Several studies examined reaction times and gait of visually impaired people during general mobility [43, 61]. Turano et al. [61] examined walking speeds and reaction times of visually impaired people to randomly emitted sounds, showing significantly higher reaction times when compared to sighted people. In contract, we focus on reaction time and speed measures during real-world assisted navigation, and not in a controlled experiment. For instance, in our application users are already informed of the upcoming navigation tasks, such as turns, before arriving to them.

2.2. Navigation for People with Visual Impairments

Technologies for mobility assistance have been extensively studied by researchers [17, 20, 68]. Current approaches include both solutions to provide spatial knowledge of the environment prior to navigation, such as maps (i.e., tactile or interactive) [15] or virtual navigation [21], and in-situ navigation assistance through turn-by-turn instructions [2, 13, 33] or landmarks about the surroundings [9, 31]. In the context of navigation for people with visual impairments, accurate systems often rely primarily on speech output (sometimes complemented with sonified and vibro-tactile cues), the preferred modality for navigational interfaces for people with visual impairments [4]

Indoor Localization.

Achieving accurate indoor localization has been extensively studied by researchers [16, 18, 38, 41]. For instance, Otsason et al. [47] presented a GSM indoor localization system in 2005 with a median accuracy of five meters. Moreover, walking detection and step counting using the smartphone sensors have been used in Pedestrian Dead Reckoning (PDR) systems to estimate the user’s movement and respective indoor location [10, 34]. Such solutions are often combined with other techniques that use sensors in the environment (e.g., Wi-Fi or Bluetooth Low Energy beacons) in order to increase the localization accuracy and robustness [25, 26]. Other approaches include using computer vision to improve the localization accuracy or to detect and avoid obstacles [11, 36, 69]. Although an important technological milestone of practical localization has been recently reached, the great majority of studies evaluate system performance in terms of either system-centered measures, such as localization error [16], or overall route statistics [16, 28, 44] and subjective user reports [16, 49]. Although in our application interaction occurs by providing frequent instruction feedback to the user so that the user may react and move accordingly, we find little research on deeper analysis of fine-grained motion measures of blind users during real-world assisted navigation.

Motion and Reaction Analysis.

Related studies have performed preliminary analysis of reactions in controlled or simplified environments. Wachaja et al. [64, 65] estimated overall reaction time and velocity during turns with blindfolded participants and a smart walker. However, it is known that blindfolded and blind people have significantly different mobility skills and strategies [54]. Adebiyi et al. [1] assessed feedback modalities provided by a human following behind participants through an obstacle course. In contrast, we emphasize inter-user variability in identifying insights in real-world user behavior and system design limitations during long navigation routes, which has been relatively unexplored (see Table 1). Our study also demonstrates the feasibility of automatic data-driven techniques for studying such motion variability.

Table 1.

Comparison to representative related research studies.

Research Study	Number of Participants	Indoor	Personalized Route Preference	Visual Impairment	Motion Tracking	Reaction Variability
Wang et al. [67]	11	✓		✓	✓
Akasaka and Onisawa [3]	11		✓		✓
Thorsten and Gerhard [63]	-	✓	✓	✓
Macik et al. [40]	-	✓	✓
Fallah et al. [16]	6	✓		✓	✓
Riehle et al. [51]	8	✓		✓	✓
Sato et al. [52]	10	✓		✓	✓
Ivanov [28]	8	✓		✓	✓
Adebiyi at al. [1]	11	✓		✓
This Work	12	✓		✓	✓	✓

Personal Needs in Navigation Interfaces

Investigation of the personal needs in navigation interfaces goes beyond route preferences [40] and different feedback modalities. In the particular case of people with visual impairments, selecting the best content and the timing of the instructions is a challenging task, yet crucial for reducing navigation errors [7, 14, 22]. While studying user reactions to the interface can enable improvements in the quality and success of the navigation, a detailed large-scale analysis of reactions among different users for different instruction types as well as overall motion patterns for groups of users is lacking. Table 1 summarizes similar studies that also have a representative number of participants, depicting that little previous work has investigated reaction variability and its implications for blind navigation.

3. EXPERIMENTAL SETTINGS

Environment.

To quantify the extent of reaction variability among users, we utilize real-world user motion data collected in a large-scale deployment of a smartphone-based turn-by-turn navigation system. While walking across floors in a public building, users traveled a total length of 400 meters, including diverse points of interest, varying spatial layouts (e.g., both large open spaces and narrow hallways), surrounding people, frequent obstacles, elevators, and automatic doors. Over 200 Bluetooth beacons, positioned at 5-10 meter intervals across multiple floors, were used for providing step-by-step navigation guidance.

Participants.

Data from 12 participants was used in the study, as detailed in Table 2. The data was collected under IRB and video consent of the participants. All participants were white cane users besides P8 who had a guide-dog. Participants age ranged 38-65, with an average 46.08 (SD: 8.46).

Table 2.

Participants’ demographics. ‘Since’ column shows blindness onset age).

ID	Gender	Age	Visual Acuity	Since	Mobility
P1	M	54	Totally Blind	2y	White Cane
P2	M	38	Totally Blind	5y	White Cane
P3	M	48	20/2000 both eyes	28y	White Cane
P4	F	48	Totally Blind	0y	White Cane
P5	F	40	Totally Blind	20y	White Cane
P6	M	42	20/500 Left, Blind Right	1y	White Cane
P7	F	53	20/500 Left, Blind Right	49y	White Cane
P8	F	44	Totally Blind	10y	Guide Dog
P9	M	65	Totally Blind	12y	White Cane
P10	M	33	Totally Blind	30y	White Cane
P11	F	42	20/2000 Both Eyes	–	White Cane
P12	F	46	20/500 Right, Blind Left	0y	White Cane

Apparatus.

We employ navigation data collected using the open-sourced turn-by-turn indoor navigation smartphone app developed in [52]. The participants received a short training session regarding the app, the instruction it generates, and its usage before the onset of the navigation. The smartphone app determines the user location along the planned path and produces turn-by-turn navigational instructions with contextual information about the surroundings. Given an annotated map of the environment, the system provides information about landmarks in the scene, such as the presence of elevators, obstacles, doors, restrooms, etc.

3.1. Semantic Clustering of Instructions

For meaningful analysis, we semantically cluster instructional guidance cues (over 100 in total provided along the route) into 8 main categories, with an example visualized in Figure 2. The clusters are, (1) Approaching notifications are presented to the users at a fixed distance before any type of turn. (2) Forward notifications include ‘proceed 80 ft forward’ and ‘you are 100 ft from your destination.’ As can be seen in Figure 2, such notifications often follow a turning point. (3) Obstacle notifications include ‘obstacle on your right side,’ as portions of the route include chairs, signage, and other types of static obstacles. (4) Info notifications provide information about the area, such as ‘automatic doors ahead’ or points of interest, such as shops, landmarks, or changes in floor types. Turn notifications are crucial components of the navigation [2]. Turns are categorized by magnitude into the following, (5) Large turns (90 degrees, e.g., turn right or left), (6) Small turns (60 degrees or less, e.g., turn diagonally or slightly to the right or left), and (7) U-turns. Users were provided information regarding the turns and their types before the onset of the experiment. We pair slight turns (i.e., when user orientation slightly needs to be adjusted to proceed into a corridor) and diagonal turns (45 degrees) as we found that users had a difficult time gauging their exact orientation changes for these and reacted in a similar manner. Moreover, combining the small turns provides a meaningful analysis as otherwise the number of events is sparse (the route contained more large turns than small turns). We also removed U-turn from the analysis as it only occurred in cases where participants veered off the path, which occurred rarely and only with a subset of the participants (6 of them). Once a turn notification is made and the user begins the turning motion, a (8) Sound and vibration follows when the user achieves the correct orientation for the turn, which is the final category for the reaction analysis. Not including this turn completion feedback, there are a total of 1,553 audio notifications in the dataset (i.e., user-interface interactions).

Fig. 2.

Example user trajectory in the floor plan, with overlaid instructional guidance categories. Notice how a user over-turns a diagonal turn and begins to veer off the planned path.

4. DATA-DRIVEN USER REACTION MEASURES

The goal of the navigation system is to assist users in their movement to the destination. To characterize user reactions along the planned route, we employ several motion measures of speed and timing, as described next. The measures are extracted following instructional cues to analyze the impact of the guidance. By employing measures which can be automatically extracted from sensor data, we are able to quantitatively analyze immediate user reaction in a scalable manner, facilitating larger-scale analysis and comparative evaluation in future studies.

4.1. Post-Instruction Average Linear and Angular Speed

The most basic characterization of user behavior involves the walking speed of users. It can be used to depict general trends in user behavior following different instruction types. The extent of variability in user speeds is therefore useful for discussing the feasibility and challenges in adaptive interface design (i.e., impact of timing) and studying user navigation strategies (i.e., angular speed when approaching a turn).

Formally, at each time step t, the user is parameterized by a position in the map and an orientation, P_t = (X_t, Y_t, θ_t). The user’s instantaneous velocity V_t can be computed from P_t at each time step. The Euclidean norm of the first two components of V_t forms the linear speed $S_t^L$ in meters per second, referred to as speed hereafter, and the magnitude of the third component provides the angular speed $S_t^A$, in radians per second. These two measures are useful for obtaining overall motion trends across users, as well analyzing variability across instructions within the same user For instance, users may walk with an overall slower speed during a ‘turn right’ notification as opposed to a ‘proceed forward.’ Based on our survey of related literature, while there has been reporting of average walking speed in general mobility, reports of walking speed variability among users during different instructional guidance are lacking. To better understand real-world usage of assistive navigation technologies, we report such values and compare their average and standard deviation (SD). To produce more easily interpretable results than the raw speed signals, we consider their averages over a temporal window following the onset of the instruction. We plot $\frac{1}{T}{\Sigma }_{i=0}^T{S}_i$, where i = 0 is the onset of the instruction, T is the length of the temporal window, and S may be $S_t^L$ or $S_t^A$.

Since reactions may last several seconds, the temporal window must allow sufficient time for the user to hear the full instruction, interpret it, and act accordingly. In the comparative analysis among instructions, we average speed values over a window of 6 seconds, beginning at the onset of the instruction. A value of 6 seconds was empirically determined based on observations from the data, in particularly for turns (see Figure 5). For ‘approaching a turn’ notifications, the window is set to be at 4 seconds due to the short notification and to avoid including the turn instruction which follows. Although these appropriate values were determined empirically, in the experiments we verified that small changes of up to a couple of seconds in the window size has minimal impact on the significance of the statistical analysis.

Fig. 5.

For in-depth study of the variability in turn task performance, we decompose a turn to components, including (a) Instruction onset to turning onset time, (b) instruction onset to turn completion time, and (c) correct heading feedback to turn completion time.

The computation of the speed measures will change in Section 5.4 to accommodate longer instructions (e.g., info), by computing speed following the completion of the instruction announcement as opposed to its onset. The original, pre-averaged temporal signals are provided in a supplementary for the interested reader.

4.2. Post-Instruction Average Speed Change

A user may maintain, slow down, or increase their speed because of an instruction. To capture the impact on user motion, we study an additional speed measure which better highlights personal variability. Based on the observation that each user’s average walking speed may be significantly different, we also analyze the speed change incurred due to the instruction. To normalize for the actual user speed, we compute $\frac{1}{T}{\Sigma }_{i=1}^T(S_i-S_{i-1})=\frac{1}{T}(S_T-S_0)$, in m/s².

In contrast with average magnitude of user acceleration, this measure can summarize the change in speed due to an instruction as well as its signed direction. Also, we note that here we are only concerned whether any impact by the instruction on the user speed, and not whether its occurring during a forward or a circular motion (i.e., acceleration). While this speed change measure can provide more meaningful inter-user comparison of reaction, it has certain limitations, in possible cases where user may slow down and then re-gain speed back within the average time window. We avoid such issues by (1) keeping the temporal window small, (2) inspecting the original temporal signals (in the supplementary). We note that in the case of angular speed, this type of speed normalization is not as meaningful, since users are generally expected to walk forward when no-instruction is provided, i.e., an angular speed of 0.

4.3. Timing

Measuring reaction and task performance time has been long studied in the research community as means for understanding interactivity. Variability in reaction time among users is a phenomenon that could be leveraged by an adaptive interface, for instance to gauge appropriate timing of instructions and reduce the chance of navigation errors. We study a complementary motion reaction measure to the speed dynamics which involves the amount of time between instruction onset and onset of the task performance by the user, i.e., beginning of a turning motion.

Specifically for turn notifications, the onset of the user reaction and the completion of the turning task can be identified automatically by inspecting the user’s orientation signal. Hence, for these notifications it is also meaningful to explore variability in the time between instruction onset and the onset or completion of the turn maneuver in seconds. Since we find large variability in turn task performance, in the analysis we decompose turns into subtasks and compute timing statistics for each subtask. These subtasks include reaction time for turning, which is the time between the onset of instruction announcement and the beginning of the turning action. Also, reaction time for the correct heading feedback before the completion of the turn. The timing measures are useful to understand how users interact with guidance cues as well as gaining insight into the optimal system design.

4.4. Implementation Details

The dataset for the analysis was collected using the turn-by-turn indoor navigation smartphone app of [52]. The app is paired with a localization algorithm to provide the instructional feedback. As discussed in the related research studies, solutions for localization based on Pedestrian Dead Reckoning (PDR) and the smartphone’s Inertial Measurement Unit (IMU) [10, 34] often suffer from drift and insufficient accuracy for providing navigation instructions for blind people. As a tradeoff between cost and accuracy, user tracking is performed with both IMU and Bluetooth Low Energy (BLE) beacon sensors placed in the environment. The accuracy will be empirically validated on the dataset in Section 5.1. The IMU and BLE sensor readings are fused using a Particle Filter [5] to track the location and heading of the user. The algorithm is efficient for continuously estimation the user’s pose and implemented using the open-source library of [53]. The smartphone app was modified to log all user pose data, recorded at 50ms intervals, as well as the time of instruction announcement onset and instruction announcement completion.

5. ANALYSIS AND RESULTS

Our analysis is organized as follows. For completeness, Section 5.1 details an error evaluation analysis of the employed sensor measurements. To validate the proposed measures and discuss general interactivity in our domain, Section 5.2 depicts overall motion patterns in the dataset across all users within each speech instruction type. Next, Section 5.3 presents a focused analysis of inter-user variability for each instruction type. Section 5.4 discusses overall trends in inter-user variability among different instructions, which are then used in a cluster analysis in Section 5.5.

Statistical testing is performed to ensure statistical significance in user behavior variability. Unless otherwise specified, for each test we fix the action type as the independent variable for studying variability across instructions, and the participant id as the independent variable when studying inter-user variability. When reporting findings, we fix a motion measure (e.g., average angular speed) as the dependent variable. In all the experiments normality is first verified using a Shapiro-Wilk test, and followed by ANOVA under normality or a t-test otherwise, with an alpha level of .05.

5.1. Measurement Error Evaluation

Our study emphasizes a data-driven approach for analyzing user behavior in response to instructional guidance. We argue that our sensor-based approach has several advantages for large-scale analysis of assistive navigation interfaces for blind users, as it does not require manual annotation. Before dwelling into meaningful trends in the user motion data, we sought to evaluate the inherent sensor error in the measurements employed (smartphone and beacon-based localization). We use the annotations for reporting error rates so that researchers performing large-scale data-driven analysis may be able to better compare with this work in the future.

We employ hand annotated user positions at fixed 1 second intervals. As we leverage sub-second timings in some of the behavior measures (e.g., reaction time to instructions), we do not employ the annotations in our main analysis, but only use them to validate our approach. The average localization error rate of the state-of-the-art indoor localization system was determined to be 1.8m on average (SD=1.28). The speed error on average is 0.10m/s (SD=0.09) (complete details, such as per-instruction error, can be found in the supplementary material). The timing measurements analyzed in our work are not influenced by the localization accuracy of the system but by the gyroscope, which we found to be accurate to about 10 degrees. While motivating us to pursue more accurate systems in the future, we generally find low error rates with similar motions trends between annotated vs. sensor-estimated data, affirming our choice of sensor-based data analysis.

5.2. Variability across Instructions

To understand overall user behavior in the data, speed measures across different instructions for all users are depicted in Figure 3. In Figure 3(a), the variation among all instructions is statistically significant (F(5,1547) = 44.55, p < .0001) for linear speed (with instruction type as the independent variable) but only due to the ‘approaching’ instruction. A further study showed no statistical significance among the other 5 types of instructions once ‘approaching’ is removed. While identifying user’s average walking speeds is important for comfortable timing of instructions, this initial finding demonstrates that there may be more variability within each instruction (i.e., different users) as opposed to among instructions. To contextualize the user-specific variability analysis which follows in the next subsections, we first explain these findings of general interactivity patterns in greater detail.

Fig. 3.

Overall motion variability across different types of instructions. (a) Average linear speed in a temporal window following each instruction type, (b) average speed change, and (c) average angular speed. Black bars show standard deviation.

Approaching Notifications Have Narrow Context.

Although user-agnostic statistics limits our analysis, we find ‘approaching’ notifications to result in significantly higher average speed than other types of instructions. The reason ‘approaching a turn’ generally results in higher speeds is that they always occur at each user’s undisturbed pace or even during an acceleration (in the case of multiple turns in close spatial proximity, as validated by inspecting the pre-averaged temporal signals in the supplementary). Moreover, since the main purpose of ‘approaching’ instructions is to prepare users to an on-coming turn, users can expect an additional time before the need to slow-down for the turn and the following ‘info’ and ‘forward’ instruction. While we do find average speeds during ‘info’ and ‘forward’ notifications to be generally higher when compared to turning and obstacle notifications, these are longer instructions, as opposed to the short and constant instruction of ‘approaching.’

Speed Reduction Following Info.

Average speed changes reveal additional insights in general interactivity under assisted navigation, as shown in Figure 3(b). In general, we find that users tend to slow down under instructional guidance, but to a varying extent (F(5,1547) = 61.3, p < .0001 among instructions). As users tend to slow down (often to a near stop) during turns, we find statistically significant variability between ‘large turns’ and all other instruction types, with the largest reduction in speed overall. Comparing ‘forward’ and ‘obstacle’ with other instructions, we also find that all differences are statistically significant. ‘Forward’ notifications often occur following turns, as users re-gain their normal walking pace, or as concise notifications during long forward stretches, hence the large difference to the other instructions. While reduction in speed during turns is expected (as users gain angular speed, Figure 3(c)), we also observe a reduction in speed during ‘info’ instructions, which can be explained in two ways. First, the ‘info’ instructions are often long to utter (unlike ‘approaching,’ which is a short instruction that always repeats in the same manner). As previously shown in literature [61], mental workload can also significantly impact reaction of blind and sighted people. Hence, the overall reduction in speeds due to instructions (even without turning) can be explained by an increased attention to the instructional guidance, which was also apparent from the video recordings. Comparing speed changes following the end of the instruction announcement (as opposed to its onset, Figure 9) aids in clarifying this phenomenon further and will be discussed in Section 5.4. A second reason is that the information provided includes cues related to the navigation (e.g., ‘elevator ahead’), and hence results in a slow-down as users adjust their mobility accordingly.

Fig. 9.

Motion statistics of the participants following instruction onset (a-c), instruction announcement completion (d-f), and the difference between the two (g-i, for each column) which highlights overall trends and user-specific variability during the realization of instructions.

A deeper analysis will reveal sharp differences on an individual basis among these instructions, which will also help explain the higher standard deviation in speed changes in some of the instruction categories, such as in ‘obstacle’ and ‘small turns’ notifications (Figure 3(c)).

Angular Speed and Navigation Tasks

Analyzing the angular speed during different instruction types (Figure 3(c)) reveals statistical significance (F(5, 1547) = 65.12, p < .0001) between the instructional groups, as expected. In pairwise comparisons we find ‘large turns’ to be significantly different to all other instruction types, including ‘small turns’. ‘Small turns’ is also statistically significant from the rest, besides ‘obstacle’ notifications, due to the small angular motion in both instances. ‘Info’ and ‘forward’ are not found to be statistically significant from each other in their average angular speed, and ‘approaching’ still significantly differs from all other groups, with minimal turning. These findings can be explained by users tending to turn slightly around obstacle notifications (e.g., ‘chair on your left’) or veer and correct when proceeding forward. Unlike ‘approaching,’ ‘forward’ notifications occur immediately after a turn completion, when users may correct their heading immediately when meeting a wall or an obstacle in a hallway, hence the increase in average angular speed.

5.3. Variability across Users

Our aim is to explore the role of user-specific differences during assisted navigation, so that interfaces can be better designed to accommodate individual differences. To study per-user variability, we will next analyze the speed measures on an individual basis within each instruction type. Throughout all the plots and tables, participants are always sorted according to their average speed change during large turns 4(b), as there is large variability in this reaction measure.

5.3.1. Turns.

We begin with turn instructions (Figure 4) as reactions to ‘turn’ notifications reveal the most significant inter-user variability, with specific insights to timing of the instructional guidance. Variability in linear speed is significant both for large turns (F(11, 174) = 4.22, p < .0001) and small turns (F(11, 135) = 4.16, p < .0001). We find that the guide-dog user P8 is generally faster than other participants, but cane users (e.g., P4) can maintain similar speed during turns, so that even when testing without the guide-dog user variability among participants is still significant (p < .001). Since reactions to ‘turn’ notifications involve the largest reduction in speed among all instruction types (Figure 3(b)), the statistical difference remains in average speed change (p < .001 for both large and small turns among users). The large spectrum includes P1, who tends to maintain speed at turns with only a slight overall reduction (Figure 4(b, e)), and P12, who reduces speed significantly. Hence, we can see that timing such notifications to minimize navigation errors is not just dependent on user speed, but also their user-specific reactions. As will be shown, users may travel several meters during their reaction, which together with the inherent localization error pose a challenge for successful turning.

Fig. 4.

Comparison of (a) average speed, (b) average speed change, and (c) average angular speed, following ‘large turn’ (90 degrees) and ‘small turn’ (less than 60 degrees) instructions. Black bars show standard deviation.

Small Turns are Difficult.

In the case of large turns, we do not find the angular speed variability to be statistically significant among users (p = .516), yet it is significant for small turns (p = .013). This can be explained by the physically easier task of gauging orientation during a sharp turn (90 degrees) compared to diagonal or slighter turns. Also, some users treat small turns with less care. While all users were instructed and trained on the differences between turn types and their extent before the study onset, small variations in orientation could easily result in over or under-turning, and consequently veering off the planned path. As a concrete example, we find that a diagonal turn in an open space area along the route caused P3, P5, and P8 to veer off the path in several occasions. Interestingly, P5 and P8 also maintain the largest angular speed average during small turns among all the participants, as shown in Figure 4(f). One example scenario is depicted in Figure 2 for P3, where the user over-turns following an instruction to turn diagonally to the left (fortunately, the presence of poles prevents the user from further veering off the path). Hence, we can see how considering personal characteristics in reactions to turn instructions can be useful for developing interfaces which better prevent user-related navigation errors. These novel observations are not found in related studies yet have significant implications on system design. While occasional over or under-turning is expected, the observation that these are user-specific differences provides several opportunities for improved interface design which will be discussed in Section 6.

Timing-based Decomposition of Turns.

Correct turning is a critical component of the navigation, involving a multi-step process. We were not able to identify related research studies with similar fine-grained insights about user behavior during assisted navigation. Hence, we further analyze reaction and task completion statistics following turning instructions, also finding significant variability among users. To study the variability in depth, we compute a variety of timing statistics (as mentioned in Section 4.3), with three main ones plotted in Figure 5. The remaining timing statistics can be found in the supplementary material. We decompose a turn into components to analyze (1) the reaction time of announcement onset to turn onset, (2) overall task performance, and (3) time from correct heading sound feedback to the end of user turning motion. For the latter, the feedback is meant to be used for proper re-orientation during the turn, yet we find reaction to it takes another 0.87s on average, SD=0.62. This system-specific insight suggests a more careful design of this functionality.

Variability in Turn Subtasks.

In general, we find that the significant variability occurs within components of the turning task, including reaction time and heading feedback reaction time. The reaction time between onset of instruction and beginning of the turning action is shown to be significantly variable among users (F(11, 298) = 3.23, p < .001). While the average reaction time (average 1.43s, SD=1.18) is close to the one reported in related literature [1], related studies are often performed in controlled settings and therefore miss the crucial finding of significant variability among users. In contrast, during real-world independent smartphone-based navigation, we show it to be highly variable in Figure 5(a). We also find that the total time to complete this task may last over 4 seconds for some users (average 3.52s, SD=1.17, p < .0001) as shown in Figure 5(b). We perform additional analysis to explain this variability.

Is It Due to Age Differences?

One possible explanation for the variability could be users’ age, but this was not confirmed by our data. While our user population is generally older (6 participants are under 46 years old), we attempted to group participants into two groups using varying age group definitions but found no statistically significant differences with respect to reaction time or overall task time. While the number of participants is representative to other studies in our application, this hypothesis needs to be further studied in the future with a larger participant pool. A similar conclusion holds for other personal attributes of the users.

Variability in Navigation Strategies During Turns.

We explain this phenomenon by inspecting each user’s mobility during the navigation. Specifically, some users are able to better anticipate upcoming turns. The insight came by inspecting the video data, in particular between the ‘fastest turner’ P1 compared to a slower one, such as P9. P1 is able to use several navigation strategies to achieve correct and quick turning. Since users are informed of the upcoming navigation turning point and turn direction after completion of the previous turn, P1 constantly checks for an available turn with the cane in the notified direction (only with the cane, at the same time while still walking forward). Hence, the moment a turn is available, P1 immediately begins turning, even before the turn notification completes. On the other hand, P9 walks in the middle of the hallway, and does not employ a similar strategy. Instead, P9 simply awaits the next instruction while using the cane to avoid obstacles, but not for anticipating or seeking turns. Also, P9 pauses before turning, and turns very cautiously as if hesitating before acting due to a notification. Inspecting videos with other users, we found each to be on a spectrum, between the behavior of P1 and P9 in terms of mobility strategies. Hence, we conclude that personal differences in mobility skill and style help explain variability in reaction to turn notifications. Section 6 discusses how such insights could inform the system design.

5.3.2. Approaching.

Based on our previous analysis in Section 5.2, overall behavior following ‘approaching’ was shown to be significantly different from all other types of instructions in terms of average speed. The higher speeds are due to the context in which this instruction occurs, which is usually while the user is at or approaching the normal walking pace.

Three Types of Users.

We find variability in average speed change to be statistically significant (shown in Figure 6(b), F(11, 298) = 3.19, p < .001). Generally, users reduce their speed as they approach an upcoming turn, but some significantly more than others, leading to roughly three types of observed behaviors (also discussed in detail through user cluster analysis in Section 5.5). Users may slow down significantly in anticipation for a turn (P2, P6, P8, P9, P12), minimally alter their speed due to an ‘approaching’ notification (P1, P4, P5), or even increase their speed when approaching a turn (P10). As these different interaction modes impact how users react to the consequent ‘turn,’ and ‘forward,’ notifications, we see how even this short and narrowly used instruction can result in significant reaction variability due to personal mobility style. Considering that ‘approaching’ notifications are immediately followed by ‘turn,’ the interface should support each user’s personal mobility to avoid incorrect turning task performance (and going down a wrong corridor).

Fig. 6.

Comparison of (a) average speed, (b) average speed change, and (c) average angular speed, following ‘approaching a turn’ instruction. Black bars show standard deviation.

A similar observation can be made by inspecting the inter-user variability in average angular speed shown in Figure 6(c). The statistically significant difference (p = .034) reflects how users approach turns in different ways. For instance, we can see from the data how some users consistently attempt to turn immediately following ‘approaching’ notifications (and hence, too early), e.g., P6, a generally cautious user that is more hesitant in their motion. We also tested for the motion measures relationship with user age, but none were found to be statistically significant (considering the previously defined group definition of 46 years old).

5.3.3. Forward and Info.

Although not as significantly as during turns, we see how the data-driven analysis reveals personal user behavior in ‘forward’ and ‘info’ notifications. We find that variability in reactions to ‘forward’ instructions is significant when considering average speed change values F(11, 430) = 5.66, p < .0001 (Figure 7(b)). This is intuitive, as most ‘forward’ notifications occur immediately following a completion of a turn and during long forward stretches.

Fig. 7.

Comparison of average speed, average speed change, and average angular speed, following informational instructions regarding points of interest in the environment and ‘forward’ instructions such as ‘go X ft forward’ or ‘destination is X ft away.’

Recognizing Cane Technique from Angular Speed.

One participant, P7, stands out in terms of average angular speed (Figure 7(c)). This single user is the reason for statistical significance (p = .031) in this case. We explain this result by looking at P7’s cane techniques, as it is relevant to understanding one particular user-specific interaction mode.

P7 does not regularly employ common cane techniques (i.e., side-to-side or wall following) when approaching or after completing a turn. Instead, she consistently turns in one swift motion immediately following a notification, without verifying the orientation or any landmarks after the turn and during the ‘forward’ notification. This is also affirmed by the fact that P7 has the lowest timing for the turn task itself, as shown in Figure 5. In navigation P7 is reserved and minimal in motion. She generally moves her cane front-back instead of side-to-side, which allows her to maintain low angular speed during the navigation, while at the same time hindering awareness of the immediate surroundings. We can see how the data reveals different cane techniques, which provide an opportunity to provide tailored support from the interface in such cases.

Identifying Cautiousness When Resuming Pace.

The interface should support the user’s unique navigation style to ensure correct and pleasant navigation. Just as we analyzed user slow down before and during turns, we can analyze how users speed up to resume pace.

A key difference among users is in the amount of time spent pausing before resuming walking speed following a turn (Figure 7(b)). Specifically, we find some users (e.g., P10, P12) slow down to a near stop regularly during turns, yet P12 resumes speed at a much quicker rate. The varying rates of resuming normal pace can also be seen by inspecting the pre-averaged temporal signal in the supplementary. We explain this difference by observing the video data and noticing a specific, post-turning strategy. Specifically, some users (e.g., P10) tend to scan their nearby environment with their cane following a turn, as to proceed carefully. Hence, this provides another example on how user strategy can be supported by interface, for instance, with additional verbosity and contextual support. As certain navigation scenarios may benefit from such a strategy, it could be elicited by the interface. A model for pausing can also be useful in determining the real-time state of the user, with additional discussion in Section 6.

Info Notifications Require Further Study.

After accounting for each user’s speed, no significant variability is found among users for reactions to ‘info’ notifications (Figure 7(e)). These notifications regarding points-of-interest along the way can be long, and at times combine with ‘forward’ notifications, e.g., ‘Proceed 42 meters and go to the first floor using the elevator on the left.’ As shown in Figure 7, users tend to reduce their speed in a similar manner following this notification. While some info notifications include relevant information to the navigation (e.g., elevators or building information), in general they may not illicit an immediate reaction from users (or one that heavily involves individual navigation skills). We note that this does not imply that such notifications have no association with user behavior and motion. For example, info notifications may prevent a user from getting lost or affirm their knowledge and confidence, and so need to be further studied in the future.

5.3.4. Obstacles.

Notifications regarding existence of objects along the path are useful for maintaining situational awareness and ensuring safe navigation. Our initial analysis in Section 5.2 revealed a higher standard deviation when compared to other types of instructions (Figure 3). Yet, due to the smaller number of events, the findings are difficult to interpret, and insights are limited. After normalization by each user’s personal walking speed, the reaction variability is not significant (p = .171), with some users accelerating to avoid obstacles, some not changing their speed, and some slowing down at times.

Navigation Aid and Obstacles.

Although statistical significance was not found among users in Figure 8(c), inspecting the figure suggests similarities to reactions’ angular speed during ‘small turns’ notifications, as users turn away from the obstacle in a variable manner. Having a guide-dog in this scenario would clearly imply a different behavior mode around obstacles. For the guide-dog user P8, we indeed observe an abnormally high (highest overall) average angular speed at .26 radians per second. Guide-dog users can quickly change their motion direction around obstacles without slowing down (e.g., due to physical contact). Hence, although further study is needed, we can still use the analysis to demonstrate the benefit of an adaptive, personalized navigation interface which can reason over the need for ‘obstacle’ notifications.

Fig. 8.

Comparison of (a) average speed, (b) average speed change, and (c) average angular speed, with standard deviation, following ‘obstacle’ notifications, such as ‘chair on your right.’

5.4. Evolution of Reaction during and after Announcements

We continue in analyzing user behavior trends that highlight the benefit of designing a more personalized interaction experience. The main aim in this subsection is to analyze a complementary component of user variability-the temporal evolution of reaction as it relates to the instruction announcement. We also study reaction patterns over multiple instructions, as opposed to in isolation as done thus far. The motion patterns will be used for clustering user interaction modes in Section 5.5. We find that users complete their reactions and navigation tasks at variable time (previously only analyzed for turns). This finding reveals novel insights regarding system-user interactivity. The main design implication is that timing certain instructions could be made adaptive, with additional insights about interactivity and user behavior for longer instructions and system verbosity.

Throughout the paper, meaningful analysis was pursued by clustering the instructions into a small set, but this process ignores the temporal realization of instructions. As users react to different instructions, the onset and completion of such reactions may occur both during, and/or following the complete announcement of the instructional guidance. For instance, ‘approaching’ and ‘turn’ notifications are concise, and hence the complete user reaction is often included in the temporal window immediately following the onset of the announcements. In contrast, other notification types may span multiple seconds when announcing, potentially delaying the reaction of the user. Hence, while comparing instructions in terms of their onset provides a natural framework for analysis across varying instruction types, in practice it is limited in comparing among instructions with highly variable length.

We complement our previous analysis by computing the motion measures following announcement completion of instructions instead of onset. We expect similar trends, but with changes for instructions which take longer to announce, such as ‘info’ and ‘obstacle.’ For direct comparison, Figure 9(a–c) depicts all of our previous analysis in a condensed form (only averages, no standard deviation), while Figure 9(d–f) depicts the motion measures computed post-announcement. In Figure 9, we emphasize the spread of the overall distribution of averages across all users and maintain the per-user coloring to identify user-specific interaction modes.

Turns.

The analysis reveals more complex patterns of interactions across users and instructional context, but several key insights can be made. As user reaction time is variable and the temporal window is now sampled during the reaction, trends show a larger distribution of the motion statistics across users (Figure 9(d–f)). We can see how large turns are still unfolding following instruction completion, while small turns are being completed for some users, e.g., P7 and P8 show an upward trend in speed in Figure 9(e). When approaching turns, the trend in speed reduction becomes more pronounced, yet some users such as P10 are still increasing speed in anticipation of the turn (and hence could benefit from tailored instructions to reduce navigation errors). These observations affirm our previous analysis of variable task time for turns.

A Better Look at Longer Instructions.

A main observation is that average speeds are now higher following the longer instructions of ‘info’ and ‘obstacle’ notifications, as seen by comparing Figure 9(b) with Figure 9(e), yet the data reveals a variety of reaction patterns among different users. The key insight in the figure is that some users stand out due to a slower or longer reaction and task performance time on the order of seconds, as previously observed for turn instructions. For ‘info,’ most users show an increase in average speed change (i.e., accelerating), yet P2, P6, P4, and P12 are still slowing down following announcement completion, as their reaction is longer and more pronounced for this instruction. Moreover, while most users are now accelerating following the completed ‘obstacle’ instruction, some users (P4, P8, and P7) stand out as decelerating and still reacting. P7, a user which does not employ a side-to-side cane scanning technique, stands out in long reaction time and large increase in average angular speed even long after the announcement. We note that the angular speed is also high for the guide-dog user P8 in this case, but is consistent during both the announcement onset and its completion as to be expected from a guide-dog user.

In summary, while previous analysis ignored the temporal realization of instructions, depicting the relationship between instruction realization and reaction demonstrated additional rare and common interaction modes between the user and the system. We see that in longer instructions, inter-user variability is pronounced as well. Understanding user behavior as a function of instruction length can be useful in dense environments that require the announcement of multiple instructions and user reactions in close proximity (e.g., ‘turn left,’ ‘there are obstacles on both sides,’ and ‘turn slightly right’ immediately) with implications to interface timing, content selection, and real-time user state modeling.

5.5. Clustering Mobility Patterns

Researchers often employ unsupervised clustering to analyze the behavior patterns of users [66]. In the previous subsections, we uncovered significant variability in reactions and explained it on an individual basis. However, in inspecting the large amount of interaction events on a per-user basis, the exploratory analysis was limited in its ability to capture mobility patterns among multiple users. To reveal reaction motion trends across user groups, we next cluster the user-specific data described in the previous sections (Figure 9). By analyzing broader interaction patterns and commonalities, we aim to provide additional design implications for navigation interfaces (i.e., accommodating reactions for groups or types of users). Beyond user categorization into different types of walkers, we expect to gain a better understanding into the multi-dimensional nature of reaction, specifically by identifying rare and prominent user interaction modes.

5.6. Clustering Approach

To identify clusters in user behavior, we represent each user as a vector of motion statistics for the six instruction types. An element in this vector corresponds to the average value of a reaction measure for a single instruction type (i.e., average speed change during ‘approaching’). The samples are standardized to have zero mean and unit variance before constructing a similarity graph.

Cluster analysis in our application domain is challenging due to the inherent diversity of the interactions and the difficulty in collecting data for many users. While the exploratory analysis involved a large number of interaction events (a total of 1,553 in the dataset), representing each user as summarized motion statistics as in Figure 9 necessarily implies dealing with a small number of data points. Yet, as the modes of interaction are assumed to be limited as well, we anticipate that clustering will result in shared behavior patterns, with users falling into certain reaction patterns (e.g., cautious users).

We follow conventional hierarchical clustering [66], which is consistent with our automatic and scalable analysis performed thus far. By leveraging a similarity graph between data points, it can be used to gain insight into patterns in our dataset. The cophenetic correlation coefficient [29] was used to measure the quality of the clustering and verify the choice of cosine similarity measure between samples and average linkage. These choices are also considered suitable for exploratory analysis [42]. A cophenetic correlation coefficient ranges of ~.88 was obtained, which is considered sufficient as a goodness-of-fit test [42].

5.7. Clustering Results

We emphasize clustering results on average speed change measures, as clustering the raw speed measures in Figure 9 would lead to an arbitrary separation based on the participants pool, i.e., low vs. high speed users. In Figure 10(a) we observe how even a simple 2-cluster partition of the data results in an interpretable representation of Figure 9(b). It appears that on a high-level, users may be divided into two categories, specifically in how they behave before, during, and after turn instructions; In C1, users reserve most of their slow-down to the actual turn, despite the ‘approaching’ notification. In contrast, this trend is inverted with most users falling in C2, where the significant slow-down occurs not during the actual turn, but before, when approaching the turn (or in a roughly equal amount). Considering that turns are the most frequent places for navigation errors [2] and that reaction could span several meters (Figure 9(c)), understanding these interaction modes is crucial for reducing navigation errors. These results suggest implications to user-specific timing of instructions, as user types interpret and leverage instructions differently. Additional conclusions can be made by inspecting Figure 10(a). In general, C1 users tend to have more volatile speed changes, including around obstacles, as opposed to their C2 counterparts, which are more stable in their reactions across different instruction types. Specifically, users in C1 slow down significantly during the turn to a near stop, which also requires a significant acceleration to match during the following ‘forward’ instruction, as shown by a relatively large positive speed change in all the C1 users. C2 users, on the other hand, slow down or speed up throughout the instructions in a more consistent manner. For instance, we see minimal speed change at ‘forward’ notifications. Similarly, C2 users also appear to slightly slow down more at ‘info’ notifications, in preparation to upcoming points of interest.

Fig. 10.

Clustering user speed change reactions. Results are shown for different numbers of clusters, with (a) 2 clusters, (b) 3 clusters, and (c) 4 clusters.

We can see how the progressive hierarchical clustering reveals relationships between common interaction patterns as well as provide insight into rarer user behavior. Several reaction patterns still appear as in-cluster outliers, so that adding a third cluster in Figure 10(b) identifies two users with highly similar reaction patterns. The users, on the extreme end of C1 who slow down almost exclusively before a large or a small turn. Separated due to their unique behavior following ‘approaching.’ Similarly, C4 includes a user which tends to accelerate in anticipation for turns, leading to a sharp reduction in speed following the turn notification onset.

We find significant variability (p < .001) among these clusters of users. Observing the significantly different patterns in Figure 10(c) implies that users falling in clusters C1-C4 can benefit from adaptively timed notifications. Specifically, users in C4 may over-shoot a turn while C3 are unlikely to do so. This observation was confirmed in inspection of the video data. For instance, P10 can be seen significantly passing a turn, leading to deviation from the planned path into an open space.

6. DISCUSSION

We present our key findings that include significant variability between users during the most critical elements of the navigation. We relate these findings with prior research in identifying user behaviors that cause navigation errors and discuss how these findings can be used to prevent such errors. We begin by discussing overall design implications in the first three subsections, followed by instruction-specific insights.

6.1. Timing of Instructional Cues

One main design implication is in the adaptive timing instructional cues. We argue that a single user model is not sufficient for successful real-world navigation. Yet, essentially all existing navigation interfaces do not consider tailoring timing of instructions to different users nor propose one can go about doing so. Specifically, we find that different users travel at different speeds, react to instructions differently (i.e., slowing down to a near stop or maintaining speed), and employ different strategies throughout the assistive navigation. To emphasize this point in a more interpretable manner, we report these findings in distance measurements in Figure 11(a, b) for turn events, and in Figure 11(c) for all events (the figure plots the difference, in meters, between the actually traversed distance over the time window T and the distance that would have been traversed assuming a constant speed model).

Fig. 11.

To better understand the relationship between user motion, instruction timing, and potential navigation errors, we plot the distance traveled by users during, (a) initial reaction in turning task (between instruction onset to turning motion onset), and (b) the entire turn task (instruction onset to turning motion completion), and (c) the impact of different instructions on the distance traveled by users following instruction announcement completion.

In the data, the most common cause of navigation errors occurred around turns, at times without ability to recovery. This finding is consistent with prior research [2] which studied the frequency of missing turns while emphasizing localization accuracy, but not the role of user behavior. As missing a turn occurs due to early or late turning by the user, our study extensively analyzed user reaction and navigation task performance during turns.

We observe opportunities for tailoring the interface when approaching a turn, during large turns, during small turns, and other instruction types as well. For instance, we can see that the distance traveled during reaction onset or task performance for turns could vary by over 1-2 meters across different users. Depending on the situational context (i.e., open space), dealing with variability over several meters can translate to errors in navigation. On top of an inherent localization error, these findings emphasize the need for adaptive timing of instructional content.

In the case of dense environments with multiple instructions or longer instructions, we see that inter-user variability in reaction should also be considered in timing the interface. We find that some users may benefit from joint announcements of multiple nearby instructions (e.g., ‘turn left and turn slightly right’ immediately after), to avoid missing a turn. This increased verbosity can be made user-specific, due to the likely associated increase in cognitive load.

6.2. Leveraging a Finer-Grained Model of Users for Content Selection

The interface should support the user’s unique navigation style to ensure correct and pleasant navigation. One benefit of our data-driven approach is that it provides a framework for supporting the real-time state of the user by observing motion and timing dynamics. For instance, Section 5.3.3 discussed how user cane techniques or states of cautiousness could be recognized in the data. In previous research, [2] reports some preliminary user-specific patterns (i.e., one participant stops 7 times while another 0 times to listen for instructions), yet statistical significance was not tested nor finer-grained user behavior such as cautiousness discussed.

We find that in unfamiliar environments and large-scale settings with diverse users, supporting a finer-grained interaction can be used to improve the quality of the navigation experience. Specifically, the system can determine if additional context and verbosity, in the form of landmarks, frequency of instructions, uncertainty of the system, or affirming user behavior, can benefit the current user. A model for pausing and resuming walking speed, also discussed in Section 5.3.3, can facilitate such a system in a more automatic manner.

Instructional guidance can be adjusted for users with varying navigation aids or visual acuity. As discussed in Section 5.3.4, the behavior of the guide-dog user around obstacles differs from most of the cane users, as expected. In the case of guide-dog users, obstacle notifications may be omitted to ensure a concise and useful interaction, yet other personal factors besides navigation aid were also found to play a role when navigating around obstacles, e.g., cane techniques. As user behavior around obstacles is somewhat similar to during slight turning tasks, an assistive interface could be designed to provide additional feedback around obstacles regarding user heading to avoid veering from the path.

As user-specific cane strategies are also present across different instructional contexts, such as when approaching turns or resuming normal walking pace following a turn, the system can use such information to determine the real-time state of the user, as well as facilitate user learning [45]. Specifically, as certain navigation scenarios may benefit from certain cane techniques, they can even be elicited by the interface.

6.3. Ambiguous and Difficult Tasks

We find that increased difficulty or ambiguity of certain navigation tasks results in larger inter-user variability. The analysis on small turns demonstrated significant variability in angular speeds. Large turns are less likely to result in over- or under-turning as 90-degree turns are prevalent in indoor and outdoor environments. As discussed in Section 5.3.1, over- and under-turning in smaller turns is a known issue. However, studying this phenomenon on a per-user basis has not been done before and suggests several design improvements.

An interface could be designed to minimize such scenarios along a route. Since a single heading feedback following turns was also shown to be insufficient in our analysis, a more useful approach would be to provide additional feedback to support the user in case of over or under turning, e.g., in the form of continuous sonification or additional information about the scene. This additional feedback may be useful for participants who often over or under turn. Hence, our analysis suggests that the interface can also support users with certain mobility skills by providing additional contextual information during turns, e.g., ‘turn after the plant on your left.’ Alerting users that they may have passed the turning point or providing feedback to support more accurate turning can therefore be done on a user-specific, scenario-specific manner.

As shown in Figure 5(c), P5 exhibits the same angular speed in both large and small turns, suggesting proneness to over-turning. While this finding was somewhat expected in navigation by people with visual impairments, depending on the surrounding context (i.e, open space or narrow hallway) even small changes in user heading may quickly lead to deviation from the path and navigation errors. In general, as over-turning in small turns is a common navigation error, a clear design implication would be to (1) choose routes that minimize small turns, if possible, and (2) provide additional re-orientation feedback in small turns (e.g., with a continuous sonified signal to guide the user slowly), (3) more verbose interface, when possible, to immediately validate correct heading using surrounding landmarks. Alternatively, considering practical limitations and inherent localization error, the users could be prompted to perform the task slowly and carefully. Although the design implications are straightforward, our analysis re-affirms the significant importance of carefully addressing interactions under ambiguous or difficult tasks, such as slight turning.

6.4. Approaching Notification Timing

Approaching notifications prepare users for an upcoming turn and were favorably reported as preferred by most of the users. Yet, variability in user reaction suggests that this instruction is leveraged in a different manner across users. Users may slow down significantly in anticipation for a turn, minimally alter their speed due to an ‘approaching’ notification, or even increase their speed in approaching a turn. Considering that ‘approaching’ notifications are immediately followed by a turn, the interface should support each user’s personal mobility to avoid incorrect turning task performance. For instance, the interface may delay the ‘approaching’ instruction for users that usually present higher angular speeds immediately after listening ‘approaching’ and therefore preventing an earlier wrong turn in certain environmental context.

6.5. Further Research is Needed for Info Instructions

Based on our analysis, informational notifications have little association with individual average speed changes. This is not entirely surprising, as many of the informational notifications provide additional context but are not essential to an immediate navigation task. Besides affirming our intuition, we conclude that such instructions may be left generic across users, but caution that further studies are needed (e.g., too many info notifications could be distracting or associate with behavior in a longer time window). Previous research reports that people with visual impairments appreciate gaining knowledge about their surroundings, but also stresses that the amount of information can be overwhelming [48]. While there may still be room for personalizing info notifications on an individual basis, the association with user reactions and cognitive load requires further study.

6.6. Variability in Reaction to Forward Instructions

We find that the way in which users pause and resume motion is indicative of personal mobility. Interfaces could accommodate and adapt to different user needs and styles as shown by the significant variability in the average speed change after ‘forward’ instructions, which often follow turns. More cautious users often pause when turning, interpret the instructional guidance, scan the path ahead with their cane to make sure they are in the correct orientation and only then resume their pace. The different nature of these users could allow the interface to convey additional information about both their immediate and long-term surroundings, rather than generic ‘proceed X feet and turn right’ type instructions. To cite one example, the interface could provide additional context on the next path segment so that the user knows what to expect (e.g., ‘You should have a table on your right. Proceed 50 feet in a very narrow corridor and turn left at the end of the corridor’). As previously discussed, we envision an interface which provides instructional content that actively supports or affirms user strategies (e.g., cane techniques, as discussed in the analysis for P7) which can be recognized in the data.

However, users such as P1, P3 and P4 usually proceed with minimal speed changes, continuing their normal pace when listening to ‘forward’ instructions. These users may have reduced time, or even need, for verbose communication and could potentially benefit from combined ‘turn’ and ‘forward’ instructions (e.g., ‘turn right and proceed 50 feet’). However, this hypothesis requires experimental validation in the future and caution is needed for instance when relevant landmarks, such as a door, can be found right after a turn. Given the users’ almost constant speed (even during turns) it may be beneficial to warn about those landmarks during or before the turn (e.g., ‘turn right and you’ll find a door’), instead of waiting for the user to complete the turn.

6.7. Obstacles and Their Impact on Angular Velocity

Despite the smaller sample size, we find a clear impact of obstacle notifications on users’ angular speed, showing that users slightly change their orientation to make sure they avoid bumping into obstacles. This finding suggests that the system can take advantage of this knowledge to foresee user behavior and reduce the chances of over/under-turning (similarly to what occurs in ‘small turns’ events) and consequently veering and deviating from the intended path. For instance, the interface may omit obstacles that are not at the reach of the user’s cane if the user is known to veer after obstacle notifications. Alternatively, the system may provide additional context (i.e., distance and direction to the obstacle feedback in a continuous manner) or instructions to prevent navigation errors (e.g., ‘there are obstacles on your right, but keep on the right side until the next turn’).

While our study only includes one guide-dog user (who also changed her orientation when notified about obstacles) previous research supports that obstacle notifications are not required for these users due to the guide-dog’s ability to avoid them [49, 68].

7. LIMITATIONS AND FUTURE DIRECTIONS

Our study finds significant motion variability across 12 participants following navigational guidance in a large-scale, real-world environment. For meaningful comparison, the navigation route and instructional cues were kept fixed. As our method can be easily extended to automatically analyze additional data, we can see how studying the motion variability distribution can benefit from additional participants with diverse characteristics, as well as additional environments (but 5-12 participants is representative in our domain). The additional data can be used to further study more rare events, such as interaction with different types of obstacles and environmental factors. Moreover, although we did discuss association with personal characteristics (i.e, age), the limited distribution in our data prevented us from gaining insights into such potentially relevant factors. Since real-world environments include surround pedestrians as well, analysis of user behavior and system design for such scenarios should be done in the future. Informed by insights from this work, another future direction would be to study the impact of different adaptive interfaces and personalization schemes on the navigation task and the user-experience.

8. CONCLUSION AND FUTURE WORK

We presented the extent of user-specific motion variability of blind users in assisted indoor navigation. Towards assistive mobility technologies with a more personalized interaction experience, our findings reveal the significant role of personal navigation style when following turn-by-turn navigational guidance. Specifically, we identified the need to not only adjust the interface to each user’s personal walking pace (i.e., speed), but also other navigation characteristics (i.e., reaction onset, reaction length, reaction type and task performance). The results were reported using automatically extracted motion measures from smartphone and beacon sensors. By describing how instructional content and timing selection could be performed on an individual basis, this work provides a myriad of opportunities for future research in independent navigation by users with visual impairments.

Fig. 1.

Examples of blind users holding a smartphone in their hand while following the turn-by-turn assistive indoor navigation app. We analyze user motion data to understand variability in reactions to different navigational context (e.g., turns, obstacles). Towards assistive mobility technologies with a more personalized interaction experience, our findings reveal the significant role of personal mobility characteristics during assisted navigation.