Effect of a looming visual cue on situation awareness and perceived urgency in response to a takeover request

Abstract

This study aimed to investigate the effect of a looming visual cue on situation awareness and perceived urgency in response to a takeover request (TOR), and to explore the underlying mechanisms of this effect through three experiments. In Experiment 1, the optimal size and speed of a red disk were determined, which were effective in capturing looming motion and conveying the urgency of the situation. The results indicated that both looming speed and size ratio had significant effects on situation awareness and perceived urgency. In Experiment 2, the effects of looming stimuli were compared with dimming stimuli, and the results showed that the looming visual cue was more effective in promoting perceived urgency and situation awareness. The results also indicated that the looming visual cue attracted more visual attention than the dimming visual cue, in line with previous studies. Experiment 3 utilized a driving simulator to test the effectiveness of the looming visual cue in promoting fast and appropriate responses to TORs in complex driving scenarios. The results showed that the looming visual cue was more effective in promoting perceived urgency and enhancing situation awareness, especially in highly complex driving situations. Overall, the findings suggest that the looming visual cue is a powerful tool for promoting fast and appropriate responses to TORs and enhancing situation awareness, particularly in complex driving scenarios. These results have important implications for designing effective TOR systems and improving driver safety on the road.

1. Introduction

The automobile industry and various IT companies such as Google are mobilizing to develop fully automated vehicles. The National Highway Traffic Safety Administration (NHTSA) classifies the development stage of self-driving vehicles into five levels, and level 5 can be referred to as a fully autonomous vehicle [1]. According to the NHTSA guidelines, partially autonomous vehicles of levels 2 to 4 should have a takeover request (TOR) alert system. Such vehicles that communicate with the road infrastructure to control the self-driving system can warn the driver to take over control before entering a communication dead zone [[2], [3], [4]]. In this case, the driver can easily take over, and is less likely to have a dangerous accident. In contrast, a conditional automated vehicle that suddenly faces technical limitations and fails to correctly recognize an upcoming object must transfer control to the driver in a very short time. The driver in turn, must respond quickly based on multiple situational data points to avoid collision. Recently, automobiles have provided sophisticated TOR alert systems that can promote situation awareness and perception of urgency in the driver [5].

Early research focused on the relative merits of visual, auditory, or tactile signals [[6], [7], [8], [9], [10], [11], [12]]. However, it is challenging to conclude simply that one modality is more advantageous in takeover situations than another is. Rather, it is considered more important to decide how to manipulate a particular modality to deliver contextual information correctly than to choose which one [6,13]. In particular, the alert system should convey the urgency of context to the driver to avoid collisions in response to the TOR, as well as in manual driving [[13], [14], [15], [16]].

Auditory modality has been manipulated previously to deliver the urgency of the context. Non-verbal abstract sounds, such as beeping sounds, increased the perceived risk and led to a faster response of the driver, even when the driver was performing non-driving related tasks (NDRTs) [17,18]. However, such simple tones, semantic voice messages, or looming sounds cannot deliver spatial information effectively in a TOR [[19], [20], [21]]. Visual signals, however, can provide spatial information, including distance and direction, and there are few limits to the type of information transmitted through visual semantic icons. However, visual signals have difficulty attracting attention when drivers perform NDRTs [22].

Other studies have therefore suggested multimodal TORs that present auditory and visual stimuli together, resulting in better response time and perceived urgency than single-modality TORs [18]. A simple tone or beep alerts the driver while a visual icon provides spatial information in auditory-visual multimodal TORs, and promotes the perception of urgency for an upcoming collision [23]. Also, a visual cue conveys the urgency of the context to attract visual attention and increase situation awareness [16]. Several studies have used various interfaces [24], shapes [18,25], and colors [16,26] to present visual stimuli for TORs. However, few studies have been conducted regarding how to present a visual stimulus in a risky situation. It is possible that even if this same stimulus is presented, there are more salient methods for capturing driver attention. In this study, we suggest a looming red dot as a visual cue for a multimodal TOR. In addition, we compared the looming stimulus with the dimming stimulus and determined the effects of the looming stimulus in various contexts.

1.1. Multimodal TOR design to promote perception of urgency

Multimodal TOR designs can provide richer information than unimodal TORs [27]. In particular, the visual single-modality alone is inferior to others because drivers are highly likely to be involved in other tasks that are not related to driving a car. A lot of studies used multimodal, and there are experiments that use changes in auditory and tactile stimuli to convey the urgency of a situation and elicit a faster response from participants. For example, presenting auditory stimuli at a high frequency and frequent repetition elicits a fast TOR response [28]. High pitch or high frequency can make people perceive faster, even if they feel less comfortable [14]. More recently, the effectiveness of visual and tactile stimuli combined with direction has also been validated to elicit fast responses in drivers [29]. The study of [18] proposed a multimodal TOR to increase perceived urgency and induce a fast response time. In their study, when a simulated auto-driving car failed to detect lanes, it flashed a semantic icon of hands-on a handle at 5 Hz. An auditory cue of a 1000 Hz sine wave was played for 1s, at the same time. The driver was guided to lay their hands on a handle to take over control. The drivers exposed to multimodal TORs were better able to put their hands on a handle and manually maintain lanes than those exposed to a visual single-modal TOR. Another study compared a similar flashing semantic icon with an audio cue in response to a TOR, and found that the audio cue was more effective with a faster response time [30].

However, apart from the visual single-condition, the effect of the visual modality associated with auditory stimuli depends on the specificity of the context. According to Ref. [30], when the driver performed a visual NDRT in the autonomous mode, the visual single-TOR-condition showed the worst performance (take-over time, hands-on time) compared to that of other modalities, such as visual-auditory and visual-tactile. Also, multimodal signals were preferred for high urgency, and auditory-only TORs were preferred for low urgency [22]. However, considering the diversity of NDRTs, it is desirable to consider various multimodal TORs [29]. For example, people who are engaging in listening to music or having a conversation with others are highly likely to neglect an auditory TOR, and a visual TOR is easily ignored when drivers are using a smartphone or playing a game. Therefore, it is desirable to study various TOR methods under different conditions.

In the case of visual modalities, visual alarms are often presented through an in-vehicle display below the windshield; however, as mixed reality technologies develop, various methods of providing TOR information via visual modalities have been studied [25,31]. If the visual modality is provided through a heads-up display on the windshield, the TOR alarm is delivered by an auditory signal, and then spatial information indicating the location of the risky object or problem can be provided through visual information. This makes it easy for drivers to attain situation awareness and be fully informed. In particular, in high-risk situations the association of various modalities is beneficial [22], and the mental mapping of spatial information provided by visual stimuli can provide a great benefit. In previous studies, the TOR was delivered using various types of visual stimuli, including different types of interfaces [24,32], stimulus shapes [18,25], and colors [16,26].

The question remains however, as to what elements of visual stimulation help drivers in urgent situations. According to Ref. [6], flashing, color, or text change failed to increase perceived urgency. Also, visual stimuli in red or green according to the urgency of the TOR situation [16]. If the situation is urgent and the visual stimulus is red, it is regarded as a matched condition; otherwise, it is an unmatched condition (a red stimulus with safety situations). They intended to find the main effect of stimulus-situation compatibility according to color and urgency level. However, there was no main effect of matched or unmatched conditions, and they discussed that manipulation of the color signal might provide a variety of information regardless of urgency. In addition, previous studies have reported that the effect of a visual TOR is reduced in environments with visual load. Another study evaluated all multimodal combinations of auditory, visual, and tactile warnings and found that such warnings induced faster response times than unimodal warnings [33]. However, multimodal does not always convey context quickly. Previous research has found that presenting TORs by adding a visual explanation on the steering wheel in addition to the auditory warning actually increases reaction time [34]. This occurs because the additional visual explanation increases information processing time [32]. Also, combinations with the visual modality (audio-visual, tactile-visual, and audio-tactile-visual) did not have any advantages in reaction time when a car in front was stopped and brought drivers' visual attention. This result indicates that a visual warning can be limited in a context with visual load. Previous study conducted a simulator study that used a high visual load to examine combinations of auditory, visual, and tactile warnings [35]. They provided a pair of red lines on each side of the screen as the visual cues. As a result, a visual warning was found to degrade the overall performance in terms of response time, hit rate, and correctness. If the situation has urgency, the cognitive load will inevitably be large, and the more complex the situation, more visual elements (various colors, shapes, sizes) of the stimuli will not help the driver's response to the TOR.

In this study, we assessed a visual warning signal that changes the driver environment and visually draws attention to other warning signals, which were manipulated in line with previous studies. We showed that by using minimal stimulus in the form of a small red dot, simply changing the presentation method affected the driver's visual attention and situation awareness.

Manipulating the physical properties of a visual cue changes the perceived urgency. A visual cue that represents urgency attracts more visual attention than other visual cues do. Rapid changes in brightness or the sudden onset of a stimulus are also perceived as signs of urgency, drawing visual attention [36,37]. According to Ref. [38], motion in specific directions attracts more visual attention than motion in other directions. In particular, this work presented the looming motion of points as a more attention-grabbing condition than the ‘onset’ condition shown in other studies [39]. They used a simple probe-detection task to find a tiny gray rectangle on the screen. At the beginning of the trial, dots on a screen moved randomly. Dots on the left or right side of the screen where a probe appeared were adjusted to move in a specific direction. They moved up, down, left, or right; and away or closer from a vanishing point in the middle. Among these six movements, looming motion away from the vanishing point attracted more visual attention than the others did. In addition, only the looming motion drew visual attention, even if it was not present at the onset. Studies on looming motion and perceived urgency have shown that visual attention to looming motion is a reaction related to visual mechanisms and emotional responses to the environment [[40], [41], [42]].

1.2. Background on looming objects

Looming signals have been recognized as clues perceived by tiny zebrafish larvae to mice, primates, and humans; triggering defensive behaviors, such as escape from and avoidance of natural enemies. Looming signals refers to a visual cue in which an object on a screen appears to be rapidly expanding in size, as if it were quickly approaching the viewer. In this study, we investigated the effectiveness of using a looming red disk as a visual cue in alerting drivers to a takeover request in a simulated driving environment.

Previous research conducted several experiments with mice trapped individually in a box [40]. Such mice either attempted to escape or froze as the size of the black disk on the display at the top of the box increased. Receding or dimming visual cues barely induced such escape reactions. In addition, the visual cue presented on the floor did not induce any escape reaction. These results show that mice only moved to protect themselves when a visual stimulus approached from above. In other words, the visual system of mice has evolved to capture this looming motion, and is an innate protection system to detect aerial predators. In addition, escape reactions to looming motion have been observed in other mammals, such as rabbits, monkeys, and humans; as well as zebrafish [40,41,[43], [44], [45]]. This reaction occurs regardless of the species, because neurons that transmit images from the retina to the brain are involved. If these neurons are deliberately damaged, the creature no longer has an escape response to the looming motion [41]. Thus, there should be a specific size and speed of expansion of a looming object that induces a more urgent escape response, depending on the species [40].

Humans also show a similar reaction to looming visual cues. Previous research showed that looming stimuli close to the face exhibit defensive behavior or tactile sensitivity and another study found that a five-month-old infant also responded similarly [43,44]. For adults, the presence of a looming signal does not often induce defensive behavior when they know that there is no threat to themselves [46]. However, adults exhibit defensive reactions in certain environments [45]. They primed the fear of being alone in a room for an adult participant concentrating on computer games in the laboratory. When a looming image was presented on the game screen, the adults showed defensive reactions by flicking their head. In addition, using threatening and non-threatening stimuli, previous study found that people perceived threatening stimuli as approaching more rapidly than non-threatening stimuli [47].

This innate reaction is a sign of the evolution of the visual system for survival against predators. Another study supported this explanation by conducting an experiment using zebrafish larvae [41]. They assumed that retinal ganglion cells (RGCs) in the brain are involved in the escape reaction of larvae to a visual looming cue. Thus, they applied targeted laser ablation to the neuropil of the optic tectum, which receives axon terminals of RGC cells, and the larvae no longer showed any escape reaction to a visual looming cue. Another researcher also suggested a specific speed of expansion and size of a looming dark disk that swept over the receptive field of the RGC of mice, and found this was associated with the size-to-speed ratio (l/v) [40]. The size-to-speed ratio (l/v) is a physical parameter that describes the relationship between the size of an approaching object (l) and its speed (v). It is calculated by dividing the size of the object by its velocity. A size-to-speed ratio of a looming black disk >1 (=30 ms to reach 0.03 cm size) induced the escape reaction of zebrafish larvae [41]. This result indicates that RGCs in a specific species do not respond to looming objects if they are smaller than a specific angle of view or expand faster than a certain velocity.

1.3. Experimental overview or summary of experiments

We conducted three experiments, with Experiments 1 and 2 being conducted online and Experiment 3 being conducted in a driving simulator to confirm the results of Experiments 1 and 2 in a more realistic driving environment.

Experiment 1 of this study was conducted as an exploratory study to determine the optimal speed of expansion and size of a red disk appropriate for the human visual mechanisms that capture looming motion. In addition, we attempted to convey the urgency of the situation by using a red color, which is perceived as a warning, and presented it with an auditory TOR. Referring to a previous study [41], we manipulated red dots with a size-to-speed ratio >1 using different size and speed combinations for visual TORs.

Experiment 2 was designed to compare looming stimuli with dimming stimuli, similar to previous studies [40,41,48]. The dimming stimulus can be presented through luminance change and is one of the most popular methods for presenting stimuli. However, previous studies have shown that, except in a unique situation, dimming is insufficient to induce escape behavior. Therefore, we attempted to determine if they had different effects on drivers' responses in driving situations.

Finally, Experiment 3 was conducted using a driving simulator in which a real traffic environment was implemented in a virtual environment. We attempted to determine whether a looming red disk could deliver the urgency of the situation and promote situation awareness, even in a simulated environment. In addition, because it is a more dynamic experiment in which manual and autonomous driving are conducted together, we examined whether the driver's situation awareness changes after the TOR according to the complexity of the driving scenario.

2. Experiment 1

2.1. Methods

2.1.1. Participants

Fifty-one participants (35 males and 16 females) between 19 and 56 years of age (M = 29.59, SD = 9.29) responded to a survey through a link posted on an online bulletin board, but two responses were excluded from the analysis due to missing values. These participants could read, understand, and discriminate between letters and images displayed on the screen. Their driving experience ranged from 0 to 26 years (M = 6.40, SD = 7.85). Participants were rewarded with a mobile coupon worth 4000 won for the survey. Experiment 1 was a 3 × 2 within-subject design, and all participants were exposed to all combinations of independent variables.

2.1.2. Apparatus

The video clips for the online survey were created using a driving simulator based on the Society of Automotive Engineers (SAE) definition of Level 3 conditional driving automation control [49]. The simulated driving environment was implemented using UNITY version 2019.4.22f1 software (Unity Technologies, SF, USA). It was designed to replicate actual driving in and around an urban area, and lasted 10 min per route. There were three paths, starting and ending at different positions. Because the conditional automated vehicle of the simulator was designed with the capability of an SAE Level 3-automated car, it transmitted control to a human driver under certain conditions via a TOR. Table 1 illustrates nine conditions (event cases) under which TORs were generated, which were categorized according to complexity level.

Table 1.

Nine event cases generating a TOR, classified according to the complexity level implemented in the driving simulator.

Complexity Level	Required Situation Awareness Level	TOR Event Case	Description
1 (low)	Perception. There is no accident situation and no driver response required.	AdvBoard	Perception of an advertisement board beside the road.
		TooClose	Perception of a car behind, following the driver's car.
		ParkedCar	Perception of a parked car in the opposite direction.
2 (moderate)	Comprehension. Only longitudinal control is required.	CarDistance	Driver is required to use the brake to adjust the velocity of the car.
		BottleNeck	Three leading cars are ahead of the driver's car.
		TooSlow	A leading car goes too slowly.
3 (high)	Projection. Both Longitudinal and Lateral Control are required to avoid collision to sudden appearance of pedestrian or Car.	Pedestrian	Pedestrian on a pavement abruptly moves toward the driver's car.
		FromSide	A car rushes toward the driver's car from the right or left at an intersection.
		FromBack	A car behind suddenly overtakes the driver's car.

In this study, we manipulated the level of complexity by applying the different situation awareness (SA) levels required for driving [50]. Situation awareness describes the process by which an operator handles situations in a dynamic environment such as driving a vehicle or operating an aircraft [50]. Drivers process the surrounding situation in the order of perception - comprehension - projection, and three levels processing occur continuously real-time. We manipulated the complexity of the scenario based on the situation awareness level. Specifically, scenarios that require only drivers’ perception (SA level 1) are low complexity, and situations that needs to understand the traffic situation in front (SA level 2) are moderate complexity, and situations where a sudden accident needs to be predicted (SA level 3) are high complexity. Therefore, complexity in this study is also related to urgency of driving situation. We created three scenarios at each complexity and produced each as a video. The detailed video flow is as follows.

The conditional automated vehicle was set to face three different event cases that would generate a TOR, which were randomly selected from each of the complexity. When the conditional automated vehicle approached the obstacle (e.g., pedestrian, leading car, parked car) by a certain distance or less, the simulator generated a TOR with looming visual cues. The auditory cue was presented at the same time as a 1-s short tone.

Participants in the online survey would see a scene in the video clip from the moment the conditional automated vehicle approached the obstacles in the TOR event case to just before colliding with the obstacle. Images from the video produced using the simulator are shown in Fig. 1. Fig. 1(a) shows the event corresponding to a Level 2 SA, which represents a situation in which the distance to the leading vehicle decreases. Fig. 1(b) and (c) illustrate Level 3 SA events. They represent a man who suddenly walks onto the roadway and a car suddenly moving through an intersection, respectively. Fig. 1(d) shows where a red disk appears in a screen.

Fig. 1.

Examples of TOR cases (a) CarDistance: The distance between cars is decreasing. (b) Pedestrian: A man suddenly begins to walk onto the roadway. (c) FromSide: The driver's car is moving forward at high speed while another is moving toward it from the right side of the intersection. (d)Appearance of red disk

In this study, the simulated videos in Experiments 1 and 2 were kept to a maximum of 30 s. Previous researcher who examined how long people need to be presented with a situation to create situation awareness in experiments using simulations, found that 7–12 s of video builds awareness of the location of nearby cars [51]. They also claimed that more than 20 s is needed to assess relative speed. In this study, since the question about the distance between participants' own vehicle and other vehicles was included, the video was created to be presented after 20 s, and the scenario ended in a maximum of 30 s.

2.1.3. Experimental variables

Since a properly manipulated looming red disk was expected to deliver urgency and promote situation awareness, these were measured as dependent variables while the size and speed of expansion of the looming red disk were independent variables. The looming red disk was presented as an alarm for the TOR in the driving simulator. Thus, the speed of the expansion was expressed as the looming speed, which can be defined as the number of times the disk fully expands in 1 s and is divided into two levels: Once per second (once) and twice per second (twice), as illustrated in Fig. 2(a) and (b). The size of the looming red disk is the final size that appears in the last frame. The red disk was shown at the center of the screen, at the end of each video clip when TOR happened. Since the participants were surveyed with their own display devices, the absolute sizes of the looming red disk shown on each device were different. The relative size, however, for each screen was the same. The size of the looming disk was expressed as the height ratio of the (disk: display screen) and consisted of three levels: 0.12, 0.32, and 0.48. Table 2 shows six combinations of the two independent variables: the looming speed and size ratio.

Fig. 2.

(a) A looming object that expands once per second. A red disk (the looming object) expands to its maximum size over 30 frames. (b) A looming object that expands twice per second. A red disk (the looming object) expands to its maximum size twice in a second.

Table 2.

Six combinations of two independent variables (looming speed × size ratio).

Looming Speed
		Once per second	Twice per second
Size (ratio)	0.12	Once/0.12	Twice/0.12
	0.32	Once/0.32	Twice/0.32
	0.48	Once/0.48	Twice/0.48

2.1.4. Procedure

Prior to the survey, all participants: (1) read the experimental description of the survey, and (2) agreed to participate. If the check box for “I agree” was not checked, then the survey was terminated. If a participant agreed, (3) a video (1912 × 1036 pixels, 30 fps) was displayed within 30 s, and (4) four questions were presented following the video. The process from steps (1) to (4) was repeated six times. Fig. 3 shows the entire survey process and the random selection of a video clip from among the six combinations of independent variables (Table 2). The takeover situation was also randomly selected from three takeover situations: CarDistance, Pedestrian, and FromSide (Table 1). After the participants experience each situation in the video, the video is immediately blacked out and four questionnaires are presented to ask about situation awareness, and perceived urgency (distance and danger). This method was devised by Situation Awareness Global Assessment Technique (SAGAT), which measures situation awareness, and SAGAT is the most sensitive and reliable method among various methods for measuring situation awareness [52].

Fig. 3.

Process diagram for the survey for Experiment 1.

Among the four questions, two (Q1 and Q2) were used to measure situation awareness. They were “What did you find when the alarm was triggered?“, and “In which direction was the (car, pedestrian) found?” Questions for perceived urgency followed: “How far did you think the distance was from what you found when the alarm was triggered?” and “How dangerous did you think the risk of a collision was?” Each participant watched a total of six videos and experienced all six types of looming visual cues. Considering that one video played for less than 30 s, the estimated time required to conduct the survey was approximately 10 min.

2.1.5. Measurement

• (1)Situation Awareness

For the measurement of Situation Awareness, the responses from the first (Q1) and second (Q2) questions were scored as 1 point if both questions were correctly answered. If either of the answers were incorrect, no score was assigned. In this experiment, we wanted to measure situation awareness as a whole, not each sub-level individually.

• (2)Perceived Urgency

Distance (Q3): Participants used a scale bar to express the perceived distance to obstacles (a pedestrian or a car), from 0 to 10 m.

Danger (Q4): Participants used a scale bar to express the perceived urgency of the situation on a 0–10 Likert scale. Zero indicated that it was not dangerous at all, and 10 indicated that it was very dangerous.

2.2. Results

The collected data were analyzed using SPSS 25.0, and Mauchly's sphericity test was performed first to test the variances of the differences between the size ratio and the looming speed of the presenting cue. Having found that the sphericity assumption was satisfied, a two-way repeated measures ANOVA test was conducted to compare the effect of visual cues on situation awareness, distance, and perceived urgency. The descriptive statistics are presented in Table 3.

Table 3.

The descriptive statistics in Experiment 1.

DV	situation awareness			distance			danger
Size	0.12	0.32	0.48	0.12	0.32	0.48	0.12	0.32	0.48
Once	0.86 (.35)	0.71 (.46)	0.69 (.47)	3.78 (2.54)	4.00 (2.23)	4.71 (2.82)	6.24 (2.41)	4.86 (2.94)	4.55 (2.51)
Twice	0.63 (.49)	0.82 (.39)	0.78 (.42)	4.43 (2.44)	4.86 (2.57)	4.96 (2.28)	4.24 (3.10)	3.49 (2.76)	3.84 (2.48)

2.2.1. Situation awareness

There were no significant main effects of looming speed, F (1, 48) = 0.82, p = .78, and size ratio, F (1, 48) = 0.16, p = .85. An interaction was found between the two independent variables, F (2, 98) = 5.89, p = .004. Fig. 4(a) shows a graph with variation in the scores achieved with the different looming speed as the size ratio increases. For the comparably small size (0.12 ratio), the situation awareness was better at once per second, but for the larger size (0.32, 0.48 ratio), the faster the twice, the greater the situation awareness.

Fig. 4.

Estimated marginal means for (a) Situation awareness, (b) Distance, (c) Danger.

2.2.2. Perceived urgency

Distance (Q3): Participants adjusted the scale bar between zero and 10 m to express how close they were to the collision when the TOR occurred. During the TOR, visual cues of different sizes were presented at different speeds (looming speed) with an auditory cue. As a result, main effects of looming speed, F (1, 48) = 6.96, p = .01 and size ratio, F (1, 48) = 4.36, p = .042 were significant. However, the interaction effect was not significant, F (2, 96) = 0.70, p = .498. As illustrated in Fig. 4(b), the participants felt closer to the collision when presenting a cue at once than twice, regardless of the size ratio of the visual cue. Also, they felt closer to the collision when a smaller cue was presented, regardless of looming speed.

Danger (Q4): Both looming speed and size had a significant effect on danger (looming speed: F (1, 49) = 18.59, p < .001; size: F (2, 98) = 7.46, p = .001), respectively. A graph of the estimated marginal means for danger (Fig. 4(c)) indicates that a looming red disk expanding once per second was perceived to be more dangerous than that twice per second. Furthermore, the smaller size of the looming red disk tended to increase perceived danger. However, there was no significant interaction effect of looming speed and size, F (2, 96) = 1.79, p = .173.

2.3. Discussion

In Experiment 1, a within-subject experiment was designed to determine the required size and speed of expansion (looming speed) of a looming cue. There were six combinations of two independent variables: the looming speed and size ratio. The results of the responses of 49 participants in the experiment under the six conditions showed that both looming speed and size ratio effected significant changes in situation awareness and perceived urgency (distance and danger).

We found that perceived distance, danger, and situation awareness changed according to the size and speed of expansion of the looming object. These results indicate the following: (1) the visual nervous system responds to a looming object, and (2) this system is involved in the perception of distance. According to Ref. [53], neurons responding to monocular looming stimuli (i.e., expanding symmetrically) affect the perception of time-to-collision (TTC). In a driving context, when the leading vehicle starts to brake, the driver in the following vehicle attempts to avoid collision due to the induction of the perception of TTC [54]. Since the time to reach a collision can be considered as distance/velocity, neurons responding to looming stimuli are directly involved in the perception of the distance to the collision. From the perspective of Experiment 1, the distance to collision was the distance to the looming stimulus. As shown in Fig. 4(b), the distance was perceived to be the shortest for the once per second and 0.12 size ratio condition. On the other hand, the perception of danger according to the size and looming speed is presented in Fig. 4(c), which is similar to the x-axis symmetry of the graph in Fig. 4(b).

Also, we manipulated the size of the looming disk to investigate the effects of size and speed on perceived danger and distance. The results showed that participants perceived the looming disk as closer as its size increased, but the perceived danger decreased as the size increased, shown in Fig. 4(b) and (c). This result indicates that a looming red disk can trigger an escape reaction due to its approaching movement. These results support findings from previous research [40,41], while also suggesting the appropriate size and speed of expansion to be used in Experiments 2 and 3.

Although the correlation between perceived distance and perceived danger was not directly measured in this study, it is possible that the different methods used to measure the two variables may have contributed to the lack of a clear relationship. Additional studies are needed to further investigate the correlation between perceived distance and perceived danger.

In situation awareness, only the interaction between the looming speed and size of the looming disk was observed. Situation awareness was highest for once per second × 0.12 and twice per second × 0.32. This experiment did not determine an optimal l/v of once per second × 0.12, or twice per second × 0.32, because the size shown to participants depends on the device used by the participant in the online survey. Instead, it provides a relative size for the angle of view. Since the looming disk size 0.32 may reduce visual information by hiding the screen, we selected a visual looming cue ratio of 0.12, which was presented once per second for Experiments 2 and 3.

The within-subject design of Experiment 1 may have decreased the effect on situation awareness and perceived urgency because a participant was repeatedly exposed to the same event. For example, one participant encountered a “Pedestrian” situation twice, in which cues of different conditions were presented. In this case, they may have answered the question easier than the first trial. In addition, the perceived urgency for the second time may be lower than the first time.

Nevertheless, the within-subject design was chosen in Experiment 1 because the impact of duplicate events for one participant would be smaller than the individual differences between participants [55]. Participants have different viewing angles, immersion in non-driving-related tasks, and individual driving patterns. Therefore, it was reasonable to choose a within-subject design to alleviate significant individual differences and to determine the size and speed of expansion of the looming stimulus. We randomized the order of events and the type of stimulus to reduce this ‘learning effect’ [55].

3. Experiment 2

The goal of Experiment 2 was to investigate the benefits of a looming red disk as a visual cue for multimodal TORs. Previous studies that used flashing icons as a visual cue for a TOR did not enhance fast responses beyond other modalities, such as auditory or haptic cues [26,56]. However, we focused on other methods of visual cue manipulation rather than flashing icons as alarm signals to deliver urgency to a situation in a driving context.

Previous research showed that a looming visual stimulus affected perceived time-to-collision (TTC) and that it was related to potential threat [47]. Also, a looming optical stimulus induced fear responses in rhesus monkeys [57]. Human infants also exhibit defensive responses to looming visual cues [58]. Another study claimed that there is a visual path in the human brain that detects looming motion [59]. In conclusion, looming stimuli can be regarded as triggers for stereotyped defensive responses and are important factors for the perception of predators. Thus, we hypothesized that the characteristic of a looming stimulus would promote the perceived urgency and situation awareness, which are important factors of a well-designed alert system [13,60,61].

To support this hypothesis, we manipulated a red disk to expand at a specific speed to a certain size, as determined in Experiment 1. We then compared this stimulus to a dimming red disk, which has been widely used to indicate urgency or notify important information in driver-vehicle interfaces [62]. In addition to both visual cues, an auditory cue was also applied to satisfy the multimodality of the suggested TOR. Finally, participants were randomly assigned to three groups: the control condition, in which the auditory cue was presented only; the dimming condition, in which a dimming visual cue was presented with an auditory cue; and the looming condition, in which a looming visual cue was presented with an auditory cue.

To investigate the effect of the TOR's visual cue on perceived urgency and situation awareness in more detail, we set the relative location of the TOR's visual cue differently depending on the TOR event case, unlike in Experiment 1. Events in which the visual cue appeared at the location where collisions occurred were categorized as ExtremelyMatched. Meanwhile, events presenting a visual cue in the location directly opposite from the collision were categorized as NotMatched. According to Ref. [39], a looming visual cue attracts visual attention because it represents potential urgency. A stimulus emerging on the opposite side from a looming stimulus that is attracting attention may cause inattentional blindness, so there is a high chance for drivers not to perceive the actual dangerous situation. Therefore, it may not always be desirable to use attention-grabbing stimuli for visual TOR. We attempted to elucidate the effect of location matching in Experiment 2.

3.1. Methods

3.1.1. Participants

A total of 90 participants (45 males and 45 females) between 20 and 49 years of age (M = 29.59, SD = 9.29) responded to the survey through a survey agency. No missing values were observed in the collected data; therefore, the data from all recruited individuals were used for the analysis. These participants could read, understand, and discriminate between letters and images displayed on the screen. Their driving experience ranged from 0 to 26 years (M = 6.40, SD = 7.11). In addition, the average driving time per day for their driving experience for the past three months was between 0 and 287 min (M = 43.66, SD = 46.24). It was announced in advance that free activities would be allowed before the TOR alarm was triggered. Participants were rewarded with 1000 points from the survey agency for the survey. The survey participants were randomly assigned to one of the three groups at the start of the survey, with 30 participants per group.

3.1.2. Experimental variables

Experiment 2 was conducted to compare the looming and dimming stimuli using an online survey. In this experiment, participants were randomly assigned to three groups: control, dimming, and looming. For the control group, the takeover situation was notified only with an alarm sound (without any visual cue). The TOR alarm for the dimming group was a dimming red disk as the visual cue. The opacity of the red disk changed from 0 to 100% when an alarm sound was presented. The TOR alarm for the looming group was a looming red disk as the visual cue. The size of the red disk was changed from zero to a 0.12 (disk: display screen height ratio) with an alarm sound. Depending on the type of visual cue, situation awareness and perceived urgency (distance and danger) were measured as dependent variables. The experimental variables are listed in Table 4.

Table 4.

Descriptions of independent variables under three conditions: control, dimming, and looming.

Independent Variables		Description
		Feature	Image
Visual Cue	Control	Alarm sound only (1 s)	No Image
	Dimming	Alarm sound (1 s) & Red disk with varying Brightness (opacity)
	Looming	Alarm sound (1 s) & Red disk with varying Size

3.1.3. Apparatus

The video clips for the survey were created using the same simulator as that in Experiment 1, but several scenes were added. Table 5 illustrates the order and complexity levels for TOR event cases occurring in each path. The situation for each TOR event case is presented in Table 1 [63]. The video clip begins from 30 s before the TOR to the moment immediately before the collision. The design of the TOR was presented differently for each group. The resolution of each video was 720 × 480 pixels with 30 fps, with a total length of 37 s, including 2 s black frames. A black frame was inserted to prevent cheating during the question-and-answer phase. Each group of people was exposed to only one type of TOR, and each participant experienced three events corresponding to each level of SA. A total of nine events were presented to each participant. Fig. 5 shows an example of a TOR with a visual cue.

Table 5.

TOR event cases with complexity levels: Events appear on each driving path, and “First, Second, and Third” in the table refer to the order in which the events appear on Path 1–3.

	TOR Event Cases with complexity levels
	First (low complexity)	Second (moderate complexity)	Third (high complexity)
Path 1	ParkedCar	FromSide	CarDistance
Path 2	TooSlow	Pedestrian	ParkedCar
Path 3	FromSide	ParkedCar	CarDistance

Fig. 5.

An example of a visual cue that appears with a TOR alarm sound.

3.1.4. Procedure

Before participating in the survey, all participants: (1) read the experimental description of the survey and (2) agreed to participate. Exclusive random selection was performed three times to determine the order of the three paths, and then video clips from each path were shown in order. The order of the video clips for each path was based on the driving path of a conditional automated vehicle in the simulator. At the end of each video clip, the participants answered questions regarding situation awareness, distance, and urgency perceived from the video. Fig. 6 illustrates the procedures for Experiment 2.

Fig. 6.

Process diagram of the survey for Experiment 2.

The questions relating to situation awareness were “Q1. What did you find when the alarm was triggered?” and “Q2. In which direction did you find?” The perceived distance question was “Q3. How far did you think the distance to the ‘answer of Q1’?” The perceived danger was determined by the answer to “Q4. When the alarm was triggered, how dangerous did you think the situation was?” As Q3 and Q4 were designed to determine a relative estimate, participants were asked to answer using a scale bar. The scale bar for Q4 was a Likert scale, where zero meant no danger at all and 10 meant very dangerous.

While most of the questions were similar to those in Experiment 1, there was a difference in that the questions following the first question were presented differently according to the correctness of the answers to Q1. If the participant chose an answer that was not shown in the video relating to Q1 or responded that nothing was found, neither the second question for situation performance (“In which direction did you find?“) nor the question on perceived distance appeared. Rather, the question regarding the perceived danger (“Q4. How dangerous did you feel when the alarm was triggered?“) appeared instead. Since Experiment 2 includes only situations without collision similar to “ParkedCar”, Q4 implied what degree of perceived urgency they felt by the situation itself when the TOR alarm was triggered.

The three dependent variables (situation awareness, perceived distance, and perceived danger) were measured for the control, dimming, and looming groups. Subsequently, a univariate ANOVA test was conducted to determine if there were significant differences for each dependent variable across the different groups.

3.2. Result

First, we categorized the complexity of driving scenarios based on situation awareness theory. To determine whether there is a difference in people's responses depending on the complexity of the scenario, Repeated Measures Analysis of Variance (Repeated Measure ANOVA) was conducted with the level of complexity of the scenario experienced by each subject as the independent variable and the level of situation awareness, perceived distance, and perceived danger as the dependent variables. Each subject experienced three scenarios of one complexity level. The subject's performance in each of the three scenarios was averaged and used in the analysis, and if the subject scored zero situation awareness in all three scenarios at a single complexity level, they were excluded from the analysis. In total, we used data from 85 subjects and found that complexity level had a significant effect on situation awareness (F = 42.54, p < .001) and perceived danger (F = 80.30, p < .001). However, there was no effect on perceived distance (F = 2.23, p = .111). Post hoc analysis using Fisher's Least Significant Difference revealed that for situation awareness, the low complexity condition (M = 0.41, SD = 0.04) was lower than the moderate complexity (M = 0.73, SD = 0.03), p < .001, 95% CI [0.40, −0.23], and high complexity condition (M = 0.76, SD = 0.03), p < .001, 95% CI [-0.43, −0.27]. For perceived danger, the low complexity condition (M = 3.31, SD = 0.28) was lower than the moderate complexity condition (M = 5.91, SD = 0.24), p < .001, 95% CI [−3,19, −2.03], and the moderate complexity condition was lower than the high complexity condition (M = 6.69, SD = 0.25), p = .003, 95% CI [−1.25, −0.28]. People showed higher levels of situation awareness and perceived the situation as more dangerous as the scenario they experienced became more complex and required more maneuvers. There were no gender and age differences or interactions. Next, we analyzed differences in responses based on the type and location of the cues, which was the main research question of this experiment.

3.2.1. Situation awareness

One point was scored for every correctly answered question regarding situation awareness. As there were two questions per video clip, the maximum score was 18 points for the nine video clips with TOR cases. We categorized all nine TOR event cases into two groups to gauge the effect of visual cue presentation on visual attention. First, TOR cases in which a visual cue was presented opposite to were categorized as NotMatched. For example, a TOR with a visual cue presented on the lower right, while a parked car was presented on the left. ExtremelyMatched indicated the category in which a potential collision could occur on the right side of the display, while a visual cue was presented on the lower right. We conducted a univariate ANOVA for situation awareness for the NotMatched, and ExtremelyMatched categories.

Results showed a marginally significant difference overall among the three participant groups receiving the different cues associated with the TORs, F (2, 87) = 3.02, p = .05, in the NotMatched category: control (M = 3.00, SD = 2.08), dimming (M = 1.67, SD = 2.04), and looming (M = 2.33, SD = 2.17). The results for points scored for situation awareness (mean ± SD) are shown in Fig. 7(a). The control group achieved significantly higher SA performance for situation awareness than the dimming group, F (1, 58) = 6.27, p = .015. The looming group achieved a lower performance than the control group, but the difference was not significant. For cases in the ExtremelyMatched category, there were no significant differences among the three visual cue groups. Fig. 7(b) illustrates the mean ± SD point scores for situation awareness for cases in the ExtremelyMatched category.

Fig. 7.

Situation Awareness for the (a) NotMatched category (b) ExtremelyMatched category (*p < .05).

3.2.2. Perceived urgency

Participants in the three groups rated perceived urgency (distance and danger) on a 0–10 Likert scale for every event. If the answer to the situation awareness question (Q1) was incorrect, the answer to the perceived distance was excluded from the analysis. A univariate ANOVA test was conducted to verify the effect of visual cue on perceived distance for the three conditions, but there was no significant effect (p > .05).

Next, we analyzed the effect of visual cue on perceived danger. All TOR event cases were grouped into two categories, NotMatched and ExtremelyMatched, and a univariate ANOVA test was conducted for each category as for the situation awareness analysis. There was a significant difference overall for the NotMatched category, F (2, 87) = 3.66, p = .03. The mean ± SD group ratings for perceived danger are shown in Fig. 8(a). Comparing the ratings of the looming and dimming conditions indicated that a looming red disk (M = 2.45, SD = 2.05) promoted perceived danger significantly more than dimming (M = 0.91, SD = 1.09), F (1, 31) = 6.83, p = .014. Fig. 8(b) illustrates the results of the univariate ANOVA test conducted on the extremely matched case. There was no significant difference between any of the groups overall, although the difference between the dimming and looming groups approached significance, F (1, 57) = 3.72, (p = .06).

Fig. 8.

Perceived Urgency for the (a) NotMatched category and (b) ExtremelyMatched category (*p < .05).

3.3. Discussion

In Experiment 2, we examined the effects of visual cues and location matching on situation awareness and perceived urgency. The results are summarized below.

First, we found an effect of a looming visual cue on perceived urgency. Analysis of perceived urgency found that a looming motion makes situations more dangerous than the other two conditions. When it comes to the ExtremelyMatched category, three group did not show any differences between them. Although the mean value is slightly high in the looming group, but those differences were not signigficant. In the NotMatched category, the looming visual cue delivered a significantly higher urgency than the dimming visual cue. These results supported the hypothesis and corroborated the findings that looming stimuli trigger defensive and escape responses [64,65].

Second, because the looming cue promoted perceived urgency more than the dimming cue, it is necessary to confirm that this looming cue attracted more visual attention than the dimming. Similar to the analysis of perceived urgency, the analysis of situation awareness was conducted for two categories: NotMatched and ExtremelyMatched. As a result, in the ExtremelyMatched category, there were no differences between groups. On the contrary, the NotMatched category showed that the dimming cue is worse than the control group. It means that if the dimming cue appeared in the opposite direction, that visual cue seems to negatively affect situation awareness, and keep the drivers' attention in opposite direction. However, different studies conducted probe detection experiments and showed that looming motion attracts more visual attention than receding or random motion [38,66]. When considering this experiment based on previous studies, the driver's attention should be fixed in the opposite position when the looming cue appears. We can interpret this as follows. Rather than focusing attention on a specific location, looming cues can increase overall alertness and prepare drivers for dangerous situations. In the case of the control condition, situation awareness may be good because attention is not focused on a specific location because there is no visual stimulus.

Although Experiment 2 was conducted as an online experiment, it supported the hypothesis that the looming object was useful as a visual cue of the TOR. However, it should also be determined whether such results can be reproduced in more realistic driving environments. The actual driving environment has many distracting factors that draw visual attention, including screens, handles, brakes, as well as the surrounding environment. In addition, NDRTs not only attract visual attention but also require cognitive resources. Therefore, a laboratory study should be conducted to determine the effect of looming visual cues in a realistic driving environment. Thus, Experiment 3 was a laboratory study, in which a simulator implemented an environment similar to an actual driving environment, that was operated by the participants themselves.

4. Experiment 3

In Experiment 3, we tried to verify the effect of the looming cue using a driving simulator. Since various complex situations can have a great influence in the actual driving situation, in the experiment, the complexity of the situation was set as an independent variable, and comparison between looming, dimming, and conditions without visual stimulation was performed. The complexity of the situation was manipulated in the same way as that produced in the image of Experiment 1 (Table 1), and visual stimuli were always appeared in a dangerous place to exclude the effect of false alarm.

4.1. Methods

4.1.1. Participants

There were 30 participants (male = 18, female = 12) between 20 and 50 years of age (M = 34, SD = 8.5), all with driver's licenses. All participants self-reported normal or corrected-to-normal vision (>50%), and they understood the instructions in Korean. Participants were recruited through an experimental agency and rewarded with points paid by the experimental agency.

4.1.2. Apparatus

Fig. 9 illustrates the laboratory setting for the auto-driving test. The driving setup consists of a steering wheel, accelerator, brake, and the same type of seat as an actual vehicle seat to prevent movement while driving. The steering wheel and pedal set were Logitech's G29 model with a resolution of 1080 pixels (FHD), 27 inch size (59.77 cm × 33.62 cm), and refresh rate 60 Hz. The average distance from the seat to the monitor was 125 cm. A driving simulator implemented using UNITY was operated on a PC-connected monitor, and the participants' responses at the time of the TOR were logged to the simulator.

Fig. 9.

Laboratory Setting for the Auto-driving Test. A steering wheel and pedals are connected to a PC and Monitor.

The simulator used three modes: manual driving, tutorial, and the main trial. In manual driving mode, participants were immersed in the driving environment by manually exploring the roads within the simulator before experiencing autonomous driving. In the tutorial and main trial, three events corresponding to low, moderate, and high complexity appeared in each path and were manipulated as in Experiment 1. The configurations of the paths and events are listed in Table 1, Table 5 The tutorial used Path 1 and the main trial used Paths 2 and 3. All participants experienced all TOR event cases that appeared in Paths 1 to 3. The auditory cue presented to the control group was a 1-s short tone, as in Experiments 1 and 2, and it was also presented with a visual cue in the dimming and looming groups.

The tutorial notified the participants that an alarm would soon sound, and manual driving would be required before entering a takeover situation. In the main trial, however, the TOR appeared with an auditory cue, indicating a takeover. At this point, one of the two visual cues was presented to the dimming and looming groups. In addition, the TOR required manual driving for 5.7 s, during which the participants were required to avoid the risk of a collision or to be aware of a situation where a TOR had occurred. To evaluate situation awareness and perception of the situation, a question window appeared at the end of the takeover situation. The questions regarding situation awareness related to the vehicle's type and location, and state in each event. The perception of urgency was rated on a one (not dangerous at all) to five (very dangerous) Likert scale using a slide bar in the simulator.

4.1.3. Experimental variables

This study was conducted to compare the type of visual stimulus for the TOR (control, looming, dimming), and participants were randomly assigned to one of these groups. The details of the three groups are presented in Table 4. The two visual cues for the dimming and looming groups were 4.53° (field of view). The presentation speed of the visual cues was determined through pilot experiments (n = 9). Finally, 30 fps was chosen as the suitable speed for situation and urgency awareness. Referring to the results of Experiment 2, a visual cue for the TOR was presented to the driver with its location relative to the obstacle as in the ExtremelyMatched category.

The TOR event cases implemented within the simulator were divided into three levels, depending on the level of complexity. A low-complexity scenario was one in which there was no significant risk even if the participant did not move the steering wheel or apply the brake (e.g., “car parked on the shoulder”). In moderate complexity, we presented a situation in which a brake response is essential to avoid collision with a car slowing down in front of it. A TOR occurred when the distance from the vehicle in front during deceleration was reduced to less than 100 m. Because the speed of a conditional automated vehicle is 60 km/h on average, a participant should apply a brake in 6 s to avoid collision. In the case of a high complexity, a TOR occurred when a car suddenly approached from the left or right side of the intersection at high speed. The terrain or buildings were designed to hide the approaching car for a while; thus, the driver was required to predict what would happen next.

4.1.4. Procedure

The participants signed a consent form after listening to a detailed explanation of the experiment. Subsequently, they adjusted the gap between the seat and driving setup for comfortable driving. As shown in Fig. 10, the participants entered the manual driving mode before the experiment and drove in the same way as an actual driving situation. Participants were instructed to press a button to start the tutorial process when they felt sufficiently immersed in the simulation. During the tutorial, participants became aware of autonomous driving and takeover situations through notices and pop-up windows while driving. At the end of the tutorial process, participants entered their details into the simulator: gender/age/driving experience (the length of time elapsed after obtaining a driver's license and the average daily driving time in the last three months); then pressed a button to begin the main trial (driving along Paths 2 and 3).

Fig. 10.

Procedure Diagram of Simulator. Participants start with manual driving. During Path 2 & Path 3 Driving, they face up to three event processes including “Auto driving, TOR, Manual Driving, and Answering” for each map.

During auto-driving in the main trial, participants were allowed to use their mobile phones. When a TOR occurred, they were guided to concentrate on manual driving for 5.7 s to avoid collisions. During this time, all responses from the brake pedal and steering wheel were logged to the simulator. The answers for the TORs were also logged to the simulator. The duration of the experiment for participants, including questions, was 50–60 min.

During each TOR event case, participants drove manually to avoid collisions, then responded to questions regarding situation awareness and the perceived urgency of the TOR context. Situation awareness was scored using two questions per case. The questions concerned the vehicle's location, type, and movement; and participants selected one of five answers, including ‘not seen’. One point was given for each correct answer, with a total of 12 points (2 paths × 3 TOR event cases × 2 questions) given if a participant answered correctly for all events. Next, participants moved the slide bar to represent their perceived urgency of the situation after the TOR. The slide bar showed 1–5 points, with one being “not dangerous at all” and five being “very dangerous”. Data with incorrect answers to situation awareness were excluded because of low immersion in the experiment. SPSS 25.0 was used to analyze both situation awareness and perception of urgency.

4.2. Results

First, we analyzed whether there was a correlation between situation awareness and perceived dander scores. For each complexity level, we averaged participants' situation awareness and perceived danger scores and calculated Pearson's correlation coefficients. We found a positive correlation between situation awareness and perceived risk at the moderate complexity level, r = 0.64, p < .001, but not at the low complexity level (r = 0.11, p = .58) and high complexity level (r = 0.24, p = .20). In addition, independent samples t-tests and Pearson correlations were conducted to determine if there were differences in scores based on gender and age, and no gender differences were found both situation awareness and perceived danger. Age was positively correlated with perceived danger in the low complexity condition (r = 0.41, p = .026), with no significant correlation found in all other conditions.

4.2.1. Situation awareness

To analyze the research question, a repeated ANOVA was conducted with visual cue, a variable between participants, and complexity level, a variable within participants, as independent variables and situation awareness as the dependent variable. As a result, the main effect of complexity level was statistically significant, F (2, 54) = 34.34, p < .001. The main effect of visual cue was not significant, F (2, 27) = 2.02, p = .152, and the interaction was not significant, F (4, 54) = 1.01, p = .41. LSD post hoc analyses of complexity level revealed that situation awareness was higher in the low complexity condition than in the moderate complexity, p < .001, 95% CI [0.93, 1.54], and higher than in the high complexity condition, p = .003, 95% CI [0.17, 70]. The moderate complexity condition was lower than the high complexity condition, p < .001, 95% CI [−1.15, −0.45].

Although there was no overall interaction effect, we wanted to further explore whether there were differences between visual cues within each complexity condition. To do so, we performed multiple comparisons using RM ANOVA with estimated surrounding means to analyze whether there were differences between visual cues within each complexity condition. We found a significant difference between dimming and looming visual cues in the high complexity condition, p = .49, 95% CI [0.00, 1.19], suggesting that looming cues increase situation awareness more than dimming cues in the high complexity environment.

Low-complexity situations were designed to identify the location and type of parked vehicles, and under these conditions, all three visual cue groups had high situation awareness scores and there was no significant effect (p > .05). Moderate complexity events were designed to identify the type and state of the decelerating vehicle. To answer the question regarding the state of the vehicle, participants had to select one of three options: stopped, decelerating, or accelerating. Most participants correctly answered the type of vehicle (98.3%); however, the correct answer rate for the state of the vehicle (38.0%) was relatively low. In particular, participants in the control group failed to distinguish ‘decelerating state’ (30%) from ‘stopped’ (60%). However, participants in the dimming and looming group showed better performance distinguishing ‘decelerating’ (45%) from ‘stopped’ (45%). However, there was no significant effect of groups in moderate complexity. Fig. 11 (a) illustrates the situation awareness of the three visual cue groups (control, dimming, and looming) according to complexity level.

Fig. 11.

(a) Results of situation awareness and (b) perceived urgency for each visual cue group, according to the different complexity levels of TOR event cases (*p < .05).

4.2.2. Perceived urgency

Similar to the analysis for situation awareness, we analyzed perceived danger by complexity level and visual cue using RM ANOVA. The main effect of complexity level was significant, F (2, 54) = 66,92, p < .001, and the main effect of visual cue was not significant, F (2, 27) = 0.40, p = 68. The interaction effect between complexity level and visual cue was also not statistically significant, F (2, 54) = 0.35, p = .85. The LSD post hoc analysis of complexity level revealed that the low complexity condition had lower perceived urgency than the high complexity condition, p < .001, 95% CI [−2.65, −2.08]. The moderate complexity condition was also lower than the high complexity condition, p < .001, 95% CI [−3.22, −2.11], and there was no significant difference between the low and moderate complexity conditions, p = .35, 95% CI [-0.34, 94].

Although there was no interaction effect, we tried to analyze whether there were differences between visual cues within each complexity condition. Same as the previous analysis, we performed multiple comparisons using RM ANOVA with estimated surrounding means to analyze whether there were differences between visual cues within each complexity condition. There were no significant differences in perceived urgency among the three groups with different visual cues for cases corresponding to low or moderate complexity (p > .05). However, we did find a significant difference between control and looming cues within the high complexity condition, p = .017, 95% CI [-0.99, −0.11]. The looming group (M = 4.50, SD = 0.41) showed significantly higher perceived urgency than the control group (M = 3.95, SD = 0 0.60). Fig. 11(b) illustrates the perceived urgency of participants in each visual cue group in TOR event cases of low complexity, moderate complexity, or high complexity.

4.3. Discussion

An important role of the TOR is to deliver contextual information to the driver for smooth changes in control [13,67]. Previous studies have explored which modality is useful for conveying the urgency of a situation because appropriate delivery of the urgency of a situation is an important factor in eliciting a fast response from a driver [7,17,68]. Specifically, multimodal TORs aim to enhance the transition of driver attention and the perception of urgency and situation awareness, by presenting visual and auditory modalities simultaneously [18,30]. In these studies, delivery of urgency was performed via the auditory modality, and visual stimuli were used only to indicate where visual attention should be directed. However, auditory cues tend to deliver not only urgency but also unnecessary annoyance, while reducing confidence in the system [8,69]. They compared auditory, visual, and tactile modalities and found that auditory modality increased annoyance when delivering high urgency. Attraction of visual attention by multimodal TORs is also based on the ‘conveyance of urgency’ [36,37]. However, visual cues, which have been used in previous studies, did not convey urgency as intended [30].

In the current context, this study attempted to design an audiovisual TOR that promoted situation awareness by promoting the perception of urgency. As we mentioned, several studies have shown that a looming object acts as a stimulus that induces urgent behavior via an animal's visual system. Thus, we manipulated the visual cues of the TOR to move in a looming or dimming motion. We expected that the characteristics of the looming motion would induce greater perception of urgency and situation awareness than the dimming motion, which has been widely used in multimodal TORs [39,41].

The takeover situations provided by the simulator consisted of three levels of complexity, from required situation awareness Level 1 SA to Level 3 SA. ParkedCar (low complexity) was the event for the Level 1 SA, in which the auto-driving vehicle passed by a car parked on the shoulder after the TOR. This event required the driver to perceive the driving context and maintain the lane during manual driving after the TOR was sounded. Thus, participants performed fewer steering- and brake-related actions than more complex events and responded to a significantly lower awareness of the risk. FromSide was an event case in which another vehicle suddenly appeared at an intersection after the TOR. In particular, the approaching vehicle was obscured by surrounding buildings and terrain to induce more urgency than in the other cases.

Comparing perceived urgency by the complexity level, the looming group presented a higher perception of urgency than the control group in an urgent situation of high complexity. The relationship between the complexity and the looming effect can be interpreted as an adult driver reacting to a looming image only in an urgent context. This result is in line with existing literature and extends previous work in the auto-driving context [45]. In contrast, a dimming red disk did not show a significant difference in the perception of urgency compared to the control group, where no visual cue was presented. This result may explain why previous studies using flashing red dots as visual cues did not cause urgency [18,30].

Previous research conducted several experiments and showed that the recognition of looming motion as an urgent cue is due to the visual mechanisms that have evolved to explore the movements of natural enemies [41]. Therefore, based on previous studies [[36], [37], [38]], we predicted that the characteristics of a looming visual cue that induces urgency would promote visual attention and enhance situation awareness. This expectation was satisfied, as illustrated in Fig. 11(a). Situation awareness showed significant differences between the dimming and looming visual cue groups in the high complexity situation level of 3. In other words, looming visual cues affected situation awareness more significantly than dimming visual cues.

Among existing studies, fast responses have been obtained using looming visual stimuli [[70], [71], [72]]. However, these studies did not conduct experiments to explain the mechanisms that contributed to the fast responses. In addition, provision of situation awareness, which is one of the most important roles of the TOR, was not evaluated during the experiments. This present study is the first to explain why looming stimuli should be used as visual cues for TORs by experimentally presenting the escape response triggered by the characteristics of the looming movement. In other words, we have shown that a looming stimulus is the most appropriate signal for conveying the urgency of the context. The urgency conveyed by this stimulus attracts visual attention, thereby increasing situation awareness. Previous studies, such as [6,16], have shown that the perception of urgency affects driving behavior and higher urgency can lead to decreased driving performance. Thus, finding a balance between signaling urgency and avoiding potential risks is crucial. Our study focused on investigating the effect of looming visual cue on the perception of urgency and situation awareness for drivers in response to TORs while recognizing the risks of excessive urgency. Future research should explore ways to balance urgency and potential risks to ensure safe and effective human-vehicle interactions.

Although this study did not include any behavioral responses during manual driving, the validity of the looming signal as a visual cue for the TOR was demonstrated. Further studies should explore how a looming red disk affects the braking response, and steering wheel and overall response. Furthermore, the emotional response of the driver should be evaluated and it will be possible to experiment by applying it to various types of stimuli.

5. General discussion

In this study, we investigated the effect of a looming red disk as a visual cue for a takeover request (TOR) on situation awareness and perceived urgency. Our primary hypothesis was that the characteristics of a looming stimulus would promote perceived urgency and situation awareness, which are essential factors for a well-designed alert system. To test our hypothesis, we conducted three experiments.

Experiment 1 aimed to determine the optimal speed of expansion and size of a red disk appropriate for the human visual mechanisms that capture looming motion. We found that a red disk induced different levels of perception of urgency and situation awareness, depending on the looming speed and final size. These results suggest that the design of TORs needs to take into account the specific parameters that will optimize the perception of urgency and situation awareness for drivers.

Experiment 2 investigated the effect of the looming visual cue on the perception of urgency and situation awareness in a driving context. We compared the looming visual cue with a dimming visual cue and found that the looming visual cue increased awareness of risks and situation awareness in a driving context, compared to the dimming visual cue. These findings suggest that the characteristics of a looming visual cue play an important role in promoting situation awareness and perceived urgency during a TOR.

Finally, Experiment 3 was conducted in a driving simulator to assess the effectiveness of the looming visual cue in raising perceived urgency and situation awareness in an urgent situation. We found that the looming visual cue was effective in raising perceived urgency and situation awareness, even in a simulated environment. These results suggest that the use of a looming visual cue can promote the perception of contextual urgency and situation awareness, even in complex driving scenarios.

One limitation of this study is that we did not measure variables such as maximum steering wheel after TOR or the time from the TOR until the driver turns the wheel more than 2° (RTS) [73]. While these measures are important indicators of driver behavior, we chose to focus specifically on the effects of the looming signal as a visual cue on situational awareness and perceived danger. Future studies could explore the effects of the looming visual cue on these additional variables, providing a more complete understanding of its impact on driving behavior.

Another limitation of this study is the short duration of the videos in Experiments 1 and 2. This may not be enough time for people to fully immerse themselves in the video, even though another study found that people develop situation awareness of the surrounding vehicles after 20 s [51]. Although this was an exploratory study to determine the stimuli, we consider that it would have been more valid if participants were exposed to the videos for a longer period of time.

In particular, it would be interesting to investigate how the looming visual cue affects participants' behavior during manual driving after a TOR. This could provide valuable insights for the design of TORs that promote safe and effective human-vehicle interactions. Furthermore, as autonomous vehicles become more prevalent, the development of effective alert systems and TORs will be critical to ensuring passenger safety and comfort.

6. Conclusion

In concluding our analysis, the prominent finding derived from our studies emphasizes the significant impact that the looming visual cue might have on enhancing situation awareness and perceived urgency in takeover request scenarios. In our study, we used the simplest possible form of stimuli to test the looming effect. In terms of practical implications, we could explore whether the looming effect is the same in practice using currently used TOR signals, and whether other modalities have similar effects.

Overall, this study provides important insights into the design of effective TORs, highlighting the importance of the specific parameters that optimize the perception of urgency and situation awareness for drivers. Further research in this area will be important to continue improving the design of TORs and other alert systems, ultimately contributing to the development of safer and more effective human-vehicle interactions.

Funding

This work was supported by the National Research Foundation of Korea Grant (NRF-2021S1A5C2A02088387) funded by the Ministry of Education of the Republic of Korea.

Data availability statement

Data will be made available on request.

CRediT authorship contribution statement

YounJung Park: Writing - original draft, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Hyunmin Kang: Writing - review & editing, Writing - original draft, Visualization, Methodology, Formal analysis, Conceptualization. Jeayeong Ji: Writing - original draft, Resources, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.