Protocol to evaluate the effectiveness of a virtual-reality-based behavioral intervention in enhancing sensory responses to real-world warning

Summary

Habituation to signals that warn of a potential danger in high-risk work environments is a critical causal factor of workplace accidents. Such habituation is hard to measure in a real-world setting, and no existing intervention can effectively curb it. Here, we present a protocol to enhance workers’ sensory responses to frequently encountered warnings at workplaces using a virtual-reality-based behavioral intervention. We describe steps for performing a virtual reality experiment and an electroencephalography (EEG) experiment with human participants.

For complete details on the use and execution of this protocol, please refer to Kim et al.¹

Graphical abstract

Highlights

• •Steps for evaluating the effect of virtual reality (VR)-based behavioral intervention
• •Protocol to elicit habituation to warning signals in a VR environment
• •Design for a behavioral intervention mitigating habituation to warning signals
• •Protocol for examining neural evidence of sensory habituation to warning signals

Publisher’s note: Undertaking any experimental protocol requires adherence to local institutional guidelines for laboratory safety and ethics.

Habituation to signals that warn of a potential danger in high-risk work environments is a critical causal factor of workplace accidents. Such habituation is hard to measure in a real-world setting, and no existing intervention can effectively curb it. Here, we present a protocol to enhance workers’ sensory responses to frequently encountered warnings at workplaces using a virtual-reality-based behavioral intervention. We describe steps for performing a virtual reality experiment and an electroencephalography (EEG) experiment with human participants.

Before you begin

The protocol below describes the specific steps for evaluating the effectiveness of experiencing virtual reality accidents in enhancing sensory responses to real-world warning alarms.¹ This protocol provides information regarding (1) how participants’ habituation to frequently presented warning alarms can be measured in a VR environment, (2) how a virtual accident can be presented contingent upon a participant showing habituated inattention to warning alarms, and (3) how the effectiveness of experiencing the virtual accident can be evaluated using electroencephalography (EEG) and event-related potentials (ERPs). This protocol also provides information about designing the VR environment that controls the trigger of the virtual accident based on the behavioral responses of a participant. For detailed information of the main behavioral study, please refer to the referenced publication: Kim et al., iScience (2023).¹

Institutional permissions

All participants in the study (n = 35; 32 males and 3 females; 20–43 years; mean age = 27.26, sd age = 6.09) were recruited from a construction company in the United States. The average years of participants’ working experience was 4.73 years (SD = 5.11 years): less than 1 year (17.14%), 1 year to less than 5 years (51.42%), 5 years to less than 10 years (17.14%), 10 years less than 20 years (8.57%), and more than 20 years (5.71%). The experimental protocol was approved by the Institutional Review Board (IRB) at Texas A&M University (IRB 2019–1270D). Informed consent was obtained from all participants. Participants were compensated with $200 for participating in the study. Those who aim to use the protocol in studies with human subjects must acquire permission from their relevant institution.

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Software and algorithms
VIVE SRanipal Eye Tracking SDK	HTC Corporation	Version 1.1.0.1
Unreal Engine	Epic Games, Inc.	Version 4.22.3
Autodesk 3dS Max	Autodesk, Inc.	Version 2019
Autodesk Maya	Autodesk, Inc.	Version 2019
PsychoPy	Open Science Tools Ltd.	Version 2021.1.4
OpenBCI GUI	OpenBCI	Version 5.0.4
OpenSync library	OpenSync	Version 3.0
Logic Pro X	Apple, Inc.	Version 10.6.3
Python	Python Software Foundation	Version 3.8.2
Python SciPy toolbox	SciPy	Version 1.4.1.33
MNE-Python package	https://doi.org/10.5281/zenodo.7314185	Version 0.24
EEGLAB	Elsevier: https://www.sciencedirect.com/science/article/pii/S0165027003003479?via%3Dihub	Version 2021.1
Artifact Subspace Reconstruction method	GitHub: https://github.com/moeinrazavi/EEG-ASR-Python	Version 1.0.0
MATLAB	MathWorks	Version R2021a
Other
HTC Vive Pro Eye	HTC Corporation
OpenBCI EEG system	OpenBCI	OpenBCI Board Kit of 32 bits and USB Dongle, a cap containing 20 electrodes

Materials and equipment

Below, additional information is provided regarding the materials and equipment required to create the experimental environment, along with a detailed description of designing the virtual environment used in the study and custom software used to implement the study.

Virtual reality headset with eye-tracking sensors

In this study, the HTC Vive Pro Eye² (HTC Corporation, New Taipei City, Taiwan; resolution: 2880 × 1600 pixels; field of view: 98 degrees horizontal of visual angle and 98 degrees of vertical visual angle; refresh rate 90 Hz) was used to display the developed VR environment. A participant’s eye movement data was collected using eye-tracking sensors embedded in the HTC Vive Pro Eye (accuracy: 0.5°–1.1°; Trackable field of view: 110°).³ HTC SRanipal SDK⁴ v1.1.0.1 was used to get filtered eye-tracking data (e.g., gaze origin and gaze direction) of each participant with a peak frequency of 90 Hz. During the experiment, the Vive Wireless Adapter was used to provide a more immersive VR experience to worker. The wireless adapter system includes Intel WiGig data transfer module (i.e., Intel Wireless Gigabit⁵) that uses a 60 GHz band to transfer data through high speed (up to 7 Gbps) wireless communication. The wireless system replaces a cable between the Vive Pro Eye and a computer system. Participants needed to continuously move to performing the task. Using a wired VR system would interrupt participants’ behaviors.

Before the beginning of the experiment, eye-tracking sensors were calibrated for each participant through the following procedure: (1) the eye-tracking sensors automatically measured each participant’s interpupillary distance (IPD) and guided a participant to adjust the HMD’s display distance using a physical dial knob; (2) the eye-tracking sensors was calibrated for each participant using a calibration module embedded in HTC SRanipal SDK.⁴ During the experiment, the eye-tracking sensors projected a ray from a participant’s gaze point and documented the name of the object that was hit by the ray. The projected ray was invisible to the participants in the experiment.

Computer system

A Dell Precision T5820 computer (Dell, Round Rock, TX, USA; CPU: Intel i9–10900 3 3.7 GHz; RAM; DDR4 128 GB; GPU: Nvidia GeForce RTX 3080) was used to operate the developed VR environment.

Virtual environment development software

The virtual environment was created using Unreal Engine v.4.22.3.⁶ All components included in the virtual environment were created using Autodesk 3dS Max v.2019⁷ and Autodesk Maya V.2019.⁸

Virtual road maintenance environment

A virtual highway maintenance working environment where a participant performs an assigned task as a part of an asphalt milling crew was created for the experiment. The virtual environment includes several types of construction vehicles such as asphalt milling machine, asphalt paver, asphalt roller, and street sweeper. A general asphalt maintenance work process was simulated in the virtual environment: (1) An asphalt milling machine removes the existing asphalt and loads the crushed asphalt onto a dump truck to haul off-site; (2) As a member of a pedestrian cleaning crew, a participant then is required to sweep out remaining debris on the surface while following closely behind the asphalt milling machine; (3) A sweeper moves back and forth; (4) The asphalt paver lays down new asphalt; (5) Lastly, an asphalt roller moves forward (see Figure 1).

Figure 1.

Construction vehicles created with Autodesk 3dS Max and Autodesk Maya

To efficiently elicit participants’ habituation to warning alarms, reflected in a decrease in participants’ responses to repeatedly presented warning alarms, the virtual environment continuously exposed participants to the risk of struck-by accident, a virtual accident associated with the street sweeper. An auditory warning alarm from the street sweeper was presented when the sweeper approached a participant. The movements of virtual construction vehicles were carefully designed to respond to a participant’s vigilance behaviors (see Figure 2). The following paragraphs provide detailed information about the movements of virtual construction vehicles.

Figure 2.

Experimental setting

(A) Schematic of the VR experimental setup with a VR headset, motion controllers, and eye-tracking sensors embedded in the VR headset. The participants’ task was removing all debris and cleaning the entire surface of the road with a broom. The participant’s actual/physical sweeping motion was synchronized in the VR environment via motion controllers attached to a real broomstick.

(B and C) Schematic of the experimental and intervention scenario. While the participants were performing the road cleaning task, construction vehicles continuously moved back and forth. Auditory warning alarms sounded to warn the proximity of the vehicles.

Movement of construction equipment

The reciprocal movement of the street sweeper that moves behind a participant is controlled by measuring its distance from a participant. When the street sweeper reaches the minimum designated distance to a participant—7.5 m, the street sweeper turns off its warning alarm and begins to reverse. This reciprocal movement of the street sweeper repeatedly exposes a participant to the risk of a potential struck-by accident while a participant is performing the virtual road sweeping task. Determining the minimum distance where the street sweeper starts to move backward is an important step. If the minimum distance is too far away, participants tend to totally ignore the movement of the street sweeper and only focus on performing the given task. On the other hand, if the minimum distance is too close, participants are likely to focus on the street sweeper and stop to perform the required task. The research team performed pilot experiments and carefully monitored participants' behaviors. The results of the pilot experiments showed that when the minimum distance was 7.5 m, participants tended to pay attention to the movement of the street sweeper right after the beginning of the experiment, and to the approaching equipment as an alarm sounded at the start of the experiment. However, as the experiment continued, they gradually shifted their focus to performing the given task and displayed a pattern of inattention to the sweeper. The designs of the movements of construction equipment are summarized in pseudocode below. For every tick, the VR system updates the functions to check the conditions of the movement of the street sweeper and the participant’s attention to the sweeper.

for every tick

P = reference to player pawn

 MP = cast P to motion controller pawn

 use MP reference to call MP.EyeFocus function

 if EyeFocus value is Sweeper

  // check whether MP was looking continuously before

  if LookingContinuously is set

   call LookBackMissCount

  if LookingContinously is not set

   set LookingContinuously to true

     // MP is looking Sweeper

   call LookBackMissCount

 If EyeFocus value is not Sweeper

  set LookingContinuously to false

CheckPlayTime:

 if PlayTime is less than 20 minutes

  call CheckWhetherToMoveBack

 else if PlayTime is greater than or equal to 20 minutes

  call QuitGame function

CheckWhetherToMoveBack:

 if MoveBack is false then

  call UpdateLookBackMissCount

 if MoveBack is true

  call get distance to MP

UpdateLookBackCount:

 if MP checks Sweeper for current forward movement cycle

  if MP already checked Sweeper during current forward movement cycle

   call UpdateLookBackCount

  else if MP checked Sweeper for first time during current forward movement cycle

   set CheckedinCurrentCycle to true

   append int 1 to CheckedinRecentCycles

   increment LookBackCount

   log LookBackCount

   call UpdateLookBackRecentCycles

UpdateLookBackRecentCycles:

 if distance between Sweeper and MP is less than 7.5m

  if MP didn’t check Sweeper in current forward movement cycle

   If the accumulated value of LookBackRecentCycles is larger than or equal to 3

    if LookBackRecentCycles is not incremented for the current forward movement cycle

     set MissedinCurrentCycle to true

    append int 0 to LookBackRecentCycles

    log LookBackRecentCycles

     call MoveForward

    else if LookBackRecentCycles is already updated

     call MoveForward

   else if the accumulated value of LookBackRecentCycles array is equal to 2

    call CheckDistancetoMill

  else if MP checked Sweeper in current forward movement cycle

   call CheckDistancetoMP While MovingBack

 else if distance is not less than 7.5 m

  call CheckDistancetoMP // Continue to move forward

Check DistancetoMill:

 Mill = get a reference to Milling Machine

 calculate distance to milling machine

 if distance is greater than or equal to 0.2m

  MoveForward

 else if distance is less than 0.2m

  set MoveBack to true // Start to move backward

Check DistatncetoMP WhileMovingBack

 get distance to MP

 if distance is less than 30m

  set MoveBack to true

  reset CheckedinCurrentCycle to false

  reset MissedinCurrentCycle to false

  stop playing the AlarmSound

  MoveBackward

 else if distance is larger than 30m

  set MoveBack to false

MoveForward:

 set the offset to be moved in (+X) direction

 if AlarmSound is not already playing

  start playing the AlarmSound

 move sweeper by the offset in the forward direction (+X)

MoveBackward:

 set offset to be moved in (-X) direction

 move sweeper by the offset in backward direction

EyeFocus:

 FocusInfo = actor name // MP is looking at

  return InEyeFocus

Task in the virtual environment

A virtual road cleaning task was designed to accelerate participants’ risk habituation within a short time period and provide immersive experiences in the experiment. During the experiment, a participant was to sweep out all debris on the surface of the working lane with a broom. A participant’s physical sweeping movement with an actual broom was captured by the motion controllers attached to the broomstick and synchronized in the virtual environment with the movements of the virtual broom.

Behavior measurement

In the experiment, a participant’s eye and head movements to check an approaching street sweeper was defined as hazard checking behaviors. In order to measure the response pattern with respect to a participant’s visual attention to an approaching street sweeper, an eye-tracking system was embedded into the virtual environment. During the experiment, eye-tracking sensors embedded in the HTC Vive Pro Eye document what a participant is looking at with a peak frequency of 90 Hz. The eye tracking system documents the vigilant behaviors (response time), and the frequency of vigilant behaviors.

Virtual accident trigger system

The virtual environment includes a system that simulates a virtual struck-by accident with the street sweeper upon a participant’s habitual ignorance of the approaching street sweeper. The virtual accident simulation consists of visual accident scenes, crash sounds, and haptic feedback via the motion controllers. To trigger the virtual accident upon a participant’s habituated ignorance, a behavior checking system with a moving window was adopted. The moving window counted the number of a participant’s successes in checking the street sweeper over the 5 most recent exposures. When a participant fails to check on the approaching street sweeper 3 out of these 5 exposures, the street sweeper starts to move forward toward a participant until it collides with the participant. If a participant recognizes the street sweeper’s this erratic movement and succeeds in evading the collision, the street sweeper makes the normal reciprocal movement, and the behavior checking system also restarts to count the participant’s vigilant behaviors.

EEG device

EEG measurements were performed with the OpenBCI Board Kit of 32 bits,⁹ a USB Dongle, and an OpenBCI EEG electrode cap containing 20 electrodes pre-organized according to the international 10–20 system.¹⁰ We used 15 channels from the 10–20 electrode placement system: C3, C4, Cz, F3, F4, Fz, Fp1, Fp2, P3, P4, Pz, T3, T4, T5, T6. The EEG was sampled and digitized at 125 Hz. We also used syringes (5 mL) with a blunt needle and electrolyte gel Electro-Caps (ECI) to ensure a good connection between the EEG sensors and the participant’s scalp.

ERP design

Twenty images of construction sites were used as backgrounds in two EEG sessions. Half of the images were presented in the first session and the second half in the second session (one week later). In each session, each of the 10 images was presented 10 times in a random order with the restriction that the same image could not be presented two times in a row. We refer to sequence as the set of events occurring during the presentation of one image. Each sequence included four sounds (of 600 ms) separated by an interstimulus interval of 3500 ms, 4500 ms, or 5500 ms. The first sound of a sequence was presented at 1750 ms or 2750 ms after the beginning of the sequence to make sure that the EEG signal related to the processing of the sound was not (or minimally) affected by the processing of the image. The last sound of a sequence terminated 1750 ms or 2500 ms before the end of the sequence. To keep constant the duration of each sequence, the five intervals that preceded and followed the sounds always lasted 18 s in total. All the possible combinations of intervals that met this requirement were presented, in a random order. Thus, each sequence lasted 20.4 s (see Figure 3). In each session, 200 alarm sounds and 200 control sounds were presented. In two consecutive sequences, four-alarm sounds and four control sounds were presented in a random order so that the EEG session never included more than eight similar sounds in a row. Participants had a self-paced break after 50 sequences in each session.

Figure 3.

Example of a sequence of events in the EEG sessions

(A) The example of EEG experiment.

(B) A sequence included four sounds (of 600 ms) separated by an interstimulus interval of 3500 ms, 4500 ms or 5500 ms. The first sound of a sequence was presented 1750 ms or 2750 ms after a background image appeared (which corresponded to the beginning of a sequence) to make sure that the EEG signal related to the processing of the sound was not (or minimally) affected by the processing of the image. The fourth sound terminated 1750 ms or 2500 ms before the end of the sequence. To keep constant the duration of each sequence, the five intervals that preceded and followed the sounds always lasted 18 s in total. All the possible combinations of intervals that met this requirement were presented in a random order. Thus, each sequence lasted 20.4 s.

EEG program

PsychoPy

During the EEG sessions, a Dell Precision T3620 computer (Dell, Round Rock, TX, USA) equipped with PsychoPy software v2021.1.4¹¹ was used to present the stimuli on a Dell P217H monitor. The participants viewed the monitor from a distance of approximately 70 cm in a dimly lit room. Participants also wore Etymotic ER4XR 45Ω high fidelity, noise-isolating in-ear earphones (Etymotic Research, Elk Grove Village, IL, USA)¹² to listen to all sounds. We embedded OpenSync library in PsychoPy to synchronize and record EEG signals with the associated alarm/control task markers.

Sound design

The two auditory stimuli (alarm and control) were made using Logic Pro X software on a 2017 MacBook Pro (Apple Inc., Cupertino, CA, USA).¹³ The original alarm sound—a truck Backing Up Beep sound—was extracted from video files.¹⁴ For creating the control sound, the alarm sound was modified using a sound equalizer. The sound distribution was equated to have the same magnitude over the spectrum of sound (from 39 Hz to 14200 Hz), which generates white ambient noise. The two sounds lasted 600 ms each and were equated using the normalize function to set the loudness to 23 LUFF.

Step-by-step method details

The first EEG experiment

Timing: 60–80 min

This step outlines technical procedures to record EEG activity for alarm and control sounds before the VR intervention.

• 1.
  Select and connect the EEG cap.

  • a.
    Measure the participant’s head circumference by placing a tape measure around the widest part of the head (usually across the forehead just above the eyebrows, above the ears and across the widest part of the back of the skull).
  • b.
    Use the participant’s head circumference to select the appropriate cap.
  • c.
    Connect the cap to the board kit (i.e., 17 electrodes: the 15 channels described above, the ground, and the reference).

Note: Ensure participants washed their hair before arriving in the lab and did not use conditioners, hair creams, sprays or gels. Hair products can make it more difficult for the electrodes to adhere to the scalp.

Note: Ensure participants removed hair extensions, weaves and toupees before arriving for the procedure.

• 2.
  Install the EEG cap and earbuds.

  • a.
    Put the cap on the participant’s head.

    Note: The Cz electrode has to be positioned at the intersection of the line connecting the nasion (bridge of the nose) and the inion (occipital protuberance), and the line connecting the left tragus and the right tragus. The tragus is the small piece of thick cartilage on the inner side of the external ear that is immediately in front of and partly closing the ear canal.

    • i.
      Make sure the cap is left-right symmetric and tight on the head.
    • ii.
      Attach the cap around the chin.
  • b.
    Ask the participant to put in the earbuds.

    • i.
      Make sure the earbuds are well secured.
    • ii.
      Test the volume.
• 3.
  Adjust the impedance of each electrode and run the task.

  • a.
    Open the Open BCI software.

    • i.
      Select System Control Panel, Cyton (live), Serial (from Dongle), 16 CHANNELS, and click on AUTO-CONNECT.
    • ii.
      Click on Start Data Stream in the new window.
    • iii.
      Make sure the switch button of the board kit is set to PC mode and the USB Dongle is connected to the computer with the switch button set to GPIO_6.
  • b.
    Apply electrolyte gel between the electrodes and the participant’s scalp using a blunt syringe.

    • i.
      Begin with the ground and the reference electrodes.
    • ii.
      Check the impedance of each electrode in the Open BCI software by clicking on the icon Ω next to the number of each channel.
    • iii.
      Click again on the icon Ω when the impedance for a specific channel is correct. iv. Restart the same process for each electrode.

      Note: The gel should be moved gently with the blunt syringe until the connection is good. The impedance has to be as low as possible (ideally, below 10 kΩ). This step may take some time (20–30 min), and playing audio or engaging in conversation can make the procedure more comfortable for the participant.

  • c.
    Turn off the Open BCI software.
  • d.
    Execute the experimental Psychopy program.

    Note: Instruct the participant to pay close attention to the visual and auditory stimuli while remaining as still as possible throughout the entire session.

    Note: Ensure the participant has a comfortable seat and position before starting.

    CRITICAL: All electrode impedances should be kept below 15 kΩ. The reference electrode is crucial to obtain a good impedance for all the channels, so take time to properly adjust the connectivity of the reference electrode.

The first virtual reality experiment (first session)

Timing: 20–30 min

This section outlines the procedures for providing the VR intervention and measuring participants’ behavioral responses to hazards.

• 4.Execute a practice VR session with a participant (5 min).

Note: Before starting the main session, a participant needs to practice to become familiar with the virtual environment and learn how to carry out the task.

Note: The practice session does not include any struck-by hazards or simulated accidents.

• 5.Give a participant a break time (5 min): After the practice session, a participant has a 5-min break.
• 6.Execute the main VR session with a participant (10–20 min).

Note: Once the main session starts, a participant needs to sweep out the debris on the surface of the road for 20 min.

Note: During the experiment, if a participant shows habituated ignorance to the approaching street sweeper, the accident is triggered, and the experiment is discontinued immediately. However, to provide a participant enough time to be aware of approaching hazards, the virtual accident is not triggered until at least 10 cycles of the street sweeper’s reciprocal movements.

Note: The experiment continues for at least approximately 10 min. If a participant shows vigilant behaviors continuously, the experiment is finished 20 min after the experiment start time.

The second virtual reality experiment (second session)

Timing: 20 min

This section outlines procedures to evaluate the sustained effect of experiencing the virtual accident on mitigating habituation to frequently exposed struck-by hazards.

• 7.Execute the main VR session again a week after the first session.

Note: In the second session, there is no practice session.

Note: The procedure of the second main VR session is identical to the first main VR session with the exception that, in the second session, the virtual accident is not triggered even with a participant’s habituated ignorance of the approaching street sweeper. The experiment is finished 20 min after the experiment start time.

The second EEG experiment (second session)

Timing: 60–80 min

Technical procedures to record EEG activity for alarm and control sounds after the VR intervention.

• 8.The next steps are identical to those of the first EEG session described above with a different script for the experimental task.

Note: The participant needs to wash and dry their hair before the second EEG session. Make sure that the lab in which the experiment is performed has appropriate equipment.

Expected outcomes

In the real world, workers’ habituated inattention to frequently presented warning alarms is difficult to monitor. Therefore, there is no established intervention by which to effectively curb these behavioral tendencies. However, in a VR environment, the developmental process of workers’ habituated inattention to warning alarms associated with workplace hazards, such as backup alarms from construction vehicles, can be monitored and intervened without risking actual injury. The integration of various sensing technologies (e.g., eye tracking, motion sensing, and biosignal sensing) with a VR environment can quantify attentiveness to warning alarms. Furthermore, the effectiveness of experiencing a virtual accident, a negative consequence of inattention to warnings, can be measured.

The presented approach mainly focuses on two event-related potential (ERP) components, one early (N1) and one late (P3), known to be sensitive to auditory habituation. We hypothesized that construction workers would show a blunted ERP response to the alarm sound compared to the control sound, and that this difference would either diminish or go away after experiencing the virtual accident. Such an outcome for the N1 component would provide direct evidence for an effect of the VR accident experience on the automatic sensory processing of warning signals. In contrast, the same outcome for the P3 component would suggest that the increase of vigilant behaviors observed after a VR accident ensues from controlled cognitive processes, such as goal-directed attention mechanisms. We supposed that an alteration of the automatic sensory processing, reflected by the N1 component, would have more long-lasting effects on behavior, and in consequence, would strongly support the utility of the VR intervention to curb risk habitation in construction workers.

Quantification and statistical analysis

Analysis of data from the VR experiment

Frequency of vigilant behaviors

To quantify the participants’ visual attention to warning alarms from construction vehicles, eye-tracking sensors are utilized. During the VR experiment, eye-tracking sensors document what a participant is looking at. The eye movement monitoring system documents the participants’ exhibition of vigilant behaviors and the frequency of vigilant behaviors. The collected data from the VR experiment should be preprocessed as follows:

In this experiment, one presentation of a warning alarm associated is defined as one exposure to the struck-by hazard. During the VR experiment, when a participant exhibits a vigilant behavior (look back and check the source of the warning alarm for the first time in each exposure), the response time is documented. The frequency of vigilant behaviors can be quantified using the equation below:

$${CR}_i=\frac{Numberofcheckingcycles}{Numberofexposures}$$

where number of checking cycles = the number of cycles that a participant succeeded in checking the approaching street sweeper by a participant i; and number of exposures = the number of exposures to the struck-by hazard of a participant i.

Quantifying the tendency of habituation to warning alarms

To quantify the participants’ habituation to frequently presented warning alarms, a bivariate linear regression model predicting response time from number of exposures to warning alarms can be used with the following equation:

$$\widehat{y_i}=B_0+B_{1}N+r$$

where $\widehat{y_i}$ is response time at number of exposures N; $B_0$ is the intercept of the regression line at N = 0; and is the slope of the regression that indicates the change in response time for each increase in number of exposures N. If the test result of the coefficient $B_1$ is significantly positive, the development of participants’ risk habituation can be determined. To avoid data manipulation, if a participant does not check the warning alarms until the vehicle reaches the minimum distance where it starts to back up, that exposure should not be included in the response time analysis.

Evaluating the effectiveness of experiencing the virtual accident

To validate the effectiveness of experiencing the virtual accident in enhancing responses to frequently presented warning alarms from construction vehicles, multiple regression analyses are recommended. Multiple regression analyses can evaluate how a participant’s experience of the virtual accident in the first VR experiment affects the response time and frequency of the participant’s vigilant behaviors in the second VR experiment. A participant’s experience of the virtual accident in the first VR experiment can be coded as a categorical variable (0 for the no accident group and 1 for the accident experience group) The following regression equation can be used:

$$\widehat{y_i}=B_0+B_{1}N+B_{2}A+B_{3}NA+r$$

where $\widehat{y_i}$ is the dependent variable (response time) at number of exposures and accident experience A; $B_0$ is the simple intercept of the regression line in the no accident group (A = 0, NAG); $B_1$ is the change in the simple intercept for each increase in number of exposures N; $B_2$ is the difference in simple intercepts, comparing the accident group (A = 1, AG) with NAG; and is the difference in simple slopes, comparing the accident experience group with the no accident group. To investigate the intervention effect of experiencing the virtual accident in the first VR experiment on checking rate in the second VR experiment, paired-samples t-tests are recommended.

Analysis of data from the EEG experiment

Python 3.8.2 and EEGLAB v2021.1 were used to do the data preprocessing and analysis.

• 1.We used Python SciPy toolbox¹⁵ and applied a forward-backward (non-causal) high-pass filter with Kaiser window, transition band of 0.5–1 Hz, and attenuation of 80 dB to remove the drifts from the signal.
• 2.We used MNE-Python v0.24 package¹⁶ and applied a bandpass filter with 0.5 Hz and 40 Hz cutoff frequencies.
• 3.We changed the reference of the signals by removing the average of all the 15 channels from the signals.
• 4.In order to remove the non-stationary artifacts (e.g., motor artifacts) from the signal, we implanted the Artifact Subspace Reconstruction (ASR) method in Python (our code is available at: https://github.com/moeinrazavi/EEG-ASR-Python).
• 5.The EEG was segmented relative to the onset of the presentation of each sound stimulus (alarm/control) onset to create stimulus-locked epochs of 1800 ms that included a 300 ms pre-stimulus period. From each epoch, we subtracted the average of the signal from −300 ms to −100 ms as the epoch baseline.
• 6.In order to remove the stationary and non-brain signal artifacts (e.g., eye blink artifacts), we used the Independent Component Analysis (ICA) toolbox in EEGLAB with MATLAB R2021a.¹⁷
• 7.We visually removed the significantly noisy epochs from the data that were not corrected by the mentioned preprocessing steps.

Given the specificity of sounds used in the present study (long, non-monofrequency), we referred to the ERP components analyzed by the time-window rather than a specific label. Indeed, auditory stimuli used in EEG studies are usually short and monofrequency. The alarm sound we used in the EEG sessions was extracted from the alarm signal used in real construction sites to increase the ecological validity of our test. This sound lasted 600 ms and included one beep with a rise and a fall. As a consequence, the identification of ERP components observed in our study requires caution.

Two components, which could correspond to N1 and P3, were measured at the time window of 540–660 ms and 804–980 ms post-stimulus onset, respectively. We also performed a post-hoc analysis at the time window of 220–420 ms post-stimulus onset, based on the apparent signal difference observed between the alarm and control sounds during this period. For these three time windows, we computed the mean amplitude separately for alarm and control sounds at the Cz and Fz electrode sites, where the deflection was maximal.

A 2 × 2 repeated-measures analysis of variance (ANOVA) was conducted on mean ERP amplitudes with type of sound (alarm, control) and session (1, 2) as within-subject variables for each time window. Subsequent t tests were performed when appropriate.

Data of participants have to be excluded from the EEG analyses if there are too many artifacts or data loss in the EEG recordings (the inclusion criterion is relatively arbitrary, but 50% of trials for each condition would be a reasonable minimum requirement) in at least one of the two EEG sessions.

Limitations

The experiment was performed with the only road construction workers (i.e., pedestrian workers), and the designed virtual accident intervention is only associated with struck-by accidents (i.e., runover by construction vehicles). The Occupational Safety and Health Administration (OSHA) defined four critical causes of fatal accidents in workplaces—falls, struck-by, caught-in/between, and electrocution hazards. The effectiveness of virtual accident experience in enhancing sensory responses to real-world warning associated with other types of hazards will be validated in future works.

In the experiment, to examine the within-subject effect, all participants participated in the EEG experiment twice. Thus, the variance in individual participants’ auditory sensitivity does not have a significant impact on the analysis results. However, estimating or measuring individual participants’ auditory sensitivity via a pure-tone audiometry test at the beginning of the experiment would provide an opportunity to examine between-subject effects.

Troubleshooting

Problem 1

The virtual accident is stimulated only when a participant shows habituated inattention to warning alarms from construction vehicles. Therefore, during the VR experiment, having evenly distributed samples is very challenging. In our previous study, most participants became habituated and exhibited less vigilant behaviors with the increase of experiment time. They focused on performing the virtual sweeping task and experienced the virtual accident. The number of participants who showed constant vigilant behaviors and did not experience the virtual accident was relatively small.

Potential solution

Although there are several statistical methods that effectively deal with imbalanced samples, obtaining balanced samples would be ideal for data analysis. Please refer to the corresponding protocol step 6 of ‘‘step-by-step method details’’.

Problem 2

Generally, workers are required to take safety training on a regular basis. Employers provide various safety training programs to their employees (i.e., workers). Each worker takes different kinds of safety training at different times. Thus, it is hard to control workers’ prior safety knowledge. The difference in each workers’ safety knowledge and prior safety training experience might affect the result of this experiment. Furthermore, individuals have different levels of risk perception and hazard recognition capabilities. However, in this experiment protocol, individual differences in risk perception and hazard recognition are not considered.

Potential solution

If participants of an experiment are experienced workers and not student participants, we recommend providing conventional classroom-based safety training to all participants to better control their prior safety knowledge. Please refer to the corresponding protocol ‘‘before you begin’’.

Problem 3

During the EEG experiment, participants have to remain attentive during the whole EEG experiment. However, the EEG experiment can be soporific for some participants.

Potential solution

To circumvent this problem, offering a cup of coffee to a participant before the EEG experiment has been widely used. Please refer to the corresponding protocol step 3 of ‘‘step-by-step method details’’.

Resource availability

Lead contact

Further information and requests for resources and reagents should be directed to and will be fulfilled by the lead contact, Changbum R. Ahn (cbahn@snu.ac.kr).

Technical contact

Questions about the technical specifics of performing the protocol should be directed to the technical contact, Namgyun Kim (nkim@tamu.edu).

Materials availability

This study did not generate new unique reagents.

Data and code availability

• •The datasets supporting the current study have not been deposited in a public repository because the employer of the participants requested that the research team not publicly disclose the original experimental data without their consent but are available from the corresponding author on request.
• •This study did not generate the original code.
• •Additional Information: Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.