A virtual reality platform for multisensory integration studies

Abstract

A unique virtual reality platform for multisensory integration studies is presented. It allows to provide multimodal sensory stimuli (i.e. auditory, visual, tactile, etc.) ensuring temporal coherence, key factor in cross-modal integration. Four infrared cameras allow to real-time track the human motion and correspondingly control a virtual avatar. A user-friendly interface allows to manipulate a great variety of features (i.e. stimulus type, duration and distance from the participants’ body, as well as avatar gender, height, arm pose, perspective, etc.) and to real-time provide quantitative measures of all the parameters. The platform has been validated on two healthy participants testing a reaction time task which combines tactile and visual stimuli, for the investigation of peripersonal space. Results proved the effectiveness of the proposed platform, showing a significant correlation (p=0.013) between the participant’s hand distance from the visual stimulus and the reaction time to the tactile stimulus. More participants will be recruited to further investigate the other measures provided by the platform.

I. Introduction

The study of multisensory integration plays a key role in understanding the brain. Cross-modal integration contributes to creating a unified and coherent representation of the environment [1] and the body; it also affects reaction behaviors [2], [3] and body ownership [4].

Renowned protocols that investigate multisensory integration are: the rubber hand illusion, the motor hand illusion, the visuo-tactile interference, etc. [5], [6]. Although those protocols investigate similar underlying process, they present very different setups and most of them requires the experimenter to manually provide stimulations. Only recently, engineers contributed to develop more accurate and automated setups [7] to improve accuracy and repeatability of the experimental paradigms.

We wondered: is it possible to have only one setup useful in any kind of multisensory integration studies? What do they have in common? The key factor seems to be the spatial-temporal coherence between the stimuli, which strongly affects the efficacy of multisensory integration (let’s think of the rubber hand illusion which does not occur if the visual and tactile stimuli are asynchronous) [8].

The key idea is to design a multi-purpose platform, exploitable in a wide range of cross-modal integration studies. We validated the platform on two healthy participants, implementing a visuo-tactile integration task to investigate the peripersonal space (PPS).

PPS can be defined as the human body field of action [9]; the human brain can differentiate between the space close to the body (PPS) and the far one, depending on the potential interaction with objects, i.e. reaching or grasping [10], [11], [12]. For example, the visual receptive field of neurons located in the ventral intraparietal cortex is linked to the arm’s tactile receptive field [13]. Those neurons code the peripersonal space integrating tactile and visual (or auditory) stimuli that occur close to the arm [14], [13], [15].

Thus, the reaction time -in response to a tactile stimulus on the hand-decreases the more a simultaneous visual (or auditory) stimulus is presented near the hand [16], [17], [18].

We aim to develop a single platform able to collect and quantify all those factors; a virtual reality platform that provides synchronous stimuli, acquires behaviour reactions and measures critical features (such as reaction times and the hand and stimulus positions) both in static and dynamic tasks.

The virtual reality allows to easily test and manipulate several conditions and features, thus granting the opportunity to delve deeper into brain plasticity and embodiment [3], [4]. The use of virtual reality is legitimated by the occurrence of cross-modal effects even using an object that mimics the appearance of a real hand (e.g. a rubber hand) [4].

The platform also includes a motion tracking system, which allows the real-time tracking of the human arm movement and the active control of virtual avatars and objects, possibly improving the embodiment and the agency.

The paper is organized as follow: section II-A describes the platform design and embedded features; the protocol implemented for the validation is presented in section II-B, whereas in section II-C and II-D the experimental setup and the data analysis are described. Results are shown in section III and discussed in section IV, where future perspectives are discussed as well.

II. Materials and Methods

A. Platform Design and Features

A custom software was developed to manage all the platform elements and their synchronization. The application includes a user-friendly interface to set the stimulus parameters: type, amplitude, position, duration, etc.

To make the virtual environment as immersive as possible, the avatar is presented in first-person perspective and animated by the participant’s movement (see Fig. 1). If required, the platform allows to select also a back-view and a mirrored perspective.

Fig. 1.

TOP: Virtual environment with the avatar seated on a chair and the arms placed on the table. The left arm is still, whereas the right arm motion is controlled by the participant’s arm. The red led appears in the right hemispace. BOTTOM: Virtual environment from the participant’s point of view (first-person perspective).

The VR headset, used to best exploit the first person perspective, can even track the participant’s head movement and the participant’s gaze.

In the user interface the avatar height, as well as the gender, can be adjusted to match the limbs’ length and the point of view of the participant.

Four infrared cameras, fixed on a structure surrounding the experimental area, track the participant’s motion through reflective passive markers attached to 3D printed rigid bodies. The motion tracking has a twofold role: i) mapping the human motion on the virtual avatar, thus improving the VR vividness and the embodiment; ii) measuring in real-time the human arms position with respect to the stimuli, to the environment and to his own thorax and head.

The rigid bodies fixed on the thorax, arms, forearms and hands allow to track the upper limb and thorax motion. In the present case only three rigid-bodies were used to track the right arm. As depicted in Fig. 2a, they were placed on the human links, far from the muscles to avoid jerky movements due to muscle contractions.

Fig. 2.

a) Scheme of the motion mapping between the human arm link orientation, defined in the experimental base reference frame, and the virtual avatar arm defined with respect to the not left handed base frame in the virtual environment; b) Scheme of the visual stimuli workspace. The blue area is the region where the red led can appear and it is limited by a led-hand distance (LHd) r = r_min ÷ r_max in a direction defined by an angle α in the range –45 ÷ 225 deg with respect to the hand. The workspace is also limited to the right hemispace with respect to the participant/avatar’s thorax.

Since the implemented protocol (see section II-B) assumes a unique position for the participant in the experimental setup, a fixed reference system was employed. The orientation of each human link was computed with respect to the base reference frame of the experimental setup (ES) and then mapped into the not left-handed virtual reality coordinate system (VR), according to:

$${}^{VR}{}_q={\left[{}^{ES}{q}_z{,}^{ES}{q}_y{,}^{ES}{q}_x,{-}^{ES}{q}_w\right]}^T$$ (1)

where q is a unit quaternion represented by three vectorial (x, y and z) and a scalar (w) component as q = [q_x,q_y,q_z,q_w]^T. This mapping requires an initial alignment between the experimental reference frame and the experimental setup, i.e. the table and the chair in this case (see section II-C).

The developed software allows to skip this step, taking as reference frame ES0 = arm0 = for0 = hand0 a starting known pose, i.e. the T pose (see Fig. 2a). The orientation of each link is computed according to the following equations:

$$\{\begin{array}{l}\begin{array}{l}{}^{arm0}{q}_{arm}{=}^b{q}_{arm0}^{-1}{\ast }^b{q}_{arm};\hfill \end{array}\hfill \\ {}^{arm}{q}_{for}{=}^{arm0}{q}_{\text{arm}}^{-1}\ast \left({}^b{q}_{for0}^{-1}{\ast }^b{q}_{\text{for}}\right);\hfill \\ {}^{for}{q}_{hand}{=}^{arm}{q}_{for}^{-1}{\ast }^{arm0}{q}_{arm}^{-1}\ast \left({}^b{q}_{hand0}^{-1}{\ast }^b{q}_{hand}\right);\hfill \end{array}$$ (2)

where arm, for and hand refer to the arm, forearm and hand links respectively; b is the unknown base frame; ^bq_link0 is the orientation of the link acquired at time 0, with the participant in the T pose (see Fig. 2a); q^–1 is the inverse quaternion. Once the orientation of each link with respect to the previous one and to the initial pose is computed, as in Eq. 2, each quaternion is converted according to Eq. 1.

The participant’s response was recorded using a keypad. The tactile stimulation was provided through an electric stimulator, controlled through serial communication by the main software developed to manage also the virtual environment and the motion tracking.

The visual stimulus was presented as a red light with a semi-sphere shape, similar to a led (see Fig. 1), that appears on the virtual table surface and lasts for 100 milliseconds. The led position was randomly selected according to the following criteria:

• the stimulus is provided only in the right hemispace (with respect to the participant/avatar’s thorax)

• the stimulus distance from the hand is randomly selected within an adjustable range

• the stimulus never appears on the hand nor the arm (i.e. the light direction is included in the –45 ÷ 225 deg range, considering the hand as the center of a reference system as shown in Fig. 2b).

Since the horizontal plane is represented by the xz plane in the VR environment (see Fig. 2a), let us consider the virtual table plane as the xz plane of a reference system. Thus, the above criteria led to the following equations:

$$x_{led}=x_{hand}+r\cos \alpha$$ (3)

$$z_{led}=z_{hand}+r\sin \alpha$$ (4)

with α = –45 ÷ 225 deg and r = d_min ÷ d_max, being d_min and d_max the minimum and maximum led-hand distances computed as the cartesian norm (see Fig. 2b).

B. Experimental Protocol

We asked the participant to react as fast as possible to a tactile stimulus provided on his right index, regardless to eventual visual stimuli.

The protocol starts with a familiarization phase where the participant, in first-person perspective, moves his arm to control the virtual one in a simple reaching task. This goal-oriented task could improve the agency and the embodiment of the virtual arm, in addition to let the participant become familiar with the VR environment. A cross in the centre of the virtual table helps the participant to fix his gaze on the same point during the whole experiment.

The familiarization is followed by four sessions of fifty trials each. A single trial consists in the presentation of a stimulus condition, that can be:

• V: visual stimulus only

• T: tactile stimulus only

• VT: visual and tactile simultaneous stimuli

The first two are control conditions to ensure that the participant is reacting to the tactile stimuli only and not to whatever stimulus he perceives.

In this pilot test each session was composed of eight T conditions, eight V conditions and thirty-four VT conditions. The protocol was tested two times: the first one with the right hand always in the same position (hereafter called “single pose” condition); the second one asking the participant to change his hand position, always in the right hemispace, ten times during each session (“multiple poses” condition). In the “multiple poses” condition, the stimulation was delivered when the hand was still, i.e. in static states.

The “multiple poses” condition was included to deeper investigate if and how the position of the hand affects the cross-modal congruency effect, being the hand the supposed center of the peripersonal space.

C. Experimental Setup

The experimental setup was composed of a table and a chair (both duplicated in the virtual environment) placed into a structure with the four Prime Optitrack cameras fixed on. The participant was seated on the chair, wearing the HTC Vive headset, with the hands and forearms placed on the table in a comfort position (see Fig. 3), and the left index placed in correspondence to the keypad.

Fig. 3.

Experimental setup. The participant is seated on a chair, wearing the VR headset. Three rigid-bodies 3D printed are placed on his right arm, forearm and hand. Each rigid-body is equipped with four reflective passive markers to be tracked by the infrared cameras placed on the structure surrounding the table. The tactile stimulus is provided by the stimulator through two electrodes placed on the participant’s right index. The response is acquired by means of a keypad, pressed with the left index.

The main application was developed in C# language using the Unity3D environment. A dedicated electronic board sent the input to the electric stimulator (Grass Astro-Med S88x stimulator), which provided the tactile stimulation through two electrodes placed on the right index. The electric stimulation intensity was set to the minimum one clearly perceivable by the participant.

The signal used for the motion tracking was acquired using the software Motive and the provided Unity plugin. The developed VR application was set with a hand-led distance in the range 8 ÷ 30cm (i.e. r_min ÷ r_max in Fig. 2b).

The following measures were computed from the recorded data:

• RT: reaction time computed as the time between the tactile stimulus and the participant’s response (keypad button pressed)

• LHd: led-hand distance computed as the cartesian norm in the xz plane (i.e. the horizontal plane in the VR environment) between the led and the hand.

• LAd: led-avatar distance computed as the cartesian norm in the xz plane (i.e. the horizontal plane in the VR environment) between the led and the thorax.

Two participants not aware of the aim of the study were enrolled to test the proposed protocol. All participants were healthy and claimed to have normal vision and proprioceptive sensations; they provided written informed consent in accordance with the declaration of Helsinki and to the Ethical Committee of the Universita Campus Bio-Medico di Roma.

D. Data Analysis

Data were analysed in Matlab 2017, removing the outliers meant as values outside the range m ± 2sd, with m representing the mean value and sd the standard deviation. The statistics software JASP was used to analyse how the reaction time is affected by the presence of visual stimuli and their position with respect to the participant’s hand or thorax (LHd, LAd).

The analysis took into account the VT trials, for both the “single pose” condition and “multiple poses” condition.

III. Results

Fig. 4 shows the results of the correlation analysis (Pearson’s coefficient) between the led distance from the hand or avatar and participants’ reaction times (RT). The two rows correspond to the two participants S1 and S2. The left block (first two columns) represents the “single pose” condition, whereas the right block (third and fourth columns) is related to the “multiple poses” condition. It is worth noting that the stimuli administration and the RT recording occurred in static conditions (with the hand still). In the first column of each block the correlation between the RT and the led-hand distance (LHd) is depicted, whereas the second column shows the correlation between the RT and the distance between the led and the avatar thorax (LAd). The analysis revealed a significant correlation (rho=0.203; p=0.013) between led-hand distance and reaction times in the “single pose” condition for participant one. The second participant shows the same trend, even if not significant.

Fig. 4.

Correlations between the distance of the LED (from hand or avatar) expressed in millimetres and reaction times (RT) expressed in milliseconds, for participant S1 and S2.

No other significant correlations were found, except for the LHd in the “multiple poses” condition of the second participant.

IV. Conclusions

We presented a virtual reality platform for the study of multisensory integration. The platform tracks the human motion and controls a virtual avatar accordingly in real-time. This let us disentangle and study the specific contribution of features that play a role in cross-modal integration studies: stimulus type, duration and distance from participants body; arm position, orientation and appearance; participant’s gaze; etc..

We validated the platform in a peripersonal space protocol, by testing two healthy participants in a reaction time task based on visuo-tactile integration.

We found a correlation between the participant’s hand distance from a visual stimulus and reaction times: the closer the visual stimulus is presented to the participant’s hand, the “faster” he responds to that stimulus. Even though the correlation coefficient is rather low (rho=0.203), the trend is coherent with expected results, based on existing literature [17]. The absence of a correlation between reaction times and led-avatar distance also confirms the previous finding on the arm-centred nature of the peripersonal space [13].

We should deeper investigate the “multiple pose” condition, enrolling more participants, to clarify how the hand movement affects the peripersonal space representation.

Summing up, the main advantage of the proposed platform is the possibility to use a single system for a great variety of experiments, real-time recording a huge set of quantitative measures, thus having the chance to deeply study the brain plasticity and the related features.

Future perspective consists in employing the platform in many additional studies, investigating how the cross-modal congruency affects the body schema representation, the embodiment of external objects, the space representation and behaviour reactions.