Understanding dual realities and more in VR

Abstract

The five commentaries on our paper (Pan & Hamilton, 2018, Br. J. Psychol.) provide a useful summary of the key issues we raise, and also bring to the foreground some new and important questions, touching on a range of issues including presence and the conceptualization of dual realities. We highlight here some important themes and directions for future investigation.

Background

The five commentaries on our paper (Pan & Hamilton, 2018) provide a useful summary of the key issues we raise, and also bring to the foreground some new and important questions, touching on a range of issues including presence and the conceptualization of dual realities. We highlight here some important themes and directions for future investigation.

First, it is worth revisiting the definition of what counts as virtual reality. Slater (2018) suggests that the ideal definition of virtual reality (VR) should include not only ‘a computer‐generated world’ but also a VR system which is perceptually surrounding, and where ‘perception is at least a function of head‐tracking’. This implies either a CAVE or head mounted display system. In our target article, we allowed for a broader definition, including systems which use augmented reality, mixed reality, and computer‐generated characters on a regular monitor. This inclusive approach allows us to consider a range of studies relevant to psychology and neuroimaging which might not meet a strict definition of VR. It will also be an interesting empirical question to test whether the same principles apply across different settings with different levels of immersion, from fully immersive VR to mixed and augmented reality.

Second, several commentaries raise the issue of the technical challenges involved in creating psychological experiments in virtual reality, in terms of realism (de la Rosa & Breidt, 2018; Kulik, 2018) and multimodal signal integration (Aglioti & Monti, 2018; de Gelder, Katsyri, & de Borst, 2018). The question of what level of realism is appropriate for a particular research project is one that must still be tested empirically (Kulik, 2018) and defining these parameters is in itself an interesting research question which can be subject to psychophysical approaches (de Gelder et al., 2018). In particular, it may be interesting to consider the role of both top‐down expectations (based on prior knowledge) and bottom‐up visual inputs in building our experience of a VR scenario. Building on predictive models of perception (Clark, 2013), it might be possible to exploit top‐down predictions so the VR system has less work to do (de Gelder et al., 2018). However, if a VR interface provides a misleading input (for instance, wooden facial expressions or odd movements), then the visual input will conflict with prior knowledge about how the face ought to move, and lead to a prediction error which could be linked to the feeling of uncanniness (Saygin, Chaminade, Ishiguro, Driver, & Frith, 2012). This aligns with the idea that ‘one cannot not communicate’ (de la Rosa & Breidt, 2018).

A common goal in creating a VR scenario is to create a strong sense of presence in participants. Slater (2009, 2018) important work on defining and understanding the place illusion and plausibility illusions underlies much of our current framework for measuring and discussing presence. In his commentary, Slater highlights our use of the term ‘belief’ in relation to presence and emphasizes that the effects of VR are not reliant on the participant's belief in the system, but rather act as a perceptual illusion, where participants feel their heart racing on a virtual clifftop despite knowing they are in the laboratory. Here, we suggest there may be an important distinction between how VR creates the illusion of a physical location and how VR can create social experiences. Visual illusions such as the Muller‐Lyer illusion are typically driven by an external stimulus, and top‐down knowledge of the illusion does little to change its power. Similarly, there is no top‐down instruction which can make a person standing with their eyes open in an ordinary laboratory feel as if they are on a clifftop, even with a compelling narrative and a good imagination. In contrast, the feeling of interacting with another human being can be independent of visual/auditory inputs and strongly modulated by top‐down instructions. For example, there are many behavioural and neuroimaging studies where participants play simple economic games or turn‐taking games via a computer interface with another person and are instructed that they are playing with another person or playing with a computer in different blocks. Even when the interface remains unchanged, this simple instruction manipulation can lead to substantial changes in both behaviour and brain activation patterns (Gallagher, Jack, Roepstorff, & Frith, 2002).

These two examples contrast extreme effects of visual input (on the clifftop) and top‐down instructions (on the computer game), but the same principles can be seen in other scenarios. In a recent study, participants saw a virtual character (VC) and could interact with it via eye movements (Caruana, McArthur, Woolgar, & Brock, 2017). Participants who were instructed that the character's behaviour was driven by a person in an adjacent room showed different patterns of gaze and reported a different subjective experience, compared to those who were instructed that the character was driven by an algorithm. In the domain of social psychology, research on mind perception has shown that both the visual appearance of a stimulus and top‐down factors such as motivation and instruction can change whether people judge an agent as having a human‐like mind (Deska & Hugenberg, 2017). Our emphasis here is on the idea that the ‘belief that I am interacting with another person’, either in VR or over any other computerized interface, is not an illusion in the same category as visual illusions, but might be an inference based on context (e.g., experimenter instructions) and the realism of the current situation. This implies that, at least in social VR, it may be useful to think about more than place and plausibility illusions and to consider what other factors drive participants to consider the VC as a human. Tricks such as appropriate experimenter instruction or even embodiment of multiple VCs in turn (Aglioti & Monti, 2018) may be helpful. Overall, understanding what features of a VC and what contextual factors are critical for realistic mind perception is an important psychological question and one that may take us closer to the ‘Mons Olympus’ of a virtual human who passes a Turing test.

The final important issue raised in several commentaries is the comparison between the different ways of experiencing a particular scenario – in VR, or via other media (watching movies, theatre, and reading a novel) or as part of a research study in a traditional laboratory, or in real life. To what extent are these different modes of experience similar or different, and to what extent can we generalize between them? de Gelder et al. (2018) introduces the useful concept of dual reality, that is, the person experiencing a VR scenario, just like the audience in a cinema, experiences a range of stimuli which can induce rich characters and emotionally engaging stories which can suck their audience in to a different world (Kulik, 2018), even one of impossible situations (de la Rosa & Breidt, 2018). Despite this richness, the user knows that this experience is not ‘real’; it is a secondary reality which can be discarded at will. However, little is known about the cognitive processes which allow us to engage in dual realities and which make them enjoyable.

Developing this concept of dual realities is important in three ways. First, it provides a useful way to compare VR to other media and understand the value of VR for research as well as training and entertainment. VR provides substantially more interactivity and more ‘genuine’ characters than other media, even immersive theatre. In a theatre or role‐play context, participants know that the actors are just actors taking on a particular role. In contrast, if participants meet a character in some VR contexts, they may treat that character as a genuine person who is not playing a role but might assume this is another research participant in another room. This might allow measurement of more genuine social interactions. There is also an inverse contrast, where participants in VR scenarios report feeling ‘less judged’ by the VCs compared to a human. For example, in therapy or health care settings where participants have the opportunity to discuss personal issues, they report finding it easier to be open with a VC compared to a human therapist (DeVault et al., 2014). Thus, VR may provide more humanization than movies/theatre but less than live interactions, and this can be exploited for both research and therapy.

The second reason for considering VR in terms of dual realities relates to the practical question of making a good and valid VR experiment. In using VR for research in psychology, it is also important to know whether the way people respond in VR is the same as how they might respond in real life (Kulik, 2018), that is, does the VR world really engage the same cognitive processes as the real world? We suggest that further empirical studies will be the best way to test this. If there are differences in behaviour between VR and live action, it is also interesting to know if these are driven by technical limitation in the VR, which could be overcome with faster and better equipment, or if they are driven by the top‐down knowledge of the participant that this virtual world is not real. Considering this question of the ecological validity of VR further, we note that the question of how behaviour in an alternate reality generalizes to performance in the real world could also be taken to apply to many non‐VR psychology studies. As soon as a participant signs a consent form and enters the artificial world of the research laboratory, it is hard to know for sure if their behaviour truly matches their real‐world behaviour outside of this context. Thus, understanding how participant's experience of the VR world or the research laboratory as an alternate reality, apart from everyday life, is important in understanding the ecological validity of many of our research results.

The final reason why we believe the concept of dual realities is important is a more theoretical and philosophical one. That is, what do we know about the cognitive capacities which allow us to engage in dual reality activities, and can VR help us in studying this? Research suggests that there are common neural processes engaged when watching a movie, which vary according to how compelling the storyline is for both films (Hasson & Landesman, 2008) and literature (Nijhof & Willems, 2015). This implies that it may also be possible to quantify immersion in a VR scenario with neuroimaging methods. As mentioned above, we suggest that novels, movies, theatre, and VR are the most obvious and compelling forms of dual reality. However, pretend play in children (Lillard, 2017), future thinking (Schacter, Addis, & Buckner, 2007), or counterfactual thinking (Van Hoeck et al., 2013) might also draw on the same type of cognitive processes. This is because these all require the ability to consider both the present reality and a possible alternative reality and might involve metarepresentation (Leslie, 1987). Understanding if engagement with VR and other media draws on similar cognitive processes to pretend and future thinking may help draw together these different strands of research and allow the neuroscientific study of the human capacity for imagination.

To conclude, we thank the authors for their helpful commentaries and look forward to seeing how future research which creates, uses, and understands virtual reality will develop over the next decades. We believe that this will require strong collaboration between researchers in computing and in psychology, with benefits for both fields. Our review suggests that VR can be a valuable tool for psychologists to create compelling social interactions and that studying how we engage in VR may give new insights into both our social behaviour and our imaginations.