Skip to main content
Dialogues in Clinical Neuroscience logoLink to Dialogues in Clinical Neuroscience
. 2020 Jun;22(2):169–177. doi: 10.31887/DCNS.2020.22.2/lvalmaggia

Virtual reality as a clinical tool in mental health research and practice


La realidad virtual como una herramienta clínica en la investigación y práctica de la salud metal

Quand la santé mentale a recours à la Réalité Virtuelle en recherche et en pratique clinique

Imogen H Bell 1, Jennifer Nicholas 2, Mario Alvarez-Jimenez 3, Andrew Thompson 4, Lucia Valmaggia 5
PMCID: PMC7366939  PMID: 32699517

Abstract

Virtual reality (VR) is a potentially powerful technology for enhancing assessment in mental health. At any time or place, individuals can be transported into immersive and interactive virtual worlds that are fully controlled by the researcher or clinician. This capability is central to recent interest in how VR might be harnessed in both treatment and assessment of mental health conditions. The current review provides a summary of the advantages of using VR for assessment in mental health, focusing on increasing ecological validity of highly controlled environments, enhancing personalization and engagement, and capturing real-time, automated data in real-world contexts. Considerations for the implementation of VR in research and clinical settings are discussed, including current issues with cost and access, developing evidence base, technical challenges, and ethical implications. The opportunities and challenges of VR are important to understand as researchers and clinicians look to harness this technology to improve mental health outcomes.


Keywords: assessment, digital technology, mental disorders, mental health, psychiatry, virtual reality

Introduction


Developments in virtual reality (VR) have the potential to radically transform the landscape of assessment in mental health. Immersive VR involves wearing an enclosed head-mounted device (HMD) that displays three dimensional images on a screen so that the person is fully immersed in a virtual environment (eg, Figure 1 ).1 Images are continuously rendered relative to the position of the head and can capture movements of the body, allowing users to explore and interact with objects and avatars (digital agents) in the virtual space. These virtual environments are either programmed using specialist software to create computer-generated, photorealistic images, or filmed with specialized cameras to create 360-degree videos of real-world scenes that can be replayed within VR. Together, these capabilities make it possible for researchers and clinicians to observe and record individuals in highly controlled and near-natural environments, in real time.


Figure 1. Examples of virtual reality environments 
(King’s College London, IoPPN Virtual Reality Research Lab, Developer: Jerome Di Pietro).

Figure 1.

VR has been applied for the delivery of exposure-based treatments, whereby individuals can experience feared situations or contexts in a safe and controlled manner, without leaving the clinical setting. Indeed, VR exposure treatments have proven effective across a range of mental health conditions. A number of reviews have been written on the topic of VR-based treatments for psychiatric conditions more broadly.2-7 Freeman et al4 conducted a systematic review in 2017, finding 154 studies on VR treatments for a range of mental health disorders. Further reviews have been published on VR treatments for schizophrenia,5 anxiety disorders,2,8 and mental disorders more generally.3,6 Evidence for the efficacy of VR treatments is noted within these reviews; however, the methodological quality of studies is generally low and the implementation of VR treatments beyond research settings is yet to be examined. Exposure therapy is notably dominant within this literature, highlighting how the ability of VR to recreate real-world environments has been leveraged to target the mechanism underpinning exposure. However, in general, innovation has not moved far from this approach, with more novel clinical applications of VR yet to be explored. One such application of VR within treatment is its use as a clinical assessment tool, a topic of relatively limited discussion within the literature. Clinical assessment is an integral element of mental health treatment, from diagnosis to treatment planning and monitoring. As such, the aim of this review is to orient the reader to the clinical application of VR as an assessment tool within mental health research and practice. The specific capabilities that underlie the utility of VR for clinical assessment is provided, as well as examples from high-quality research studies, finishing with a discussion of current considerations and limitations in the field.


Benefits of virtual reality for assessment


From symptoms and cognition to functioning and capacity, the measurement of psychological phenomena is central to research and practice in mental health.9 Although we can have confidence in current assessment instruments to a certain degree, there are many threats to reliability and validity.10 Differences between the assessment context (eg, a lab or clinic) and the real world generate multiple sources of potential bias, threatening the accuracy of results. Real-world assessments are possible, but can be costly and time consuming, and access can be limited by location and mobility. VR may overcome many of these limitations through the ability to generate highly controlled, real-world experiences.11

Enhancing ecological validity


With technological advances over the past decade in particular, VR has become increasingly immersive. By immersing individuals in real-world situations through VR, it is possible to conduct assessments that more closely emulate what happens in daily life. This capability overcomes the issue of ecological validity, that is, the degree to which the findings of research studies generalize to real-world settings.11 Research has consistently shown that individuals respond to virtual environments as if they were experiencing them in real life.12-15 Virtual environments are known to produce physiological changes consistent with emotional responses to real-world scenarios13,16,17 and have the ability to elicit symptoms such as paranoia,18,19 cravings,20,21 anxiety,14,15,22 and fear.23 A meta-analysis of nine randomized controlled trials comparing VR exposure with in vivo exposure for phobias found equivalent effects for both interventions, suggesting similar processes occur.24 Experiences in VR have also been found to elicit predictable behaviors, with one study finding that people with higher levels of paranoia kept a greater interpersonal distance from avatars within a virtual environment, which was considered a reflection of perceived trust and social threat.25 These findings highlight one of the main advantages of VR within mental health: the ability to simulate experiences in everyday life.


Whereas validation studies are still lacking, research has demonstrated that VR-based assessments can perform comparatively to assessments conducted in the real world.12,26-28 Gorini et al12 examined emotional reactions to food in a virtual kitchen, a real kitchen, and in photographs, among individuals with eating disorders. They found that both real and virtual exposure to food cues elicited the same emotional reactions, which were greater than those elicited by the photographs. VR also enables access to situations and experiences previously difficult to attain in research, such as hard to reach or dangerous environments. For example, VR has been used in neuroimaging research to study brain activation in naturalistic scenarios, a method previously impossible from inside the scanner.29 Furthermore, since recent developments have resulted in VR becoming completely mobile, assessments do not need to occur within the clinic or lab, allowing people to access them remotely.30 The possibility to deliver automated assessments in people’s homes, independent from a clinician, is an exciting opportunity to increase efficiency, improve accessibility, and reduce cost. 


Control and manipulation of the virtual environment


VR offers the ability to control and manipulate features of the environment that can be used to test and assess relevant variables, such as eliciting paranoia in social situations18,31,32 or examining responses to cues within the environment.12,20,21 Experimental control is a cornerstone of psychological research, enabling direct comparison between conditions to determine causal relationships between variables. Strong methodological rigor can be achieved in VR through careful manipulation of variables across conditions in virtual environments.11 For example, Veling et al31 randomized participants with psychosis, siblings, and health controls to conditions with varying levels of social stress within a virtual environment. They found a dose-response relationship between social stress and paranoia, which was associated with psychosis liability, supporting the theory that social stress may account for the relationship between the environment and psychosis. Another example comes from Freeman et al33 who examined experiences of paranoia and perceptions of social rank within a virtual underground train. Participants were placed in two conditions: one where they were taller than others on the train and another where they were shorter. Findings showed that under the condition where they were shorter, participants had a more negative view of themselves relative to others and greater levels of paranoia. The differences in paranoia were fully mediated by social comparison, suggesting that negative views of the self, relative to others, may drive feelings of mistrust.


Another capability of VR is that individuals are able to interact with objects within the system, rather than simply observing or imagining different scenarios. This not only enhances ecological validity, as interaction is inherent to the real world, but also allows researchers to examine behaviors of individuals within the virtual environment and their impact by manipulating different contingencies.34 Previously, this type of research was only possible using actors or “confederates” who performed certain roles within situations, which was costly and limited in ecological validity. Furthermore, the controlled nature of the experience enables greater reproducibility relative to field studies where the environment is constantly changing. These capabilities have important implications for social psychology,35 but also for identifying differences characteristic of mental disorders.5 For example, previous studies have demonstrated that exposure of each participant to the same virtual scenario can allow controlled examination of the determinants of paranoid ideation.18,33,36,37 As such, the enhanced environmental control and interactivity of VR allows for standardization of otherwise dynamic variables, ensuring a consistent assessment experience, both over time and across individuals.


Personalization and tailoring


Because virtual environments are computer-generated or recorded, it is also possible to program tailored VR experiences that match individual needs, abilities, or preferences (eg, slowing down a sequence, using text or audio instructions, minimizing distraction). Rizzo et al38 developed a VR-based exposure therapy for posttraumatic stress disorder (PTSD) that enabled the therapist to customize various features of a combat scenario most relevant to the trauma experienced by soldiers. Since effective treatment of PTSD requires exposure to cues of highly idiosyncratic experiences, customization of the virtual environment is an important feature. Such customization also has important utility for assessment purposes, though very little research has examined this to date. For example, functional analysis involves the examination of how symptoms change in relation to different triggers and responses.39 VR could be used to examine changes in symptoms across different situations and in relation to certain cues, for example visual, auditory, and olfactory, allowing precise insight into the determinants of relevant clinical events. Gatti et al40 describe a VR protocol whereby clinicians were able to customize the virtual environment with personally relevant cues to alcohol craving and behaviors for the purposes of clinical formulation. The ability to perform psychiatric assessment via personalized tests and tasks meaningful to the individual and their situation is powerful but is currently understudied within VR.


Real-time, automated data capture


The benefits of using technology for automatic data capture has been well recognized.41 Mobile apps for tracking symptoms and other clinically relevant information over time42 has been a major area of progress in digital mental health, with findings suggesting that they are feasible and acceptable to individuals.43,44 Researchers are starting to consider how data collected from devices such as smartphone apps and VR might be clinically informative. Patterns in data collected from technology devices, such as movement, speech, and geolocation, have been associated with changes in symptoms45 and may even be used to predict relapse among people with psychosis.42

It is possible to capture data automatically from users during VR experiences. Eye-tracking software can be integrated with VR,30 capturing a source of data commonly used to identify markers of psychiatric disorders and cognition.46 It is also possible to measure behaviors within the virtual environment, such as decisions about how to navigate and interact with different objects and agents.25 Capturing the temporal relationship between variables in real time (ie, how thoughts, emotions, and behaviors unfold in relation to changes in the environment), makes it possible to test hypothesized processes and causal interactions. Furthermore, physiological measures commonly used as objective indicators of psychological states, such as heart rate and galvanic skin response, can be recorded and synced with virtual content or even eye gaze. Mühlberger et al47 developed a VR-delivered behavioral avoidance test (VR-BAT) to assess fear in specific phobia while automatically collecting heart rate, skin conductance, subjective rating of discomfort, and stimulus-approach distance. Results indicated that the physiological measures were good predictors of fear intensity. Others have begun to investigate integrating more sophisticated biosensors, such as wireless electroencephalography (EEG), with VR to assess psychologically relevant constructs such as emotional state, with the aim to feed the data back to clinicians for enhanced decision making48 or even within VR itself as biofeedback.49

Increasing engagement


Another benefit of delivering clinical evaluation via VR is the potential to enhance an individual’s engagement with the test or assessment. Traditional testing procedures undertaken in mental health can be lengthy, repetitive, and/or laborious, which may impact individual performance, especially for measures of attention or memory. Therefore, replacing traditional assessment with more engaging, meaningful, or enjoyable methods has substantial appeal. This is particularly important given that symptoms such as poor concentration and lack of motivation are common across mental health conditions.50 This application of VR may also be of benefit in young people experiencing mental ill-health, a population where digital technologies are common and their use for mental health is promising.51

VR may enhance engagement with clinical evaluations through the immersive, realistic, enjoyable environments enabled by the technology.52,53 The term “presence” has been used in VR research to refer to the subjective experience of being in a place or environment, even when one is physically situated in another.1 The feeling of presence is influenced by the degree of immersion, defined as the extent to which the system generates sensory stimulation consistent with sensorimotor experience (eg, images are updated as the head moves). Sense of presence has traditionally been linked to an individual’s level of engagement54 and motivation.55 As such, high rates of presence and immersion reported within investigations of clinical uses of VR point to the ability of the technology to enhance engagement with clinical evaluation.56-58 However, it is currently difficult to create VR environments with a high level of realism, which can create a strong aversion to the experience.59

VR could further increase engagement with clinical evaluation by adding elements of digital games to the assessments, a process known as “gamification.”60 Incorporating features of games, such as rewards and feedback, within VR may engage individuals more fully in the evaluation process, limiting distractions that may compete for attention. Pollak et al53 reported that young participants (aged 9 to 17) rated a VR attention-deficit/hyperactivity disorder (ADHD) assessment significantly more enjoyable than a computerized test of attention. The affordances inherent within VR technology have the potential to increase engagement with clinical evaluation for all individuals across presenting disorders, potentially improving the reliability and validity of the assessments.


Applications of VR for assessment in clinical research studies


Owing to the ecological validity, immersive capabilities, and ease of standardized data collection as discussed above, the field has begun to explore the use of VR for the assessment of clinically relevant outcomes. To date, these have fallen into three main areas: social functioning, cognition, and symptomatology. Social functioning can be assessed using automatic data capture such as eye gaze, proximity to VR avatars, and recording of responses to simulated social situations.25,56 Across studies, individuals with mental health conditions known to impact social functioning differed from control participants on VR-recorded measures. Cognition was among the first outcomes to be assessed with early-stage VR programs,61-63 focusing on memory and executive function, commonly assessed using maze navigation and attention tasks. For example, response to a VR-administered continuous performance task assessed attention and response inhibition in children and teens with ADHD compared with controls.64 Finally, VR environments have also been used to elicit and assess symptoms, such as paranoid ideation in the general population,19,65 individuals at risk for psychosis,32,37 those with a first-episode psychosis,57 and individuals with long-standing persecutory delusions.66 Other symptoms such as auditory hallucinations,67 disordered eating,68 addiction,69 and phobia47 have also been studied using VR.


Considerations and limitations


Whereas VR offers exciting opportunities to advance multiple areas of mental health, it is important to remain cautiously optimistic. Mirroring broader issues within the field of digital mental health, the following are major areas where caution is needed.


Cost and access


Although VR has been around for several decades, only recently has the technology advanced to the point of commercial readiness. A major milestone occurred in 2010 with the release of the Oculus Rift , a relatively affordable VR device directed at consumers. Prior to this, VR technology mainly existed behind the closed doors of software companies and specialized research labs.1 Over the past decade, we have seen rapid technological advancement and proliferation of the marketplace, with various companies offering a range of devices suited to different consumers and budgets. Critical to VRs viability, 2019 saw the release of the fully mobile HMD Oculus Quest .70 Mobile HMDs can be run without a cable connecting them to a computer, overcoming earlier problems with mobility and ease of use. It is now easier to imagine VR becoming commonplace in clinics, hospitals, and people’s homes because it is easier to set up and more convenient to use. However, the cost of these devices is still prohibitive for many, with the most recent version of the Oculus Quest costing around US $400. 


The biggest limiting factor to the implementation of VR into clinical practice at present is the lack of evidence-based VR programs that can be bought off the shelf and used by clinicians and researchers. A number of labs around the world are developing their own software packages and testing them, but they are not yet commercially available. The few commercially available products developed by software companies have not been tested to show whether they are safe and effective. Additionally, because the technology is advancing so rapidly, hardware becomes outdated quickly and proprietary issues limit the availability of VR applications across newer platforms. Consequently, we are yet to see VR have the same market penetration as smartphones, with only a small proportion of consumers currently owning these devices. Some clinicians have integrated VR into their practice,71 but again we are yet to see these treatments become widely available despite good evidence base for some approaches, particularly exposure therapy for anxiety disorders.2 In comparison, there has been a more steady increase in the use of VR within mental health research.72 The slow integration of VR into clinical care is due to many factors. These include a lack in infrastructure to support the technology within services, absence of training and standardized evidence-based VR packages, the learning curve and costs associated with adopting new technologies, and more broadly, fears that technology may hinder engagement or even replace mental health professionals’ roles.73 For these reasons, it is imperative that new VR applications, and in fact any digital technology, are designed with considerations for the systems and context in which they will be implemented.74,75

Developing evidence base


Traditionally, medical research is painstakingly slow at developing new evidence.76 Research is costly, time consuming, and in many ways, inefficient. This is particularly problematic in digital mental health, where research operates at a much slower pace compared with technology development. Therefore, research often lags behind in providing the evidence base necessary to justify the hype surrounding many technologies.74 As a result, the marketplace is largely dominated by mental health technologies that lack the evidence to substantiate their claims.77 VR is no exception, with a booming market of HMDs, games, and applications ready for consumer use.


In addition to the complexities in developing and validating new VR technologies, it is also difficult to maintain them. Updates to software and hardware require ongoing maintenance costs that are not afforded in the traditional funding model offered by research grants. Potential solutions are still unfolding; however, currently, it is rare for the technologies and applications developed by researchers to be widely available for consumers and clinicians.78 They are also often developed in isolation, with collaboration across research centers lacking. Better partnerships across institutions as well as with commercial companies may offer solutions to this problem, but until this time, clinicians and researchers may find themselves frustrated by the lack of readily available VR assessment tools.


Technical requirements and issues


Technology can be complex and unstable, with teams of experts required for effective development and maintenance. Anyone interested in developing new VR applications for research should be warned that considerable technical, design, and computational expertise is required.79 Enhancing ecological validity is a major offering of VR; however, the degree of immersion necessary to generate the sense of presence is a matter of ongoing research. Features that affect this sense of immersion include the display parameters of the VR system (eg, frame rate and resolution), design features (eg, realism of visual objects), and multisensory feedback.1 An interesting phenomenon related to this that is often observed in VR is termed the “uncanny” valley.59 This refers to the way in which humanoid characters in a virtual world can elicit a feeling of unease or revulsion if they look very human-like but are imperfect (eg, their breathing movements are not realistic), reducing the overall realism of the environment. Interestingly, avatars with more cartoon-like feature might overcome this problem as they are easily accepted and do not seem to elicit the same sense of unease as more real-like humanoids.80 Clearly, the design of reliable and immersive VR experiences is complex. This, combined with the computing power and hardware required to build and run VR experiences, is a challenge for those in the field. 


Ethical issues


A number of reviews and commentaries have been written about the ethical landscape of digital mental health.81-83 Common themes across this literature include privacy, confidentiality, transparency, security, and ownership of data. These issues are also relevant to VR, especially when the application involves the collection of personal information and is accessed via the Internet. VR brings about more specific ethical challenges, however. More conceptually, the blurring of realities may have undue consequences on how we relate to and understand the world, perhaps especially so for conditions such as psychosis where reality distortion is already a challenge. As discussed previously, research has demonstrated that experiences in VR can have the same impact as if they occurred in the real world, necessitating considerable caution when conducting VR experiments designed to manipulate behavior.84 Some have commented on the dangers of enabling continual access to alternate realities, potentially perpetuating escape from the discomfort of the real world.85 The side effects of VR—namely, eye strain, cybersickness, and reality distortion1 —are also important considerations, and limited research exists on their long-term effects. In considering these factors, Madary and Metzinger,84 and Rizzo and Schulheis,85 provide recommendations for the ethical use of VR within research, drawing on general principles underpinning respect for rights and protection from harm.


Conclusion


Sufficient evidence has accumulated to support the benefits of VR for a variety of assessment purposes in mental health. VR elicits similar psychological and physiological reactions to real-world environments, extending the reach of current assessments beyond the lab or clinic. Superior capabilities for experimental manipulation and controlled exposure could significantly advance the field of mental health by improving methodological rigor, as well as enabling more accurate and individualized assessment. Automatic data capture of behaviors and signals from VR experiences can reveal important insights that might improve our understanding of mental health conditions and inform more tailored treatments. Advances in hardware, software, and research evidence has progressed in recent years; however, more studies are clearly needed to establish the reliability and validity of VR-based assessments, and issues with access to these resources and ethics require attention, thought, and research as the field develops.


Acknowledgments

The authors declare there are no conflicts of interest

Contributor Information

Imogen H. Bell, Orygen, Melbourne, Australia; Centre for Youth Mental Health, The University of Melbourne, Australia.

Jennifer Nicholas, Orygen, Melbourne, Australia; Centre for Youth Mental Health, The University of Melbourne, Australia.

Mario Alvarez-Jimenez, Orygen, Melbourne, Australia; Centre for Youth Mental Health, The University of Melbourne, Australia.

Andrew Thompson, Orygen, Melbourne, Australia; Centre for Youth Mental Health, The University of Melbourne, Australia; University of Warwick – Division of Mental Health and Wellbeing, University of Warwick, UK.

Lucia Valmaggia, King’s College London, Institute of Psychiatry, Psychology & Neuroscience, London, UK; South London and Maudsley NHS Foundation Trust, London, UK.

REFERENCES

  • 1.Slater M, Sanchez-Vives MV. Enhancing our lives with immersive virtual reality. Front Robot AI. 2016;3:74. doi: 10.3389/frobt.2016.00074. [DOI] [Google Scholar]
  • 2.Carl E, Stein AT, Levihn-Coon A, et al Virtual reality exposure therapy for anxiety and related disorders: a meta-analysis of randomized controlled trials. J Anxiety Disord. 2019;61:27–36. doi: 10.1016/j.janxdis.2018.08.003. [DOI] [PubMed] [Google Scholar]
  • 3.Eichenberg C, Wolters C, Eichenberg E. IntechOpen; 2012. Virtual realities in the treatment of mental disorders: a review of the current state of research Virtual Reality in Psychological, Medical and Pedagogical Applications pp. 35–64. [Google Scholar]
  • 4.Freeman D, Reeve S, Robinson A, et al Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychol Med. 2017;47(14):2393–2400. doi: 10.1017/S003329171700040X. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Rus-Calafell M, Garety P, Sason E, Craig TJ, Valmaggia LR. Virtual reality in the assessment and treatment of psychosis: a systematic review of its utility, acceptability and effectiveness. Psychol Med. 2018;48(3):362–391. doi: 10.1017/S0033291717001945. [DOI] [PubMed] [Google Scholar]
  • 6.Valmaggia LR, Latif L, Kempton MJ, Rus-Calafell M. Virtual reality in the psychological treatment for mental health problems: a systematic review of recent evidence. Psychiatry Res. 2016;236:189–195. doi: 10.1016/j.psychres.2016.01.015. [DOI] [PubMed] [Google Scholar]
  • 7.Veling W, Moritz S, van der Gaag M. Brave new worlds—review and update on virtual reality assessment and treatment in psychosis. Schizophr Bull. 2014;40(6):1194–1197. doi: 10.1093/schbul/sbu125. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Maples-Keller JL, Bunnell BE, Kim SJ, Rothbaum BO. The use of virtual reality technology in the treatment of anxiety and other psychiatric disorders. Harv Rev Psychiatry. 2017;25(3):103–113. doi: 10.1097/HRP.0000000000000138. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Baer L, Blais MA. New York, NY: Humana Press; 2010. Handbook of Clinical Rating Scales and Assessment in Psychiatry and Mental Health [Google Scholar]
  • 10.Myers K, Winters NC. Ten-year review of rating scales. I: overview of scale functioning, psychometric properties, and selection. J Am Acad Child Adolesc Psychiatry. 2002;41(2):114–122. doi: 10.1097/00004583-200202000-00004. [DOI] [PubMed] [Google Scholar]
  • 11.Parsons TD. Virtual reality for enhanced ecological validity and experimental control in the clinical, affective and social neurosciences. Front Hum Neurosci. 2015;9:660. doi: 10.3389/fnhum.2015.00660. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Gorini A, Griez E, Petrova A, Riva G. Assessment of the emotional responses produced by exposure to real food, virtual food and photographs of food in patients affected by eating disorders. Ann Gen Psychiatry. 2010;9:30. doi: 10.1186/1744-859X-9-30. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Meehan M, Insko B, Whitton M, Brooks FP Jr. Physiological measures of presence in stressful virtual environments. ACM Trans Graph. 2002;21(3):645–652. doi: 10.1145/566570.566630. [DOI] [Google Scholar]
  • 14.Riva G, Mantovani F, Capideville CS, et al Affective interactions using virtual reality: the link between presence and emotions. Cyber Psychol Behav. 2007;10(1):45–56. doi: 10.1089/cpb.2006.9993. [DOI] [PubMed] [Google Scholar]
  • 15.Slater M, Pertaub DP, Steed A. Public speaking in virtual reality: facing an audience of avatars. IEEE Comput Graph Appl. 1999;19(2):6–9. [Google Scholar]
  • 16.Costanzo ME, Leaman S, Jovanovic T, et al Psychophysiological response to virtual reality and subthreshold posttraumatic stress disorder symptoms in recently deployed military. Psychosom Med. 2014;76(9):670–677. doi: 10.1097/PSY.0000000000000109. [DOI] [PubMed] [Google Scholar]
  • 17.Diemer J, Alpers GW, Peperkorn HM, Shiban Y, Mühlberger A. The impact of perception and presence on emotional reactions: a review of research in virtual reality. Front Psychol. 2015;6:26. doi: 10.3389/fpsyg.2015.00026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Freeman D, Slater M, Bebbington PE, et al Can virtual reality be used to investigate persecutory ideation? J Nerv Ment Dis. 2003;191(8):509–514. doi: 10.1097/01.nmd.0000082212.83842.fe. [DOI] [PubMed] [Google Scholar]
  • 19.Riches S, Garety P, Rus-Calafell M, et al Using virtual reality to assess associations between paranoid ideation and components of social performance: a pilot validation study. Cyberpsychol Behav Soc Netw. 2019;22(1):51–59. doi: 10.1089/cyber.2017.0656. [DOI] [PubMed] [Google Scholar]
  • 20.Lee JH, Ku J, Kim K, et al Experimental application of virtual reality for nicotine craving through cue exposure. Cyberpsychol Behav. 2003;6(3):275–280. doi: 10.1089/109493103322011560. [DOI] [PubMed] [Google Scholar]
  • 21.Kuntze MF, Stoermer R, Mager R, Roessler A, Mueller-Spahn F, Bullinger AH. Immersive virtual environments in cue exposure. Cyberpsychol Behav. 2001;4(4):497–501. doi: 10.1089/109493101750527051. [DOI] [PubMed] [Google Scholar]
  • 22.Owens ME, Beidel DC. Can virtual reality effectively elicit distress associated with social anxiety disorder? J Psychopathol Behav Assess. 2015;37(2):296–305. [Google Scholar]
  • 23.Diemer J, Lohkamp N, Mühlberger A, Zwanzger P. Fear and physiological arousal during a virtual height challenge—effects in patients with acrophobia and healthy controls. J Anxiety Disord. 2016;37:30–39. doi: 10.1016/j.janxdis.2015.10.007. [DOI] [PubMed] [Google Scholar]
  • 24.Wechsler TF, Mühlberger A, Kümpers F. Inferiority or even superiority of virtual reality exposure therapy in phobias? A systematic review and quantitative meta-analysis on randomized controlled trials specifically comparing the efficacy of virtual reality exposure to gold standard in vivo exposure in agoraphobia, specific phobia and social phobia. Front Psycol. 2019;10:1758. doi: 10.3389/fpsyg.2019.01758. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Fornells-Ambrojo M, Elenbaas M, Barker C, et al Hypersensitivity to contingent behavior in paranoia: a new virtual reality paradigm. J Nerv Ment Dis. 2016;204(2):148–152. doi: 10.1097/NMD.0000000000000414. [DOI] [PubMed] [Google Scholar]
  • 26.Koenig S, Crucian G, Dalrymple-Alford J, Dünser A. Assessing navigation in real and virtual environments: a validation study. Int J Disabil Hum Dev. 2011;10(4):325–330. [Google Scholar]
  • 27.Renison B, Ponsford J, Testa R, Richardson B, Brownfield K. The ecological and construct validity of a newly developed measure of executive function: the virtual library task. J Int Neuropsychol Soc. 2012;18(3):440–450. doi: 10.1017/S1355617711001883. [DOI] [PubMed] [Google Scholar]
  • 28.Rose FD, Attree EA, Brooks BM, Parslow DM, Penn P. Training in virtual environments: transfer to real world tasks and equivalence to real task training. Ergonomics. 2000;43(4):494–511. doi: 10.1080/001401300184378. [DOI] [PubMed] [Google Scholar]
  • 29.Reggente N, Essoe JK, Aghajan ZM, et al Enhancing the ecological validity of fMRI memory research using virtual reality. Front Neurosci. 2018;12:408. doi: 10.3389/fnins.2018.00408. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Juvrud J, Gredebäck G, Åhs F, et al The immersive virtual reality lab: possibilities for remote experimental manipulations of autonomic activity on a large scale. Front Neurosci. 2018;12:305. doi: 10.3389/fnins.2018.00305. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Veling W, Pot-Kolder R, Counotte J, van Os J, van der Gaag M. Environmental social stress, paranoia and psychosis liability: a virtual reality study. Schizophr Bull. 2016;42(6):1363–1371. doi: 10.1093/schbul/sbw031. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Valmaggia LR, Freeman D, Green C, et al Virtual reality and paranoid ideations in people with an ‘at-risk mental state’ for psychosis. BJPsych. 2007;191(S51):s63–s68. doi: 10.1192/bjp.191.51.s63. [DOI] [PubMed] [Google Scholar]
  • 33.Freeman D, Evans N, Lister R, Antley A, Dunn G, Slater M. Height, social comparison, and paranoia: an immersive virtual reality experimental study. Psychiatry Res. 2014;218(3):348–352. doi: 10.1016/j.psychres.2013.12.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Wieser MJ, Pauli P, Grosseibl M, Molzow I, Mühlberger A. Virtual social interactions in social anxiety—the impact of sex, gaze, and interpersonal distance. Cyberpsychol Behav Soc Netw. 2010;13(5):547–554. doi: 10.1089/cyber.2009.0432. [DOI] [PubMed] [Google Scholar]
  • 35.Pan X, Hamilton AFC. Why and how to use virtual reality to study human social interaction: the challenges of exploring a new research landscape. Br J Psych. 2018;109(3):395–417. doi: 10.1111/bjop.12290. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Freeman D, Garety PA, Bebbington P, et al The psychology of persecutory ideation II: a virtual reality experimental study. J Nerv Ment Dis. 2005;193(5):309–315. doi: 10.1097/01.nmd.0000161686.53245.70. [DOI] [PubMed] [Google Scholar]
  • 37.Valmaggia LR, Day F, Rus-Calafell M. Using virtual reality to investigate psychological processes and mechanisms associated with the onset and maintenance of psychosis: a systematic review. Soc Psychiatry Psychiatr Epidemiol. 2016;51(7):921–936. doi: 10.1007/s00127-016-1245-0. [DOI] [PubMed] [Google Scholar]
  • 38.Rizzo AS, Difede J, Rothbaum BO, et al Development and early evaluation of the Virtual Iraq/Afghanistan exposure therapy system for combat-related PTSD. Ann N Y Acad Sci. 2010;1208(1):114–125. doi: 10.1111/j.1749-6632.2010.05755.x. [DOI] [PubMed] [Google Scholar]
  • 39.Sturmey P. Elsevier; 2011. Functional Analysis in Clinical Treatment [Google Scholar]
  • 40.Gatti E, Massari R, Sacchelli C, Riva G. Virtual reality protocol: an instrument to assess alcohol-dependent individuals. Annu Rev Cyberther Telemed. 2007:55–63. [Google Scholar]
  • 41.Torous J, Kiang MV, Lorme J, Onnela JP. New tools for new research in psychiatry: a scalable and customizable platform to empower data driven smartphone research. JMIR Ment Health. 2016;3(2):e16. doi: 10.2196/mental.5165. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Ben-Zeev D, Brian R, Wang R, et al CrossCheck: integrating self-report, behavioral sensing, and smartphone use to identify digital indicators of psychotic relapse. Psychiatr Rehabil J. 2017;40(3):266–275. doi: 10.1037/prj0000243. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Wang K, Varma DS, Prosperi M. A systematic review of the effectiveness of mobile apps for monitoring and management of mental health symptoms or disorders. J Psychiatr Res. 2018;107:73–78. doi: 10.1016/j.jpsychires.2018.10.006. [DOI] [PubMed] [Google Scholar]
  • 44.Bell IH, Lim MH, Rossell SL, Thomas N. Ecological momentary assessment and intervention in the treatment of psychotic disorders: a systematic review. Psychiatr Serv. 2017;68(11):1172–1181. doi: 10.1176/appi.ps.201600523. [DOI] [PubMed] [Google Scholar]
  • 45.Torous J, Staples P, Barnett I, Sandoval LR, Keshavan M, Onnela JP. Characterizing the clinical relevance of digital phenotyping data quality with applications to a cohort with schizophrenia. NPJ Digit Med. 2018;1:15. doi: 10.1038/s41746-018-0022-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Trillenberg P, Lencer R, Heide W. Eye movements and psychiatric disease. Curr Opin Neurol. 2004;17(1):43–47. doi: 10.1097/00019052-200402000-00008. [DOI] [PubMed] [Google Scholar]
  • 47.Mühlberger A, Sperber M, Wieser MJ, Pauli P. A virtual reality behavior avoidance test (VR-BAT) for the assessment of spider phobia. J Cyber Ther Rehabil. 2008;1(2):147–158. [Google Scholar]
  • 48.Gaggioli A, Cipresso P, Serino S, et al A decision support system for real-time stress detection during virtual reality exposure. Stud Health Technol Inform. 2014;196:114–120. [PubMed] [Google Scholar]
  • 49.Pallavicini F, Algeri D, Repetto C, Gorini A, Riva G. Biofeedback virtual reality and mobile phones in the treatment of generalized anxiety disorder (GAD): a phase-2 controlled clinical trial. J Cyber Ther Rehabil. 2009;2:315–327. [Google Scholar]
  • 50.Fioravanti M, Bianchi V, Cinti ME. Cognitive deficits in schizophrenia: an updated metanalysis of the scientific evidence. BMC Psychiatry. 2012;12(1):64. doi: 10.1186/1471-244X-12-64. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Bell IH, Alvarez-Jimenez M. Digital technology to enhance clinical care of early psychosis. Curr Treat Options Psychiatry. 2019;6(3):256–270. [Google Scholar]
  • 52.Hodge J, Balaam M, Hastings S, Morrissey K. Montreal, QC, Canada: Association for Computing Machinery’s Special Interest Group on Computer Human Interaction (SIGCHI); 2018. Exploring the design of tailored virtual reality experiences for people with dementia Proceedings of CHI Conference on Human Factors in Computing Systems Paper 514. [DOI] [Google Scholar]
  • 53.Pollak Y, Weiss PL, Rizzo AA, et al The utility of a continuous performance test embedded in virtual reality in measuring ADHD-related deficits. J Dev Behav Pediatr. 2009;30(1):2–6. doi: 10.1097/DBP.0b013e3181969b22. [DOI] [PubMed] [Google Scholar]
  • 54.Lombard M, Ditton T. At the heart of it all: the concept of presence. J Comput Mediat Commun. 1997;3(2):JCMC321. doi: 10.1111/j.1083-6101.1997.tb00072.x. [DOI] [Google Scholar]
  • 55.Retaux X. Presence in the environment: theories, methodologies and applications to video games. PsychNology J. 2003;1(3):283–309. [Google Scholar]
  • 56.Choi SH, Ku J, Han K, et al Deficits in eye gaze during negative social interactions in patients with schizophrenia. J Nerv Ment Dis. 2010;198(11):829–835. doi: 10.1097/NMD.0b013e3181f97c0d. [DOI] [PubMed] [Google Scholar]
  • 57.Veling W, Brinkman WP, Dorrestijn E, Van Der Gaag M. Virtual reality experiments linking social environment and psychosis: a pilot study. Cyberpsychol Behav Soc Netw. 2014;17(3):191–195. doi: 10.1089/cyber.2012.0497. [DOI] [PubMed] [Google Scholar]
  • 58.Slater M, Lotto B, Arnold MM, Sánchez-Vives MV. How we experience immersive virtual environments: the concept of presence and its measurement. Anuario de Psicología. 2009;40:193–210. [Google Scholar]
  • 59.Mori M, MacDorman KF, Kageki N. The uncanny valley [from the field] IEEE Robot Autom Mag. 2012;19(2):98–100. [Google Scholar]
  • 60.Deterding S, Dixon D, Khaled R, Nacke L. Tampere, Finland: Association for Computing Machinery; 2011. From game design elements to gamefulness: defining “gamification.” Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments [DOI] [Google Scholar]
  • 61.Sorkin A, Weinshall D, Modai I, Peled A. Improving the accuracy of the diagnosis of schizophrenia by means of virtual reality. Am J Psychiatry. 2006;163(3):512–520. doi: 10.1176/appi.ajp.163.3.512. [DOI] [PubMed] [Google Scholar]
  • 62.Ku J, Cho W, Kim JJ, et al A virtual environment for investigating schizophrenic patients’ characteristics: assessment of cognitive and navigation ability. Cyberpsychol Behav. 2003;6(4):397–404. doi: 10.1089/109493103322278781. [DOI] [PubMed] [Google Scholar]
  • 63.Spieker EA, Astur RS, West JT, Griego JA, Rowland LM. Spatial memory deficits in a virtual reality eight-arm radial maze in schizophrenia. Schizophr Res. 2012;135(1-3):84–89. doi: 10.1016/j.schres.2011.11.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 64.Neguț A, Jurma AM, David D. Virtual-reality-based attention assessment of ADHD: ClinicaVR: Classroom-CPT versus a traditional continuous performance test. Child Neuropsychol. 2017;23(6):692–712. doi: 10.1080/09297049.2016.1186617. [DOI] [PubMed] [Google Scholar]
  • 65.Freeman D. Studying and treating schizophrenia using virtual reality: a new paradigm. Schizophr Bull. 2008;34(4):605–610. doi: 10.1093/schbul/sbn020. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 66.Freeman D, Pugh K, Vorontsova N, Antley A, Slater M. Testing the continuum of delusional beliefs: an experimental study using virtual reality. J Abnorm Psychol. 2010;119(1):83–92. doi: 10.1037/a0017514. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 67.Stinson K, Valmaggia L, Antley A, Slater M, Freeman D. Cognitive triggers of auditory hallucinations: an experimental investigation. J Behav Ther Exp Psychiatry. 2010;41(3):179–184. doi: 10.1016/j.jbtep.2009.12.003. [DOI] [PubMed] [Google Scholar]
  • 68.Ferrer-Garcia M, Gutierrez-Maldonado J, Treasure J, Vilalta-Abella F. Craving for food in virtual reality scenarios in non-clinical sample: analysis of its relationship with body mass index and eating disorder symptoms. Eur Eat Disord Rev. 2015;23(5):371–378. doi: 10.1002/erv.2375. [DOI] [PubMed] [Google Scholar]
  • 69.Detez L, Greenwood LM, Segrave R, et al A psychophysiological and behavioural study of slot machine near-misses using immersive virtual reality. J Gambl Stud. 2019;35(3):929–944. doi: 10.1007/s10899-018-09822-z. [DOI] [PubMed] [Google Scholar]
  • 70. Oculus; Oculus Quest 2019 Available at: [Google Scholar]
  • 71.Lindner P, Miloff A, Zetterlund E, Reuterskiöld L, Andersson G, Carlbring P. Attitudes toward and familiarity with virtual reality therapy among practicing cognitive behavior therapists: a cross-sectional survey study in the era of consumer VR platforms. Front Psychol. 2019;10:176. doi: 10.3389/fpsyg.2019.00176. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 72.Cipresso P, Chicchi Giglioli IA, Alcañiz Raya M, Riva G. The past, present, and future of virtual and augmented reality research: a network and cluster analysis of the literature. Front Psychol. 2018;9:2086. doi: 10.3389/fpsyg.2018.02086. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 73.Riva G. Virtual reality: an experiential tool for clinical psychology. Br J Guid Counc. 2009;37(3):337–345. [Google Scholar]
  • 74.Mohr DC, Lyon AR, Lattie EG, Reddy M, Schueller SM. Accelerating digital mental health research from early design and creation to successful implementation and sustainment. J Med Internet Res. 2017;19(5):e153. doi: 10.2196/jmir.7725. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 75.Mohr DC, Riper H, Schueller SM. A solution-focused research approach to achieve an implementable revolution in digital mental health. JAMA Psychiatry. 2018;75(2):113–114. doi: 10.1001/jamapsychiatry.2017.3838. [DOI] [PubMed] [Google Scholar]
  • 76.Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med. 2011;104(12):510–520. doi: 10.1258/jrsm.2011.110180. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 77.Larsen ME, Huckvale K, Nicholas J, et al Using science to sell apps: evaluation of mental health app store quality claims. NPJ Digit Med. 2019;2:18. doi: 10.1038/s41746-019-0093-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 78.Firth J, Torous J, Nicholas J, et al The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials. World Psychiatry. 2017;16(3):287–298. doi: 10.1002/wps.20472. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 79.Lindner P, Miloff A, Hamilton W, et al Creating state of the art, next-generation Virtual Reality exposure therapies for anxiety disorders using consumer hardware platforms: design considerations and future directions. Cogn Behav Ther. 2017;46(5):404–420. doi: 10.1080/16506073.2017.1280843. [DOI] [PubMed] [Google Scholar]
  • 80.de Borst AW, de Gelder B. Is it the real deal? Perception of virtual characters versus humans: an affective cognitive neuroscience perspective. Front Psychol. 2015;6:576. doi: 10.3389/fpsyg.2015.00576. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 81.D’Alfonso S, Phillips J, Valentine L, Gleeson J, Alvarez-Jimenez M. Moderated online social therapy: viewpoint on the ethics and design principles of a web-based therapy system. JMIR Ment Health. 2019;6(12):e14866. doi: 10.2196/14866. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 82.Torous J, Roberts LW. The ethical use of mobile health technology in clinical psychiatry. J Nerv Ment Dis. 2017;205(1):4–8. doi: 10.1097/NMD.0000000000000596. [DOI] [PubMed] [Google Scholar]
  • 83.Wykes T, Lipshitz J, Schueller SM. Towards the design of ethical standards related to digital mental health and all its applications. Curr Treat Options Psychiatry. 2019;6(3):232–242. [Google Scholar]
  • 84.Madary M, Metzinger TK. Real virtuality: a code of ethical conduct. Recommendations for good scientific practice and the consumers of VR-technology. Front Robot AI. 2016;3:3. doi: 10.3389/frobt.2016.00003. [DOI] [Google Scholar]
  • 85.Rizzo A, Schultheis MT, Bush SS, Drexler ML. Lisse, Netherlands: Swets & Zeitlinger; 2002. Ethical issues for the use of virtual reality in the psychological sciences Studies on Neuropsychology, Development, and Cognition, vol 4. Ethical Issues in Clinical Neuropsychology pp. 243–277. [Google Scholar]

Articles from Dialogues in Clinical Neuroscience are provided here courtesy of Taylor & Francis

RESOURCES