Unreal that feels real: artificial intelligence-enhanced augmented reality for treating social and occupational dysfunction in post-traumatic stress disorder and anxiety disorders

ABSTRACT

Background: Fear- and trauma-related conditions, such as post-traumatic stress disorder (PTSD) and social phobia, often manifest as socially avoidant behaviours, which commonly contribute to social and occupational disability transdiagnostically. While gold-standard treatments (i.e. exposure therapy, psychotropic medications) are effective, they are hindered by high dropout rates and limited impact on real-world functioning. Furthermore, most existing interventions only target symptom reduction, with few addressing avoidance-related deficits in social and occupational functioning.

Objectives: This methods paper introduces an innovative augmented reality exposure therapy (ARET) technology designed to address the limitations of traditional interventions for anxiety disorders and PTSD, by directly targeting social and occupational dysfunction through exposure to real-life social contexts.

Method: We introduce an ARET system, using artificial intelligence (AI)-driven, augmented reality (AR) technology, that enables exposure to realistic scenarios within the patient's real-world environment, fostering contextual generalization and functional improvement. Featuring holographic three-dimensional humans, precise surface mapping, wireless mobility, and telemedicine capabilities, the software provides customizable exposure scenarios to transform an environment into various spaces (e.g. grocery store, house party) with diverse human characters, as well as flexible AI-driven human interactions tailored to individual needs.

Results: We share observations and feedback from the treatment of first responders with PTSD. Patients found the technology easy to use, with immersive realism, active engagement, and strong emotional responses needed for effective exposure therapy. Advances in AI-driven character development and AR hardware accessibility support the wider adoption of ARET by clinicians.

Conclusion: By bridging the gap between clinical interventions and real-world functioning, ARET offers a transformative approach to addressing the pervasive impact of psychiatric disorders on social and occupational outcomes.

HIGHLIGHTS

• AI-driven augmented reality exposure therapy (ARET) offers a unique opportunity for in vivo exposure therapy for PTSD by incorporating realistic social and occupational scenarios into patients’ real-world environments.

• This innovative technology supports customizable exposure scenarios with diverse human characters, enabling more effective and immersive therapy that addresses social avoidance and improves real-life functioning.

• Preliminary observations show promising efficacy in reducing social and occupational impairments, with patient feedback indicating high engagement and improved confidence in managing real-world situations.

1. Introduction

Considering the substantial, transdiagnostic impact of numerous psychiatric conditions on social and occupational functioning, interventions designed to specifically target and support these abilities – as opposed to targeting symptoms or prior adversities – remain limited. Fear- and trauma-related disorders, for instance, commonly result in deficits in social and occupational functioning due to socially avoidant behaviours (Kelly et al., 2019; Solomon & Mikulincer, 2007). Such disorders are highly prevalent and disabling, affecting nearly one in three people during their lifetime, with the most prevalent conditions including phobias, social phobia, and post-traumatic stress disorder (PTSD). (Kessler et al., 2012). Indeed, 8% of Americans suffer from PTSD, whereas the prevalence is as high as 30% among special groups such as refugees, frontline health workers, veterans, and first responders (e.g. firefighters, police) (Abu Suhaiban et al., 2019; Jetelina et al., 2020; Marmar et al., 2006; Preti et al., 2020; Ruscio et al., 2008), and 90% of the urban population is exposed to trauma resulting in clinically significant symptoms (Rahman et al., 2018).

Despite the monumental need, current gold-standard treatments for trauma-related disorders, i.e. exposure therapy (ET) and psychiatric medications, remain limited in their approach. Although these approaches can be effective when treatment adherence is high, many patients do not complete trauma-focused ET protocols owing to discomfort (Najavits, 2015), prescribed medications often have unwanted side effects (Cochran et al., 2008; Reger et al., 2013), and these treatments primarily focus on symptom reduction, with less improvement in real-life social and occupational functioning. While in vivo exposure is an important element of exposure-based treatments for PTSD, owing to in-office limitations (mainly, an inability to replicate avoided situations), this form of exposure is commonly left to patients to complete on their own. Nonetheless, patients are often too anxious to venture into these avoided, unpredictable life situations, or to independently identify and create exposures to those routine life situations (Hundt et al., 2020). As a result, despite symptom improvement, some patients remain avoidant after conventional therapies are completed (Rodriguez et al., 1999). To prompt improvements in social and occupational functioning, additional treatment options that directly target overall functioning offer another avenue to recovery and well-being (Scoglio et al., 2022). Although existing interventions remain important strategies available to clinicians and patients, transdiagnostic interventions centred on improving social and occupational real-world functioning (i.e. functioning-based approaches), as opposed to targeting other symptoms, would have broad implications for a variety of fear- and trauma-related disorders.

Social avoidance in PTSD represents a key barrier to improved social functioning, and is a common outcome associated with fear- and trauma-related disorders (Kelly et al., 2019; Scoglio et al., 2022). Particularly in cases of interpersonal trauma, avoidance of social interaction, public spaces, and crowded environments is prevalent (Bjornsson et al., 2020; Dworkin, 2020). This avoidance leads to significant disability in conducting daily tasks, including running errands, attending family or community functions, and maintaining employment. Indeed, such avoidance often leads to confinement to the home or an inability to function unaccompanied (Lépine & Lellouch, 1995). While conventional ET is primarily focused on addressing specific trauma-related memories, the methodology discussed herein offers a novel approach to providing exposure to real-life situations that are typically avoided by affected individuals. In other words, rather than focusing on the trauma memories, this exposure focuses on the safe life situations that the patient avoids because of their traumatic experience.

Rather than provide an extensive review of mixed-reality technology, as such reviews are already available (Gonçalves et al., 2012), this methods paper aims to describe, provide rationale for, and generate ideas for future applications of artificial intelligence (AI)-motivated, wireless augmented reality (AR) technologies for ET to real-world scenarios. To our knowledge, this is the only technology of this type in existence. Exposure-based AR treatment models that target social functioning by resolving social avoidance have broad implications, not only for fear- and trauma-related disorders, but also for any condition in which social/occupational functioning is a major impairment or there is a need for learning new social skills.

The following sections will: (1) give a brief overview of the underlying neurological mechanisms of ET; (2) discuss, in greater depth, limitations to existing ET models; (3) introduce current treatment options that employ virtual reality (VR) and AR, as well as outline the strengths and weaknesses of each; (4) describe a novel AR technology and model for ET, including AI-assisted features, feasibility of use, and patient feedback; and (5) discuss implications and options for expanded implementation.

1.1. Mechanisms of ET

Extinction learning is the dominant laboratory model for ET. In brief, in PTSD, an association is made between a safe cue or context and a threat via experiential fear conditioning, observation of others experiencing the threat, or instruction by others (Hartley & Phelps, 2010). During extinction learning, the feared object or context is presented repeatedly without the threat, until new memories are formed in which the cue is no longer linked with the threat (Rodriguez et al., 1999). While laboratory models include only the animal and the feared cue, ET in humans also involves the therapist – a social safety cue – as a crucial element of the therapeutic process (Buchholz & Abramowitz, 2020). The therapist signals safety via both instruction and behavioural modelling of safety and non-avoidance. The core principle of ET is, therefore, gradual exposure to feared objects/situations with guidance from a clinician until the fear response has been extinguished. Successful ET includes two key elements: cue generalization and contextualization.

1.1.1. Cue generalization

An important element of effective ET is generalization of safety learning to relevant (i.e. real-life) cues. The parallel core pathology in PTSD is that fear responses are generalized to cues that share qualitative features with the original feared stimulus (Lis et al., 2020). For example, a survivor of sexual trauma may generalize fear of the perpetrator to all people resembling him, or to all males. It is also essential to create situations that include people whose physical characteristics resemble those that the patient avoids owing to their unique experience. Many present-day veterans with combat trauma, for instance, avoid or are anxious around people of Middle Eastern descent; therefore, exposure hierarchies that include such racial and cultural cues are important in helping veterans to achieve optimal functionality.

1.1.2. Contextualization

Extinction learning is context dependent, i.e. associations between the previously feared cue and safety are linked to the context within which the exposure takes place (Hartley & Phelps, 2010). As a result, extinction learning taking place in a clinic may not extend to a patient’s real-world context. For instance, a parking lot resembling the context of an assault or shooting could trigger a fear response, even after treatment has taken place in the office (Rodriguez et al., 1999). Therefore, it is important that exposures be completed in as many real-life contexts as possible.

1.2. Limitations of traditional ET

Despite great efficacy, multiple factors limit access and adherence to ET, as well as the real-life value of ET (Becker et al., 2004; Eftekhari et al., 2013; Foa et al., 1999; Tarrier et al., 1999). First, there is a shortage of mental health providers, especially in areas located far from academic centre (Andrilla et al., 2018; Morales et al., 2020; World Health Organization, 2018). Too often, patients receive only medication or general support therapy for years before seeing a trained specialist. Further avoidance and low motivational features of PTSD make it increasingly difficult to leave one’s home to seek clinical care, especially since the coronavirus disease 2019 (COVID-19) pandemic (Kelly et al., 2019; Scoglio et al., 2022). Secondly, the feared/avoided situations are often inaccessible in the clinic, and thus therapeutic approach behaviours cannot be practised under supervision. Clinicians do not have access to scenarios associated with trauma in first responders (e.g. the scene of a troubling cardiopulmonary resuscitation) or a diverse set of people in the clinic. Even when these are available, clinicians have very limited control over the within-session behaviour of the feared objects (e.g. the behaviour of people at a mall where exposure to the crowd is taking place). For these reasons, ET is often limited to pictures, video clips, other untested and less salient proxies, or imagination of the feared situations or traumatic experience. Imaginal exposure lacks the level of arousal required for the development of safety learning (Lane et al., 2015); therefore, traditional ET otherwise relies on the patient’s ability to self-expose in between sessions.

Thirdly, in a clinical setting, generalization of safety learning to relevant cues can rarely be accomplished satisfactorily with the limited proxy stimuli available to the clinician. It is almost impossible to create in vivo exposure scenarios that include diverse people of different sexes, ages, races, and body types, which is a necessary facet in promoting safety generalization by practising exposures relevant to the types of people or cues avoided by the patient.

Fourthly, patients may experience anxiety in a variety of social situations, but exposure often occurs in the context of the clinic, limiting contextualization of safety learning. As discussed, safety learning in one social context may not map on to other contexts (Hartley & Phelps, 2010). Reduced anxiety in response to social interactions in a sterile clinic often does not apply to other, unpredictable, environments (e.g. workplaces, grocery stores) (Rodriguez et al., 1999). This emphasizes the need for treatment that can be delivered across contexts.

Fifthly, when patients complete exposures independently, therapists cannot watch for subtle rituals or safety behaviours. Safety behaviours interfere with ET, as patients misattribute the absence of a feared outcome to their use of safety behaviours (Blakey & Abramowitz, 2016).

Finally, patients too often find ET homework overwhelming and are consequently unable to voluntarily initiate exposures. Fewer than 50% of those eligible for prolonged exposure (PE) therapy complete treatment, and 25–40% of those who attend one session drop out (Davis et al., 2013; Gros et al., 2011; Jeffreys et al., 2014; Kehle-Forbes et al., 2016; Mott et al., 2014; Rauch et al., 2012). It is also often difficult for patients to independently identify and create exposures to the routine life situations that they avoid. When exposures are identified, it is then challenging to elicit the optimal level of anxiety for effective extinction of fear, particularly in real-world circumstances where the intensity of exposure is less predictable.

In summary, current treatments focus less on real-world social and occupational functioning outside the clinic, often because of notable barriers to the effective implementation of in vivo exposure towards addressing real-life impairment. Although patients often experience significant symptom reduction in other domains after psychopharmacological or psychotherapeutic intervention, many remain socially and occupationally impaired (Kelly et al., 2019; Rodriguez et al., 1999). In our work with first responders, our team has frequently observed such deficits, wherein patients remain disabled even years after losing a PTSD diagnosis.

A way of providing ET that would allow the clinician to present the patient with a range of different stimuli, with varied characteristics in multiple real-life contexts, while facilitating communication with patients to address behavioural hurdles, would largely overcome existing barriers.

1.3. Virtual reality ET

Over the past few decades, VR has been used to surmount several limitations of traditional ET (Gonçalves et al., 2012). VR is ideal when the creation of an entire exposure scenario, including context, is required. For example, in cases of combat PTSD, VR scenarios focusing on exposure to the trauma context (e.g. virtual Iraq or Afghanistan) have proven to be as effective as PE therapy (Rizzo & Shilling, 2017; Rothbaum et al., 2001). These treatments are largely based on the PE model, focusing on exposure to trauma memories. The heterogeneity of trauma experiences and high programming costs for developing exposure scenarios, however, have limited VR therapies to primarily combat-related scenarios (Kothgassner et al., 2019). Although the capacity to create combat-related exposures that are not possible to replicate in vivo is indeed valuable, extending this model towards the creation of VR scenarios relevant to traumas such as shootings or rape is not reasonable (Kothgassner et al., 2019; Rothbaum et al., 2001). VR also has several limitations that reduce its clinical utility: (1) VR requires the creation of the whole environment, and is therefore, temporally and financially, less efficient; (2) exposure is limited to predefined scenarios that are not always reflective of a given patient’s experience; (3) patients are not able to navigate and interact with their actual physical environment as they view a virtual environment, limiting a realistic immersive experience; (4) because VR exposure is to feared cues within virtual environments, contextual generalization of safety learning to lived environments and situations is limited; (5) tolerability is a drawback to the clinical implementation of VR, as it is often associated with discomfort (e.g. motion sickness) (Kim et al., 2022; Vovk et al., 2018); and (6) extant literature indicates that VR therapies are only as effective as existing ETs for PTSD and fear-based disorders (Emmelkamp et al., 2020; Eshuis et al., 2021; Gonçalves et al., 2012). Thus, the dissemination of VR is challenging, as therapists must learn new technology only to achieve the same clinical outcome.

2. Augmented reality

The technology described herein aims to address many of these limitations to treatment by offering diverse exposure stimuli/scenarios, and through enhanced interaction with patients to facilitate fear extinction and functional improvement.

AR is the newest wave of interactive human–computer technology that enables exposure to avoided stimuli within a patient’s actual physical reality, thereby assuring contextual relevance (Eshuis et al., 2021). Whereas VR substitutes a virtual environment for the real one, AR adds specific virtual elements to the real environment, thereby requiring less processing power and fewer costs than VR (Dünser et al., 2011). In contrast to more primitive versions of this technology (e.g. mobile AR apps), novel wireless AR head-mounted systems support both a high degree of accuracy regarding the position of high-resolution three-dimensional (3D) objects within real surroundings, and the ability to customize and play stereo sound.

The application of this technology to ET, i.e. augmented reality exposure therapy (ARET), has several advantages over traditional exposure therapies. For one, ARET can provide a diverse range of avoided objects and scenarios for ET, thereby assuring cue generalization within a real-life context. As any real environment can be used as a backdrop for AR, ARET can also be combined with telemedicine. Furthermore, combining virtual objects with a real environment ensures better immersion than in VR, as the patient can see and freely navigate their real environment while interacting with the virtual objects and having visual access to therapist simultaneously. AR also uses the actual environment to apply ET within real-life contexts, as opposed to re-creating virtual situations that may or may not represent a given individual’s experience. This is particularly important as it allows the real-life contextualization of extinction learning. Fittingly, early data suggest that AR is superior to VR in creating the level of arousal needed for successful ET (Tsai et al., 2018).

Earlier versions of AR technology have been used to treat simple phobias; for instance, by positioning AR-generated insects on or near a patient’s hand or body (Carlin et al., 1997). Our group has developed a patented technology to this effect (US Patent Office non-provisional: 15/259,844) and demonstrated efficacy in using advanced wireless AR headsets to treats simple phobias (Javanbakht et al., 2021).

This technology was developed within an academic institution, but in close collaboration with leading AR industry partners. Our initial studies on the treatment of phobias have been promising. In these platforms, the clinician can see a live, 3D map of the patient’s environment; by use of drop-down menus, select a spider, snake, or dog (of diverse types and breeds), place them in patient’s environment, and define their behaviour (e.g. spider crawling at a specific speed from the floor to the wall to the ceiling; dog barking, jumping, walking, sitting). In a published clinical trial, all patients with fear of spiders were able to touch a real tarantula, or the tank containing it, after a single treatment session lasting less than 1 hour (Javanbakht et al., 2021). In an ongoing clinical trial, we are producing the same results in participants with a fear of dogs (Javanbakht, 2024).

To date, however, these and similar treatments have been limited to specific phobias. The following section will detail an easily scalable ARET intervention for PTSD and other fear- and avoidance-related conditions by addressing the avoidance of common, real-life environments. Initially, this intervention has been targeted towards treating first responders with PTSD by providing ARET to work-related environments (e.g. police station, fire station), in addition to commonly avoided societal environments (e.g. crowds, busy grocery stores). Under a project funded by the US Department of Defense, we are currently fine-tuning the technology for use in veterans with PTSD. Further applications and expansions of this technology to treat a broad range of populations and conditions are also discussed.

3. Method

3.1. Interactive AR technology for addressing social and occupational disability

This technology includes realistic human characters for use in addressing social and occupational disability in first responders with PTSD. This allows for contextual flexibility and the possibility of use in telemedicine, owing to easy remote connectivity between the patient and the therapist, as well as adjustability to diverse physical contexts.

Commercially available AR devices (e.g. Microsoft HoloLens or Apple Vision Pro) have capabilities ideal for use in ET. These devices display 3D objects, provide excellent surface mapping needed for the accurate positioning of objects in the real physical environment, enable interaction with objects and allow patient mobility through wireless capability, and provide stereo sound. Such devices are also affordable (currently $3500 apiece) for in-clinic use. Theoretically, they could also be shipped to patients for remote use, but that would require the purchase of multiple devices by each clinic. However, as AR becomes more publicly available and manufactured on a large scale, the expectation is that, over time, the price will decline.

AR has immense potential for revolutionizing the treatment of fear- and trauma-related disorders, for several reasons: (1) AR requires less programming as it does not require the creation of the whole environment (e.g. our technology places virtual humans proportional to the confines of a real room); (2) it allows for cue generalization (e.g. people of different sexes, ages, races, body types, and behaviours); (3) it allows for contextualization of safety learning in real environments (Bouton & Bolles, 1979); (4) our experiences indicate that, as patients cognitively ‘know’ that the objects are not real, treatment acceptance is higher; (5) AR offers a sense of immersion, as patients interact with 3D virtual objects in their real environment, while being able to see their own bodies (hands, feet, etc.), which creates a higher degree of physical presence; (6) AR provides a sense of control (a pivotal aspect of trauma treatment in particular) as patients can walk freely around their environment and interact with virtual objects; (7) this technology allows for direct communication with the therapist, as patients have visual/audio access to the therapist concurrent with the virtual objects; (8) given that the patient is in continuous visual connection with the real environment, the common side effect of dizziness during VR use does not occur within AR; and (9) the visibility of the real-life environment and adaptability with different contexts allows the delivery of in vivo ET with telemedicine. This can help to address limitations in geographical access to ET experts.

While studies investigating AR have been limited, accruing evidence reveals a convincing trend that AR enhances ET from both an objective and a subjective perspective (Albakri et al., 2022). For instance, evidence suggests that AR results in significantly greater anxious engagement (e.g. attenuated heart-rate variability) compared to VR (Tsai et al., 2018). Studies have also shown that AR for spider phobia significantly elevates skin conductance response (SCR) and self-reported anxiety during the peak of the exposure, compared to baseline, even in a non-clinical sample (De Witte et al., 2020). Our own findings showed that AR objects could elicit an SCR similar to that of real feared objects (Javanbakht et al., 2021). Our preliminary data using our advanced AR technology have replicated these effects for a variety of phobias and fear-based conditions; at the end of AR treatment, our patients have successfully interacted with live insects and large animals after only one treatment session (Javanbakht et al., 2021). This is far more efficient and effective than similar studies of VR use in similar clinical conditions (Carlin et al., 1997; Garcia-Palacios et al., 2002). Patients’ confidence in treatment also improved to 80% post-intervention, as did their confidence in recommending the treatment to others with similar conditions (Javanbakht et al., 2021), which is a paramount factor for a novel technology in overcoming resistance to change.

3.2. Apparatus

The software ExpandXR (following ExposXR, which was created for the treatment of simple phobias), was created as a collaboration between academia [the Stress, Trauma, and Anxiety Research Clinic (STARC)] and leading industry partners in AR and movie industry programming. This collaboration has ensured clinical relevance, industry levels of realism of the characters and scenarios, and ease of use of the technology for therapists with average computer skills (see Section 4.1).

The connection between the AR headsets and the clinician’s computer can take place through a Wi-Fi connection or a local network. Both devices could be on the same or different Wi-Fi networks, allowing versatility of use in diverse contexts. Once the headset worn by the patient has connected to the therapist’s computer, a 3D map of the patient’s environment is acquired via the headset and reflected on the therapist’s screen. Using that map, the therapist draws the boundaries of the virtual environment within the patient’s real context. A virtual door opens on the real wall in front of the patient, through which AR characters will enter their environment. The therapist will see a live feed of the patient’s position in the environment, and their point of view (Figure 1). This is especially helpful during remote use of the technology.

Figure 1.

Clinician's desktop view. The clinician can see patient's point of view (large image), as well as their position in the room and in relation to the AR characters (small image on the top left). By clicking +, the clinician can advance through preset scenario hierarchies. The clinician may also choose from a list of sounds to add to the scenario for customization of triggers. Clinician can enter patient's reported level of distress at any time under (SUDS Value).

The software offers three options for use: preset scenarios, a scene generator, and AI-driven humans.

3.2.1. Preset scenarios

These scenarios allow exposure to a crowd of human characters of diverse age, sex, race, body type, outfits, and behaviour within diverse, commonly encountered contexts. By the addition of digital props and background sounds to patient’s real environment, an empty room can turn into a living room, a bar/restaurant, a grocery store, a sports event, a police station roll-call room, or a fire station (Figures 2–4). Each scenario starts with one or two least intimidating characters (e.g. one or two petite women) entering the far corner of the room. By clicking ‘next’, the therapist can summon more characters into the room at each level, until the room is filled by up to 15 characters of diverse appearances and behaviours. The conversations between the characters increase in intensity. One scenario, for instance, ends with two characters arguing with each other, and, by way of spatial recognition, staring at the patient. In the police station, the final level involves a radio announcing a shooting incident, where two police officers leave the room to respond to the incident. Sound is spatially relevant: as the patient nears each group of characters, they become louder. The scenarios are designed for efficient use in the most encountered – and therefore most needed for social functioning – environments. Table 1 presents an example of the exposure hierarchies in the ‘Social Gathering’ scene.

Figure 2.

Snapshot of patient's point of view in the restaurant scenario, collected via the AR headset. Note the experience is much more realistic, 3D, and immersive through the headset. The character in a suit and tie is an AI-generated avatar of the paper's first author.

Figure 3.

Snapshot of patient's point of view in grocery store scenario, collected via the AR headset. Note the experience is much more realistic, 3D, and immersive through the headset.

Figure 4.

Snapshot of researcher acting as patient in police roll call scenario, collected via the AR headset. Note the experience is much more realistic, 3D, and immersive through the headset.

Table 1.

Example of graded exposure to crowds in the simulated gathering scenario. With the addition of props, this could happen at a house party, in a restaurant/bar, or in a sports event concession area. Clinicians can add background noise ranging from a knock on the door to different loud noises, guns, people, and animals.

Level	Visual component	Audio component
1	Two females walk through the virtual door and stand in the far-left corner, having a neutral conversation	Softly spoken voices, talking about vacation and planning a concert event
2	A taller male joins them and stands, quietly listening to them
3	A white female and a black male come closer to the viewer on the left side, having a neutral conversation	Talking about traffic and work
4	A white male and a black female come and join the second group	Making introductions and talking about their jobs
5	A white and a black older male enter and stand on the left side of the patient, and start a neutral conversation	Talking about fishing
6	A white female on her phone passes the patient and exits the room	Talking with an angry tone
7	A tall, heavy-set black male walks in, finds the patient, asks a question, then turns around and leaves the room	Nicely introduces himself and asks for Thomas
8	Four younger males and females, white, black, and Asian, enter the room and start a benign conversation, but with some complaining	Talking about work and city-life difficulties
9	A tall white male and a tall black male enter, stand in front of the patient, and argue. Occasionally, they turn their heads and stare at the patient	Arguing about a cryptocurrency investment gone wrong
10	A Middle Eastern male in a suit enters and announces the fireworks	Loud noise of fireworks

3.2.2. Scene generator

This environment allows for more customized interactions between the patient and the AR characters. The therapist can choose from a list of nearly 100 diverse characters, position up to 12 characters in the patient’s environment, and command their behaviour. Behaviours include realistic facial expressions (e.g. frowning, smiling, surprised, sad) and a range of body gestures (e.g. shrugging shoulders, nodding, tapping foot). There are diverse characters sitting on chairs or at a desk (e.g. for replicating encounters with an authority or job interviews) and characters with a gun or bruises (e.g. triggers for first responders). Therapists can also add background sounds and props, including chairs, tables, food, a bar, dead bodies (for first responders), and concession stands, to customize various environments. The text-to-speech capability allows for live, dynamic conversations between each character and the patient. Clinicians can choose from AI-generated voices indistinguishable from real human voices, type in the text they want the character to speak, and have a fully customized conversation with the patient through each character.

3.2.3. AI-driven humans

This environment allows for the automation of conversations between patients and highly realistic, customized AR-AI human characters. Therapists can write a ‘brain’ for a desired character by describing their characteristics, flaws, motivations, dialogue style, and personality. For example, our first AR-AI human, Amy, is a 38-year-old female teacher from Detroit, married to a librarian, with two children, whose father was a police officer. Her hobbies include watching TV, baking, and going to football games, as she is a Detroit Lions fan. Amy is friendly and supportive, and does not talk about politics or religion. Her character is inclined towards trust, joy, positivity, openness, and extraversion. Ross, on the other hand, is an older police officer who has ‘seen a lot’; he is more formal and distant, but still friendly and trustworthy. Once the characters are placed in a patient’s environment, they will initiate a fully automated, unscripted conversation with the patient, using large language models. The patient’s voice is sent to the cloud, where the response is determined and sent back, and an AI-generated, realistic voice indistinguishable from humans will then speak through the headset speakers. The response time is around one second, allowing for further realism. AI may be directed to have a specifically motivated role (e.g. an interviewer, a difficult boss, or a patient with severe mental illness for de-escalation training). This AI-driven feature is a step up, allowing the patient to practise more fluid and less predictable social encounters. Therapist can make further edits to the AI brain, or choose a specific AR character for that brain, based on each patient’s needs.

The following links contain video demos, recorded via the AR headsets. The first link demonstrates the preset scenarios and scene generator, and the second link shows interactions with a fully automated AI character with role transition throughout the video: https://www.youtube.com/watch?v=fQCw-35S5Vs and https://www.youtube.com/watch?v=PJpM5YZAwI4.

Of note, each AR character was created using AI, based on a reference image or photograph of the specific race, sex, and age that was shared with the programming team, to ensure diversity.

The clinician can enter the patient’s subjective units of distress in real time, on the Subjective Units of Distress Scale (SUDS; ranging from 1 to 10). These data, along with exposure hierarchy level, background noise, and props presented, are automatically saved in a spreadsheet. Throughout the treatment, clinicians can choose from a list of sound effects (e.g. different types of gunshot sounds, barking dog, crying person or baby, people arguing, fireworks, slamming door, and many more) that could be a trigger for the specific patient. To allow a better sense of control, the patient can pause the scene such that all characters will disappear, by saying a safe word (i.e. ‘Holy Cow’).

4. Results

4.1. Feasibility of use

New technologies often struggle against inertia among target users, as learning new protocols increases the clinician/patient burden. With this in mind, hundreds of hours were spent on optimizing software fluidity and the user interface to ensure ease of use by therapists with different levels of technological skills, in consultation with several dozen therapists from different disciplines (e.g. psychiatrists, psychologists, social workers) and settings (e.g. academic, private, military), to improve effortless use and maximal deployment. In all stages of development, decision making was conducted with the technologically lay-therapist in mind, to ensure future dissemination. In addition, we have greatly benefited from working with highly skilled and experienced top-tier industry partners to create state-of-the-art technology, while considering practicality and relevance to large-scale use. First responders were involved in the development, based on their specific real-world needs, and the same approach was applied to revisions for use in veterans with PTSD.

4.2. Patient feedback

We have previously detailed the treatment session of one patient (a middle-aged female police officer and survivor of assault with PTSD), along with her psychophysiological data (Javanbakht et al., in press). Here, we provide feedback from patients treated at STARC, using this technology. This included clinician-observed behavioural responses, patients’ self-reports during the sessions, and feedback following the completion of each treatment session. Patients tolerated the AR headset easily, and there were no complaints of dizziness or about the weight of the headset. Similarly to ARET for simple phobias, most patients with PTSD find participation in the treatment easier than real-life ET, as they cognitively know that the exposures are not real. During treatment, however, the psychophysiological, subjective, and objective fear response is strong and reaches the necessary level for treatment effects. All patients who used this technology found it as triggering as real-life environments. Even when they were repeatedly reminded the characters were not real, patients demonstrated significant avoidance behaviour at the beginning of the treatment. Often, patients take almost an entire session, sometimes more, to complete exposure hierarchies in only one scenario. Another police officer survivor of sexual assault with PTSD, for instance, could only tolerate two female characters talking with each other in the far corner of the room in her first treatment session. Another officer with PTSD stood in a corner of the room with her arms crossed on her chest, with sweaty hands, for the majority of the first session. For most patients, it was difficult to have their back to any of the AR characters. The diversity of characters has also been tremendously helpful in triggering different emotions. Each patient demonstrates their own individual response to different characters in the room. For example, the female patient survivor of sexual assault found it specifically difficult to tolerate a male character quietly listening to two female characters talking, despite the simultaneous presence of other, more traditionally intimidating or arguing characters in the room. Upon further exploration, we found that this was because the perpetrator of her assault was a quiet, benign-looking male, whose behaviour was unpredictable. Despite completing other scenarios, another police officer did not agree to enter the police station roll-call room, as their trauma was work related. This patient felt more comfortable in the fire station scenario, as they found fire fighters safe and protecting. The same patient finally agreed to allow one petite female police character in the room during session 6 of the treatment, but as soon as she saw the character, she felt extremely anxious, afraid that the character would attack her, although that she cognitively knew that this was not possible. She described the experience as ‘They feel like real humans. This is so psychological, I cannot feel they are not real.’ For most police officers with work-related trauma, exposure to the police station was the most difficult, occasionally prompting significant emotional reaction. One female police officer with PTSD, for instance, suddenly began crying when she heard a character complaining about another officer getting promoted over her, as she had had a similar experience. The characters also triggered cognitive patterns and transferences. One of the above officers felt that one of the characters ‘reminds me of myself, she is reserved, she is uncomfortable, she wants to leave, I can see that. She is just saying enough to just get by. This makes me fill claustrophobic, it seems like she is pinned, she cannot move. She is passive to where she cannot say anything. That is how I am.’ About another character, she said: ‘He looks and acts the same as my sergeant. His outfit is terrible. He seems like a farmer who likes outdoors and has a lot of horses.’ Towards the end of each exposure scenario, most patients were comfortable with the clinician leaving the room to allow an independent sense of control.

Overall, patients believed that there was a benefit to the presence of diverse environments, and that the props and the sound ambiance accurately created a perception of these diverse contexts. Patients found the AI characters the most engaging, frequently remaining in the conversation for long periods. Many were initially nervous, had long pauses, and allowed the AI character to take the lead. With encouragement, they became more direct and engaged. Occasionally, patients forgot about the presence of the therapist and were fully immersed in the conversation. The AI characters also demonstrated realistic reactions, with associated emotions and pauses. For example, one patient was planning a trip to Tennessee, and the AI character told them about her own experience in that state, including good restaurants and trails that the patient could visit. Once, Amy asked a patient whether they had pets; when they responded, ‘Can kids be considered pets?’, Amy sighed, and answered with appropriate empathy that one mother would have for another. Another patient, a current firefighter and former police officer, shared the difficulties of their job with Ross; he offered empathy and shared his own experiences as a lifetime first responder. One patient came asking for the grocery shopping exposure session, as ‘I am tired of buying grocery online, it is costing me too much’. This patient was then able to do grocery shopping, spending 30 minutes in the grocery store twice in the following week.

5. Discussion

5.1. Implications and future directions

While ET is a first line treatment for avoidance in fear- and trauma-related disorders, its meaningful use, especially in relation to avoided real-life situations, has been highly limited owing to a lack of access to avoided cues and contexts within clinical settings. VR technologies have been useful in creating the context of trauma-related exposures based on PE models of PTSD treatment (Gonçalves et al., 2012; Kothgassner et al., 2019). AR offers an opportunity to address the gap between the clinic and the real-world avoidances. AR allows for sophisticated merging of the real and virtual, as well as immersive, in vivo exposure to a wide range of avoided stimuli (Eshuis et al., 2021). Given this integration of virtual objects and real contexts, AR allows for flexibility of use in diverse physical contexts, including remote use during telepsychiatry. The patient’s ability to see their real world both promotes mobility and realistic, immersive interactions with the AR objects and the therapist, and provides real-world contextualization and generalization of treatment effects (Rodriguez et al., 1999).

The majority of current treatments for PTSD are focused on symptom reduction, with far less emphasis on real-world functioning. Indeed, in most studies of these treatments, social and occupational functioning is not a primary target of treatment outcomes. ARET can address that gap in available interventions by focusing not on trauma-related content specifically, but rather on safe situations that are avoided because of trauma, thereby promoting social and occupational functioning. As such, this is one of few treatment methods specifically targeting avoidance-related disability, with potential for use in conjunction with other established treatments.

As a result of its focus on the patient’s ability to engage in a diverse range of social functions – from simply being at a grocery store to engaging in a conversation with a fully automated AR-AI human with different levels of difficulty – this technology has transdiagnostic utility. It may be employed in treating a range of psychiatric conditions, such as social anxiety disorder, PTSD, autism spectrum disorders, and severe mental disorders that require social interaction skills training. The clinician’s ability to determine the content of speech and behaviour of AR characters allows for treatments customized to the specific needs of the individual patient, regardless of the type of diagnosis.

Highly malleable AI allows for automation of treatment and social skills learning, by motivating each AI character to address the special needs of a patient. An AI brain can be assigned to any character of a determined age, sex, race, and physical appearance to address the avoidances of a patient. For example, a patient with a history of sexual assault by a tall person of one race can first practise safety near benign male characters of other races, before eventually practising exposure to a tall person of the race of the perpetrator, at the highest level of exposure hierarchy. AI brains are adjustable by clinicians, so they can demonstrate specific motivations; for example, for use in scenarios involving job interviews, police de-escalation training, or practising assertiveness with a bully. The diversity of available characters will also allow for equity in treatment, as some patients may initially feel more comfortable with characters of their own race, sex, or other characteristics. With the rate at which AI technology is expanding in the realm of human character development (e.g. in the gaming industry), therapists may soon be able to describe desired settings, props, environments, and specific characters, which could be created by AI in the moment to meet the specific needs of each patient. Flexible AI-driven characters have the potential to evolve through their interactions with each patient, supporting graded exposure to their specific avoidances and fears. AI could also direct the exposure scenarios based on the patient’s responses, with customizable degrees of predictability. More advanced AI character engines will also allow for the involvement of multiple AI-driven characters in interactions with patients.

To date, AR technologies have been mostly available to developers or for corporate use, remaining relatively unknown to clinicians and the community at large. This has limited their use in clinical treatment, particularly given financial concerns and initial resistance in acquiring new technology; however, these dynamics are changing. In 2024, Apple’s first-wave production of its Vision Pro headsets quickly sold out (released in March 2024; 500,000 sold). As the technology becomes less costly and more readily available, and is more widely used in entertainment, gaming, and other areas of life, home use of this technology for between-session practice will become more feasible. While, for many patients, initiating treatment is difficult owing to an avoidance-based inability to leave their home (Kelly et al., 2019; Solomon & Mikulincer, 2007), the growing availability of telepsychiatry ARET will provide a means of scaffolding patients towards the next level of in vivo exposure.

Modern AR technologies also allow for eye tracking and even measuring pupil diameter. Along with tools such as SCR measures, AI technology will allow for the greater prediction of treatment response, particularly for algorithmic treatments such as ET. This highlights opportunities for the automation of ARET in the near future, particularly given highly limited access to therapists skilled in ET.

6. Conclusion

We have developed an AI-powered ARET technology that allows users to interact with digital humans in their real surroundings, assisting otherwise reticent patients to engage with exposure stimuli. The technology focuses on exposure to common – albeit avoided – civilian-life scenarios, fear of which often limits functioning and is a broad cause of disability. Our pilot data indicate great success in overcoming avoidance using the ARET approach. This technology offers varying complex applications based on therapist and patient comfort, as well as on individual needs. These include predefined, scalable scenarios, a library of characters whose behaviour and content of speech may be fully controlled and defined by the therapist, and characters with AI speech-generation capabilities. While most currently available treatments for fear- and trauma-related disorders focus primarily on symptom reduction, this technology directly targets avoidance and functional disability, thereby facilitating broad transdiagnostic use and greater impact. As AI advances and AR becomes more available to the public, this line of work has the potential to contribute towards shaping the future of treatments for fear- and trauma-related disorders.

Funding Statement

This work was supported by the State of Michigan Department of Health and Human Services.

Disclosure statement

No potential conflict of interest was reported by the authors. The patent (US 10,839,707) and the technology licence are owned by Wayne State University.

Authors’ contributions

Arash Javanbakht: inventor of the technology, conceptualization, writing – reviewing and editing, funding acquisition, investigation, and supervision; Liza Hinchey: conceptualization, methodology, writing – original draft, and writing – reviewing and editing; Kathleen Gorski: data curation, project administration; Alex Ballard: design, collecting data, testing and use of technology, and writing; Luke Ritchie: technical development of the technology, investigation, conceptualization, and writing; Alireza Amirsadri: conceptualization, development, funding acquisition, investigation, supervision, and writing.