TherapyTrainer: Using AI to Train Therapists in Written Exposure Therapy

Abstract

Though evidence-based treatments for mental disorders are effective, existing implementation efforts are expensive and difficult to scale. Novel solutions— especially those that offer active learning strategies, repeat skill practice and personalized feedback to therapists — are needed to fill this gap. We developed TherapyTrainer, which uses large language models (LLMs) to allow therapists to practice delivering written exposure therapy (WET) for PTSD to AI-Patients while receiving expert feedback from an AI-Consultant. Here we present initial feasibility, acceptability and usability data for TherapyTrainer gathered from therapists, supervisors, and WET expert-consultants across iterative rounds of development. In Phase 1, we rapidly prototyped and developed TherapyTrainer based on ongoing feedback from WET clinicians and experts (n = 4). In Phase 2, mixed methods data from therapists engaged in an otherwise-routine WET workshop (n = 14) indicated that TherapyTrainer is feasible and acceptable and may help therapists feel prepared to deliver WET. In Phase 3, therapists (n = 6) completed structured user testing interviews to identify key issues impacting usability for subsequent rounds of development. AI and large language models hold potential to provide ongoing support to therapists in a cost-effective and scalable manner, and may help close the research-practice gap.

Decades of implementation science research has not bridged the research-practice gap in evidence-based psychotherapy. Despite treatments being effective, their uptake in clinical settings – even those that invest heavily in training – is low, leading to calls for novel solutions (Kazdin & Blase, 2011; Maguen et al., 2020). Existing therapist training practices – such as reading a treatment manual or attending a workshop – do not lead to gains in therapists’ uptake or delivery of these treatments, including for the exposure-based treatments (Herschell et al., 2010; Trivasse et al., 2020; Valenstein-Mah et al., 2020). Providing therapists ongoing support after learning an evidence-based treatment (EBT), such regular consultation on their delivery of the EBT, may help ensure the treatment’s uptake; consultation improves therapist self-efficacy and fidelity as well as patient outcomes (Edmunds et al., 2013; Valenstein-Mah et al., 2020; Wiltsey Stirman et al., 2022).

However, issues with consultation limit its potential impact. Recruiting and maintaining adequate pools of trainer-consultants is resource-intensive (Rosen et al., 2017), consultation can be prohibitively expensive (Okamura et al., 2018), and therapists struggle to fit it into their schedules (Chard et al., 2012), challenges which together limit access to consultation. Therapists who do have access to consultation are typically required to complete a set number of cases, and can face challenges recruiting or retaining patients within the required time frame (Stuart et al., 2018; Timmer et al., 2016; Worley et al., 2020). This results in therapists facing long waiting periods before they can practice the EBT they just learned, or having practice periods that are infrequent, interrupted, or incomplete. Consultation also typically involves a delay between delivery of treatment and receiving feedback. Innovations are needed to improve therapists’ ability to quickly and effectively practice their new skills while receiving expert support after EBT training.

Desired Qualities of a Novel Therapist-Training Program

An ideal therapist training technology would promote active and experiential learning, allow timely and repetitive EBT skill practice, and facilitate timely and tailored feedback.

Active and Experiential Learning

Experts recommend that therapist training incorporate active learning techniques, such as behavioral role plays, which afford therapists opportunities to engage with the material directly rather than serving as passive knowledge recipients (Beidas & Kendall, 2010). In education generally, active learning improves learners’ motivation, attitudes, and outcomes (Michael, 2006; Owens et al., 2020); when used to train exposure therapy, active learning enhances therapist self-efficacy and treatment perceptions (Frank et al., 2023). An ideal therapist training program would therefore involve active learning techniques.

Repetitive Skill Practice

Methods that involve repetitive skill practice have been suggested to enhance the acquisition of therapy skills (Mahon, 2023). Role play is commonly used to allow skill practice during workshops (Beidas & Kendall, 2010), yet is time consuming, requires a willing partner, and may not predict real-world therapist behavior (Decker et al., 2013).

A version of role play uses “simulated patients” – lay individuals trained to portray specific cases symptom presentations – to allow therapists opportunities for practice. Commonly used in medical training (Kühne et al., 2022; Pheister et al., 2017) and relatively new to psychology, simulated patients can facilitate active learning as well as assessments of therapist competency or fidelity (Edwards et al., 2016; Kühne et al., 2022; Imel et al., 2014), and may increase provider confidence when learning new treatments (Rogers et al., 2014).

However, practical barriers limit the use of simulated patients for EBT training. It is time-consuming and costly to train and employ standardized patients (Kühne et al., 2022), and significant effort is needed to ensure they seem believable, stay in character, and depict psychopathology authentically (Kühne et al., 2022). Given resource and availability constraints, they are generally used to train and assess providers on a single session (e.g., Rogers et al., 2014), restricting therapists’ access to practice across the entire treatment protocol.

An ideal training program would provide the opportunities for immediate skill practice offered by simulated patients, while also enabling unlimited, on-demand practice of complete protocols—benefits not feasible with actor-based simulations.

Timely Feedback on Full Sessions

Experts have also highlighted the value of targeted feedback, a core part of consultation which is theorized to improve therapist skill and understanding vis-à-vis EBTs (Edmunds et al., 2013; Mahon, 2023). Yet current consultation models leave gaps in feedback. During consultation, therapists commonly receive feedback based on self-selected segments of session recordings (e.g., Johnson et al., 2022; Mallard Swanson et al., 2021) or self-reports regarding what occurred in the session, which may be inaccurate (Caron & Dozier, 2022; Wiltsey Stirman et al., 2013). More intensive methods like live or retrospective observation of complete sessions are typically too time-consuming to execute at scale, meaning that therapists rarely receive feedback on full sessions. Therapists also rarely receive feedback in real time (Westra & Di Bartolomeo, 2024), which may be particularly concerning for low-fidelity behaviors that could derail a session. Scalable innovations that deliver real-time feedback on complete therapy sessions may improve therapists’ ability to deliver EBTs.

To summarize, innovations that provide post-training support for therapists could help bridge the research-practice gap. Ideal qualities of this support include active learning strategies, opportunities for repetitive skill practice, and timely feedback on full sessions.

A Generative AI Solution for Therapist Training

Generative AI, particularly large language models (LLMs) – sophisticated computer models which can flexibly generate human-like text and obtain domain-specific expertise (Yao et al., 2023), hold promise for offering EBT training solutions at scale (Stade et al., 2024a). Researchers have used LLMs to support interpersonal skill training (Fang et al., 2025; Shaikh et al., 2023); this notion could be extended to allow therapists EBT skill practice.

Properties of generative AI make it well-suited for use in a therapist training tool. Generative AI enables the creation of AI-simulated patients (Holderried et al., 2024; Louie et al., 2024) by realistically simulating human behavior (Park et al., 2023). Interacting with AI-simulated patients could give therapists a low-stakes, controlled environment in which to practice delivering EBTs. Given its 24/7 availability, generative AI supports immediate, on-demand, and unlimited skill practice, reducing delays in applying new skills and allowing therapists to rehearse full treatment protocols without relying on actors or role-play partners. Models incorporating multiple generative AI “agents” playing different roles – such as an EBT expert (Held et al., 2024) – could support training solutions that provide real-time feedback on therapist performance.

LLMs have been used for training basic counseling skills (Louie et al., 2024; Tanana et al., 2019) and to facilitate practice and feedback on psychological assessment or individual therapy skills (Elyoseph et al., 2024; Lin et al., 2024; Wang et al., 2024). To our knowledge, LLMs have not yet been used for training in complete EBT protocols – likely due to the substantial model tailoring required to accommodate the dynamic changes in patient and therapist behaviors present across a multi-session treatment protocol.

The Current Study

Here we introduce TherapyTrainer, which allows therapists to practice delivering the full protocol of written exposure therapy (WET), a form of cognitive behavioral therapy, with AI-based simulated patients, while receiving real-time, expert-level feedback from an AI-consultant. In WET, patients write about their traumatic event in a repeated and prolonged fashion to integrate the trauma memory and reduce their distress (Sloan & Marx, 2019). There is currently a high demand for therapist training in WET due to its efficacy, abbreviated, five-session format, and low dropout rate (Sloan & Marx, 2024). WET’s relatively brief format and narrow therapeutic scope (primarily guiding and providing feedback on the written exposure) reduce the complexity of model customization and make it well-suited for a proof-of-concept LLM-based therapist training application. Following a three-phase user-centered design approach, we developed and refined TherapyTrainer through iterative feedback from therapists, focusing on usability, feasibility, and acceptability.

Method

Development of TherapyTrainer

TherapyTrainer is an AI-powered, web-based tool that allows therapists or trainees to practice delivering WET to simulated patients while receiving guidance from a simulated consultant. The “AI-Patients” and “AI-Consultant” are large language model-based conversational agents (“chatbots”) developed by applying prompt engineering to an OpenAI foundation model (e.g., gpt-4o). Foundation models encode a wide range of human behaviors during pretraining (Park et al, 2023), and prompt engineering uses written instructions to direct the model’s responses, enabling it to generate context-specific outputs in real time without additional fine-tuning or datasets of real interactions (Stade et al., 2024a). In this approach, the AI-Patients and AI-Consultant generate responses dynamically, simulating realistic patient expert-consultant responses based on the therapist’s input.

We developed early prompt prototypes using WET training materials and input from individuals with PTSD, WET, and prompt engineering expertise (E.C.S., J.C.E., D.L.K., A.S., and S.W.S.). We iteratively refined the TherapyTrainer program based on ongoing feedback from therapists and WET trainer-consultants, aiming to enhance the tool’s suitability for training and ensure that the AI agents behaved as intended (see Phases 1 and 2 below). Additional details on TherapyTrainer’s technical development are presented in the supplemental text.

Simulated Patient Agents

The first AI-Patient is a Latino male combat veteran in his early thirties with no psychological comorbidities, representing a beginner level of clinical complexity. This case was adapted from materials originally developed for a WET therapist training program.

The second AI-Patient is a middle-aged Black woman with a history of sexual trauma, intimate partner violence, and substance use disorder, representing an intermediate level of clinical complexity. This case was developed by authors with expertise in WET and PTSD (D.L.K. and S.W.S) to introduce important clinical challenges therapists may experience while administering WET, including reluctance to engage in writing assignments, risk of return to substance use, and suicidal ideation.

Consultant Agent

The AI-Consultant provides feedback on the therapist’s delivery of WET to the AI-Patient. The AI-Consultant was instructed to prompt the therapist to do the unique elements of WET (e.g., provide the rationale for WET, provide trauma narrative writing instructions, provide feedback on the narrative) in a culturally responsive manner. Structured prompts based on the WET treatment manual (Sloan & Marx, 2019) and therapist training materials help ensure the AI-Consultant’s advice aligns with the WET protocol.

Using TherapyTrainer

On the TherapyTrainer website, users interact with each AI-Patient at their own pace to simulate a full course of treatment, divided into the five manual-defined sessions of WET. When the user sends a message (e.g., “How did the writing session go for you?”) the AI-Patient generates a custom response in the style of the patient (e.g., “It was really hard… writing it down brought back all those feelings and memories”). Users simultaneously receive feedback and guidance from the AI-Consultant (e.g., “Remember to end the session with instructions about allowing thoughts and feelings related to the trauma to come up”). Users can also direct questions to the AI-Consultant or choose to practice individual therapy skills.

Participants

Phase 1

In Phase 1, Masters- and doctoral-level clinicians provided feedback on the TherapyTrainer prototype. This included authors ranging from having no prior experience with written exposure therapy to being WET trainer-consultants (E.C.S., A.G., D.L.K., and S.W.S.) and four other WET clinicians and experts.

Phase 2

Masters-, doctoral, and medical resident-level clinicians (n = 14) used TherapyTrainer as part of an otherwise routine WET training workshop. Participants were predominantly female (86%) and had achieved postgraduate levels of education. The sample was somewhat racially diverse: 64% of participants were White, 29% were Asian, and 7% were Black. Most therapists were from community or hospital-based mental health settings (79%), and the remainder were from Veterans Affairs settings (21%). Four therapists were native Spanish speakers. Half of the sample (n = 7) additionally received consultation in WET; remaining therapists (n = 7) did not receive consultation due to capacity issues and were offered support via TherapyTrainer alone.

Phase 3

Participants were Masters- and doctoral-level therapists (n = 6), with varying levels of WET expertise, ranging from no exposure to having completed a WET workshop. This group was mostly female (67%) and had a mean age of 34.17 (SD = 6.43). Half of therapists were White (n = 3); one was Asian, one was mixed race, and one self-described as “other” race. Most therapists were from Veterans Affairs settings (67%, n = 4); the rest were from community or hospital-based mental health settings (33%, n = 2).

Measures

Phase 1

User Feedback.

Following Lyon and Koerner (2016), we gathered feedback from clinicians on their general impressions of the TherapyTrainer prototype, as well their ideal attributes of a WET therapist training platform.

Phase 2

Usage.

We examined the number of therapists who used TherapyTrainer during and immediately following the workshop, operationalized as sending at least one message through the platform.

Feasibility.

We operationalized feasibility via the following therapist-level engagement metrics, again requiring therapists to have sent at least one message through the platform to be included in analyses. These metrics included (1) distinct instances of platform use, (2) the number of WET sessions completed (out of five), (3) the number of AI-Patients interacted with, time spent using TherapyTrainer, (4) across all interactions, (5) per interaction, and (6) per AI-Patient, (7) total number of messages sent, and (8) total number of words written.

Acceptability.

We assessed therapist perceptions of TherapyTrainer’s acceptability via quantitative rating scales administered immediately following the workshop. Three items assessed the perceived realisticness of the two AI-patients and the extent to which TherapyTrainer increased users’ preparedness and confidence in using WET, respectively. Items were rated from 0 (not at all) to 4 (very great extent). In addition to calculating average scores, we also re-coded the items to obtain a binary measure of agreement. Scores of three (“moderate agreement”) and higher reflected “agreement” with the statement. Using these binary-coded items, we present percent agreement for each statement.

User Experience.

We took detailed notes on verbal feedback provided by therapists during the workshop. Therapists were for their general impressions and to suggest ways to make TherapyTrainer more engaging. Additionally, in the post-workshop survey, therapists provided written responses to the question, “Please share your feedback and suggestions about the [TherapyTrainer] exercise and how it influenced your experience using WET with your clients.”

Adopting WET into Clinical Practice.

About one month post-workshop, the lead trainer contacted therapists who received access to TherapyTrainer alone, and inquired whether they had begun treating patients in their own clinical practice with WET, serving as our measure of WET adoption.

Phase 3

User Experience.

We interviewed therapists about their experiences with TherapyTrainer using a structured user experience guide, which contained open-ended questions eliciting both general feedback (e.g., “What were your first impressions when you opened the app?”) and targeted questions about components of the app highlighted by users during previous rounds of testing (e.g., “How do you feel about the appearance and placement of AI consultant’s feedback on the screen? How about the frequency, length, and tone of the feedback?”).

Procedure

This study was approved by the Stanford University Institutional Review Board. The WET training program activities described in Phase 2 were classified as program evaluation.

Phase 1: Prototyping and Rapid Iteration

Following human-centered design principles, we conducted prototyping and rapid iteration of TherapyTrainer (Lyon & Koerner, 2016), including the embedded AI-Patients and AI-Consultant. We began by creating early prototypes of TherapyTrainer – including the AI-Patients and the AI-Consultant – and iteratively testing and refining the tool internally. Once sufficiently developed, we presented the prototype to additional therapists and WET trainer-consultants. Feedback focused primarily on the tool’s suitability for WET training, the consistency of the AI-Consultant’s responses with those of a real WET consultant, and the perceived realism of the AI-Patients’ behavior.

E.C.S., D.L.K., S.W.S., and A.S. reviewed and discussed the feedback to inform prompt engineering revisions for AI-Patients and the AI Consultant as well as to prioritize, define, and implement program-level improvements to TherapyTrainer (e.g., new features, design decisions). We continued this process in an iterative fashion – developing the prototype, reviewing it with therapists and WET trainer-consultants, and refining it based on their feedback – until a WET trainer-consultant (D.L.K.) deemed the version of TherapyTrainer (version 1) ready to be deployed to a naive group of therapists.

Phase 2: Proof-of-Concept Deployment of TherapyTrainer to Therapists Following Workshop

Therapists participating in an otherwise routine WET training workshop were given access to TherapyTrainer, version 1. During the workshop, one cumulative hour (across two time points) was set aside for therapists to familiarize themselves with TherapyTrainer and practice delivering WET. Therapists tested TherapyTrainer from their personal computers or phones while in a group video conferencing call. Therapists were encouraged, but not required, to begin with the first simulated patient, given that this was designed to be an introductory level case.

Therapists provided feedback (i.e., verbal user experience feedback) on the video call during and immediately after testing TherapyTrainer. Two clinical psychologists and one software engineer on the call (S.W.S., D.L.K., and A.S.) took detailed notes and used this feedback to rapidly implement improvements to TherapyTrainer.

Immediately following the workshop, participants completed quantitative acceptability measures (i.e., items assessing realisticness, preparedness, and confidence) and provided written user experience feedback. The qualitative and quantitative feedback was used for development until two authors (E.C.S. and S.W.S.) deemed the version of TherapyTrainer (version 2; see Figure 1) ready for subsequent user testing. Therapists were given the option to continue using TherapyTrainer after the workshop. About one month after the workshop, the lead trainer contacted therapists to inquire whether they had begun treating patients in their own clinical practice with WET, which served as our measure of WET adoption.

Figure 1.

TherapyTrainer Prototype

Phase 3: Structured User Testing

Therapists naïve to TherapyTrainer were recruited to engage in one-hour long user testing sessions, with the goal of identifying and prioritizing key features impacting usability for subsequent rounds of development. We presented therapists with version 2 of TherapyTrainer, and asked them to use the product as they would if they were encountering it on their own. Following this, therapists were interviewed using the interview guide. The user testing sessions were video recorded and subsequently transcribed. Common themes were extracted from the therapist feedback by two clinical psychologists (E.C.S. and S.W.S) and a research coordinator with experience in product management and user-centered design (S.S.).

Results

Phase 1

Here we report high-level themes and improvements made during the prototyping and rapid iteration phase of TherapyTrainer’s development. The bulk of the comments centered on the AI-Consultant’s feedback. Therapists found the AI-Consultant’s messages to be overly verbose and formal, and while they appreciated getting positive feedback, they found the AI-Consultant’s praise ingratiating. On the other hand, therapists noted that the AI-Consultant often focused on basic counseling skills and professionalism at the expense of exposure therapy techniques, and recommended reserving such feedback for cases where therapist behavior was clearly inappropriate or unprofessional. We also learned that the AI-Consultant frequently gave directions on several therapy skills at once, but therapists preferred to receive guidance focused on the next, most-important WET skill. These comments formed the basis for changes we made to the AI-Consultant via prompt engineering (see Method section and the supplemental text).

Therapists also recommended adding features to TherapyTrainer, such as accessible descriptions of the AI-Patients and clear overviews of the WET skills needed for each session. In response, we embedded patient descriptions on the application homepage and a session-specific “task list” within the chat-based user interface. Therapists also wanted the ability to speak to the AI-Consultant to ask questions or clarify feedback, so we added this functionality and allowed users to choose between interacting with the AI-Consultant or AI-Patient.

Lastly, therapists wanted opportunities to practice handling challenging scenarios, such as managing suicidal ideation during WET, and to practice subcomponents of WET in an ad hoc manner, like providing the treatment rationale or feedback on trauma narratives. This prompted us to add suicidal ideation to the second AI-Patient’s case description, and to ensure TherapyTrainer enabled practice of individual skills or session subcomponents rather than requiring the completion of entire sessions.

Phase 2

Usage

Of 14 therapists who participated in the pilot program, 93% (n = 13) engaged with TherapyTrainer on the day of the workshop, and one of those therapists also used it the following day. One therapist was not able to access TherapyTrainer due to restrictions on her work computer. One therapist, a native Spanish speaker, interacted with TherapyTrainer in Spanish.

Feasibility

We present feasibility data in Table 1. Users interacted with TherapyTrainer on 1 to 4 separate occasions (M = 1.77, Mdn = 2, SD = 0.83). During these interactions, therapists engaged with between 1 and 4 of the five sessions comprising the WET protocol (M = 2, SD = 1.15). Most therapists (57%; n = 8) interacted with the first AI-Patient, and roughly a third of therapists (36%; n = 5) interacted with both AI-Patients.

Table 1.

Feasibility Statistics for Therapists Using TherapyTrainer During a Routine Written Exposure Therapy Workshop

Category	M	Mdn	SD	Min.	Max	n
Unique program uses	1.77	2	0.83	1	4	13
Number of WET sessions	2.00	2	1.15	1	4	13
AI-Patients used	1.38	1	0.51	1	2	13
Usage time (total), minutes	14.51	16.62	9.6	0.33	32.78	13
Usage time (per use), minutes	7.66	8.31	4.15	0.33	16.39	13
Usage time (AI-Patient 1), minutes	11.58	10.35	8.67	0.33	32.78	13
Usage time (AI-Patient 2), minutes	6.80	6.27	5.13	0.28	12.03	5
Message count	11.69	9	10.28	2	40	13
Word count	417.38	237	541.88	13	1,643	13

Note. Use is defined as the therapist sending at least one message in the specified context.

Cumulative usage time ranged from 20 seconds and 32.78 minutes (M = 14.51, SD = 9.60, both in minutes). For each unique session, usage time ranged from 20 seconds to 16.39 minutes (M = 7.66, SD = 8.31, both in minutes). On average, therapists spent 11.58 minutes (SD = 8.67) with the first AI-Patient and 6.80 minutes (SD = 5.13) with the second AI-Patient.

Therapists sent between 2 and 40 messages through the TherapyTrainer platform (M = 11.69, Mdn = 9, SD = 10.28) and the number of words written ranged from 13 to 1,643 (M = 417, Mdn = 237, SD = 542).

Acceptability

Half of therapists who completed the workshop (50%; n = 7) provided quantitative acceptability data. These therapists rated both the degree to which the AI-patients seemed realistic, and the extent to which TherapyTrainer prepared them to start working with WET patients, as 3.00 (both SD = 0.82), representing a “great extent.” Lastly, therapists rated the extent to which TherapyTrainer gave them confidence to start using WET as 2.57 (SD = 0.98), in the “moderate” to “great” range.

When we examined the same data coded in a binary fashion, 71% percent of therapists (n = 5) felt that the AI-patients were realistic; 71% (n = 5) felt that TherapyTrainer prepared them to start using WET with patients; 57% (n = 4) felt that TherapyTrainer increased their confidence in using WET. In their qualitative feedback (Table 2), therapists reported finding TherapyTrainer realistic and helpful.

Table 2.

Selected Therapist Feedback on TherapyTrainer

Phase	Constructive	Positive
2	The AI-Consultant’s feedback was tangential to the skills I was trying to practice (e.g., suggested I assess for suicide when the Simulated-Patient mentioned they were "struggling," which felt tangential to the WET skills I was practicing) I didn’t like that the AI-Consultant told me what feedback to give the AI-Patient without first asking me what I thought. It felt like it took some work to figure out how to use TherapyTrainer, to the extent that the time spent actually practicing the skills with TherapyTrainer during the workshop was minimal.	I've never seen anything like this that can offer such targeted feedback on learning a specific therapy. I love that I could interact with it in Spanish! I like this training addition! What a great use of technology.
3	I'm getting distracted by seeing both the AI-Consultant’s feedback and AI-Patient’s responses at the same time. It’s hard to know what to focus on. I didn’t know what balance to strike between responding to the AI-Consultant’s feedback and moving on to the next task. Every comment from the AI-Consultant said, “Good job, however…”; after the sixth comment, the “good job” felt a little invalidating.	I really like the AI-Patient’s responses – they seem very straightforward. The points the AI-Consultant made were substantive and hinted at things I hadn’t done. The AI responded right away - that was amazing. I was not expecting that. I like the task list, it’s kind of like how I already do remote therapy so it’s very reinforcing.

Note. Feedback has been edited for clarity.

Adopting WET into Clinical Practice

Of the therapists who received access to TherapyTrainer alone (n = 7), the majority chose to begin treating patients with WET within a month of the workshop (71%; n = 5), providing preliminary indication that practicing with TherapyTrainer may help therapists feel adequately prepared to begin using WET even in the absence of consultation.

User Experience

Here we summarize themes from Phase 2 user testing; as in Phase 1, user experience feedback informed current and future changes we will make to the AI-Consultant via prompt engineering. Selected therapist feedback from Phases 2 and 3 is presented in Table 2. Users appreciated the AI-Consultant’s knowledge of therapy process factors (e.g., building rapport, checking patient understanding) and its detailed understanding of WET. However, users also highlighted instances in which the AI-Consultant gave guidance that seemed misaligned with the WET approach (e.g., suggesting coping skills or suicidality assessment in response to AI-Patient distress). Users appreciated the AI-Consultant’s emphasis on culturally competent treatment, and one user liked that she could interact with TherapyTrainer in Spanish.

Some therapists found it unnatural to type messages to the AI-patient, stating a preference to communicate via voice. Therapists stated a desire to revise their previously sent messages so they would better align with feedback received from the AI-Consultant. Some users found TherapyTrainer difficult to learn to use, which limited their practice time. Lastly, therapists expressed a desire for a way to access WET training materials (e.g., the manual) from within TherapyTrainer.

Phase 3

To gain more insight into usability challenges faced by therapists, particularly their first time using TherapyTrainer, we conducted structured user testing interviews. During these interviews, several therapists wanted clearer introductory guidance on how to use TherapyTrainer. Several therapists found it distracting and cognitively-burdensome that the AI-Consultant’s messages appeared at the same time and within the same dialogue as the AI-Patient’s messages. Most therapists stated a preference for speaking rather than typing messages, but found the voice-to-text feature inaccurate or difficult to use.

While therapists appreciated the speed and accuracy of the AI-Consultant, some felt overwhelmed by the degree of critical feedback (“It always has something to point out that I did wrong”) and were unsure whether to address feedback in subsequent messages or move on to the next therapy skill.

Therapists appreciated having access to the session task list, but wanted it to have more clinical content, such as written exposure instructions, important psychoeducation points, or video recordings of therapists demonstrating WET skills. Additionally, therapists noted instances in which the order of skills in the task list was misaligned with the AI-Consultant’s instructions.

On the positive side, some therapists stated that TherapyTrainer would be a valuable resource following a WET workshop, and that they found the AI-Patient realistic.

Discussion

Current approaches to training therapists in evidence-based treatments don’t reliably impact therapist use of these treatments, hindering efforts to implement EBTs. Such findings highlight the need for novel therapist training methods that improve rates of EBT delivery and therapist fidelity to the treatments as designed. Simulated patient approaches can allow therapists opportunities to practice therapy skills, receive feedback, and tackle challenging clinical scenarios in a low-stakes environment, which may impact self-efficacy and increase use of exposure-based therapies. We developed TherapyTrainer, a large language model (LLM)-based tool that enables therapists to experientially learn written exposure therapy (WET) through personalized interactions with AI-based simulated patients and an AI-based consultant. Using a three-phase approach, we rapidly prototyped and iteratively developed TherapyTrainer and then gathered usability, feasibility, acceptability, and WET adoption data from both therapists and WET trainer-consultants. Initial results indicate that TherapyTrainer may be a useful training tool to improve therapists’ level of comfort delivering WET. More broadly, this work highlights the potential for generative AI-based therapist training tools to help disseminate EBTs and address the science-practice gap.

What We Learned Developing an AI-Based Therapist Training Tool

Much of the feedback we received during the product prototyping and rapid iteration phase (Phase 1) centered on the AI-Consultant’s feedback. Therapists wanted feedback that was succinct, centered on one skill rather than many, and focused on WET rather than general therapy skills. Therapists appreciated having treatment-specific information (e.g., session-level overviews) and patient-specific details (e.g., demographics, trauma history) embedded in the user interface, perhaps mimicking notes they might bring to session. Feedback also underscored the value therapists place on skill practice: Therapists wanted to tackle challenging scenarios, like patient suicidality, and to have a venue for repeating individual WET skills on an ad hoc basis. Taken together, this feedback provides valuable insight into therapists’ values and priorities when practicing EBTs and receiving feedback.

During Phase 2 and 3 user testing, a central theme that emerged was the challenges therapists faced figuring out how to use TherapyTrainer. This underscored the importance of enhancing learnability and reducing cognitive load, two aspects of usability identified by the human-centered design field (​​Lyon & Koerner, 2016). However, prioritizing usability while responding to user suggestions at times proved challenging. In response to user suggestions, we added a voice-to-text feature, but its usability was poor, owing to our choice to make use of quickly-implementable rather than state-of-the-art technology. This tension highlights the challenge of maintaining high usability while engaging in rapid, responsive development.

During earlier rounds of testing, users noted instances in which the AI-Consultant recommended content that conflicted with the WET protocol, an impact of LLMs’ pre-training on wide-ranging internet text (Stade et al., 2024b). Following prompt engineering refinements, users no longer noted these issues, seeming to indicate that our changes improved the AI-Consultant’s fidelity to WET. However, during subsequent testing, several users perceived the AI-Consultant as overly critical; though the feedback itself was appropriate to WET, the AI-Consultant frequently gave instruction on too many different WET skills at once, and didn’t give users a chance to first practice skills themselves. This highlights a tension in building AI consultants: They must possess detailed, treatment-specific knowledge while delivering feedback with the right degree, form, and timing to effectively guide trainee development.

The supervision literature emphasizes tailoring teaching to the trainee’s current knowledge base, and eliciting trainees’ ideas before providing feedback, a technique known as scaffolding (Cummings et al., 2015). Future improvements to TherapyTrainer could align the AI-Consultant more closely with scaffolding and other supervision techniques believed to encourage deeper learning (Reiser et al., 2018). In particular, engineering the AI-Consultant to be more Socratic would align with user preferences for feedback fostering reflection and discovery, rather than merely providing instruction.

Strengths of an AI-Based Therapist Training Tool

Ideal Training Qualities Made Scalable

We developed TherapyTrainer to have qualities expected to improve therapist EBT outcomes, including using active learning strategies, allowing for frequent, personalized practice, and providing therapists with feedback on their work. Given its use of generative AI, TherapyTrainer offers training opportunities rarely possible using human consultants or simulated patients, such as simulating full EBT cases and receiving detailed, utterance-level feedback on entire simulated therapy sessions, rather than on recorded session segments or therapist verbal reports.

Flexibility of Language

TherapyTrainer also offers the ability for therapists to interact with it in multiple languages. Although further testing is needed to ensure that TherapyTrainer’s accuracy and culturally-responsiveness across languages, leverging LLMs’ flexibility of language could offer a major advantage by requiring less time and fewer resources than training multilingual consultants or translating content for web-based trainings.

Patient Outcomes

AI-based therapist training solutions may benefit patients outcomes, especially given that therapists’ second and subsequent patients in an EBT improve more than their first (Johnson et al., 2022; Mallard Swanson et al., 2021, Wiltsey Stirman, 2022). Thus, models like TherapyTrainer, which allow therapists to complete simulated “practice cases,” may accelerate therapist skill development and improve outcomes for actual patients treated.

Therapist Adoption of WET

Notably, a high proportion of clinicians in our study who received TherapyTrainer alone (without traditional consultation) began seeing WET cases immediately after the workshop. Though more rigorous research would be needed before drawing firm conclusions, this hints that using TherapyTrainer could improve therapists’ confidence and could shorten their time to begin using WET with patients. Reducing therapist latency without the use of traditional consultation would be particularly impactful, as previous work in the cognitive processing therapy (CPT) literature has highlighted that therapists who don’t receive consultation are less likely to use CPT with patients (Monson et al., 2018). However, research is needed to test whether receiving TherapyTrainer alone actually reduces latency, or if other reasons might explain why clinicians in our study who didn’t receive consultation had relatively high rates of WET uptake. It is possible that consultation plays a larger role in therapist-uptake of treatments that are relatively complex, like CPT, which requires mastery of more therapist behaviors than WET.

Future Directions

Improvements to TherapyTrainer

Usability.

All three phases of user testing highlighted key usability issues (Lyon & Koerner, 2016) that likely limited therapists’ ability to practice and improve their WET skills using TherapyTrainer. To address this, planned future development of TherapyTrainer will focus on usability issues, including streamlining user onboarding, improving speech-based interaction, simplifying the user interface, and titrating the degree of critical feedback from AI-Consultant.

Assessing Therapist-Level Behavior Change Mechanisms.

Prior research has identified candidate therapist-level mechanisms that enhance EBT use and patient outcomes, including self-efficacy, competency, and positive expectations about patient outcomes (McLean et al., 2024; McLeod et al., 2018; Ruzek et al., 2016). To test whether TherapyTrainer engages these mechanisms, future work could monitor these mechanisms using built-in questionnaires or passive, language-based assessment methods (Park et al., 2015). Automated measuring of mechanisms would also allow for tailoring content based on results (e.g., offering a second beginner-level practice case to a therapist with low self-efficacy). Future research could also examine the relationship between use of TherapyTrainer and these mechanisms, exploring potential dose-response relationships between usage metrics (e.g., program usage time, number of WET sessions completed) and therapist-level outcomes.

Automated Fidelity Monitoring.

One therapist-level mechanism that may be particularly important to monitor within TherapyTrainer is treatment fidelity. Measuring fidelity using simulated patients has been tested previously (Imel et al., 2014), and automated fidelity evaluation may be possible by applying language-based fidelity assessment (e.g., Shah et al., 2022) to therapist language recorded by TherapyTrainer. Adding therapist-level fidelity monitoring could aid in identifying therapists needing additional support, enabling more parsimonious use of consultation calls in a stepped-care-like model. Similarly, incorporating a separate agent to monitor the AI-Consultant’s fidelity could serve as a quality assurance mechanism, helping ensure its feedback remains consistent with the WET protocol.

Levels of Feedback.

TherapyTrainer allows therapists access to “bug in the eye” style (Vezer, 2021) fine-grained feedback on their performance delivering WET. Having access to scalable yet intensive feedback raises the issue of optimal feedback levels. It may be possible to adjust feedback intensity along a continuum, allowing supervisors, trainers, or therapists themselves to tailor it as needed. However, it is an empirical question whether titrating levels of feedback will result in improved outcomes.

Adding Clinical Content.

Therapists requested the ability to review treatment-specific concepts and view video clips of workshops and sample therapy sessions within TherapyTrainer. As the WET manual specifies that therapists should read specific scripts aloud, such as the psychoeducation content or the introduction to the writing assignments, other users requested the WET “scripts” to be built into TherapyTrainer. Including additional challenging scenarios -- such as acute suicide risk, substance use, dissociation, difficulties selecting an index trauma, and client ambivalence about continuing the protocol – may help therapists overcome apprehension and boost their confidence and willingness to deliver trauma-focused treatments to more clients. Adding these features could make TherapyTrainer a “one stop shop” for WET training.

On Demand Training “Cases.”

TherapyTrainer, if proven to be a sufficient replacement for human training cases in building therapist competence, may someday play a role in provider certification processes. Therapists could complete additional AI-based cases, circumventing the need to recruit and retain human patients for training cases and eliminating the risk of drop out. This could help address an existing inefficiency in therapist certification: In WET training programs, therapists may attend consultation for over a month before identifying training cases, and may require extensions to complete these cases (Worley et al., 2020).

To realize this vision, a higher degree of trainer or consultant oversight over training cases would likely be beneficial. A trainer-facing “dashboard,” containing metrics like number of sessions completed and response latency as well as summaries of therapists’ strengths and weaknesses, could give trainers insight into therapist performance. If implemented, this could surpass current EBT training practices in comprehensiveness, as TherapyTrainer records all interactions between the therapist and AI-Patient. Adding automated fidelity monitoring (described above) to the dashboard could give consultants real-time insights into therapists’ understanding and application of the protocol. Ultimately, this approach could provide a more structured, scalable, and thorough method for supporting therapist skill assessment.

Extension to Other EBTs

Though we developed TherapyTrainer using WET, we plan to extend it to other EBTs. Candidate treatments include other highly structured protocols, whose clear steps facilitate training the AI-Consultant to deliver high-fidelity, concrete feedback. Brief interventions (Schleider & Weisz, 2017) are preferable, due to the significant prompt engineering efforts needed to train the AI-Consultant on each individual treatment element. Behaviorally-focused treatments like CBT for insomnia (Perlis, 2008) may also be preferable, as the criteria for what constitutes high-quality behavioral interventions are often relatively straightforward. However, early evidence suggests that LLMs can successfully guide therapists to deliver difficult-to-master cognitive interventions, like Socratic dialogue (Held et al., 2024).

Other treatments that require therapists to provide targeted, client-facing feedback may integrate well with TherapyTrainer. Here, we created a prototypical client work sample (the trauma narrative), outlined the ideal feedback a therapist should give about the narrative, and trained the AI-Consultant to guide therapists towards that ideal feedback. This process could extend to treatments like parent-child interaction therapy, where therapists give parents feedback on their interactions with their child (McNeil & Hembree-Kigin, 2011).

Bridging the Gap: From Prototype to Practice

While our findings suggest that TherapyTrainer has potential as a training tool for WET, further development is needed to support its effective integration into therapist training efforts.

Engagement.

Therapists in our study engaged with TherapyTrainer only modestly: On average, they completed 2 out of 5 WET sessions, used the platform for approximately 15 minutes of the allotted time, and sent just 12 messages. While more research is needed to understand the drivers of this limited engagement, results from our Phase 3 in-depth user testing suggests that therapists faced difficulties orienting to the tool. These startup challenges may have consumed much of therapist’s allotted practice time, reducing opportunities to engage deeply with the platform.

To support practice with full cases and better learning outcomes, therapists may need additional scaffolding and structure. As discussed above, a trainer-facing dashboard could help consultants to monitor therapist progress and view therapist fidelity to WET. Making case completion a formal requirement within WET workshops may also improve usage. Additionally, our study offered two separate small blocks of time for practice, but it may be more effective to provide a single, uninterrupted block to help therapists build momentum with the tool. Alternatively, integrating smaller, skill-specific practice segments into training programs might be preferable, allowing therapists to build mastery over time. Future work should explore the optimal way to integrate TherapyTrainer into EBT workshops, balancing active skill practice with time needed for didactic instruction.

Implementation challenges.

Before TherapyTrainer can be more fully integrated into EBT training processes, other system-level barriers will have to be addressed. These include access restrictions on systems such as VA computers and limited time for additional training due to high clinical caseloads (Chard et al., 2012). One potential strategy to promote adoption is to offer continuing education (CE) credits for practice using TherapyTrainer, providing a concrete incentive for participation.

Future Empirical Studies

Down the line, randomized clinical trials will be needed to establish the effectiveness of TherapyTrainer. Assessing several different outcomes will be important. Ideally users would show improvement in subjective metrics, like therapists’ sense of preparedness and confidence in delivering the EBT, as well as objective metrics, like number of EBT cases initiated and completed, and latency to EBT adoption. More distal metrics, such as patient symptom reduction, would also be of interest.

Ideal comparison conditions will need to be determined. In one sense, it would be ideal to randomize therapists to use TherapyTrainer or consultation-as-usual, with the hypothesis that TherapyTrainer could achieve noninferiority with far fewer resources. A robust investigation of the effects of TherapyTrainer would also compare this product to access to a non-specialized generative AI chatbot (i.e., one that has not been tailored to provide specialized training and consultation in an EBT). This would allow us to rule out the possibility that effects of TherapyTrainer are attributable to nonspecific aspects of generative AI, such as 24/7 accessibility or the helpful nature of generative AI. On the other hand, there are some reasons to suggest that using TherapyTrainer could result in superior outcomes. Given the effect of therapist practice on patient outcomes (Wiltsey Stirman, 2022), therapists who use TherapyTrainer before integrating WET into practice may achieve better patient outcomes with their first case than those who do not.

Conclusion

In conclusion, TherapyTrainer represents a promising advancement in therapist training, leveraging generative AI to address persistent gaps in the implementation of evidence-based treatments for PTSD. By enabling therapists to practice delivering written exposure therapy in a realistic, low-stakes, and flexible environment, TherapyTrainer may help address critical barriers such as insufficient ongoing consultation and limited opportunities for skill practice. Initial results indicate that TherapyTrainer is feasible and acceptable and may have the potential to reduce the latency between training and clinical implementation. Limitations of our work include modest therapist engagement and a lack of experimental evidence demonstrating TherapyTrainer’s efficacy; more research is needed to rigorously evaluate TherapyTrainer, including its impact on therapist fidelity, EBT adoption rates, and patient outcomes. TherapyTrainer holds promise as a scalable, cost-effective solution for bridging the science-practice gap and ultimately improving patient access to effective treatments.