The impact of fine-tuning LLMs on the quality of automated therapy assessed by digital patients

Abstract

The use of generative large language models (LLMs) in mental health applications is gaining traction, with some proposals even suggesting LLM-based automated therapists. In this study, we assess the impact of fine-tuning therapist LLMs to improve the quality of therapy sessions, addressing a critical question in LLM-based mental health research. Specifically, we demonstrate that fine-tuning with datasets focused on specific therapeutic techniques significantly enhances the performance of LLM therapists. To facilitate this assessment, we introduce a novel evaluation system based on digital patients, powered by LLMs, which engage in text-based therapy sessions and provide session evaluations through questionnaires designed for human patients. This method addresses the inadequacies of traditional text-similarity metrics, which are insufficient for assessing the quality of therapeutic interactions. This study centers on motivational interviewing (MI), a structured and goal-oriented therapeutic approach. However, our digital therapists and patients can be adapted to work in other forms of therapy. We believe that our digital therapists offer a standardized method for assessing automated therapists and showcasing the potential of LLMs in mental health care.

Introduction

The rapid advancements in large language models (LLMs) have introduced new opportunities for their application in clinical psychology. Some recent studies have proposed that LLMs could function as automated therapists¹, and certain commercial applications have already started using them in this capacity; however, it is not yet clear to what extent these models are fully capable of fulfilling this role.

The most common approach to using an LLM in a conversation, such as in therapeutic settings, is to simply start chatting with it by prompting the LLM with relevant instructions on how to respond. However, this approach is only effective in therapeutic settings if there is confidence that LLMs have the skills and knowledge required to offer therapy. An alternative approach is to fine-tune the LLM using relevant datasets, which can provide the appropriate context for improving its therapeutic skills. In this study, we examine the impact of this fine-tuning approach, using different types of datasets, on the performance of the LLM as an automated therapist. To evaluate the performance of the LLM, we initially employed standard metrics commonly used to measure the quality of LLM-generated text. However, we found that these metrics are not well-suited for therapeutic settings. Therefore, we developed a new approach for evaluating chat-based therapy sessions, utilizing LLMs to play the role of patients. Each digital patient is created with specific characteristics that come into play during the therapy session with the therapist. Once the session concludes, the digital patients are asked to answer a series of questions about the quality of the session. We convert their answers into a score, which we demonstrate to be correlated with clinicians’ evaluations of the session.

Throughout the study, we focus on motivational interviewing (MI), a psychotherapeutic technique designed to help individuals resolve their ambivalence toward behavioral change through a collaborative and client-centered approach². We chose MI because it is structured and can be used in one or two sessions, aligning with our goal to focus on a one-session therapy technique for this study. More complex techniques requiring multiple sessions are reserved for future work. Traditionally, MI sessions are evaluated by mental-health professionals using specific coding and evaluation frameworks, like the Motivational Interviewing Skills Code (MISC) (https://casaa.unm.edu/tools/misc.html) and the Motivational Interviewing Treatment Integrity (MITI) (https://motivationalinterviewing.org/sites/default/files/miti4_2.pdf). These coding frameworks are designed to capture the nature of responses given by the therapist during their conversation with the patient. Using these coding frameworks for evaluation is labor-intensive, as it requires professionals to read through the conversations and assign codes to each utterance. Furthermore, randomized control trials intended to evaluate clinical protocols are exceedingly costly and time-consuming due to the human burden. Given this context, LLM-based evaluation appears timely.

There is growing interest in applying LLMs in psychology, opening up exciting new opportunities for innovation in mental health and psychological counseling. In a recent perspective, Demszky et al.³ provide an overview of how LLMs can be beneficial in the field of psychology, particularly for improving measurement, diagnosis, and treatment methods. The authors address several challenges associated with the use of LLMs in this context and emphasize the necessity for further research to fully realize their potential in psychological applications.

In a recent paper, Kang et al.⁴ focused on improving LLMs for providing psychotherapy by fine-tuning them on domain-specific instructions derived from therapy and counseling data. They evaluate the effectiveness of these fine-tuned models in providing therapy-related responses and demonstrates the potential of LLMs in automating certain aspects of psychotherapy. However, the evaluation is done using standard metrics like text similarity and coherence, along with human assessments based on questionnaires typically used in psychotherapy to measure the effectiveness of therapist-patient interactions; our work indicates the limitations of this approach, as well as suggesting a solution. On the therapeutic side, Liu et al.⁵ introduced “ChatCounselor”, an LLM developed to provide support in diverse mental health contexts. To assess its performance, the authors used OpenAI’s GPT-4 as a benchmark. They curated a set of targeted questions to evaluate ChatCounselor’s capabilities by comparing its responses to those of GPT-4. Chiu et al.⁶ proposed BOLT, a computational framework to study the conversational behavior of LLMs in therapy. Their analysis of GPT-3 and some other LLMs revealed that these models often resemble behaviors more commonly seen in low-quality therapy. They also demonstrated some unique characteristics, such as reflecting more upon clients’ needs and strengths. The study suggested that while LLMs have potential as therapists, further research is needed to ensure quality care.

As highlighted in the overview by Demszky et al.³ and later expanded by Stade et al.⁷, a significant challenge in using LLMs for mental health applications is evaluation.

Traditional evaluation techniques, which focus on text generation tasks using similarity functions, are deemed insufficient for psychology-related applications. Demszky et al.³ thus propose two alternative methods for a more effective evaluation: (1) Expert evaluation, which involves mental-health professionals assessing the model’s output, considering their expertise and professional judgment; and (2) Impact evaluation, a method to evaluate the model’s output based on its effect within the context of a specific psychological task, focusing on the practical impact of the language model’s contributions. Ji et al.⁸ drew similar conclusions, particularly focusing on the application of LLMs in mental health. They stressed the importance of a judicious and considerate approach when utilizing LLMs in this domain. Their perspective is that LLMs should be seen as tools that compliment, rather than seek to replace, human expertise in mental health.

Lai et al.⁹ presented Psy-LLM, a framework designed to use LLMs as assistive tools for mental health professionals. The framework provides mental-health professionals a useful interface for asking professional questions about relevant topics. Recent studies have begun exploring the possibility of LLMs as substitutes for human participants in psychological settings, mainly for training and evaluation purposes. Dillion et al.¹⁰ explore the potential and caveats of replacing human participants by LLMs, and provide an example case study indicating that LLMs are highly correlated with humans in moral judgment. They discuss the need to simulate multiple “personalities”, which we address further below. Aher et al.¹¹ demonstrate a range of such studies in which LLMs replace human participants, such as ultimatum game, linguistics, replicating Milgram’s obedience studies, and “wisdom of the crowds”. Similarly to our approach, the input to the model includes some demographic information and some instructions about the task, and the model is expected to perform the task using a relatively simple zero-shot prompting technique. In our work, we suggest that this line of research, investigating the viability and effectiveness of LLMs in roles traditionally filled by humans, can be extended to areas such as therapeutic interactions, diagnostic processes, or other mental-health text-based tasks.

Evaluating the performance of LLMs in dialog generation raises some unique challenges. Unlike tasks with clear-cut answers, dialogs inherently involve subjectivity, nuance, and a need for contextual understanding. The complexity of dialog evaluation is compounded by the necessity to assess not just factual accuracy, but also the relevance, coherence, and emotional intelligence of the responses. While there are established metrics for evaluating various aspects of language models, their applicability to dialog generation, especially in therapeutic contexts like our study on LLM-generated motivational conversations, is limited.

BERTScore¹², a widely used metric for text similarity, leverages the BERT¹³ language model to calculate a similarity score between the generated text and a reference text. This metric evaluates similarity by comparing the contextual embeddings of words in both the reference and generated texts, calculating their cosine similarity. It is effective for tasks where reference texts are available for comparison. However, in our case of generating therapeutic conversations from scratch, we lack these reference points. Similarly, MAUVE¹⁴ analyzes the quality of generated text by comparing the distribution of latent representations of the generated text with a set of reference texts. It uses statistical techniques to measure how closely the generated text aligns with the style and content of the references. While insightful for tasks with ample reference material, MAUVE’s effectiveness diminishes in our scenario. Given the unique and individualized nature of each therapeutic conversation, assembling thousands of accurate reference examples is impractical.

Giorgi et al.¹⁵ propose metrics grounded in established psychological theories of human communication and relationships. They show that these metrics are uncorrelated with traditional natural language processing (NLP) metrics, such as BERTScore, indicating that they capture complementary information.

Across these studies, we found no established method for evaluating therapy sessions that could be applied consistently across different therapeutic approaches and conditions. To measure the impact of fine-tuning a therapist LLM on related datasets in order to establish more concrete therapeutic knowledge within the model, we address this gap by proposing a new evaluation platform for therapist LLMs, which is an extended and more validated version of our previous work¹⁶. This platform utilizes digital patients powered by LLMs and is validated within the context of motivational interviewing.

In this work, we assume that an automatic therapist is built using a generative LLM. To evaluate its performance, we begin by reviewing existing automated methods for assessing the quality of machine-generated text. After highlighting the limitations of these metrics in therapeutic contexts, we introduce a novel approach for evaluating automatic therapists. Lastly, we develop ten distinct therapist LLMs and assess their therapeutic capabilities using our proposed evaluation method.

In summary, our contributions include the following: (1) We measured the impact of fine-tuning an LLM on various types and sizes of datasets, assessing its performance as an automated therapist, with a specific focus on MI. Our findings indicate that fine-tuning with datasets centered on specific therapeutic techniques yields the most effective results. (2) We developed and validated a new evaluation approach for chat-based therapy sessions using digital patients, initially focusing on MI.

Methods

Evaluating therapist LLMs using established automated metrics

As a first try to assess the quality of the text generated by the therapist LLM during a therapeutic session, we used several existing automatic text-similarity metrics commonly used in text-generation tasks. Such metrics work by comparing the generated text to a reference text. However, it is difficult to compare a full conversation between a patient and a therapist to a reference transcription, since the conversation may change depending on the response generated by the therapist. Therefore, we came up with a method that allows us to use text-similarity metrics, based on the following recipe:

• We generated conversation fragments from reference therapeutic sessions.

• Each fragment consisted of an even number of utterances, where an utterance refers to a single turn in the conversation. The therapist always initiated the first utterance, followed by the patient, with the two alternating throughout, and the final utterance always made by the patient.

• Each therapist is then given the fragment as input and asked to generate the next response. The generated utterance is then evaluated against the reference utterance, taken from the original session, using a text-similarity metric.

The dataset that we use for generating those conversation fragments is called MI-TAGS¹⁷. It contains 242 MI conversations, originally made for demonstration and training purposes. The conversations are fully annotated using the known MITI coding scheme; however, we only use the text and do not use the codes. Overall, from the 242 sessions, we generated 6652 conversation fragments.

To evaluate the sensitivity of each automatic text-similarity metric, we developed three therapist agents, each incorporating a different level of MI skill. These agents are not meant to be fully functional or state-of-the-art LLM-based professional therapists; instead, they are designed to represent extreme cases that we believe an evaluation system should be able to distinguish.

The three types of therapists are categorized as poor, average, and expert, each tailored to demonstrate varying levels of empathy, understanding of the client’s perspective, and therapist expertise relevant to MI. These distinctions specifically address the therapists’ attitude toward the client’s ambivalence about change, ranging from judgmental to acceptance, in accordance with the definition of therapist expertise outlined in ref. ². These dimensions are typically assessed by professionals using the coding frameworks we previously discussed. Below are more details on the therapist categories we created for this study:

• Poor therapist: programmed to exhibit a poor understanding of the client’s perspective, a lack of empathy, and displays judgmental attitudes towards clients’ ambivalence.

• Average therapist: represents an average level of understanding regarding the client’s perspective, typically partnering with the client to explore his/her ambivalence, yet with occasional expressions of judgment and lapses in empathy.

• Expert therapist: exemplifies ideal understanding of the client’s perspective, characterized by deep empathy, acceptance, and non-judgmental support.

Each therapist characteristic profile is created using detailed prompt engineering, ensuring consistent and distinct behavior aligned with their designated skill level. The prompt is designed to facilitate dynamic interactions, allowing the therapist to respond to a wide range of patient characteristic profiles and scenarios. One of the authors (M.Z.), an experienced clinician in MI (hereafter referred to as the expert), evaluated 10 conversations conducted by each of the three therapists with a single patient, also modeled by an LLM (as described in “Evaluating therapists with digital patients”). This process was used to iteratively refine the prompts until the three expertise levels were accurately distinguished in alignment with their definitions. The final system prompts given to the LLM to create the therapists appear in Supplementary Fig. 1, 2 and 3. For evaluation, we use five text-similarity metrics.

BLEU (Bilingual Evaluation Understudy)¹⁸. Used in machine translation. It compares candidate translations to the reference gold translation, assigning a score between 0 and 1 (higher is better).

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)¹⁹. Typically used to assess the quality of automated summaries or translations. It works by comparing the generated text to a set of reference summaries. We use the ROUGE-1, ROUGE-2, and ROUGE-L variations, based on unigrams, bigrams, and longest common subsequence, respectively.

METEOR (Metric for Evaluation of Translation with Explicit Ordering)²⁰. Typically used in machine translation. Unlike BLEU, it considers synonyms, word stems, and sentence structure when comparing machine-generated translations to human references. This often results in scores that better reflect human judgments of translation quality.

BERTScore¹². Generally used in text-generation tasks. It uses BERT²¹, an encoder language model, to compare the generated text to a reference text. Instead of relying on exact word matches, it uses text embeddings, which make it more sensitive to synonyms and paraphrases, often resulting in scores that better match human judgment.

MAUVE¹⁴. Generally used in open-ended text generation tasks, MAUVE directly compares the distribution of machine-generated text to the reference text. Essentially, it quantifies the difference between these two distributions, providing a nuanced assessment of the quality and diversity of the generated text. This makes MAUVE more sensitive to capturing stylistic and distributional nuances in the generated text, aligning its scores closer to human evaluations. To use MAUVE, one needs to generate the output distributions typically used by the decoder algorithm to generate the next token. These distributions are usually referred to as soft targets. Therefore, only models that can provide these soft targets can be evaluated with MAUVE. In our work, we typically use GPT-3.5, which only provides hard targets. To obtain soft targets, we use other models, such as GPT-2 and LLama-2, by running them on the decoded text generated by GPT-3.5.

The results are detailed in the “Results” section.

Evaluating therapists with digital patients

Our alternative approach for therapy assessment involves creating distinct patient characteristic profiles through prompt engineering for an LLM.

Altogether, we constructed 96 unique patient characteristic profiles, varying across multiple parameters such as gender, age, targeted behavioral change (such as smoking or obesity), the duration of the habit, previous attempts at managing it, and the level of cooperation in motivational sessions.

To test the validity of our approach, the patients engaged with three types of therapists, identical to the ones described in the previous section.

The conversations between a therapist and a patient were generated carefully, with each interaction generated utterance by utterance. In this context, an utterance refers to one speech turn in the conversation. This approach ensures that every utterance not only logically follows the previous one but also stays true to the distinct patient characteristic profiles. Importantly, the LLMs used for the therapist and patient in each interaction were different and independent, allowing for authentic responses in line with their predefined role in the conversation. Therefore, substituting the LLM-based therapist with a human therapist who interacts through a chat console represents the logical progression and something we aim to explore in future work.

Unless stated otherwise, we use OpenAI’s GPT-3.5-turbo-1106 to run the experiments.

We included instructions for ending the conversation in both the patient and therapist prompts. Once the conversation was concluded, we submited it for evaluation. This evaluation involves providing the entire conversation transcript to a separate patient LLM, along with specific instructions, as follows. This LLM is tasked with completing two questionnaires: one assessing session satisfaction and another evaluating the working alliance between the patient and therapist. These questionnaires are similar to those typically filled out by human patients after therapy sessions. To clarify, the patient LLM receives the full conversation transcript as input, plus guidelines on how to respond to the two questionnaires. We provide more details on the evaluation step below.

The patients in our study are designed with a set of key parameters, each contributing to the distinctiveness of the patient’s characteristic profile. These parameters include:

• Gender: male or female.

• Age: old or young.

• Problem: the patient is dealing with: smoking or obesity.

• Duration of the problem: a few months or many years.

• Efforts to solve the problem: never attempted or attempted many times.

• Cooperation level: low, high, and starts low and gradually increases during the conversation.

There are 96 combinations of parameter settings, with each one representing a unique patient characteristic profile, characterized by a distinct set of challenges and attitudes towards counseling.

The system prompt is implemented with a template such that the options above are filled in; we provide an example of a system prompt for a patient with a specific characteristic profile in Supplementary Fig. 4.

The conversation is generated step-by-step, where each step produces one utterance. The process begins with providing the therapist’s system prompt to the LLM, which then generates the first utterance. After the first utterance is produced, we provide the patient’s system prompt to the LLM, but this time it is concatenated with the therapist’s initial utterance. Importantly, each step involves a fresh instance of the LLM without any memory from the previous step, except the conversation generated so far. In other words, the complete context needed for each step is contained within that step’s specific prompt. In the third step, we use the therapist’s system prompt again, now adding it to the entire conversation generated so far. We continue this process step by step, alternating between the system prompts of the therapist and the patient, each time appending the full ongoing conversation. The generation process is over when either the therapist or the patient indicates the end of the session, as per the guidelines set out in the prompt. Figure 1 presents the start of a conversation, illustrating a specific patient beginning therapy with the digital expert therapist.

Fig. 1.

The beginning of a therapy conversation generated by a digital patient and a therapist LLM.

As mentioned above, once a conversation is concluded, we change the prompt and pass it on to a fresh LLM, instructed to respond to two questionnaires aimed at evaluating the quality of the treatment provided by the therapist. For each questionnaire, all the questions are combined into a single, comprehensive prompt. The prompt includes a detailed description of each question and some general instructions for the model, which are concatenated with the content of the therapy conversation. The LLM is directed to provide its responses on a scale ranging from 1 to 5. These responses are then extracted from the LLM’s output and systematically organized into a questionnaire results sheet, enabling a structured assessment of the therapy session’s quality. The first questionnaire contains five questions regarding the overall satisfaction of the session, inspired by²². The second questionnaire assesses the working alliance between patient and therapist, and it contains 17 questions taken from²³.

The full set of questions is provided in Supplementary Table 1. In Supplementary Fig. 5, we provide an example of the prompt used for evaluation. The prompt is written for the LLM as if it were a professional therapist evaluating a conversation between a therapist and a patient. We designed it this way to align with how the human clinicians evaluated the same conversations, ensuring comparability for evaluation purposes.

In order to demonstrate the performance of our digital patients, we conducted an experiment where these patients were involved in MI sessions with the three therapist LLMs, introduced in a previous section.

In our study, every possible combination of the six patient characteristics is utilized, resulting in 96 distinct patient prompts. For each unique set of patient characteristics, we conduct three separate dialogs, corresponding to each category of therapist skills: poor, average, and expert. In total, this approach leads to the generation of 288 conversations, calculated as 96 × 3 = 288.

We collected the responses to the two questionnaires and organize them into a structured table. We then analyzed the results, focusing on reliability and validity measures.

To assess reliability, we report Cronbach’s alpha, calculated from the responses the LLM provided to the questions in the two questionnaires. This statistical measure is commonly used to evaluate the reliability of a questionnaire. In our study it offers insights into the internal consistency of the LLM’s responses. Importantly, we included two reversed questions in the model test, a technique often used in questionnaires to check for response bias and ensure participant attention.

For validity, we first tested whether the model can distinguish among the three levels of therapist skills. In other words, we examined whether the answers given to the questions for conversations with poor, average, and expert digital therapists reflect these quality differences, and if so, are the differences among the levels significant. Additionally, three psychologists with expertise in MI reviewed a randomly selected subset of 30 conversations and responded to the same questions from the two questionnaires, on behalf of the (digital) patients. The evaluators were not aware of the category of the digital therapist in each conversation. Next, we compared the responses of the evaluators with those generated by the LLM, using basic correlation metrics to assess the alignment between the evaluators’ responses and those of the model. This comparison helps in determining the extent to which the LLM’s responses are valid and aligned with professional judgments in the context of MI.

Fine-tuning a therapist LLM

There has been ongoing debate about the most effective approach for applying LLMs to a specific downstream task-whether it is better to fine-tune the model, rely solely on prompting, or combine prompts with external knowledge^24–28. Inspired by these discussions, we investigated the impact of fine-tuning a therapist LLM on the quality of the therapy it provides. We employed the same evaluation technique described in the previous section, using 96 patient LLMs, each designed with a unique profile, to engage in conversations with the therapist LLM, fine-tuned under different conditions. After each interaction, the patient LLMs respond to the same two questionnaires as before.

For this use case, we utilized Meta’s open-source LLama-2 model, experimenting with two sizes: 7 billion and 13 billion parameters. We chose not to use OpenAI’s closed models because, for this set of experiments, we require transparency in knowing that the LLM was not previously trained on the datasets we intend to fine-tune it on. Additionally, we needed the flexibility to fine-tune the models multiple times without incurring high costs.

In total, we created ten digital therapist LLM variations: the two original pre-trained models, used as baselines, and eight fine-tuned versions. All ten variations are configured with the same prompt we used for the expert digital therapist from the previous section. The fine-tuning step is performed by taking the text completion approach, where we provided a conversation fragment to the therapist model and adjust its weights to generate the response that follows in the actual session from which the fragment was extracted. Further details on the fine-tuning process are provided in a subsequent section.

A key component of our approach was the use of two strategically selected datasets for the fine-tuning process. The first dataset, MI-TAGS¹⁷ (denoted as MI), which we have already mentioned in a previous section, was recently released by members of our lab and was chosen due to its close alignment with MI, our primary therapeutic task. The second dataset was sourced from Alexander Street (denoted as AS), featuring a collection of transcripts of more general psychotherapy sessions. These transcripts encompass a wide range of therapeutic approaches and client populations, offering a broad perspective on real-world therapeutic interactions. The dataset’s strength lies in its diversity, capturing the nuances of different therapeutic modalities and client presentations. Typically, this dataset is used as a valuable resource for researchers and clinicians interested in studying the dynamics of psychotherapy, identifying effective therapeutic techniques, and understanding the complexities of client-therapist interactions. By analyzing the transcripts, researchers can gain insights into the factors that contribute to successful therapeutic outcomes, as well as the challenges and obstacles that may arise during the course of therapy. Additionally, the dataset can be used to train and evaluate automated therapy systems, ensuring that these systems are equipped to handle the diverse range of client needs and therapeutic scenarios encountered in real-world practice.

This deliberate contrast between the two datasets was designed to explore the impact of using highly relevant versus mildly relevant data on the performance of the therapist LLM.

For evaluation, we used our digital patients described in the previous section, hosting 96 unique patient LLMs, still implemented using OpenAI’s GPT-3.5-turbo-1106 model.

We followed OpenAI’s guidelines and used three message roles in constructing our chat session: system, user, and assistant. The system role is specifically designed to initiate the therapeutic session, establish a clear framework, and provide overarching guidance to the therapist. The user role simulates a patient interacting with the therapist, and at the core of the interaction lies the assistant role, which embodies our therapist LLM. In our approach to fine-tuning, each entry in the training set comprises a sliding window of a therapy session of six utterances, equivalent to three conversational exchanges (Fig. 2). In our fine-tuning process, we provided the model with the first five utterances of the window. The model then generated the subsequent, final utterance token by token, during which we update the model’s weights based on the generated response to align with the actual response from the session.

Fig. 2.

An example of a single instance for fine tuning the therapist LLM. Each instance comprises a sliding window of six utterances from the original conversation.

We fine-tuned our models on 2 Galax 3090 GPUs, employing FlashAttention-2 for accelerated attention mechanisms and QLoRA²⁹ for efficient model adaptation. We used the hyperparameters outlined in Tables 1 and 2 to guide the training process.

Table 1.

QLoRA hyperparameters and trainable parameter reduction during fine-tuning

	Llama-2 7B	Llama-2 13B
Alpha	128	128
Dropout	0.05	0.05
Rank	256	256
Trainable params	43M (0.61%)	59M (0.45%)

Table 2.

Hyperparameters used during fine-tuning of both LLM sizes

Epochs	2
Batch size	2
Learning rate	2e-4
Precision	bfloat16
Attention	Flash-Attention 2

Our primary focus in this use case is to study the impact of fine-tuning strategies aimed at optimizing the therapists’ dialog proficiency within the therapeutic context. Therefore, we fine-tune both the 7B and 13B parameter variants of Llama-2 in order to evaluate the impact of the model size. Additionally, we use four dataset combinations, as described below:

• Training set 1 (MI Focused): This training set is composed exclusively of 6000 samples (here, a sample is a window of six utterance, as described above) from the MI-TAGS dataset. Its focus on language associated with MI aims to provide therapists with a deep understanding of this therapeutic method’s core communication dynamics.

• Training set 2 (AS Focused): To introduce a broader therapy foundation, this training set utilizes 6000 samples drawn exclusively from the AS dataset, which contains conversations from various therapeutic disciplines. By fine-tuning the therapist model on this dataset, it is exposed to diverse conversational styles, potentially enhancing its overall fluency and communicative adaptability across a wide range of psychological interventions, rather than being limited to MI.

• Training set 3 (AS [Ext.]ended): To investigate whether a larger volume of general counseling data translates to even greater therapeutic proficiency, this training set expands upon the previous one, comprising 20,000 samples from the AS dataset.

• Training set 4 (MI & AS): Designed to strike a balance between targeted and general therapy exposure, this training set combines 5000 samples from MI-TAGS with 5000 samples from the AS dataset, summing to 10,000 samples in total. This allows us to test whether hybrid training sets foster a synergy, resulting in therapists who excel in both the nuances of MI and broader therapeutic skills.

Once fine-tuned, the LLM can be used as a therapist in the same way as in the previous section, using the same prompts provided to the digital expert therapist. To assess the performance of the models, we employ our digital-patient evaluation approach, following the same patient characteristic definitions as outlined in earlier sections.

Results

We divide this section into three parts, each reporting results corresponding to the three parts in the “methods” section.

Evaluating therapist LLMs using established automated metrics

Figure 3 presents the results obtained using the five text-similarity metrics introduced in “Evaluating therapist LLMs using established automated metrics.” The findings suggest that none of the metrics effectively differentiate among the three levels of expertise—most metrics show no significant differences, while MAUVE yields results in the opposite direction than expected. Thus, although a human MI expert validated that the three digital therapists do indeed reflect three levels of expertise as intended, this was not captured by any of the text similarity metrics.

Fig. 3.

Evaluation results of the three therapists using some known text-similarity metrics.

This observation prompted us to develop and explore an alternative approach for therapy assessment. In this new method, we utilize digital patients, implemented by LLMs, to simulate real patients who provide feedback on the quality of the therapy; these are described in the next section.

Evaluating therapists with digital patients

In order to assess the reliability of the model’s ratings, we computed Cronbach’s alpha, a common practice to assess the reliability of a questionnaire in social science, which measures the internal consistency in rating similar questions. Reliability analysis of the two questionnaires demonstrated exceptionally high Cronbach’s alpha coefficients, indicating strong internal consistency. Specifically, Cronbach’s alpha was 0.97 for Questionnaire 1 and 0.98 for Questionnaire 2. This confirms that the rating model is consistent in filling on questionnaires regarding the generated conversations.

As described in “Evaluating therapists with digital patients,” we let the models decide when to stop the session; otherwise, we forced the session to terminate after 50 turns (100 utterances), which only happened on 2 out of 288 occasions.

Tables 3 and 4 provide the average utterance and word count over the 288 generated conversations. Analysis indicates an extension in session duration concurrent with therapist improvement. A one-way ANOVA analysis yielded a statistically significant variance in utterance counts across the three proficiency categories (F = 81.6, p < 0.001). Subsequent post-hoc comparisons using Tukey’s HSD test revealed that each category pair (poor vs. average, average vs. expert, and poor vs expert) demonstrated significant differences (p = 0.0001, p = 0.003, p = 0, respectively).

Table 3.

Count of utterances in a conversation, categorized by the therapist level

Level	Mean	Std
Poor	12.92	2.69
Average	17.80	4.85
Expert	19.81	3.63

Table 4.

Word count in a conversation broken down by therapist level

	Therapist		Patient
Level	Mean	Std	Mean	Std
Poor	374.58	94.75	329.44	98.13
Average	507.78	131.00	430.16	135.57
Expert	619.10	138.35	439.62	113.23

Comparable trends were noted in the analysis of word count. The one-way ANOVA indicated a statistically significant difference in word counts between therapist categories (F = 94.3, p < 0.001). Furthermore, post-hoc comparisons employing Tukey’s HSD test revealed significant differences between all pairs of therapist categories (p < 0.001 for each comparison). The ANOVA analysis reveals significant differences in patient word count across different therapist categories (F = 26.0, p < 0.001). Post-hoc analyses indicate a significantly lower word count in patient responses to the poor digital therapist than in the average and expert digital therapists (both p < 0.001). However, the difference in word count between the expert and average digital therapists is not significant.

The patient LLMs were asked to complete two questionnaires per conversation, with prompts framed as if acting in the role of professional therapists (as explained in “Evaluating therapists with digital patients”), to ensure comparability with the human clinician evaluations. Both questionnaires in our study are structured to maintain a consistent response scale throughout: a response of 1 always signifies an aspect of the treatment that was ineffective or unsatisfactory, while a response of 5 indicates that an aspect of the treatment was highly successful. The opposite questions were reversed to ensure standardization.

Figure 4 displays the mean and standard error across all responses to Questionnaires 1 (session satisfaction) and 2 (therapist-patient working alliance).

Fig. 4.

Mean response values of patient models to Questionnaire 1 (session satisfaction) and 2 (therapist-patient working alliance), categorized by the therapist skill level, with standard error bars indicating variability across sessions.

The differences among the three therapist categories were found to be highly significant, as established by a one-way ANOVA and subsequent Tukey post-hoc tests. Specifically, for Questionnaire 1, the ANOVA analysis yielded F = 67.6 (p < 0.001), and post-hoc analysis also indicated p-values less than 0.001. Similarly, for Questionnaire 2, the ANOVA analysis showed F = 169.3 (p < 0.001), and the post-hoc tests mirrored these results with p values less than 0.001.

A clinical psychologist (M.Z., co-author of this paper), an expert in MI, along with two social psychology PhD candidates, who received training on MI from our MI expert, reviewed 30 randomly selected sessions and completed the questionnaires on behalf of the digital patient, in a manner similar to that of the LLM. The evaluators provided answers to the questionnaires “blindly,” meaning they were unaware of the category of the therapist associated with each session, and without viewing each other’s results. Additionally, we did not coordinate between the evaluators and did not provide any formal guidelines.

We independently calculated Cronbach’s alpha for the three evaluators, as summarized in Table 5. For comparison, the digital patients achieved scores of 0.95 and 0.97 for Questionnaires 1 and 2, respectively.

Table 5.

Internal consistency of the three evaluators individually, expressed by Cronbach’s alpha values, calculated for the two questionnaires

Evaluator	Questionnaire 1	Questionnaire 2
1	0.97	0.97
2	0.97	0.99
3	0.85	0.90

Evaluator 1 is the MI expert, and evaluators 2 and 3 are the two additional social psychologists.

Next, we concatenated the 22 questions from both questionnaires, despite their disparate origins. The combined Cronbach’s alpha values for the two questionnaires indicate a very high level of internal consistency across both human evaluators and the LLM, with scores of 0.98, 0.95, 0.93, and 0.99 for evaluator 1, evaluator 2, evaluator 3, and the digital patients, respectively. This high level of internal consistency indicates that the items within each questionnaire are reliably related. It is possible that both questionnaires are measuring the same single construct, or that they are measuring two different constructs that are highly correlated. Consequently, their integration into a single metric appears justified, which we propose to interpret as an indicator of therapist quality.

To assess agreement between the three human evaluators, we calculated Krippendorff’s alpha, obtaining values of 0.63 for Questionnaire 1 and 0.67 for Questionnaire 2, indicating near-moderate and moderate agreement, respectively. We further examined the confusion matrices between each pair of evaluators (Figs. 5 and 6) and found that disagreements were predominantly local: when one evaluator assigned a rating of 1, disagreements most often involved another evaluator giving a 2, and similar patterns occurred for other adjacent ratings. Because the questionnaire score is computed as the average of ratings across all questions, these small local differences are attenuated in the final score.

Fig. 5. Confusion matrices for Questionnaire 1 (session satisfaction) showing pairwise comparisons between the three human evaluators.

Darker colors indicate higher counts. Disagreements are predominantly local, with most differences occurring between adjacent ratings.

Fig. 6. Confusion matrices for Questionnaire 2 (working alliance) showing pairwise comparisons between the three human evaluators.

Darker colors indicate higher counts. As in Questionnaire 1, most disagreements are local, typically involving adjacent rating values.

Given that agreement was not perfect, we compared the LLM-based digital patients with each clinician separately by computing Pearson correlations between their averaged questionnaire scores. In Questionnaire 1, which measured session satisfaction, a Pearson correlation analysis revealed a moderate positive correlation between the human MI expert and the digital patients, with a coefficient of 0.65 (p < 0.001), as shown in Fig. 7. Similarly, moderate positive correlations were observed for the two other evaluators, with coefficients of 0.69 and 0.63 (p < 0.001), respectively, also depicted in Fig. 7. Stronger positive correlations were observed in Questionnaire 2, which assessed the working alliance: the human MI expert achieved a coefficient of 0.84 (p < 0.001), while the two social psychologists showed coefficients of 0.78 and 0.83 (p < 0.001), as illustrated in Fig. 8.

Fig. 7. Correlation between human evaluators and model for a subset of 30 sessions; Questionnaire 1 (session satisfaction).

Panel a shows the correlation between the model and the MI expert; panel b shows the correlation between the model and clinician 1; and panel c shows the correlation between the model and clinician 2.

Fig. 8. Correlation between human evaluators and model for a subset of 30 sessions; Questionnaire 2 (working alliance).

Panel a shows the correlation between the model and the MI expert; panel b shows the correlation between the model and clinician 1; and panel c shows the correlation between the model and clinician 2.

Finally, we conducted an additional experiment using Amazon Mechanical Turk with human evaluators who do not have a clinical background. However, a small pilot study revealed problems, which we discuss in Supplementary Note 1.

Along with the quantitative analysis, a qualitative examination was performed by the human MI expert, revealing noteworthy themes pertaining to the sessions. Notably, the patient LLMs demonstrated a consistent tendency to respond courteously even in situations where the digital therapist exhibited dismissive or offensive behavior. Furthermore, the advice offered by the therapists (regardless of their intended “quality”) exhibited a repetitiveness in all sessions characterized by a limited scope; for example, primarily focusing on breathing techniques and exercise as means of alleviating anxiety without elaborating alternative options. This lack of tailored recommendations was evident across diverse patient profiles, indicating a uniformity in the therapeutic guidance provided by the LLM. Both of these identified themes align with expectations associated with LLMs.

Encouraged by the ability of our digital patients to differentiate between various therapy skills, we decided to use these patient LLMs to evaluate the impact of fine-tuning a therapist LLM on a relevant dataset of therapy sessions, as a case study. We discuss this evaluation in the next section.

Fine-tuning a therapist LLM

As above (Evaluating therapists with digital patients), we verify the reliability of our approach by calculating Cronbach’s alpha. This analysis focuses on the coherence of the LLM-generated ratings across the two questionnaires. The Cronbach’s alpha values are 0.9 for Questionnaire 1 and 0.94 for Questionnaire 2.

As expected, the questionnaires could consistently distinguish among different qualities of automated therapists. Figures 9, 10 visualize mean ratings from patient model responses to Questionnaires 1 and 2, respectively, grouped by therapist LLMs.

Fig. 9. LLM-based therapist scores on Questionnaires 1.

a Mean scores of LLM-based therapist responses on Questionnaires 1, demonstrating the impact of model size, fine-tuning, and their combination on performance. Each bar represents the median (middle line), 25th and 75th percentiles (box edges), whiskers (typically showing 1.5 times the inter-quartile range), and outliers. b summarizes the statistical significance scores of the differences between the groups. Those scores were calculated using one-way ANOVA followed by Tukey post-hoc tests.

Fig. 10. LLM-based therapist scores on Questionnaires 2.

a Mean scores of LLM-based therapist responses on Questionnaires 2, demonstrating the impact of model size, fine-tuning, and their combination on performance. Each bar represents the median (middle line), 25th and 75th percentiles (box edges), whiskers (typically showing 1.5 times the inter-quartile range), and outliers. b summarizes the statistical significance scores of the differences between the groups. Those scores were calculated using one-way ANOVA followed by Tukey post-hoc tests.

There were several differences among the LLM therapists. Specifically, the 13B model fine-tuned with the MI-TAGs dataset was superior compared to all others, as confirmed by a one-way ANOVA and following Tukey post-hoc tests. On Questionnaire 1, ANOVA yielded F = 8.6 (p < 0.001), with post-hoc analysis demonstrating p values below 0.001. Questionnaire 2 results are similar, with ANOVA showing F = 13.3(p < 0.001).

Models 13B MI, 13B MI and AS, and 7B MI performed best across both questionnaires. No significant differences exist among those three variations.

In our prompts to both the therapist and the patient, we instructed the models to determine the natural conclusion of the session within an upper bound of 25 turns, which is equivalent to 50 utterances.

Table 6 and Fig. 11 illustrate the average number of utterances and words generated across the 96 conversations for each model. Our analysis shows a significant reduction in word count after fine-tuning, regardless of the training dataset. This finding suggests that fine-tuning significantly impacts the verbosity of the therapist LLM, potentially impacting the dynamics of the therapeutic exchange.

Table 6.

Count of utterances in a conversation, categorized by therapist LLM variation

Model	Mean	Std
7B Baseline	14.25	9.24
7B MI	25.31	10.78
7B AS	26.54	5.67
7B AS Ext.	24.81	10.02
7B MI & AS	25.92	9.70
13B Baseline	24.03	19.03
13B MI	27.93	10.92
13B AS	29.93	12.31
13B AS Ext.	28.67	11.16
13B MI & AS	25.10	10.03

Fig. 11.

Comparison of mean word count between therapist and patient responses across different therapist LLM variations, with standard error bars indicating variability across sessions.

A one-way ANOVA confirmed statistically significant differences in word counts across all ten models (F = 61.4, p < 0.001). Employing Tukey’s HSD test for post-hoc comparisons, we found significant differences between each baseline model and all fine-tuned models (p < 0.001). Interestingly, we also observed significant differences between two of the baseline models themselves.

In summary, all fine-tuned models outperformed the baseline LLMs on Questionnaire 1, and most did so on Questionnaire 2. Furthermore, larger LLMs (13B) generally performed better than smaller ones (7B). Training the model on a highly specific dataset (MI) yielded better performance than training on a diverse dataset (AS, general counseling), while the hybrid datasets (MI + AS) outperformed the more general dataset.

Discussion

There is growing interest in using LLMs as automated therapists (e.g.,^9,30). However, before we proceed further with real patients, it is crucial to develop robust, reliable, and valid methods for evaluating their therapeutic skills. We demonstrated that existing text-similarity metrics are not sufficient and propose using digital patients to simulate therapy sessions, providing feedback that can be used as a new evaluation metric. The digital patients are implemented using LLMs, each characterized by different profile categories. We engaged them in an MI therapy session with digital therapists, also implemented with an LLM. Importantly, our results showed that these digital patients could complete questionnaires related to the therapy, with their ratings proving to be both reliable and valid, thus making it appropriate as an evaluation metric.

We note that the variance in the rating by the LLM is very low. This is reflected in very high Cronbach’s alpha values, close to 1, whereas human studies rarely yield values over 0.9. Thus, while we provide a wide range of digital characteristic profiles, we do not claim that our platform replaces a complete human population. Increasing variance in the model responses, to obtain a better approximation of a human population, can be achieved by simple statistical methods such as adding noise or model temperature. However, our goal in this study is not to replace human participants for psychological studies (e.g., as discussed by ref. ¹⁰); rather, our main goal is to allow for a standard, reliable, and valid method for evaluating digital therapists.

As a next step, we utilized the same digital patient platform to evaluate the impact of fine-tuning LLM therapists. The results demonstrate that fine-tuning strategies significantly affect the dialog proficiency and therapeutic skills of LLM-based therapists. Our analysis revealed that fine-tuned models showed distinct changes in their conversational patterns compared to the baseline models. Notably, we observed a reduction in verbosity after fine-tuning, indicating that targeted training can effectively shape the dynamics of therapeutic exchanges.

Our findings also highlight that the choice of training dataset plays an important role in influencing the therapist LLM’s performance. Therapists fine-tuned with datasets explicitly focused on MI generally demonstrated better alignment with MI principles, as evidenced by higher scores on our new evaluation metric. This emphasizes the importance of domain-specific training data in developing effective LLM-based therapeutic solutions. Furthermore, our results indicate that incorporating a broader therapeutic foundation through datasets like Alexander Street, in addition to MI-related datasets, but not instead, can enhance the overall therapy skills.

We make the digital-patient platform available, and hope it can be further extended, explored, and utilized by the community to ensure responsible use of AI in mental health and clinical psychology.

Our work has certain limitations. Our method provides an evaluation on two dimensions: session satisfaction and working alliance. While we consider these two the most critical measurements, it might be beneficial to broaden the method to include additional evaluation dimensions and employ a more diverse range of questionnaires. Furthermore, our quality measurement of the therapist is only based on three levels of quality, poor, average, and expert; a more refined scale may be desired.

Additionally, the numeric ratings assigned by the digital patients to the questions show low variance, which differs from the typical response patterns seen in human agents when interacting with such questionnaires.

In “Fine-tuning a therapist LLM,” we present evaluation results generated by the virtual patients and use these findings to draw conclusions about the optimal fine-tuning settings for the therapist. However, these results were not manually validated; instead, we rely on the earlier evaluation, which demonstrated that the virtual patients assess sessions in a manner closely aligned with human evaluators.

We acknowledge the potential for generating problematic text, particularly in sensitive areas such as mental health. This includes the possibility of producing content that may be offensive, inaccurate, or inadvertently harmful. Although our experiments did not encounter this issue, it is important to recognize that contemporary LLMS (GPT-3.5 in our case), despite its built-in safeguards, can still occasionally generate inappropriate or problematic responses. Addressing these challenges requires continuous monitoring and improvements. Such limitations are typical across current LLMs, and the extent of their impact, along with the need for ongoing research, will vary depending on the specific application.

All data utilized in this study were generated through artificial intelligence, except AS, which was taken from a curated open source. Using synthetic conversations ensures the complete anonymization and privacy of individuals, as the conversations between the digital therapist and the digital patient, along with their distinct characteristic profiles, were entirely synthetic and not based on real human interactions. By employing prompt engineering to construct varied therapist and patient characteristic profiles, we avoided the ethical complexities and privacy concerns associated with the use of personal, sensitive, or identifiable data often encountered in clinical research. Furthermore, our methodology sidesteps the potential risks of inadvertently revealing personal health information, ensuring compliance with privacy regulations and ethical research standards.

AI research in mental health raises major ethical concerns, especially with regard to the privacy of patients. As a result, there is a scarcity of real-life datasets, which in turn constrains the development of generative models and evaluation methods. Using synthetic patients can be an important step in overcoming these challenges, if indeed it can be shown that they replace human patients, at least in specific aspects.

Finally, caution is necessary when utilizing LLMs for mental health, as they are often unpredictable and not fully understood or fully controlled. Careful measurement, such as suggested here, should be exercised, yet for real-world deployment, additional measures may be required.

Author contributions

K.B., D.F. conceived the study and developed the methodology; S.Y. contributed to the implementation of the methodology; M.Z., A.BK., and B.C. contributed to the curation and validation; M.Z. and A.BK. provided clinical advice; All authors performed formal analyses; S.Y. generated visualizations; K.B., D.F., and S.Y. wrote the original draft of the manuscript with inputs from all other authors. All authors had full access to all the data in the study; All authors contributed to manuscript review and editing, and approved the final manuscript.

Data availability

We used two datasets in our study. (1) MI-TAGS: B.C., M.Z., S.Y., Doron Friedman, and Kfir Bar. 2024. Motivational Interviewing Transcripts Annotated with Global Scores. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 11642–11657, Torino, Italia. ELRA and ICCL. (2) Alexander Street: counseling and Psychotherapy Transcripts: Volume I https://search.alexanderstreet.com/psyc Counseling and Psychotherapy Transcripts: Volume II https://search.alexanderstreet.com/ctrn.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s44184-025-00159-1.