Opening the Black Box of Family-Based Treatments: An Artificial Intelligence Framework to Examine Therapeutic Alliance and Therapist Empathy

Abstract

The evidence-based treatment (EBT) movement has primarily focused on core intervention content or treatment fidelity and has largely ignored practitioner skills to manage interpersonal process issues that emerge during treatment, especially with difficult-to-treat adolescents (delinquent, substance-using, medical non-adherence) and those of color. A chief complaint of “real world” practitioners about manualized treatments is the lack of correspondence between following a manual and managing microsocial interpersonal processes (e.g. negative affect) that arise in treating “real world clients.” Although family-based EBTs share core similarities (e.g. focus on family interactions, emphasis on practitioner engagement, family involvement), most of these treatments do not have an evidence base regarding common implementation and treatment process problems that practitioners experience in delivering particular models, especially in mid-treatment when demands on families to change their behavior is greatest in treatment – a lack that characterizes the field as a whole. Failure to effectively address common interpersonal processes with difficult-to-treat families likely undermines treatment fidelity and sustained use of EBTs, treatment outcome, and contributes to treatment dropout and treatment nonadherence. Recent advancements in wearables, sensing technologies, multivariate time-series analyses, and machine learning allow scientists to make significant advancements in the study of psychotherapy processes by looking “under the skin” of the provider–client interpersonal interactions that define therapeutic alliance, empathy, and empathic accuracy, along with the predictive validity of these therapy processes (therapeutic alliance, therapist empathy) to treatment outcome. Moreover, assessment of these processes can be extended to develop procedures for training providers to manage difficult interpersonal processes while maintaining a physiological profile that is consistent with astute skills in psychotherapeutic processes. This paper argues for opening the “black box” of therapy to advance the science of evidence-based psychotherapy by examining the clinical interior of evidence-based treatments to develop the next generation of audit- and feedback- (i.e., systemic review of professional performance) supervision systems.

Introduction

Psychotherapy process research is essential for developing and refining effective interventions for multi-problem, multi-stressed families dealing with mental and physical health problems. However, traditional methods of measuring psychotherapy processes have been limited by the "Black Box" of therapy sessions, which are difficult to measure and understand. Recent advancements in artificial intelligence (AI) methodologies, such as machine learning and wearable technologies that can facilitate the examination of reciprocal and evolving biopsychosocial processes that occur within and between psychotherapy sessions. In this position paper, we conduct a review of the psychotherapy process constructs of therapeutic alliance and rupture and repair, and empathy, within the context of family-based treatments. This review of psychotherapy process constructs serves as a foundation for our primary aim, which is to explore the potential benefits of incorporating wearable technologies and AI methodologies in the study of psychotherapy processes in family-based treatments. We do not address the ongoing debate regarding the distinction between common and unique factors in psychotherapy (Wampold, 2015). Rather, we concentrate on outlining the current state and efficacy of technological and AI solutions that hold promise for examining the internal workings of evidence-based family therapies.

Scope of the Problem

Family-based psychosocial treatments (e.g. Multisystemic Therapy, Functional Family Therapy) have shown efficacy in controlled research settings for treating children with psychological problems and uncontrolled chronic medical conditions such as asthma and diabetes (Baldwin et al., 2012; Borduin et al., 2009; Latimer, 2001; Naar-King et al., 2016; von Sydow et al., 2013). Several of these treatments have shown large effect sizes and are designated as evidence-based (EBT) and well-established for children in the mental health and chronic illness sectors (Chronis et al., 2006; Friedlander et al., 2021; Kaslow et al., 2012; Law et al., 2014; McCart & Sheidow, 2016). However, the efficacy of these EBTs in real-world practice settings remains far from optimal (Curtis et al., 2019; Weisz et al., 2013, 2014), especially for underserved, multi-problem, and multi-stressed families dealing with serious mental and medical problems (Guerrero et al., 2013; Lu & Zhang, 2021; Reardon et al., 2017). The discrepancy between the outcomes achieved in controlled settings and those in real-world contexts, even when therapies are implemented with fidelity (Collyer et al., 2020), signals a pressing need for refining implementation and training strategies to amplify the “voltage” (Chambliss, 2013) of EBTs in real world contexts.

The modest success of EBTs in real-world settings could be, in part, attributed to the emphasis placed on providers mastering the tasks outlined in treatment manuals during implementation (Beidas & Kendall, 2010; Brickman & Fristad, 2021; Lochman et al., 2022), rather than providing guidance on how to manage the complex, nuanced interpersonal processes that inevitably occur during the delivery of EBTs (Cunningham et al., 2010). Indeed, many treatment manuals and counselor trainings in EBTs provide guidance on issues that can arise during treatment, most do so in a largely anecdotal way rather than being data driven. This content gap is likely due to the assumption that therapists possess basic facilitative skills, such as the ability to establish a strong therapeutic alliance, therapeutic empathy, and the ability to handle difficult encounters. Concerning the latter, community health workers have begun to be used to provide mental health services as they bring their cultural knowledge and shared linguistic skills to best manage difficult encounters (Schaaf et al., 2020). Unfortunately, many counselors in community settings fail to receive adequate training in non-specific therapeutic skills in work with multi-problem families, where culture matching has its limitations, which limits their ability to effectively engage with and address the needs of high-need clients (Schaaf et al., 2020). Moreover, counselors in community settings often do not know how to manage interpersonal process difficulties within the context of the theoretical framework of the treatment protocol in a way that does not derail the session and the delivery of critical content while maintaining strong alliance and empathy (Guan et al., 2019). Provider struggles with navigating challenging interpersonal dynamics may contribute to less effective treatment implementation and subsequent lower treatment retention and outcomes (Knox et al., 2023).

These training gaps in EBT implementation may be attributed, in part, to the underdeveloped science of psychotherapy processes research. However, recent advancements in innovative methods offers an opportunity to address these gaps by providing data-driven techniques for training therapists on what to say, do, and when, during challenging interactions, while building or preserving an alliance and expressing accurate empathy. Our team hopes to revolutionize the field of family-based psychotherapy process research by utilizing new technologies that can open the “Black Box” of treatment and take advantage of the large amount of untapped information available within each treatment session (e.g. complex interplay of dynamic biopsychosocial processes) to fully and comprehensively, using multimodal data (e.g. text, audio, psychophysiology) and artificial intelligence methodologies, examine microsocial and psychotherapy processes within and across sessions that are crucial for effective therapy that’s at the “heart and soul of behavior change.” In subsequent sections, we discuss therapeutic alliance, rupture and repair of therapeutic alliance, and therapist empathy while highlighting the complexity of these constructs for counselors working to deliver family-based treatments or interventions. We then explore physiological measurements as well as artificial intelligence technologies that could be used to quantify these intangible elements for training purposes. By doing so we are creating a platform for discussion about how best to innovatively use these tools to close EBT training implementation gaps moving forward.

Psychotherapy Processes

Therapeutic Alliance and Rupture and Repair of the Therapeutic Alliance

The most studied process variable in psychotherapy research is the therapeutic or working alliance, usually defined as the working relationship that develops between clients and therapists based on an agreement on therapy goals, collaboration on therapy tasks, and the presence of a bond of mutual trust and respect (Bordin, 1979; Wampold & Flückiger, 2023). The alliance is a robust predictor of therapy outcome in treatment for adults (Flückiger et al., 2018), children and adolescents (Karver et al., 2018), and couples and families (Friedlander et al., 2018). Research on the alliance in individual therapy with adults has demonstrated that the alliance makes a unique contribution to outcome above and beyond client intake characteristics, other process variables such as homework compliance, and adherence and competence ratings (Flückiger et al., 2020a). The association between higher alliance and better outcome is not only found when examining differences between clients, but also within-clients: a meta-analysis examining client ratings of alliance and symptoms session-by-session in the early phase of treatment found a reciprocal relationship in which improvements in alliance were associated with subsequent improvements in symptoms, which were then associated with further improvements in subsequent alliance (Flückiger et al., 2020b).

Research on alliance with families has shown similar results to research on alliance with adults and suggests that building a strong working alliance may be even more critical in family therapy, where multiple family members may be involved, and the complexity of the issues may require collaboration and coordination among family members and the counselor. For example, therapeutic alliance is associated with outcomes between family members and the entire family. Specifically, results show that a strong therapeutic alliance in family-based treatments is associated with positive outcomes and better treatment retention across different treatment approaches (Friedlander et al., 2018). Therapists contribute to the alliance in family-based treatments, both through positive in-session behaviors such as engagement and emotional connection between family members that predict a positive alliance (Welmers-van de Poll et al., 2021), but also through problematic behaviors such as competing for control, which contributes to a poor alliance (de la Peña et al., 2012). Additionally, a recent meta-analysis found that the association between alliance and treatment outcomes was stronger for family-involved treatments compared to individual treatments (Welmers-van de Poll et al., 2018). Importantly, there is evidence that the beneficial impact of the alliance on outcome is mainly due to the therapist's contribution (Baldwin et al., 2007; Del Re et al., 2012, 2021), suggesting that the therapist's contribution to the alliance is critical.

Given the importance of the alliance in therapy, it is not surprising that problems that may arise in the alliance can adversely impact treatment progress. Difficulties agreeing on goals, collaborating on tasks, and building and maintaining a bond are referred to as alliance ruptures (Muran & Eubanks, 2020). A meta-analysis has demonstrated that ruptures are prevalent in individual therapy and that unrepaired ruptures predict poor outcome or dropout (Eubanks et al., 2018). Thus, while ruptures present challenges in treatment, they also present opportunities for repair, which appears to be an important process, as rupture repair predicts positive outcome and treatment retention (Eubanks et al., 2018). In fact, one study found that clients whose self-reported alliance scores were consistent with a rupture repair pattern achieved greater improvement in therapy than clients who reported no alliance ruptures (Stiles et al., 2004).

A current conceptualization is that the process of recognizing and repairing an alliance rupture gives therapists and clients an opportunity to better understand each other’s experience and how the treatment may not be meeting the patient’s needs or preferences, to become more aware of one’s interpersonal behaviors and their impact on others, and to work through an interpersonal conflict in an adaptive way, which may facilitate a corrective experience for clients with a history of conflictual relationships (Muran & Eubanks, 2020). However, in order to repair a rupture, therapists need to recognize that a rupture has occurred, and several research studies have found evidence that therapist recognition of a rupture is related to subsequent improvements in alliance or outcome (Atzil-Slonim et al., 2015; Chen et al., 2018; Rubel et al., 2018; Zilcha-Mano et al., 2017).

Training counselors to recognize and successfully navigate ruptures is challenging, with a small meta-analysis of rupture resolution training and supervision studies finding only a small, nonsignificant effect on client outcome (Eubanks et al., 2018). However, a more recent randomized controlled trial of an alliance-focused training for therapists conducting cognitive behavior therapy found that the training facilitated decreases in negative in-session processes (therapists’ controlling and blaming behaviors and clients’ dependent behaviors) and increases in positive in-session processes (patient and therapist expressiveness and therapist affirmation), and several of these positive changes were linked to good treatment outcome (Muran et al., 2018).

Within the context of family-based treatments, only a small body of literature has begun to examine alliance ruptures and repairs (Friedlander et al., 2021). Several evidence-based case studies that tracked in-session behavioral markers of rupture and repair concluded that successful repair required the therapist to strengthen their bond with each client (i.e., family member), to attend to each client’s sense of safety within the therapeutic system, and to foster family members’ attachments with each other (Escudero et al., 2022). Research on rupture and repair in family-based treatments presents particular challenges due to the complexity of the alliance in a systemic context. In addition to the personal alliances between the therapist and each family member, there is a group alliance between the therapist and the family as a whole, and a within-system alliance among family members (Pinsof & Catherall, 1986). Unbalanced or split alliances, in which family members view their alliances with the therapist very differently, present challenges for therapists and are associated with worse outcome and increased risk of dropout (Friedlander et al., 2018). However, the importance of maintaining balance across family members’ alliances may depend on the treatment format: research suggests that balance in alliance predicts retention in conjoint family-based treatment, but overall strength in alliance predicts retention in treatments using both individual and conjoint formats (Robbins et al., 2008).

While the research on rupture and repair in family-based treatments is limited to date, the concept of rupture and repair is consonant with the well-established “struggle- and-working through” (Chamberlain et al., 1984) hypothesis, which proposed that in early sessions of family-based treatment, therapists limit how much they challenge family members to navigate caregiver resistance and build a good working relationship (Chamberlain et al., 1984). However, in mid-treatment, therapists begin to ask more of caregivers, and these increasing demands elicit increased resistance from caregivers, resulting in a period of “struggle.” The families who have successful outcomes “work through” this resistance and as the families employ the skills learned in treatment, they begin to see improvement and resistance decreases. Stoolmiller and colleagues (Stoolmiller et al., 1993) found a curvilinear association between caregiver resistance and positive treatment outcome that was consistent with this hypothesis.

Empathy

Therapist working alliance and therapist empathy are closely related constructs in the psychotherapy process literature (Wampold & Flückiger, 2023). As suggested in the review above, therapist empathy is a key factor in building and maintaining a strong working alliance, and the association between therapist empathy and working alliance is bidirectional (Flückiger et al., 2020b). Given this established association between alliance and empathy, it is not surprising that empathy is one of the most consistent predictors of treatment outcome across disparate therapies (Wampold & Imel, 2015) and has been repeatedly shown to be positively associated with treatment effectiveness, accounting for more variance in client change than specific treatments or treatment components (Watson et al., 2014).

For example, the impact of therapist empathy has been extensively studied and major reviews of the literature show it to be a strong predictor of treatment outcome. A meta-analysis of over 6000 clients across 82 samples of therapist-client interactions found that ratings of the therapist understanding of the clients’ feelings moderately predicted therapy outcome (d = 0.58), more so than empathic accuracy per se (Elliott et al., 2018). Using objective medical outcomes, across 21 studies, patients with diabetes had better disease-relevant scores when their physician scored higher on empathy; patient satisfaction was also higher when their doctor had more empathic communication skills, and malpractice claims were lower (Hojat, 2016a). A meta-analysis of psychotherapy found effect sizes around 0.20 for therapist empathy predicting outcomes; and low empathy predicted higher subsequent dropout and relapse and a weaker therapeutic alliance, which predicted less positive patient change (Hojat, 2016b).

Research suggests relations between therapist empathy and outcome is due to the impact therapist empathy has on client engagement. For example, therapist empathy is often rated by clients as one of the "most helpful" treatment-related experiences (Morris & Suckerman, 1974; Rorty et al., 1993), and is positively associated with client disclosure, which is a key factor in successful therapy outcomes. Within the context of family-based interventions, Thompson and colleagues found that building a strong therapeutic alliance is crucial to engaging families in home-based therapy and suggested that processes for building this alliance occurred through therapist being empathic toward cultural and contextual factors that influence family engagement in the therapeutic process (2007). In another study, researchers found caregivers, who participated in an MST study, reported that they valued their therapists' support and empathy, and perceived their therapists as being non-judgmental and respectful (Kaur et al., 2017). This study also showed that the empathy which facilitated alliance was an important factor in sustaining the positive changes achieved through MST, even after the therapy had ended. These results and interpretations were supported by a systematic review that explored the determinants and outcomes of the therapeutic alliance in treating justice-involved youth (Papalia et al., 2022). Based on 33 studies, the results indicated that the therapeutic alliance between justice-involved youth and their therapists is related to the youth's engagement, satisfaction, and overall positive treatment outcomes. Several determinants were identified as crucial for the therapeutic alliance, including trust, empathy, communication, and authenticity, among others.

Despite the well-established role of therapist empathy on outcomes, developing strategies to identify and improve empathy within clinical contexts is technically challenging because the intent of another person’s behavior is nuanced, highly contextual, and deeply interactive; and can be expressed in multiple modalities (e.g. text, audio, psychophysiology). Neuroscientists contend that humans have learned to recognize empathy over time through mirror neurons in the premotor area of the brain, which are activated when performing a specific empathetic behavior or while observing another individual perform a similar action (Iacoboni et al., 2005; Kohler et al., 2002). Better understanding of empathy in clinical contexts, toward improved therapist training methods, requires empirical approaches and analytical tools that can discern empathetic behaviors exhibited during actual clinical encounters that lead to behavior change.

Empathic Accuracy

Therapeutic empathy is defined as the ability of the therapist to understand and experience their client's emotions, thoughts, and experiences, while remaining objective and non-judgmental (Bohart & Greenberg, 1997). This involves not only the therapist's ability to perceive and understand their client's emotions, but also to communicate this understanding in a way that validates and supports the client's experiences. Empathetic accuracy, on the other hand, goes beyond the therapist's ability to simply perceive and experience the client's emotions, but to also accurately interpret and respond to those emotions in a way that is helpful and effective (Ickes, 1993).

Empathic accuracy assessed using a methodology developed by David Ickes (Ickes & Tooke, 1988; Ickes et al., 1990) involves a video-cued procedure that compares an observer’s emotional ratings with that of a target’s self-report of their emotional state. In a clinical context, the therapist is the observer, and the client is the target, watch a playback of a video recording of their session and use a rating dial to provide a continuous ratings of the client’s emotional state during the session. Moment to moment agreement between the therapist and client provides an objective measure of a therapists’ accurate empathy. The empathic accuracy methodology (Ickes & Tooke, 1988; Ickes et al., 1990) has been validated many times, but seldom applied in clinical studies examining the role of empathy and therapeutic alliance in the implementation of various family-based EBTs. Yet, empathetic accuracy may be foundational to therapy outcomes since therapists must first accurately identify the client’s state before they can address their clients’ related concern(s). For example, an analysis of dozens of therapist-client sessions found that therapists who could accurately rate their client’s affect, were more accurate for negative than positive affect (due to interference from their own affect); the affect between therapist and client also matched during sessions and when therapists were less accurate about client positive affect, symptoms were higher in the following session (Atzil-Slonim et al., 2019). In an experimental setting, empathic accuracy for another’s negative affect predicted higher prosocial helping for a confederate in need (Eckland et al., 2020). It should be noted that a similar methodology, Interpersonal Process Recall (Kagan, 1980; Kagan et al., 1969), often used for clinical supervision to elicit treatment participants internal experiences, also uses a review of session video tapes, may be used to supplement the empathic accuracy methodology. In this position paper, we argue that the empathic accuracy methodology is particularly useful as an empirical tool for studying psychotherapy processes in family-based EBTs as it more fully captures the complexity of dyadic interpersonal interactions compared to subjective reporting (i.e., self-report measures) to include communicative features such as voice tone, words used, facial expressions and body posture, fundamental frequency of speech signals, and other nonverbal features and it does so over the course of a session.

Psychotherapy Processes with Multi-problem, Multi-stressed Families

Research on rupture and repair in family-based treatments presents particular challenges due to establishing a working alliance in psychotherapy with families is a complex and multi-faceted process. The therapeutic relationship between the therapist and family members can vary greatly depending on the family’s structure, individual personalities, and dynamics. Additionally, each family member may have different objectives for therapy and different ways of communicating their needs. This can often be challenging as the goals of therapy may differ between individual family members, tasks may need to be tailored to the unique dynamics of the family system and trust can take time to develop with individual family members and with the family as a unit. There are also multiple power dynamics that can play out between family members, which can further complicate the therapeutic relationship. Finally, it is important to remember that families often have a shared history of experiences that influence their current relationships, which can make it even more challenging for a therapist to understand everyone’s perspective.

With all these complexities in mind, it is important for the therapist to take an active role in understanding and addressing these complexities. For a strong working alliance, the therapist must form alliances with each family member, a group alliance between the family as a whole, and a within-system alliance among family members (Pinsof & Catherall, 1986). Unbalanced or split alliances, in which family members view their alliances with the therapist very differently, present challenges for therapists and are associated with worse outcome and increased risk of dropout (Friedlander et al., 2018). However, the importance of maintaining balance across family members’ alliances may depend on the treatment format: research suggests that balance in alliance predicts retention in conjoint family-based treatment, but overall strength in alliance predicts retention in treatments using both individual and conjoint formats (Robbins et al., 2008).

Therapists who work with multi-problem, multi-stressed families in family-based therapies face unique challenges in establishing a strong therapeutic alliance and empathetic connection. These families often experience a complex array of stressors that can lead to increased tension and conflict during treatment sessions. As a result, therapists may struggle to effectively connect with their clients and understand the nuances of their experiences. One factor that can exacerbate this challenge is the presence of cultural differences between the therapist and the family. Studies have shown that differences in cultural background can impact the development of a strong therapeutic alliance and may lead to ruptures in the therapeutic relationship. Additionally, cultural differences can affect the perception and expression of emotions, which can make it even more difficult for therapists to accurately interpret and respond to the emotions of the family members.

Furthermore, therapist working with multi-problem, multi-stress families must navigate the complexities of forming a strong alliance, while remaining empathetic and supportive, as these families typically require greater attention to the establishment of empathy in therapy. Empathy can be particularly challenging to establish in this context, as family members may have different emotional responses and needs and may not always be able to express their emotions in a clear and direct manner. Moreover, families may have a history of trauma, which can affect their emotional regulation and interpersonal functioning.

Despite these challenges, therapists have been able to build strong therapeutic alliances with multi-problem and multi-stress families, as evidenced by the successful implementation and dissemination of family EBTs such MST and Functional Family Therapy in treating serious clinical problems (e.g. delinquency, substance abuse) in real-world practice settings. However, even within the context of EBTs, we still know relatively little about what makes some therapists more effective than others. It is widely assumed that some therapists are better able to develop cultural competence, establish clear and effective communication, and are attuned to the family's emotions and needs. For empathy to foster a strong therapeutic alliance and improve outcomes, however, therapists must first accurately identify the client’s feelings (i.e., empathic accuracy) and make the client feel empathized with (communicative skills), while simultaneously managing their own emotional reactivity to what’s happening in the session (e.g. high negative affect between a parent and child). That is, therapists are not only impacting ongoing interactions during a clinical session, they are also being influenced by participants which is often imperceptible to observers and unacknowledged by the therapist. Thus, it is not surprising that there is wide consensus among psychotherapy researchers and theorists that “emotions should be studied as dynamic systems that interact over time not only within the client or therapist (i.e., intrapersonally) but also between the two individuals (i.e., impersonally)” (pp. 4; Atzil-Slonim et al., 2019) Teasing apart the moment-bymoment interactions between effective therapists and the families that they effectively treat with is quite nuanced and complex-a complexity that heretofore has been largely ignored by psychotherapy process researchers given the limitations of our contemporary scientific toolkit. It is our contention that to fully capture the dynamic, bidirectional, and synergistic nature of psychotherapy relationships that work, requires expanding the measurement of therapy processes beyond existing methodologies such as self-reports of therapists and clients or ratings by expert observers.

Traditional Examination and Limits of Measurement and Conceptualizations of Psychotherapy Processes

Despite the vast literature on the therapeutic alliance and outcomes, the study of psychotherapy processes has been limited by a lack of innovative research methods that can fully capture the inherent complexity of moment-by-moment interpersonal interactions that occur within and across treatment sessions. Current methods for studying psychotherapy processes have emphasized qualitative data-driven methods that rely on therapist and client completion of self-report measures, various process rating systems by domain experts or clinical supervisors, and behavioral coding by trained human coders (Garfield, 1990). Behavioral coding, the gold standard, is a process that is quite repetitive, resourceintensive (i.e., time and human capital), and cognitively demanding (Idalski et al., 2019). Behavioral coding poses a particular challenge for wide-spread dissemination of EBTs because of its associated costs; costs that heretofore have not been reimbursable (Kessler, 2008). Additionally, psychotherapy process researchers and clinical supervisors have relied heavily on the examination of client-therapist verbal interactions (via reviewing session recordings, transcripts or direct observations) when an estimated 60–65 percent of interpersonal communication (Mehrabian & Ferris, 1967, 1967) is conveyed nonverbally (e.g. eye contact/gaze, facial expression, hand gestures, etc.).

Limited progress in development of methods to study psychotherapeutic processes is in part due to the complexity inherent in measuring interpersonal interactions particularly the bidirectional and synergistic nature of such interactions (Imel et al., 2015). Although qualitative methods have been determined to be rigorous through statistical methods such as inter-rater reliability, they lack the quantitative elements that could make observed results and complexities more concrete. The data revolution has made it possible to incorporate quantitative methodology into research which can give a fuller understanding of clinical interactions while mitigating any ambiguities caused by manual coding, supervisors' observations, or experts’ opinions.

Given results that suggest empathetic accuracy is at the foundation of high-quality therapeutic interactions, one promising avenue for establishing an empirical basis for psychotherapeutic processes is through the use of artificial intelligence and machine learning to examine therapist empathic accuracy and interpersonal autonomic physiological synchrony—“an evolutionary-based mechanism for facilitating affiliation, bonding, cooperation, and social cohesion” (Mayo et al., 2021). Translational artificial intelligence and machine learning approaches have been lagging in research on the development of family EBTs. Nonetheless, machine learning approaches utilizing speech and/or wearable sensor data have begun to find utility, and the ability of machine learning models to leverage expert labeling for auto-classification of large data has become an attractive option, and we think is crucial to examine therapist empathic accuracy along with unobservable, non-linguistic neurobiological correlates of empathy, such as interpersonal autonomic physiological measures.

A focus on empathetic accuracy is also ideal as it is a more objective measure of empathy, has been validated many times, and prior research on empathic accuracy and interpersonal autonomic physiological synchrony has laid the groundwork for exploring the use of wearable sensing technologies and natural language processing to examine therapeutic alliance and empathy in dyadic clinical interactions. Empathic accuracy research has shown that clinicians who accurately identify and empathize with their clients' emotional states are better able to form and maintain a strong therapeutic alliance. Research in the area of interpersonal autonomic physiological synchrony has demonstrated the importance of nonverbal cues in communication and has suggested that the synchronization of physiological responses (e.g. heart rate) between clients and therapists may be indicative of an empathic connection. This research has provided a foundation for the use of wearable sensing technologies (such as heart rate monitors and accelerometers) and natural language processing techniques to capture and analyze nonverbal cues and linguistic data in real-time during clinical interactions. With these methods, one could build machine learning models that can automatically classify instances and degrees of therapist empathy based on validated measures of therapeutic alliance and empathic accuracy, allowing for a more comprehensive understanding of psychotherapeutic processes.

Physiology and Psychotherapy Processes

Physiological data can provide much needed information on the biological correlates of interpersonal processes (e.g. thoughts, feelings) occurring “under the skin” that can influence therapy outcome (Deits-Lebehn et al., 2020). Within this context of integrating AI and physiology to better understand empathy, physiological signals of interest include heart rate, heart rate variability, and galvanic skin response. The inclusion of these signals is important to the measurement of empathy because it provides insight into aspects of the therapy process not consciously controlled by the client or the provider.

Heartrate and Heartrate Variability

Heart rate is controlled by the autonomic nervous and endocrine systems. In response to neurological or physical stress, or hormone release (Gwathmey et al., 1994), heart rate can fluctuate as a reflection of the body’s state of stress or relaxation. The sympathetic nervous system, specifically, triggers an increase in heart rate under stressful conditions, and the parasympathetic nervous system helps to decrease heart rate when a stressor is removed or when the body perceives that it is safe (Gordan et al., 2015). The interplay between the sympathetic and parasympathetic nervous systems forms the foundation of heart rate regulation.

Heart rate variability is the measure of regulation of heart rate over time and has recently been associated with quantifying an individual’s response to stress (Boonnithi & Phongsuphap, 2011) and emotional regulation (Godfrey et al., 2019). Heart rate variability is commonly depicted as the interbeat interval or space between consecutive heart beats, giving insight into the relative balance between the sympathetic and parasympathetic nervous system activities. A branch of the parasympathetic nervous system, the myelinated vagus nerve (to the heart), inhibits parasympathetic nervous system driven behaviors such as fight-or-flight, hypothalamic–pituitary–adrenal axis activity and promotes a calm physiological state that promotes positive social affiliations such as sharing or caring (Porges, 2007). Therefore, heart rate variability can be thought of as a metric of the dynamic balancing between the sympathetic nervous system and parasympathetic nervous system, and the vagal nerve can be considered a brake that slows down heart rate and dampens threat processing. In moments of stress and reduced feelings of safeness, the heart rate increases (i.e. the vagal nerve “brake” is disengaged) but the capacity to disengage and re-engage the vagal brake is indicated by higher heart rate variability (Petrocchi & Cheli, 2019). Thus, increased heart rate variability is associated with the ability to downregulate physiological arousal and regulate stress (Thayer & Lane, 2009; Thayer et al., 2012), while lower resting heart rate variability has been associated with chronic reduced ability to downregulate psychophysiological arousal and stress responses (Thayer et al., 2012).

With the advancement of unobtrusive wearable technologies that can continuously monitor heart rate, we can gain access to vast amounts of data during treatment sessions that not only provide a snapshot of a patient's physiological state at a given moment but also track their changes over time. Machine learning algorithms can analyze these complex and copious amounts of data to detect patterns and trends that have largely gone undetected up to this point due to a reliance on human observers. These patterns, in turn, can offer insights into how the physiological responses of a patient or therapist evolve over the course of therapy, enabling an objective, data-driven understanding of the psychotherapy process.

Artificial intelligence can be leveraged to model the interplay between heart rate variability and various elements of psychotherapy. Higher heart rate variability, for instance, is associated with better emotion regulation (Luecken & Appelhans, 2006; Mather & Thayer, 2018; Yoo et al., 2018) enhanced metacognitive awareness (Lischke et al., 2017; Meessen et al., 2018), and increased empathy (Lischke et al., 2018). By using artificial intelligence models trained on heart rate and heart rate variability data and corresponding therapy outcomes, we can start to predict how changes in physiological states might influence or signal changes in the therapeutic relationship. Results from this kind of analysis can provide evidence-based feedback to therapists, helping them cultivate a stronger therapeutic alliance, build empathy, and more effectively repair ruptures in the therapeutic relationship. It can also provide real-time insights during sessions, enabling therapists to adjust their approach based on objective physiological data.

Electrodermal Activity: Galvanic Skin Response

Galvanic skin response, a method of measuring electrodermal activity linked to emotional and stress reactions (Neumann & Blanton, 1970) has been studied in psychotherapy research for over six decades (Riess, 2011). However, results pertaining to galvanic skin response data in relation to empathy has yielded inconsistent findings in the literature, with some studies showing a negative correlation and others showing a positive correlation between galvanic skin response and self-reported emotional empathy (Deuter et al., 2018; Messina et al., 2013). Although empirical studies have produced varying results, it is clear that there is a measurable association between electrodermal activity and the emotional responsiveness of therapists, perceived empathy by clients, physiological concordance/synchrony during emotional stimuli, and many self-reported measures of empathy (Del Piccolo & Finset, 2018). Given current measures make it difficult to assess empathy both qualitatively and quantitatively, there is a need for research that incorporates innovative techniques to further explore the association between electrodermal activity and empathy.

Mixed findings could be due to factors that current analytical methods are unable to consider. As previously discussed, machine learning algorithms can dissect large and complex datasets, unearthing patterns that may elude traditional analytic methods. These high-dimensional models can also integrate galvanic skin response data with additional physiological parameters such as heart rate and heart rate variability to paint a more comprehensive picture of a person's emotional and stress responses. This multimodal approach capitalizes on the strengths of each measure. Heart rate and heart rate variability shed light on the dynamics of sympathetic and parasympathetic nervous system activity, while galvanic skin response can offer insight into skin conductivity changes reflective of stress reactions. Together, using machine learning to examine heart rate, heart rate variability, and galvanic skin response could provide a richer understanding of the physiological underpinnings of psychotherapeutic processes. In the context of empathy, this multi-faceted data analytic approach could offer fresh insights. For instance, machine learning models could parse the nuanced interplay of heart rate, heart rate variability, and galvanic skin response in response to empathic encounters, potentially illuminating new physiological signatures of empathy.

Physiological Synchrony and Psychotherapy Process

While examining the distinct role of client and therapist heart rate, heart rate variability, and galvanic skin response with artificial intelligence and machine learning techniques has the potential to offer considerable insights into psychotherapeutic processes, gaining understanding of the shared emotional experience between counselors and clients while they are interacting is critical for psychotherapy process research. We argue that by examining heart rate, heart rate variability, and galvanic skin response in conjunction with physiological synchrony (Palumbo et al., 2017) using methods of empathetic accuracy within the context of machine learning and artificial intelligence techniques has the potential to open the 'Black Box' of psychotherapy (Chaspari et al., 2015).

Empirical findings demonstrate that physiological synchrony exists between individuals in various intimate interpersonal contexts (e.g. empathic exchanges; Palumbo et al., 2017), and the therapeutic dyad is no exception (Palumbo et al., 2017; Stratford et al., 2012; Tschacher & Meier, 2020). Meta-analytic studies reveal that higher levels of synchrony in heart rate and skin conductance response are associated with increased empathy and attention during therapeutic sessions (Palumbo et al., 2017), as well as overall higher ratings of alliance (Stratford et al., 2012; Tschacher & Meier, 2020).

While these findings are encouraging, the current approaches to evaluating physiological synchrony and associated provider–client dynamics has at least two limitations. One, traditional measurement and statistical approaches are unable to fully evaluate the complexity and richness of the physiological and interaction data to robustly study physiological synchrony. Related, current methodologies are not able to account for multiple measures of physiology to understand how synchrony or asynchrony among the physiological measures is important for capturing key moments in the psychotherapeutic process. Only a handful of studies have tried to use feedback regarding the other’s physiology during a live interaction to change behavior. For example, when people receive haptic feedback (i.e., touch, tactile information, nonverbal communication) of their romantic partner’s skin conductance response increasing or decreasing beyond a set threshold, they use more feeling words (Rojas et al., 2020). A master’s thesis found that when participants thought their heart rate was more similar to another’s, they rated that person as having higher empathic understanding, social connection, and shared knowing and understanding compared to those with dissimilar biosignals; this effect increased for more trait empathic participants (Dam, n.d.). However, the feedback was falsified in this case. Clearly, work on using feedback to adjust empathy in the moment is only beginning.

Machine learning and artificial intelligence techniques could sift through multivariate and time-series data to discern patterns that may otherwise remain hidden. By integrating heart rate, heart rate variability, galvanic skin response, and physiological synchrony, machine learning algorithms could learn the physiological signatures associate with successful therapeutic interactions. For instance, machine learning models could be trained to detect periods of emotional synchrony between the therapist and client, and then correlate these periods with successful therapeutic outcomes. This would pave the way for developing data-driven strategies for building therapeutic alliances, enhancing empathy, and repairing ruptures in the therapeutic relationship.

Furthermore, artificial intelligence and machine learning can extend beyond the realm of research to practical applications. Real-time monitoring and analysis of physiological indices and synchrony could provide immediate feedback to the therapists. Therapists could then adjust their approach in “real time” in response to the physiological cues of their clients, thus enhancing their empathic accuracy and strengthening the therapeutic alliance. Moreover, these sophisticated algorithms could identify personalized therapeutic strategies by learning from each unique therapeutic dyad's physiological patterns and interaction dynamics. This could lead to more tailored and effective approaches to build alliances, express empathy, and mend ruptures.

Artificial Intelligence and Machine Learning in Psychotherapy Processes

Developing software that can detect empathy is technically challenging because the interpretation of the intent of another person’s behavior is highly contextual and can be expressed in multiple modalities (e.g. text, audio, psychophysiology). Although multiple modalities are involved, most of the research in this area is limited to the analysis of text captured from conversations (Dwyer et al., 2018). Machine learning provides a solution in that models can be trained to classify the therapist’s behaviors as empathetic or non-empathetic based on observations and interpretations of the therapist’s behaviors during a session.

As suggested above, artificial intelligence and machine learning applications in psychotherapy have largely focused on ways to accelerate research through automated coding (behavioral or linguistic) (Aafjes-van Doorn et al., 2021). In this way, these approaches can be thought of as tools to make the current practice of psychotherapy process research easier and more efficient. We are suggesting a paradigm shift in which artificial intelligence and machine learning applications are used to provide automated classification of previously elusive (difficult to objectively discern) therapy process variables (i.e., empathy) and to inform therapist training and feedback systems through real-time predictive analytics that add new dimensions to the psychotherapy process state-of-art.

While there are several artificial intelligence technologies and machine learning algorithms, such as support vector machines, Bayesian classifiers, or decision trees (Hasan et al., 2016), central to this position paper are artificial neural network-based algorithms using machine learning and deep learning approaches because these approaches facilitate the incorporation of multimodal data and data fusion approaches that more completely capture the complexities of psychotherapy sessions. While current artificial intelligence approaches are an advance on traditional methods, most are limited to one data type (i.e., text, audio, physio, etc.). We assert that artificial intelligence-based approaches must gather data from multiple vantage points to get a more complete picture of psychotherapy process variables. Just as the human therapist gathers information from multiple patient indicators (i.e., body language, word choice, eye contact, situational context, or treatment history), the machine should be able to discern how multiple aspects of communication (i.e., speech patterns via natural language processing, word choice, body language) or physiological response (i.e., heart rate, galvanic skin response, cortisol or other hormonal responses to stress) to treatment may be leveraged for better understanding of therapy processes.

The key difference between machine learning and deep learning approaches is the incorporation of the human expert in determining input features. In machine learning, the human expert decides which input features are important in the final decision (classification or prediction), whereas deep learning approaches allow the machine (using copious amounts of data) to determine which input variables are the most important. Both approaches may be valuable in understanding psychotherapy process as some tasks, such as behavioral coding, may have very defined input variables known to the human expert for reasons of validation or reproducibility and be more suited to machine learning. While tasks such as classification of ambiguous therapy process variables such as empathy, trust, or alliance may be less clear to the human expert with respect to the correct input features, particularly when considering the inclusion of physiological data. These tasks may be more suited for deep learning approaches.

Supervised and Unsupervised Techniques

Supervised and unsupervised machine learning techniques offer distinct approaches to modeling the association between physiological data and psychotherapeutic processes such as empathy. Supervised learning is a method where an algorithm learns the relationship between labeled output data (usually derived from experts with validated processes to determine statistical reliability of labels), establishes relationships between inputs (e.g. physiological data components and their interrelationships), and outputs (e.g. psychotherapeutic outcomes like empathy). This technique offers an opportunity to train machine learning models to autonomously group combinations of input data into output classes (i.e., empathy or non-empathy). Similarly, if empathy was defined on a quantitative scale (ex. 0 – 10 where 10 is most empathic), a machine learning model could then be trained to predict which value or empathy score would be assigned to a given set of input data. (Chetouani et al., 2017). By providing models with many examples of input–output pairs during training, the machine can learn to make accurate predictions or classifications about new input data that it has not seen before.

On the other hand, while unsupervised learning techniques such as correlation analysis, recurrence analysis, and clustering analysis (e.g. K-Means) (Aafjes-van Doorn et al., 2021) is the most common approach used, it offers limited utility for developing algorithms that integrate physiological data for the prediction of pyrotherapeutic processes like empathy. These techniques are limited for our context for one main reason. Unsupervised approaches fail to include expert knowledge of context and outcomes of interest. While we argue that current research on empathy needs to be expanded to other data types that we know are physiologically or cognitively relevant, the previous work in elucidating empathy is well-established with validated measures. This work should not be disregarded to satisfy a completely data-driven approach. Rather, combinations of these approaches should be the optimal solution.

Although supervised learning techniques have not been applied in the same capacity as unsupervised ones, there is a solid foundation for using sub-categories of supervised learning techniques, namely neural networks, deep learning techniques, convolutional neural networks, and recurrent neural networks. These methods have shown potential for accurate prediction and classification problems involving audio and physiological data, as well as for extracting relevant spatial features from sensor data, reducing the need for manual feature labeling (Kanjo et al., 2019; LeCun et al., 2015; Li et al., 2020; Tripathi et al., 2019).

Previous research has successfully applied supervised machine learning to label and classify segments of session transcripts according to established codebooks of treatment processes (Hasan et al., 2016, 2019; Idalski et al., 2019). Various machine learning and deep learning algorithms have been employed, such as Support Vector Machine, Naïve Bayes, Conditional Random Fields, Decision Tree – J48, Adaptive Boosting (AdaBoost), Random Forest, DiscLDA, and Convolutional Neural Network. The Support Vector Machine has emerged as a particularly robust tool for automated coding (Hasan et al., 2016).

Considering the potential of these machine learning techniques, we propose a more integrated approach to psychotherapy process research. By combining expert knowledge, physiological measures, and advanced machine learning techniques, we can develop more accurate models for identifying negative interpersonal processes and devising effective strategies to address them. This integrative approach holds promise for advancing the study of psychotherapy processes in a manner that is data-driven and directly applicable to clinical practice.

Natural Language Processing

As we traverse the expanding frontiers of psychotherapy process research, the role of advanced Natural Language Processing techniques and machine learning methodologies is becoming progressively vital. Employing the Generative Pre-trained Transformer (GPT) model from OpenAI, NLP has become a powerful tool, especially if combined with supervised techniques (Azaria, 2022). Natural language processing (NLP) is a subfield of artificial intelligence that involves processing and analyzing large amounts of written text and spoken language to extract meaningful information, and is one of the primary ways artificial intelligence is applied in psychotherapy process research (Le Glaz et al., 2021).

Conceptually, NLP can be used to detect specific therapeutic processes, such as empathy, emotional regulation, and therapeutic alliance, and quantify their frequency and intensity. By using machine learning algorithms to identify patterns in the data, patterns that represent synchrony can be quantified and predictions about therapeutic outcomes can then be made (Goldberg et al., 2020). Common NLP algorithms that can be used for empathy detection are sentiment analysis, discourse analysis, word embeddings, and named entity recognition. Neural networks (discussed above) are also used to analyze digital data such as physiological signals or self-report data. Studies have shown the potential of artificial intelligence to outperform human raters in certain aspects of psychotherapy process analysis and provide a more objective and reliable assessment of therapeutic processes.

Emotional Speech Analysis (Bertero et al., 2016) algorithms are a group of NLP algorithms that are specifically used to detect empathy and other emotions from audio signals in psychotherapy (Flemotomos et al., 2021; Imel et al., 2015, 2017). These algorithms analyze audio data from therapeutic sessions to determine the emotional state of the patient and monitor therapy progress and effectiveness. Nevertheless, it is critical to be cognizant that these algorithms can potentially become inaccurate due to nuances such as accent, style of speaking, and environmental noise. Therefore, it is important to evaluate their performance across a variety of cultural contexts in order to consider any potential biases or constraints.

The research by Imel et al. (2015) over the past five years represents a major leap in this domain, illustrating the power of NLP and machine learning to supersede conventional dictionary-based methods, like linguistic inquiry and word count. This work highlights the ability of ML to explore clinically relevant psychotherapy constructs, which are often significant predictors of outcomes but difficult to measure qualitatively in traditional psychotherapy process research. A study exploring the interplay between therapeutic alliance and NLP found a direct association between a strong alliance and improved treatment outcomes (Goldberg et al., 2020). This growing body of evidence underscores the potential for integrating NLP and ML techniques in advancing our understanding of psychotherapy processes and improving client outcomes.

Recent Advances in the Use of Artificial Intelligence, Machine Learning and Physiology

Recent research integrating artificial intelligence and machine learning with physiology data are particularly relevant for this position paper, as they demonstrate the significant potential for improving psychotherapy process research applied to family-based EBTs. By utilizing multimodal ambulatory assessments, Timmons and colleagues (A. Timmons et al., 2017a, 2017b) were able to gain insight into couples' emotional states, vocalizations, as well as their physiological responses including heart rate variability, skin conductance response and respiration during daily life. The study used big data methods and machine learning techniques to analyze their extensive dataset and revealed that physiological synchrony was associated with positive emotional interactions and that positive vocalizations were associated with increases in heart rate variability. The study highlights the importance of examining both physiological and affective factors and the potential of big data methods for capturing and analyzing complex and dynamic relationships in real-world settings.

In a follow-up study, researchers examined how physiological, vocal, and motion data can be used to detect conflict among couples in real-time (Timmons et al., 2017a, 2017b). The authors utilized various wearable technologies to capture and process the data. This study employed machine learning tactics, for example support vector machines and decision tree analysis, to dissect the data and recognize patterns that may be associated with conflict. The findings indicated that the combination of physiological signals and speech provided more accurate detection of conflict compared to using only physiological signals or only speech. Additionally, the study found that physiological signals alone were not reliable indicators of conflict. These results highlight the potential of wearable technology in detecting conflict, which may be relevant to the examination of rupture and repair.

Two additional studies provide compelling results that demonstrate the potential for studies that integrated artificial intelligence, machine learning, and physiology data to improve the effectiveness of therapy. A study on provider behaviors that predict motivational statements in adolescents and young adults with HIV used the motivational interviewing (MI) framework to analyze audio-recorded clinical encounters between the providers and patients (Idalski et al., 2019). The audio recordings were transcribed and coded using a coding scheme designed to capture the provider's MI-consistent and MI-inconsistent behaviors. The researchers used natural language processing and a support vector machine algorithm to identify and classify instances of provider communication as either motivational or non-motivational, based on the framework of motivational interviewing. The algorithm achieved high accuracy in predicting the presence or absence of motivational statements, and the study demonstrated the potential of machine learning to facilitate the analysis of large volumes of clinical data and identify patterns of communication that are associated with positive health outcomes.

Miner et al. (2022) used NLP to develop a computational approach to measuring three linguistic characteristics of psychotherapy—timing, responsiveness, and consistency—and tested the approach using a dataset of transcribed therapy sessions. Specifically, NLP algorithms were used to identify linguistic features such as word choice, sentence length, and frequency of specific words, which were then used to calculate measures of timing, responsiveness, and consistency in therapist-patient interactions. The results showed that the approach was effective in measuring these linguistic characteristics and could provide a useful tool for evaluating therapy quality and identifying areas for improvement. This research suggested that therapists who expressed themselves with consistent and responsive language had a better success rate when working with clients.

Advancing Psychotherapy Process Research Toward Integration of Physiological Data and AI into Quantitative Training for Therapists

Significance

As discussed above, the current operationalization of therapeutic alliance and empathy was born from research that primarily used self-report measures, observational measures, and behavioral coding. Except for linguistic analysis and physiological measures, artificial intelligence and machine learning have mainly been used in psychotherapy process research to automate coding of self-report and observer ratings and make the process of assessing therapeutic alliance and empathy more efficient and has yet to incorporate largely unobservable correlates. We assert that the data revolution and subsequent inclusion of quantitative data-driven methodology in psychotherapy process research would provide a more complete picture of clinical interactions and removes the inherent ambiguity of reliance of expert opinion, supervisor observation, and manualized coding. Specifically, to fully understand the dynamic and synergistic nature of interpersonal interactions that facilitate therapeutic alliance and empathy in therapy broadly, and family-based therapy narrowly, we assert that it is essential to expand beyond self-report and expert observer ratings to include examining therapeutic alliance and empathy (defined as interpersonal synchrony and accuracy), as well as unobservable, non-linguistic neurobiological correlates of empathy such as interpersonal autonomic physiological data. This will open the "black box" of psychotherapy processes.

Our Current Work

One such example of the advancement of AI-driven approaches in psychotherapy is our current work, which integrates wearable sensing technologies, machine learning, natural language processing, and validated measures of therapeutic alliance and empathy. Within the context of a behaviorally focused, family-based intervention for children with obesity and behavior problems, and their obese or overweight, and largely uninvolved caregivers, to target activity level and weight loss, this approach will use machine learning to automatically classify instances and degrees of therapist empathy in family-based clinical interactions, utilizing artificial intelligence-derived predictive and classification models and the physiological data available from wearable devices (heart rate monitors, skin conductance sensors, and accelerometers). In addition to selecting physiological indicators of emotion, a key feature of the resulting data is volume. For example, a standard one-hour session using a heart rate monitor alone, collecting data at a modest rate of once per second, would result in over 3600 individual data points. With multiple sources of physiological data being recorded simultaneously, the data rapidly multiples, forming a rich dataset that is ideal for ML and AI methodologies. We hypothesize that this approach will provide a more complete picture of empathy and enable machine learning to find connections that may not be obvious. The elucidation of empathy in quantitative, less time-consuming ways, will enable the development of just-in-time therapist feedback as our work makes more clear the specific patterns and constraints of empathy on therapeutic alliance. However, the first part of this work is a more fundamental understanding of empathy and therapy process variables from a physiological perspective.

Implementation of our proposed approach requires several specific steps, including 1) data collection, 2) data preprocessing, 3) feature engineering, 4) model development and validation, and 5) model interpretation. We briefly describe those steps here.

1. Data Collection. We will audio-record sessions to collect linguistic data. We will also affix galvanic skin response and electrocardiogram sensors to youth, parents, and therapists to collect electrical conductivity of the skin due to sweating, heart rate and heart rate variability. Additionally, validated self-report measures and expert observer ratings of alliance and empathy will be used.

2. Data Preprocessing. We will pre-process these data, which involves sampling the data to determine what will go into the models and identifying and removing noise from the data. More specifically, for sampling the sensor data, this will involve ensuring that all time-series data is of equal length via interpolation. For sensor data, noise typically includes motion artifacts and temporary disruptions in connectivity. These will be removed via statistical identification of outliers (± 2 standard deviations). For the speech data, noise typically includes extraneous conversations and/or background noise. This will be removed by a series of digital filters that isolate frequencies related to the speech of interest.

3. Feature Engineering. The audio data will undergo feature extraction, isolating the common voice features (Arruti et al., 2014; Gold et al., 2012; Partila et al., 2015) of intonation, pitch, mel-frequency cepstral coefficients, and speaking rate. These features are commonly used in audio analysis to determine emotional state (Wang et al., 2022). There are over 30 prosodic and spectral features that could be explored.

4. Model Development and Validation. Our approach uses empathy measures (empathic accuracy, interpersonal synchrony, results of natural language processing) as labels for empathy within recorded therapy sessions. During empathic accuracy, interpersonal synchrony, and natural language processing data collection, the moments in which accurate empathy or synchrony is detected serve as time stamps for points in the session where empathy has occurred. We use these time stamps as labels in the training of the machine model. We will integrate the sensor data, audio data, and artificial intelligence and machine learning algorithms to generate three models: text and audio only, physiology only, and text, audio, and physiological. Generating three models will allow us to compare the contribution of each data type to the empathic context. Each of the audio features, will be represented as a time-series and the empathy labels from either empathic accuracy or synchrony will provide time stamps at which the model will interrogate these time-series. A time window will be devised around the labeled time stamp to encapsulate the signals active during the empathic exchange. After training, we will test the accuracy of each model (text only vs text + audio vs text + audio + physiological) with a test data set that will consist of session data not previously exposed to the model.

5. Model Interpretation. The developed recurrent neural network models will take the spatial features and use them to build a dynamic model of how these features changes over time. This means that the model can learn to selectively store and retrieve important past information based on the current input and the previous outcomes. The model then uses the learned spatial and temporal features to classify the session data as empathic or non-empathic. We will collect multivariate time-series data both in synchronous and asynchronous modes. We plan to use a cross-correlation approach via the Pearson product-moment correlation coefficient to demonstrate statistically relevant relationships between variables.

Ethical Considerations of Artificial Intelligence in Psychotherapy Research

It is important to note that the research proposed here is in its infancy, and there is much to be learned about potential limitations and challenges of combining the techniques and methodologies proposed. For example, some studies have highlighted the importance of human expertise in the interpretation of artificial intelligence results, as artificial intelligence-generated results may be subject to error or misinterpretation (Horn & Weisz, 2020). Additionally, the ethics of artificial intelligence -based research in psychotherapy is an important and ongoing area of discussion, as artificial intelligence algorithms may have the potential to affect therapeutic relationships and patient privacy. Moreover, we must consider the potential for researchers to unintentionally embed code that disadvantages marginalized groups (Timmons et al., 2022). There are copious amounts of research on the potential for bias in data-driven artificial intelligence approaches (Jobin et al., 2019). Care should be taken to include diverse groups in the collection of input data to improve the robustness of model predictions or classifications, as well as reduction in harmful bias. Focus on mitigating bias in input data and over-generalized model predictions will also improve the explainability (Nor et al., 2021) and transparency (Wagenmakers et al., 2021) of model decisions. Finally, it is important to engage participants as co-researchers in the process of data collection, analysis, and interpretation. By embracing a participatory approach in this new line of research, clients can become collaborators in the research process, which can help counteract objectification and potential ethical issues discussed above. In fact, our study protocol includes a pilot phase that involves interviewing providers and their clients about their perceptions of the study aims, their participation, and the risks and benefits to providing their physiological data for the stated aims.

Discussion

Children and adolescents with severe behavioral, emotional, or medical problems are at risk for short- and long-term negative outcomes. Family-based psychosocial treatments have been shown to be effective for treating these problems; however, their effect sizes tend to drop once they are taken out of academic settings and implemented in real-world practice settings. This issue, known as the implementation cliff (Harvey & Gumport, 2015), is in part, due to therapist-level factors, such as limited training and resources and inadequate psychotherapy process skills in treating multi-stressed, multi-problem families. Unfortunately, EBT trainings and manuals fail to provide guidance on effectively managing interpersonal process problems as they arise with these families. These gaps in training and implementation are, in part, due to limits in our knowledge of current psychotherapy process research. We think these gaps in training and implementation can be addressed through innovative methods in psychotherapy process research, such as physiological monitoring (in real-time) and AI (e.g. machine learning), to provide a more comprehensive understanding of the nuances of clinical interactions and improve the accuracy of psychotherapy process measurement. The use of AI and machine learning (ML) could also aid in effectively disseminating evidence-based practices into real-world clinical settings, ultimately improving mental health care for those in need.

The use of interpersonal processes to evaluate psychotherapy sessions has been a longstanding practice, but it has limitations in terms of capturing the full range of information available in a therapy session. For example, traditional self-report measures and gold standard methodologies (behavioral coding) fail to fully capture the nuances of interpersonal processes, which require observing factors such as voice tone, facial expression, and other metrics (e.g. eye gaze, body position) that are often overlooked or ignored. However, the application of AI and machine learning technologies has the potential to address this limitation. By analyzing multiple features of communication, such as intonation, pitch, and sequencing, these technologies can better capture the complexity of personal interactions and improve the prediction and training of empathy and therapeutic alliance. Current analytical tools lack the capacity to capture this information in a sustainable manner, leaving researchers without a complete understanding of the importance of these factors. Therefore, the integration of AI and ML in psychotherapy process research has the potential to transform the field and provide a more comprehensive understanding of these complex interactions. In addition to our current work, the framework presented in this manuscript can be used with other physiological data such as provider–client eye tracking to further move the needle on psychotherapeutic process research forward.

To wrap up this discussion, the potential benefits of creating a pre-trained model for psychotherapy process research are briefly highlighted. Although the initial effort to measure empathic accuracy and alliance may be comparable to ongoing studies, the end result will be a model that researchers and trainers can use without having to replicate the same work. The question arises of whether this pre-trained model can be applied to new data with the same input variables. The simple answer is yes, as long as the input variables are the same, the pre-trained model can be readily applied to new data as test data. This pre-trained model will be able to generate a classification or prediction of the targeted outcome. This will save significant time and resources for researchers and trainers who can then focus on the unique aspects of their study rather than replicating the same foundational work. Overall, the creation of a pre-trained model has significant potential to advance the field of psychotherapy process research.

The use of AI and ML is proposed as a potential vehicle to achieve these goals. The pre-trained model generated in our study of empathy and its physiological or natural language correlates can serve as a foundation for future research. By using transfer learning, new data such as cortisol or EEG measurements can be integrated into the pre-trained model, resulting in comparable accuracy with less training data. This approach is not limited to studies of empathy, but can also be applied to other psychotherapy process research topics, such as trust genuineness, or cultural humility. In addition, the generated algorithm can function as a real-time feedback tool for therapists to enhance their empathic skills. The reliable measure of empathy can provide immediate feedback during therapy sessions, leading to the development of improve therapeutic skills. This tool could have a significant impact on the training of therapists and ultimately improve patient outcomes. For example, this tool could be used to give real-time empathy feedback to a therapist that could adjust their application of the treatment, thus maintaining positive therapeutic alliance and improving outcomes for the patient. Similarly, a clinical supervisor could apply this tool as a quantitative metric of therapists’ development during training. In summary, future audit and feedback systems can be designed to provide just-in-time feedback, by unobtrusively gathering ecological momentary assessment data of within-session interactions, using multimodal wearable biosensing technologies, in-real time, will scale-up clinical training for the mental health workforce at substantial cost savings.

Data Availability

Data sharing is not applicable to this article as no new data were created or analyzed for this position paper.