Effect of nonverbal behaviors and speech characteristics of online medical ECAs on patients’ satisfaction: The mediating roles of empathy and trust

Abstract

Objective

Embodied conversational agents (ECAs) play a crucial role in digital healthcare. While existing studies have examined the effect of ECAs’ appearances and nonverbal behaviors on acceptance, the mechanism by which nonverbal behaviors and speech characteristics jointly influence patients’ satisfaction remains unclear. This study explores the impact of medical ECAs’ nonverbal behaviors and speech characteristics on patients’ perceived empathy, trust, and satisfaction.

Methods

This study adopted a mixed factorial experimental design. Independent variables included nonverbal behaviors (facial expressions: smiling/painful; body postures: open/closed) and verbal behaviors (speech rate and pitch: low/medium/high). Dependent variables included perceived empathy, trust, and satisfaction. Speech rate served as a between-subjects variable and pitch as a within-subjects variable. 75 participants were recruited to watch and interact with the 12 pre-recorded ECA videos presented in a random order. Perceived empathy, trust and satisfaction were measured.

Results

Body postures significantly affected satisfaction (p< .001), while facial expressions showed no significant impact (p=.302). Speech rate significantly influenced satisfaction (p=.034), with low rate yielding higher satisfaction than high rate (p= .044); pitch showed no significant main effect (p=.295). Trust fully mediated the relationship between nonverbal behaviors and satisfaction, while empathy and trust serially mediated this effect. Trust partially mediated speech characteristics’ effect on satisfaction. The combination of painful expression, open posture, low speech rate, and high pitch yielded significantly higher satisfaction than other conditions (p< .001).

Conclusions

Open body postures and low speech rate directly enhanced patients’ satisfaction. Trust mediated the effects of both nonverbal behaviors and speech characteristics on satisfaction. The optimal combination of open postures, painful expressions, high pitch, and low speech rate most effectively improved patient satisfaction. These findings provides empirical evidence for multimodal medical ECA design and enhancing digital health service experience.

Introduction

Embodied Conversational Agents (ECAs) are artificial intelligence systems integrating language and non-verbal interaction capabilities, ¹ particularly by combining large language models and techniques such as retrieval-enhanced generation to enhance the accuracy, transparency, and adaptability of interactions. ² As an emerging human-computer interaction technology, its application in medical support has become increasingly widespread. ³ ECAs can communicate with patients in real time and in a multimodal manner through human-like avatars, bringing new possibilities to traditional healthcare service models, particularly demonstrating significant potential in disease management, mental health support, and patient education.^4,5 They offer functions such as emergency medical consultations, symptom checks, appointment scheduling, and post-treatment follow-ups, all without time or location constraints. ⁶

These capabilities position ECAs as a central component of the Digital Front Door strategy—an approach in which technology serves as the primary gateway between patients and healthcare providers. The COVID-19 pandemic accelerated this structural shift from face-to-face care toward distributed, remote-first delivery models, with evidence suggesting that patients’ experiences at this initial digital touchpoint meaningfully shape their subsequent engagement, adherence, and healthcare utilization.^7,8 Yet this transition also introduces a critical design challenge: without the physical presence of a clinician, the communicative warmth and nonverbal cues that naturally foster empathy and trust must be deliberately embedded within the virtual interface. ⁹ In this context, the behavioral and communicative design of ECAs as the first point of digital contact may determine not only immediate patient experience, but also longer-term engagement with the broader care system.

However, the unique nature of medical contexts presents distinct challenges for ECAs in practical applications. The successful implementation of digital healthcare systems or technologies depends not only on the technological sophistication itself, but also on users’ understanding and willingness to adopt the technology, as well as meeting their performance expectations, effort expectations, and social impact. ¹⁰ Furthermore, Unlike customer service or e-commerce, medical consultations involve health-related anxiety, privacy concerns, and emotional vulnerability,^11,12 making patient trust and perceived empathy key factors in the success of medical ECAs.^13,14 Traditional online medical consultations primarily relied on text-based dialogue, lacking the emotional expression and non-verbal cues in face-to-face interactions. It could lead to inadequate information transmission and insufficient emotional resonance. ¹⁵ In contrast, ECAs can provide a richer and more intuitive communication experience through multimodal interaction, such as visual imagery, facial expressions, gestures, and speech characteristics. ¹⁶ The appearance and communication style of ECAs significantly influence trust, and the sensitivity of medical contexts makes trust-building particularly important.^13,14 Optimizing ECAs’ visual appearance, speech characteristics, and interactive behaviors could effectively enhance patients’ trust and acceptance of the healthcare system. ¹⁷

Trust and empathy are widely recognized in doctor-patient interactions as core factors influencing patients’ satisfaction and treatment adherence. Trust reflects patients’ beliefs in healthcare professionals’ competence, kindness, and integrity, ¹⁸ while empathy reflects a doctor’s ability to understand, feel, and share patients’ emotional experiences. ¹⁹ Researches indicated that a doctor’s expression of empathy could significantly enhance patient trust, thereby improving overall satisfaction.^20,21 For example, direct eye contact and a forward-leaning or upright posture, ²² mimicking others’ facial expressions, using facial expressions that convey concern, smiling, nodding, using a warm tone of voice, a lower pitch, and touching others’ arms or hands could influence patients’ perceived empathy.^23–25 Therefore, the behavioral characteristics of ECAs influence patients’ perceptions of trust and empathy.

Existing researches suggest that the design features of ECAs show a significant impact on patients’ psychological perceptions. These design features primarily include two dimensions: nonverbal behaviors, including facial expressions, body posture, and eye contact; and speech characteristics, such as speech rate, pitch and voice quality. ²⁶ Regarding nonverbal behaviors, facial expressions could rapidly influence patients’ trust judgments toward ECAs, with expressions of pain more effective than smiles in eliciting perceived empathy in medical contexts. ²² Body posture also plays a significant role, with open body postures such as uncrossed arms and leaning forward typically interpreted as more approachable and trustworthy. ²⁷ In particular, empathetic ECAs could improve patients’ experience during the consultation. ²⁸ According to studies on speech characteristics, speech rate and pitch together could greatly affect users’ emotional reactions and levels of trust; better trust was generally linked to lower pitch.^29,30 Gao et al. ³¹ also explored the impact of virtual agents’ stylization and speech characteristics on trust building, providing new insights for optimizing the multimodal design of ECAs.

Most existing studies have focused on the influence mechanisms of ECAs in a single modality, such as studying facial expressions or speech characteristics alone, lacking systematic studies on the comprehensive effects of nonverbal behaviors and speech characteristics. Second, research on the mechanisms through which these design features influence patient satisfaction via the mediating effects of trust and perceived empathy remains limited. Third, the optimal configuration of different multimodal behavioral cues in medical contexts has not been sufficiently validated.

This study aims to explore how nonverbal behaviors and speech characteristics of medical ECAs influence patient satisfaction through the mediating effects of trust and perceived empathy. Specifically, this study will: (1) examine the independent effects of ECAs’ facial expressions, body posture, pitch, and speech rate on patients’ satisfaction; (2) reveal the mediating role of trust and perceived empathy in the relationship between design features and satisfaction; (3) validate the interactive effects of nonverbal behavior and speech characteristics, and explore the optimal multimodal design combination. Furthermore, the results are expected to be applied in empathy training for healthcare professionals by analyzing effective empathy expression patterns, improving medical professionals’ communication skills, and ultimately improving patients’ overall healthcare experience and satisfaction. The findings can also guide the optimized design of medical ECAs, assisting developers in creating more trustworthy and patient-friendly virtual medical assistants.

Research model and hypothesis development

Nonverbal behaviors

Nonverbal behaviors refer to conveying information through nonverbal means such as facial expressions, body posture, and gestures. ³² In healthcare settings, the nonverbal behaviors are crucial for shaping patient-clinician interactions and influence patients’ perceptions of care quality. ³³ Related studies have identified specific nonverbal behaviors that influence how patients perceive clinicians.^34,35 These behaviors include eye contact, maintaining eye level with the patient, leaning forward, adopting an open posture, mirroring others’ expressions, displaying caring facial expressions, smiling, nodding, using a warm tone of voice, and appropriate physical contact. ²⁴ These nonverbal behaviors have a significant impact on offline medical consultations, affecting patients’ satisfaction and healthcare service utilization. ³⁶ Rosenthal-von Der Pütten et al. ³⁷ found that the nonverbal behaviors of ECAs significantly affected patient acceptance, with approximately 65% of patients’ acceptance stemming from nonverbal behaviors. However, the absence of vocal inflection and natural body movements may also affect patient acceptance of ECAs. Therefore, ensuring consistency between verbal and nonverbal cues is crucial for patient experience.

Facial expressions represent one critical aspect of nonverbal behavior. Facial expression interaction can improve subjective happiness with the patient experience by increasing their emotional engagement. ³⁸ Previous investigations have revealed that smiling can indirectly increase happiness by generating satisfaction and elevating patients’ sense of respect. ³⁹ Smiling in the medical settings can improve overall satisfaction by increasing patients’ comfort, trust, and positive perception of doctors. ⁴⁰ Doctors who display smiling expressions are perceived as more empathetic, enthusiastic, and capable of improving patient satisfaction than those without nonverbal behaviors. ²⁷ Patients often perceive doctors who lack smiles as less patient-centered and less willing to seek for medical care. ⁴¹ It was also found that negative expressions reduce patients’ satisfaction. ⁴²

Consistent with facial expressions, body posture is another significant nonverbal cue that affects the initial impression of doctor-patient interactions. ⁴³ Open and relaxed body postures are generally perceived as conveying greater warmth, calmness, informality, and approachability than closed or tense body postures. ⁴⁴ In particular, nodding, leaning forward, facing the patient directly, and keeping arms and legs uncrossed could improve patients’ satisfaction. ⁴⁵ The research on power posing of body posture by Carney et al. ⁴⁶ was applied explicitly to doctor-patient interactions by Forkin et al. ⁴⁷ They demonstrated that doctors who adopted high-power positions, or open postures, were more likely to be seen as self-assured, wise, and focused on leadership than those who adopted low-power postures, or closed postures. As a result, they were more likely to gain the trust and satisfaction of their patients. In light of these results, the following hypotheses are proposed:

H1a

ECAs displaying smiling expressions will significantly increase patient satisfaction compared to painful expressions.

H1b

ECAs exhibiting an open body posture will significantly increase patient satisfaction compared to a closed posture.

Chain mediation mechanism through trust and empathy

The ECAs’ nonverbal behaviors directly influence patients’ perceptions, and the effects can also be mediated through psychological variables. Trust, as a core concept in interpersonal relationships, has been widely applied in the field of interaction research between humans and virtual agents.^48,49 Based on the computer-as-social-actor paradigm, researchers have confirmed the applicability of interpersonal trust dimensions. ⁵⁰ Existing studies indicated that patients remained more skeptical in interactions with virtual agents compared with interpersonal interactions, ⁵¹ resulting in varying satisfaction levels. Patient satisfaction with virtual agents largely depends on their trust. ⁵² The transparency of interactive intent demonstrated by digital agents during interactions can effectively reduce patients’ perceived vulnerability, thereby fostering trust and promoting the adoption of new technologies.^53,54

Facial expressions are an essential part of nonverbal behaviors when building trust. People could establish trust judgments about others in a very short time, about 38 milliseconds. ⁵⁵ People make unconscious decisions about whether or not to trust others based on their observations of facial expressions. ⁵⁵ According to studies based on the theory of emotional generalization, positive expressions were regarded as the most trustworthy, while negative emotional expressions diminished trust and positive expressions significantly increased trust. ⁵⁶ Nevertheless, the function of facial expressions differed in varied situations.^44,57 Patients in discomfort were more likely to believe that a doctor is sincere and emotionally engaged, particularly when combined with open body postures. ⁴⁷ In contrast, smiling could make people appear more friendly, but its positive effects might be reduced when addressing serious health issues, as it might be perceived as lacking the necessary seriousness. According to Lennart Seitz et al., ⁵⁸ lacking an emotional foundation for trust restrict patients’ confidence in ECAs in medical contexts. While open postures conveyed a sense of acceptance, ⁴⁷ virtual agents consistently displaying pain alongside patients demonstrated empathy. ⁵⁹ Therefore, we propose a hypothesis that painful facial expression with open posture could be the most effective in building trust and improving satisfaction.

Empathy is another significant mediating factor that is associated with trust. Eklund and Meranius ⁶⁰ defined empathy as the capacity to feel, comprehend, and share others’ experiences. In other words, it is the process of recognizing and comprehending patients’ cognitive and emotional requirements and reacting appropriately.^51,61,62 Previous studies have primarily focused on two categories of empathetic reactions. The first is a cognitive response, sometimes referred to as perspective-taking, which is the capacity to comprehend the intentions, feelings, and ideas of others. ⁶³ The second is emotional response, which is demonstrated by trying to experience other people’s emotions and reacting to them in a way that aligns with how they perceive them. ⁶⁴ This study primarily focuses on the emotional responses elicited by ECAs. Although ECAs may lack empathy, ⁶⁵ they possessed the ability to express empathy, primarily through nonverbal behaviors that reflected human emotions. ⁶⁶ Facial expressions, eye contact, and open body postures that demonstrated concern or alignment with the patient were considered expressions of empathy. ²²

Trust and empathy are intimately linked. When ECAs exhibited both a forward-leaning posture and a painful facial expression at the same time, participants felt the most empathy. ²² Kraft-Todd found that physicians who adopted open, patient-facing positions were seen as more capable and sympathetic than those who wore more introverted with closed positions. ²⁷ Doctors’ empathy could facilitate positive doctor-patient interaction development, and previous studies and theoretical frameworks suggested that empathy was essential to patient trust. ⁶⁷ According to patients, doctors’ empathic actions, such as listening to their thoughts, comprehending their viewpoints, providing comfort, and providing individualized care were the cornerstone of trust. ²⁰ Additionally, empathic actions significantly increased patient happiness, and patients’ opinions of doctors’ empathy directly influenced their trust.^68,69 In light of this analysis, this study proposes the following hypotheses:

H2

The effect of ECAs’ nonverbal behaviors on satisfaction is mediated by enhanced patients’ trust. Specifically, the combination of painful expressions and open posture can increase patient trust, thereby improving satisfaction.

H3

ECAs’ nonverbal behavior combining painful expressions and open posture enhances patients’ satisfaction through increased perceived empathy, which subsequently strengthens trust.

The role of speech characteristics and their mediating effect

In virtual medical interactions, auditory speech characteristics are as important as visible nonverbal behaviors. Voice, the most basic form of human communication, is vital to the sense of service quality. The speech characteristics of service providers had a major impact on consumers’ perceptions of service quality, product assessments, level of satisfaction, and willingness to use.^70,71 In their study on voice-based consultation satisfaction, Gray et al. found that virtual and in-person consultations had very high patients’ satisfaction levels and performed comparably, providing crucial support for the application of speech characteristics in medical settings. ⁷²

Among speech characteristics, the influence mechanism of pitch requires further investigation. As an essential perceptual feature of speech, pitch enables listeners to identify pitch attributes such as high and low. ⁷³ Given the uncertainty of disease information, patients often feel anxious and seek positive cues from doctors. ⁷⁴ Researches indicated that high pitch could convey physicians’ positive emotions, reduce psychological stress through emotional contagion, and enhance patient confidence.^75–77 Therefore, pitch positively influenced patients’ satisfaction, notably higher pitch. ⁷⁸ However, some psychological studies also indicated that when voice pitch was higher, it might be perceived as emotionally immature ⁷⁹ or lacking calmness, ⁸⁰ thereby reducing patients’ satisfaction. However, in medical consultation settings, patients’ expectations of improved health often led to overlook those negative factors. In contrast, low pitch was typically associated with sadness or lack of joy, ⁸¹ aggression, and threat. Thus, low pitch might evoke negative emotions, exacerbate patients’ psychological burden, and lead to low satisfaction.

Another essential speech characteristic that correlates with pitch is speech rate, which is the quantity of speech units generated in a given amount of time and is strongly associated with information support. ⁸² People who speak more quickly are regarded as more compelling than those who speak more slowly, and a faster speech rate usually means that the doctor can give more information, demonstrating the degree of concern for the patient. ⁸³ However, rapid speech shortens the time that patients use to acquire and retain information. It may decrease their capacity to process and interpret information. ⁸⁴ A decrease in speech rate was found to be a way of expressing sadness. ⁸⁵ Doctors can help patients relax in medical situations by speaking more slowly. ⁸⁶

Additionally, ECAs’ speech characteristics may influence trust perception of patients. ⁸⁷ Researches on the relationship between voice and trust has produced mixed results. Some studies found that people are more likely to trust individuals with lower-pitched voices,^88,89 while other studies have found that individuals are more likely to trust people with higher-pitched voices. ⁹⁰ During doctor-patient interactions, doctors communicate with patients through voice, and the interactive sounds contain emotional and attitudinal information, which is conveyed nonverbally through emotional vocal cues. ⁹¹ Doctors who slow down their speech rate were perceived as more caring and empathetic when conveying information. Slowing down speech rate can help patients relax and alleviate anxiety, and it significantly influences patients’ perceptions when delivering bad news.^86,92 Based on these findings, this study proposes the following hypotheses:

H4a

ECAs with higher pitch will significantly increase patients’ satisfaction compared to medium or lower pitch.

H4b

ECAs with a slower speech rate will be associated with higher patients’ satisfaction compared to faster speech rates.

H5

The effect of speech characteristics on patient satisfaction is mediated by enhanced patients’ trust. Specifically, the combination of low speech rate and high pitch increases patients’ trust, thereby improving patients’ satisfaction.

H6

Combining a slower speech rate and higher pitch in ECAs further enhances patients’ trust by increasing their perceived empathy, thereby improving satisfaction.

Integrative effects of multimodal behavioral cues

The preceding analysis focused on the independent effects of nonverbal behaviors and speech characteristics. However, the integrative effects of ECAs’ behavioral cues during the interaction needed to be investigated in the online medical settings. The theory of emotional consistency proposes that when facial expressions, body posture, and speech characteristics align emotionally, observers are more likely to perceive authenticity and reliability, thereby enhancing emotional perception.^93–95 Additionally, de Gelder and Vroomen ⁹³ found in their multimodal emotional integration study that visual and auditory channels exhibited interactive effects in emotional perception, and inconsistent facial expressions and speech reduced the accurate identification of emotional information. Wang and Gratch ⁹⁴ further found that when the vocal emotions of ECAs align with their facial expressions, patients were more likely to perceive them as empathetic individuals with a sense of social presence. Behavioral consistency helped evoke patients’ psychological safety, enhancing trust and perceived empathy. ⁹⁶ Specifically analyzing the multimodal interaction combinations in this study, in a medical setting, virtual agents and patients displaying consistent expressions of pain can express concern and empathy for the patient’s predicament, ⁵⁹ conveying understanding and attention to patient suffering. Open postures, as a form of body language, convey an attitude of acceptance, support, and willingness to listen. ⁴⁷ Together, these two nonverbal cues create a pattern of caring emotional expression. According to speech characteristics, a high pitch could be used to express warmth and positive emotions, giving patients emotional comfort and enhancing their trust.^75,76,78 On the other hand, a slower speech rate can help patients relax ⁸⁶ and improve comprehension of information. ⁸⁴ Additionally, this combination of speech characteristics creates a pattern of care-oriented emotional expression.

Current research predominantly focused on single-modal interaction behaviors of ECAs, with limited attention paid to identifying which specific combinations of speech characteristics and nonverbal behaviors most effectively convey empathy and trust. Studies on multimodal emotion integration suggest that behaviors from different sensory channels undergo mandatory integration, which occurs automatically and unconsciously. ⁹³ Individuals simultaneously integrate visual and auditory information to form impression-based judgments of others. ⁹⁷ On the visual behavioral cues, combinations of nonverbal behaviors are crucial for communicating empathy. Riess and Kraft-Todd ³⁵ proposed the E.M.P.A.T.H.Y. tool, highlighting posture and facial expressions as key nonverbal elements in medical communication. Specifically, painful facial expressions effectively convey empathy and emotional resonance with patients’ distress, ⁵⁹ while open postures signal receptiveness and willingness to listen. ⁴⁷ Recent studies indicate that slower speech rates and higher pitch are reliably recognized by listeners as conveying greater empathy ⁹⁸ and trust, ⁸⁷ thereby enhancing satisfaction. Thus, when nonverbal behaviors and speech characteristics are presented simultaneously, they may produce a synergistic enhancement effect, more comprehensively addressing patients’ dual needs for empathy and trust during medical consultations.

Rahsepar Meadi et al. ⁹⁹ also examined the difficulties associated with virtual consultations, stressing the value of multimodal behavioral cues in delivering individualized and reliable healthcare services. Their findings revealed that patients’ perceptions of physician empathy and dependability significantly influenced satisfaction with virtual medical consultations. Drawing on these insights, the present study investigates how specific combinations of behavioral cues influence patient perceptions of ECAs in virtual healthcare contexts. This study proposes the following hypotheses:

H7

The combination of ECAs’ nonverbal behaviors featuring painful expressions and open posture with speech characteristics featuring lower speech rate and higher pitch will significantly enhance patient satisfaction.

Therefore, a research model was developed based on hypotheses H1-H7, as shown in Figure 1.

Figure 1.

Research model.

Methods

Study design

ECAs’ speech characteristics (speech rate: low, medium, high; pitch: low, medium, high) and nonverbal behaviors (facial expressions: smiling and painful; body posture: open and closed) were used as independent variables in a mixed factorial experimental design to test the research hypotheses. To reduce the possibility of fatigue effects, speech rate was manipulated as a between-subjects factor, while nonverbal behaviors and speech pitch were manipulated as within-subjects factors. For each between-subjects experiment, 25 participants were assigned to watch 12 video clips and complete a self-report scale following each clip. This study followed the APA Journal Article Reporting Standards (JARS) for experimental research in reporting the methods and results, ¹⁰⁰ incorporating the AGReMA guidelines as a reference for reporting the process and outcomes of the mediation analysis. ¹⁰¹

All data were collected in an independent laboratory with a quiet and private environment to minimize external interference. Participants sat alone in front of a computer screen and randomly watched and interacted with 12 pre-recorded ECAs videos. Each video depicted an ECA conducting a virtual consultation with a patient in a telemedicine setting. Three common mild symptoms were presented to simulate a real medical consultation, coughing, rash, and abdominal discomfort. The ECAs provided two standardized statements, the patient asking about symptoms and the ECA offering medical treatment recommendations, as shown in Table 1. After the inquiry, participants had a 7-second waiting time to describe their symptoms, enhancing immersion, shown as Figure 2.

Table 1.

Standardized verbal descriptions provided by ECAs.

Inquiry	Treatment suggestions
Hello, I am the physician responsible for your treatment. Please describe your symptoms, such as pain, fever, or discomfort.	Rash — Don’t worry. Based on the symptoms you described, you may have urticaria. This is a common skin condition characterized by itchy red welts or rashes. It is recommended to avoid known allergens as much as possible to relieve the symptoms and keep your skin cool and dry.
	Cough — Don’t worry. Based on the symptoms you described, you may have a cold or pharyngitis with fever. This is a common respiratory infection. You can take medications for a dry cough, such as amoxicillin, to alleviate the symptoms, and drink sufficient water.
	Abdominal discomfort — Don’t worry. You may have indigestion, bloating, or other gastrointestinal issues based on the symptoms you described. You should avoid greasy and spicy foods, and may consider taking antacids or prokinetic agents, such as omeprazole. After meals, avoid lying down immediately or engaging in strenuous exercise.

Figure 2.

Schematic diagram of ECA diagnosis and treatment process in each video clip.

Materials

The experiment employed a video-based paradigm in which participants watched pre-recorded stimuli. Following the experimental design, 36 video clips were created using DAZ Studio 4.23 and Blender 3.4. The ECAs were created using DAZ Studio 4.23 (64-bit) human models and clothing, with facial modifications, animations, and scene setup performed in Blender 3.4. The video clips were created using the keyframe method in Blender 3.4. Various keyframe interpolation methods were used to adjust transitions between different postures and expressions, with referenced open and closed postures by Forkin et al. ⁴⁷ Finally, the professional video editing software was used for voice synthesis and editing.

The ECA featured a male ECA wearing medical attire, set in a hospital environment. The choice of a male ECA was based on researches indicating that while female-voiced assistants might be perceived as more trustworthy, there was no significant difference in the degree to which medical advice was accepted between the male and female agent. ¹⁰² All ECAs displayed 36 combinations of two dynamic nonverbal behaviors (ie, facial expressions, body posture) and three variations in pitch and speed rate (low, medium, high). Facial expressions either aligned with the patient’s described symptoms (moderate painful expression) or not aligned (smiling). Body posture followed open and closed posture standards. ⁴⁷

The voice was created using the Alibaba Cloud platform, with pitch and speech rate adjusted by modifying basic parameters. The default medium pitch and speech rate were set to 0, high pitch adjusted to +40, low pitch to -40, high speech rate to 80, and low speech rate to -120. This study employed AI-based lip-sync technology (Lip-sync by Sieve platform) to enhance the naturalness and immersion of the ECAs’ speech in the videos and ensure synchronization between speech and lip movements, enabling the ECA to exhibit lip movements consistent with the speech contents. In total, 36 video vignettes were pre-recorded and an example was presented in supplementary file. Each video was approximately 30 seconds long, showcasing a front-view perspective of the ECA from the waist up, shown as Figure 3. A representative video clip has been provided as supplementary material.

Figure 3.

Schematic diagram of nonverbal behaviors of medical ECAs. (a) Smiling facial expression with an open body posture; (b) smiling facial expression with a closed body posture; (c) painful facial expression with an open body posture; and (d) painful facial expression with a closed body posture.

Experiment procedure

Before the formal experiment, a detailed description and consent of the study were given to the participants. Participants’ demographic data (name, age, and gender) and their experience with mobile medical consultation services (ie, whether they had previously used online medical services, how satisfied they were with those services, whether the responses from those services were useful, and whether they would use those services again) were collected at the beginning of the experiment. Each participant was randomly assigned to one of three speech rate conditions and E-Prime 3.0 software was used to present 12 videos in a random order. Participants were asked to fill out a questionnaire assessing their perceptions of empathy, trust, and satisfaction following each video. After every four video parts, participants were asked to identify the ECAs’ facial expressions to make sure that they were focused. Those who gave a wrong response were deemed inattentive and were not included in the analysis. After completing the about 30-minute-long experiments, participants were given a gift to compensate for their time.

Participants

This study used the GPower 3.1 to estimate the sample size, with a moderate effect size of 0.25, a significance level of 0.05, and a statistical power of 80%. ¹⁰³ It was estimated that at least 60 valid participants were needed. A total of 80 participants were recruited from the university community. Five participants were excluded due to failed attention checks, resulting in a final sample of 75 participants (47 females, 28 males; M_age=23.0 years, SD=2.5). Detailed demographic characteristics were presented in Table 2. Written informed consent forms were obtained from all the participants before the experiment. This study was approved by Ningbo University Ethics Committee (approval number: NBU-2025-483) and conducted in accordance with the Declaration of Helsinki.

Table 2.

Demographic characteristics and online healthcare service experience of participants (N = 75).

Variable	Category	Frequency (n)	Percentage(%)
Gender	Female	47	62.7
	Male	28	37.3
	Total	75	100.0
Ever used online medical services	Yes	47	62.7
	No	28	37.3
Satisfaction (users only, n = 47)	Very dissatisfied	2	4.3
	Dissatisfied	4	8.5
	Neutral	27	57.4
	Satisfied	12	25.5
	Very satisfied	2	4.3
Perceived helpfulness of the responses (n = 47)	Helpful	34	72.3
	Not Helpful	13	27.7
Willingness to reuse (n = 47)	No	4	8.5
	Depends	24	51.1
	Yes	19	40.4

Measures

To assess the participants’ trust, empathy, and satisfaction with ECA, a 10-item scale was used for trust assessment, adapted from the studies by McKnight et al. ¹⁰⁴ and Guo et al., ¹⁰⁵ including three dimensions: competence, benevolence, and integrity. Perceived empathy was measured using a 9-item scale adapted from Paiva et al. ¹⁰⁶ Satisfaction was measured using a 7-point single-item scale ranging from very dissatisfied to very satisfied. All measurements utilized a 7-point Likert scale. The detailed contents of the scales were appended as Table 3.

Table 3.

Trust and perceived empathy measurement scales.

Scale/Measurement	References	Items
Trust – Competence	McKnight et al., 2002. 104	The ECAs perform their work very well.
		The ECAs are very knowledgeable and experienced in the medical field.
		The ECAs fulfill the physician’s role very well.
	Guo et al., 2016. 105	The ECAs are competent and able to provide helpful advice.
Trust – Benevolence	McKnight et al., 2002. 104	The ECAs genuinely care about my health and try to help me.
		If I have questions, the ECAs will do their best to answer them.
		I believe the ECAs act in my best interest.
Trust – Integrity	McKnight et al., 2002. 104	I believe the ECAs are very sincere.
		I believe the ECAs’ responses are reliable.
		I believe all the ECAs’ responses are honest.
Perceived Empathy	Paiva et al., 2017. 106	The ECAs treat me with warmth and care.
		The ECAs see things from my perspective.
		The ECA feels sorry for my illness.
		The ECA considers various aspects when responding.
		If the ECA saw me injured, they would want to protect me.
		The ECA seems to understand my concerns.
		The ECA answers my questions thoroughly and from multiple perspectives.
		I think the ECA is a compassionate person.
		The ECA adopts a positive approach and attitude, putting themselves in my position.

Data analysis

First, the collected data was organized and screened to ensure all questionnaires were fully completed and data quality was maintained. Scale data was entered based on the scoring of each item. During data entry, reverse-scored items were processed accordingly, and metrics such as the mean and median values for each variable were calculated to serve as indicators for subsequent analysis. Then, normality and reliability tests were performed prior to data analysis. Except for satisfaction, which was measured using a single 7-point Likert scale without assessing reliability, all other items demonstrated excellent internal reliability. Normality was assessed using the Kolmogorov-Smirnov and Shapiro-Wilk tests, and all variables significantly deviated from a normal distribution (ps < .05). Therefore, nonparametric tests were employed: Friedman tests were used to examine within-subjects effects of facial expression and body posture (H1a, H1b) and pitch (H4a), while a Kruskal-Wallis test with Dunn-Bonferroni post hoc comparisons was used for the between-subjects effect of speech rate (H4b). No demographic covariates were included in these analyses; the potential influence of age, gender, and prior online medical service experience is acknowledged as a limitation.

All three dependent variables (perceived empathy, trust, and satisfaction) were measured simultaneously at a single time point immediately following each video clip. Although this cross-sectional measurement limits causal inference, the hypothesized serial mediation pathway was established based on established theoretical frameworks in doctor-patient interaction research.^20,21 Mediation analyses were conducted using the PROCESS macro (Model 4 for simple mediation; Model 6 for serial mediation) with 5,000 bootstrap samples, and significance was determined by 95% bias-corrected confidence intervals excluding zero. All data processing was performed in IBM SPSS Statistics 27, and graphs were drawn using OriginLab 2024.

Results

Descriptive statistics and reliability analysis

Reliability referred to the consistency of a set of measurements, meaning that when similar methods are used to measure the same psychological construct, the two results show a high degree of correlation. ¹⁰⁷ Descriptive statistical analysis and reliability measurements were performed for all variables. The results showed that satisfaction had a median of 5.00 (IQR = 2.00, range = 1-8), trust had a median of 5.15 (IQR= 1.50, range = 1.10-7.40, α = 0.945), and empathy had a median of 4.50 (IQR = 1.88, range = 1.00-7.63, α = 0.938). Satisfaction was measured using a single item on a 7-point Likert scale. Since it only included one item, internal consistency was not assessed. All other variables demonstrated acceptable internal reliability (αs>.76). Exploratory factor analyses confirmed the construct validity of both multi-item scales, with single-factor structures explaining 67.32% and 70.25% of the variance for trust and empathy, respectively (KMOs >.92, all ps<.001). Normality was assessed using the Kolmogorov-Smirnov and Shapiro-Wilk tests. Since all variables deviated significantly from normal distribution, nonparametric tests were used to ensure the robustness of the results.

Effects of nonverbal behaviors and speech characteristics on satisfaction

To investigate the effects of nonverbal behaviors and speech characteristics on satisfaction, nonparametric tests were conducted to examine the main effects of nonverbal behaviors (facial expressions, posture) and speech characteristics (pitch, speech rate) on satisfaction. The results were presented in Table 4. The Friedman test results indicated that the main effect of facial expressions (smiling vs. painful) on satisfaction was not significant (χ²(1) = 1.07, p=.302). Although the mean rank of the smiling expression (M=1.52) was slightly higher than that of the painful expression (M = 1.48), the difference was insufficient to support statistical significance, suggesting that facial emotional cues had a limited impact on patient satisfaction in medical settings, and thus did not support hypothesis H1a. In contrast, under the body posture condition, satisfaction with open postures (M=1.66) was significantly higher than that with closed postures (M=1.34), with the difference being statistically significant (χ²(1)=69.66, p < .001), supporting hypothesis H1b. Regarding speech characteristics, there was no significant difference in satisfaction levels across pitch levels (low, medium, high) (χ²(2)=2.44, p=.295), thus not supporting hypothesis H4a. The Kruskal-Wallis test revealed a significant main effect of speech rate on satisfaction (H(2)=6.77, p =.034). Further Dunn-Bonferroni post hoc analysis indicated that satisfaction with low speech rate (mean rank=481.08) was significantly higher than that with high speech rate (mean rank=430.68, p=.044), supporting hypothesis H4b.

Table 4.

Statistical analysis results of the effects of different nonverbal behaviors and speech characteristics on patients’ satisfaction.

Variable	Conditions	Mean ranks	Test statistic	p
Facial Expression	Smiling, Painful	1.52 (smiling), 1.48 (painful)	χ2(1) = 1.07	.302
Body Posture	Open, Closed	1.66 (open), 1.34 (closed)	χ2(1) = 69.66	< .001
Vocal Pitch	Low, Medium, High	1.94, 2.03, 2.03	χ2(2) = 2.44	.295
Speech Rate	Low, Medium, High	481.08, 439.74, 430.68	H(2) = 6.77	.034*

*Note. Only the comparison between low and high speech rate was significant after Bonferroni adjustment (p=.044).

The mediating role of trust

To examine whether the nonverbal behaviors of ECAs (the combination of painful expressions and open postures) indirectly influenced patient’ satisfaction by enhancing their trust, a mediation analysis was conducted using the PROCESS Model 4, ¹⁰⁸ with 5,000 bootstrap resampling iterations. The independent variables were dichotomized, with the combination of painful expressions and open postures coded as 1 and all other combinations as 0. The results were shown in Table 5.

Table 5.

Mediation and serial mediation analysis results.

Path	B	SE	t	p	95% CI
Mediation Results: Nonverbal Behavior → Satisfaction
Nonverbal Behavior → Trust	0.291	0.084	3.44	0.001	[.125, .456]
Trust →Satisfaction	1.012	0.025	40.17	<.001	[.962, 1.061]
Total Effect	0.388	0.106	3.64	0.001	[.179, .597]
Direct Effect	0.094	0.064	1.47	0.143	[–.032, .220]
Indirect Effect	0.294	0.077	-	-	[.147, .448]
Mediation Results: Speech Characteristics → Satisfaction
Speech Characteristics → Trust	0.285	0.117	2.44	0.015	[0.056, 0.514]
Trust → Satisfaction	1.02	0.025	40.65	< .001	[0.971, 1.069]
Total Effect	0.129	0.148	0.87	0.384	[-0.161, 0.419]
Direct Effect	-0.162	0.088	-1.84	0.066	[-0.334, 0.011]
Indirect Effect	0.291	0.1	-	-	[0.095, 0.483]
Serial Mediation: Nonverbal Behavior → Empathy → Trust → Satisfaction
Nonverbal Behavior→ Empathy	0.519	0.103	5.06	< .001	[0.318, 0.720]
Empathy → Trust	0.646	0.017	37.9	< .001	[0.612, 0.679]
Nonverbal Behavior→ Trust	0.177	0.072	2.45	0.014	[0.035, 0.318]
Trust → Satisfaction	0.623	0.037	16.78	< .001	[0.550, 0.696]
Direct Effect	-0.005	0.059	-0.08	0.937	[-0.121, 0.111]
Indirect Effect	0.209	0.043	-	-	[0.126, 0.296]
Serial Mediation: Speech Characteristics → Empathy → Trust → Satisfaction
Speech Characteristics → Empathy	0.169	0.143	1.18	0.24	[-0.112, 0.450]
Empathy → Trust	0.642	0.017	38.29	< .001	[0.609, 0.675]
Speech Characteristics → Trust	0.177	0.072	2.45	0.014	[0.035, 0.318]
Trust → Satisfaction	0.627	0.037	16.87	< .001	[0.554, 0.700]
Direct Effect	-0.118	0.08	-1.47	0.141	[-0.276, 0.039]
Indirect Effect	0.247	0.106	-	-	[0.042, 0.462]

Note. Nonverbal behavior combinations (painful expression and open posture combination), Speech characteristics combination (low speech rate and high pitch), Bootstrap sample size = 5,000.

The mediation analysis revealed that the nonverbal combination of painful expressions and open postures significantly and positively predicted patients’ trust in ECAs (B=0.291, p < .001), and trust further positively predicted satisfaction (B=1.012, p < .001). Although the direct effect of nonverbal behaviors on satisfaction was not significant (B=0.094, p=.143), the indirect effect was significant (B = 0.294, 95% CI [0.147, 0.448]), supporting hypothesis H2, indicating that trust fully mediated the relationship between nonverbal behaviors and satisfaction.

To test whether the combination of speech rate and pitch influenced patients’ satisfaction by mediation role of trust, a mediation analysis was conducted using the PROCESS Model 4. The results were shown in Table 5. The combination of low speech rate and high tone of voice significantly enhanced patients’ trust (B=0.285, p = .015). Trust also had a significant positive predictive effect on satisfaction (B=1.020, p < .001). Although the direct effect of the combination on satisfaction was marginally significant (B=–0.162, p = .066), its indirect path through trust was significant (B = 0.291, 95% CI [0.095, 0.483]), supporting hypothesis H5 and validating the partial mediating effect of verbal behavior combinations by a mediation role of trust.

Serial mediation effects of perceived empathy and trust

A serial mediation analysis was conducted using PROCESS Model 6 to examine whether nonverbal behaviors (painful expressions and open postures) influenced patient satisfaction through perceived empathy and trust. The results were shown in Table 5. Nonverbal behaviors significantly enhanced perceived empathy (B=0.519, p < .001), empathy strengthened trust (B=0.646, p < .001), and trust further enhanced satisfaction (B=0.623, p <.001). The serial path was significant (B=0.209, 95% CI [0.126, 0.296]), while the direct path was not significant (B=-0.005, p=.937). Empathy and trust played a complete mediating role in this path, supporting hypothesis H3.

Further examining whether empathy influenced satisfaction through a sequential path leading to trust. The PROCESS Model 6 results, as shown in Table 5, indicated that the combination of low speech rate and high pitch did not significantly affect perceived empathy (β = 0.169, p = .24). Still, it had a significant positive effect on trust (β = 0.177, p = .014). Both empathy and trust significantly predicted satisfaction. Although the total impact of voice on satisfaction was not significant (β = -0.118, p = .141), the separate mediating path through trust was significant (Boot 95% CI = [0.034, 0.195]). The total effect was significant through the indirect impact of empathy and trust but not through the direct effect, thus partially supporting H6, indicating that only trust played a significant mediating role, while empathy did not, suggesting that speech characteristics primarily influenced satisfaction through trust.

Interaction effects of nonverbal behaviors and speech characteristics

Since the data did not follow a normal distribution, the Mann-Whitney U test was used to compare patient satisfaction differences between the group with speech characteristics and nonverbal behavior (coded as 1) and the other groups (coded as 0). The analysis results were shown in Table 6, indicating that the combination group’s satisfaction scores were significantly higher (p < .001). Specifically, participants in the group exhibiting a combination of painful facial expressions, open posture, slow speech rate, and high pitch had significantly higher satisfaction scores than those in the non-combination group (U=17598.000, Z=-3.442, p < .001). This result supported hypothesis H7.

Table 6.

Mann-Whitney U test results for the effect of behavioral combinations on patients’ satisfaction.

Group	Sample size (N)	Mean rank	Sum of ranks
Non-combination Group (coded = 0)	875	458.11	400848.00
Combination Group (coded = 1)	55	583.04	32067.00

Based on all analyses, the results of hypothesis tests regarding the effects of nonverbal behaviors (facial expressions, body posture) and speech characteristics (speech rate, pitch) on patients’ satisfaction were summarized, and the mediating role of empathy and trust in this process was demonstrated, as shown in Table 7.

Table 7.

Hypothesis test results.

Hypothesis	Findings	Test/model used	Significance (p < .05)	Supported
H1a (Facial Expression → Satisfaction)	Facial expression had no significant effect on satisfaction.	Friedman test	Not significant (p = .302)	Not supported
H1b (Body Posture → Satisfaction)	An open posture significantly increased satisfaction.	Friedman test	Significant (p< .001)	Supported
H4a (Vocal Pitch → Satisfaction)	Vocal pitch had no significant effect on satisfaction.	Friedman test	Not significant (p = .295)	Not supported
H4b (Speech Rate → Satisfaction)	A low speech rate significantly increased satisfaction.	Kruskal–Wallis test & Dunn–Bonferroni	Significant (p = .034, only low vs high significant)	Supported
H2 (Nonverbal Behaviors a → Trust→ Satisfaction)	Trust played a full mediating role in the effect of nonverbal behaviors on satisfaction.	PROCESS Model 4	Significant (indirect effect: p < .001)	Supported
H3(Nonverbal Behaviors a →Empathy → Trust → Satisfaction)	Empathy and trust played a full mediating role in the effect of nonverbal behaviors on satisfaction.	PROCESS Model 6	Significant (serial mediation: p < .001)	Supported
H5 (Speech Characteristics b → Trust → Satisfaction)	Trust played a partial mediating role in the effect of speech characteristics on satisfaction.	PROCESS Model 4	Significant (indirect effect: p = .015)	Supported
H6 (Speech Characteristics b → Empathy → Trust → Satisfaction)	Only trust was a significant mediator; empathy was not significant.	PROCESS Model 6	Partially significant (Trust: p = .014, Empathy: p = .240)	Only trust mediation supported
H7 (Combined Behaviors c → Satisfaction)	Combined behaviors significantly increased satisfaction.	Mann-Whitney U test	Significant (p < .001)	Supported

^aNonverbal behaviors: Painful expression and open posture.

^bSpeech characteristics: Low speech rate and high pitch.

^cCombined behaviors: Painful expression, open posture, low speech rate and high pitch combination.

Discussion

This study systematically examined how nonverbal behaviors and speech characteristics of medical ECAs influenced patients’ satisfaction, focusing on perceived empathy and trust as mediators. This study found that the body posture of ECAs significantly affected patient satisfaction, whereas the facial expression’s main effect was insignificant. This result contrasted to some extent with established studies. Empirical studies such as Kraft-Todd et al. ²⁷ and Forkin et al. ⁴⁷ showed that open body posture significantly enhanced patients’ first impressions, trust and satisfaction with their physicians. The present study similarly found that patients reported higher levels of trust and satisfaction when the ECA adopted an open posture. In contrast, facial expression’s effect on patients’ satisfaction did not reach significant levels, which was not entirely consistent with the finding that smiling enhanced satisfaction in some studies.^39,40,109 A possible explanation is that body posture affects participants’ whole range perception. ¹¹⁰ Future studies could utilize eye-tracking technology to more accurately capture the distribution of patients’ attention during interactions with ECAs.

Furthermore, when paired with an open posture, painful expressions considerably improved patients’ perceptions of ECAs empathy without lowering satisfaction. This finding was in line with the study by Marcoux et al. ²² and Choi et al., ¹¹¹ which implied that the nonverbal combination of open postures and painful expressions was interpreted as a sign of greater empathy. Specifically, individuals with high empathy traits were more likely than people with low empathy to unconsciously display facial responses that matched the facial expressions of the observed subject.^112–116 This automatic facial consistency reflected emotional empathy with the observed subject.^117–119 For example, observers could feel another person’s pain and display matching painful expressions. Mediation analyses further revealed that the effect of nonverbal behaviors on satisfaction was fully mediated by trust, an essential antecedent of perceived empathy. This finding is further consistent with UTAUT-based models applied in digital health contexts, in which trust has been identified as a pivotal variable mediating the relationship between system-level features and adoption or satisfaction outcomes.^53,54 Notably, the role of social influence documented in AI system adoption studies ² aligns with our finding that the social and emotional cues conveyed by ECAs such as open postures and empathic vocal characteristics function as social presence signals that strengthen trust and ultimately drive satisfaction.

The present study furtherly validated the interaction effects among different nonverbal behaviors to enhance their perceived empathy and trust of ECAs. In contrast, a single smiling facial expression that lacked support from other behaviors might instead be interpreted as unserious or out of place, ¹²⁰ weakening its positive effect on satisfaction. Multidimensional combinations of nonverbal behaviors increased patients’ satisfaction more than single dimensions, and their path of action was primarily through influencing perceived empathy and trust.

Unlike earlier studies, pitch had no direct impact on patients’ satisfaction when it came to speech characteristics. Our study found that slower speech rate resulted in higher levels of satisfaction, inconsistent with the findings of Liu et al.’s ⁷⁷ study, which found that the speed at which a doctor spoke in their questioning tone was positively correlated with patients’ satisfaction and that doctors who spoke at a high speed tended to have high patients’ satisfaction, while faster speech rate decreased patients’ satisfaction. For patients did not have enough time to receive and analyze the information, fast speech might diminish cognitive trust, making it more difficult for patients to understand and perhaps leading to reduced satisfaction. ¹²¹

Although past studies have explored the effects of a single dimension of speech rate or pitch on patients’ trust and satisfaction,^76,77 the interaction effect between pitch and speech rate has not been systematically examined in healthcare contexts. In this study, we found that the combination featuring low speech rate and high pitch significantly enhanced patients’ trust in ECAs compared to other combinations, thereby increasing overall satisfaction. The serial mediation analysis also indicated that although this combination failed to enhance perceived empathy directly, it indirectly affected satisfaction through enhanced trust. Previous studies showed that low speech rate made utterances more explicit, reducing patients’ cognitive load and information anxiety.^85,86 Meanwhile, high pitch stimulated positive emotional responses, conveying rapport and warmth while making it easier for patients to establish social connections with physicians. ⁷⁸ This combination helped to weaken the sense of distance caused by virtual identity and enhance the judgment of ECAs’ trust in virtual medical scenarios. Prior researches have shown that a lower pitch was often perceived as more authoritative and enhanced listeners’ trust in the speaker.^88,122 However, our findings suggested that patients’ trust when interacting with medical ECAs relied more on the emotional cues conveyed by the voice than on the power cues. Specifically, the combination of low speech rate and high pitch was more in line with the need to build social connection and emotional support during medical consultations, and had a greater impact on enhancing ECAs’ trust and satisfaction of ECAs. The interaction between pitch and speech rate was not only a tool for information transfer, but also constituted an essential tool for physicians’ emotional expression and social presence construction.

The findings of this study also revealed the importance of multimodal emotional integration. It has been shown that mandatory integration of emotional information from different sensory channels occurs and that this integration process was automatic and unconscious.^93,123 The theory of emotional congruence states that when facial expressions, body postures, and speech characteristics are emotionally congruent, it is easier for observers to perceive authenticity and trust.^93,94 This study also validated this theory, which found that combining nonverbal behaviors (painful expressions and open postures) with speech characteristics (low speech rate and high pitch) of ECAs significantly enhanced patient satisfaction. This study empirically confirmed that this combination of nonverbal behaviors and speech characteristics significantly impacted patient satisfaction in digital healthcare consultation, and future ECA design should fully consider combining the two to enhance patients’ experience.

Limitations and future research

While this study revealed how nonverbal behaviors and speech characteristics of ECAs in digital health settings affected patient satisfaction through empathy and trust, its scope of study also demonstrated several limitations. In terms of experimental design, this study used a relatively simplified medical consultation scenario in which participants watched video clips and assessed them. It was a single-exposure experimental design that may not fully reflect the complexity and dynamics of fundamental medical interactions. Moreover, nonverbal communication is an ongoing and dynamic process. ¹²⁴ As Lal and Neduncheliyan ⁵ pointed out, patients’ perceptions of virtual healthcare agents change as the duration of the interaction increases. Thus, future researches should adopt a longitudinal design to explore the persistence of the effects of virtual agents in long-term use situations. Additionally, all dependent variables were measured simultaneously at a single time point, which limits causal inference in the observed mediation pathways.

Another limitation is the use of questionnaires for measurement only. Although questionnaires provide essential results, they may not capture subconscious or transient impressions triggered by nonverbal behaviors. Future studies may introduce physiological measures such as heart rate variability, galvanic responses, or eye tracking that can provide a more nuanced understanding of patients’ emotional responses to physician cues. ¹²⁵ The sample was drawn primarily from the university community, and the participants were mostly young students, which may have limited the external validity of the results. Since age, medical experience, and digital literacy can influence patients’ responses to medical ECAs,^126,127 and these variables were not included as covariates in the current analyses, future studies should both expand the sample and account for these demographic factors in statistical models.

Beyond methodological limitations, this study raises ethical concerns that warrant attention. While optimized combinations of nonverbal cues painful expressions, open postures, low speech rate, and high pitch enhance perceived empathy and trust, their persuasive effectiveness also carries risks. Overly convincing ECAs may foster uncritical patient reliance on AI agents, potentially impairing critical evaluation of medical information. The WHO’s ethics guidelines on AI for health caution that emotionally engaging systems can induce automation bias, wherein patients inappropriately defer medical judgment to automated agents. ¹²⁸ Furthermore, ECAs that exhibit near-human behavioral characteristics may precipitate the Uncanny Valley Effect, which can paradoxically induce discomfort or evasive responses in certain users. ¹²⁹ Bach and Männikkö furtherly emphasize that justified patient trust in AI healthcare must rest on transparency and human oversight rather than emotional cues alone. ¹³⁰ Future work should therefore balance behavioral optimization with clear ethical boundaries—including safeguards against over-reliance, accurate information delivery, disclosure of artificial identity, and alignment with national and international AI ethics frameworks.^128,131

Conclusions

This study focused on the nonverbal behaviors and speech characteristics of medical ECAs in digital healthcare contexts, exploring how they influenced patients’ perceptions of empathy and trust, thus affecting patients’ satisfaction. Through a mixed experimental design and empirical analysis, the study revealed the independent effects of various behavioral cues and clarifies their interaction mechanisms and mediating pathways. Results showed that open body postures and low speech rate significantly improved patient satisfaction, whereas facial expressions and vocal pitch alone did not produce significant effects. Trust served as a key mediator in the relationship between ECAs’ behaviors and patient satisfaction, with empathy and trust jointly forming a serial mediation pathway. The findings indicated that combining ECAs’ nonverbal behaviors, specifically painful facial expressions and open body posture, with speech characteristics, such as low speech rate and high pitch, significantly enhanced patient satisfaction. This combination could lead to the highest level of patient satisfaction compared to other combinations, ultimately improving the patients’ consultation experience. As AI healthcare systems become more widespread, the emotional expression capabilities of virtual agents will become a key factor in influencing patients’ perceptions of empathy, trust, and satisfaction. The behavioral design framework and validated pathways proposed in this study provide theoretical guidance and design standards for developing future digital healthcare systems.

ORCID iD

Fei Fang https://orcid.org/0009-0001-3835-3907

Ethical considerations

Ethical approval was received from Ningbo University Ethics Committee (approval number: NBU-2025-483).

Consent to participate

Written informed consent was obtained from all participants prior to participation.

Data Availability Statement

De-identified data will be made available upon reasonable requests to the corresponding author and with data use agreements in place.*