Effects of personalization and source expertise on users’ health beliefs and usage intention toward health chatbots: Evidence from an online experiment

Abstract

Objective

Based on the heuristic–systematic model (HSM) and health belief model (HBM), this study aims to investigate how personalization and source expertise in responses from a health chatbot influence users’ health belief-related factors (i.e. perceived benefits, self-efficacy and privacy concerns) as well as usage intention.

Methods

A 2 (personalization vs. non-personalization) × 2 (source expertise vs. non-source expertise) online between-subject experiment was designed. Participants were recruited in China between April and May 2021. Data from 260 valid observations were used for the data analysis.

Results

Source expertise moderated the effects of personalization on health belief factors. Perceived benefits and self-efficacy mediated the relationship between personalization and usage intention when the source expertise cue was presented. However, the privacy concerns were not influenced by personalization and source expertise and did not significantly affect usage intention toward the health chatbot.

Discussion

This study verified that in the health chatbot context, source expertise as a heuristic cue may be a necessary condition for effects of the systematic cue (i.e. personalization), which supports the HSM's arguments. By introducing the HBM in the chatbot experiment, this study is expected to provide new insights into the acceptance of healthcare AI consulting services.

Introduction

From enhancing the user experience and assisting healthcare professionals, to improving healthcare processes and unlocking potential benefits, health chatbots are being harnessed to achieve various goals, thereby demonstrating the power of conversational AI. ¹ In the global market, many health chatbots are emerging, such as Ping An Good Doctor (PAGD) from China, Woebot from the USA and Babylon from the UK. ² In particular, during the COVID-19 pandemic, the demand for virtual health services has grown substantially. For example, PAGD experienced phenomenal growth at the peak of the pandemic period with more than 1.27 billion consultations in 2021. ³ With the advances in artificial intelligence (AI) and natural language processing (NLP) technologies, some health chatbots can combine medical databases and users’ personal information to generate health advice automatically. Chatbots are helping clinicians in the UK to provide patient-centered medical care, while smoothing out differences across the nation and aiding patients in self-management. ⁴ In China, a substantial number of manufacturers are working on optimizing the interactions between users and health chatbots and attracting more users. ⁵

Improving usage intention is crucial for health chatbots to realize its beneficial outcomes. One of the major approaches adopted to enhance usage intention is to provide more tailored and professional health-related information to the users.^6,7 Since Awad and Krishnan ⁸ proposed the personalization-privacy paradox, the number of discussions on how the presence of personalized information elicits more positive perceptions via the Internet has increased.^9,10 Especially in the healthcare context, one of the characteristics of a health diagnosis is that it is tailored to users according to their demographic information, symptoms or lifestyle habits. ¹¹ Hence, how to balance users’ concerns about privacy and the provision of personalized health services has become an issue of widespread concern among practitioners. In addition, previous studies have shown that the source of information is one of the aspects of users’ judgments regarding the trustworthiness and quality of chatbots.^4,12 Among them, source expertise has exhibited its effect in terms of perceptions regarding the credibility of messages when users are processing online health-related information. ¹³ However, in the context of health chatbots, how the source expertise can affect users’ health-related perceptions still deserves an in-depth analysis.

From a theoretical standpoint, the HSM has provided an explanatory framework for online and offline health communication campaigns.^14,15 Exploring users’ heuristic processing and systematic processing is helpful in understanding their health information decision-making and belief changes in the context of health chatbots. When users’ cognitive motivation is relatively high, it is possible to simultaneously process both the heuristic cue and the systematic cue.^16,17 Given that users tend to utilize health chatbots for specific needs, we conducted an online experiment with health chatbots to explore the interaction effect of source expertise and personalized cues on users’ beliefs and behavorial intention. In addition, the HBM has also been found in the previous literature to be relevant and effective in predicting users’ health-related perceptions and attitudes for e-health interventions.^18,19 We also examined the mediating role of three factors from health beliefs (i.e. perceived benefits, self-efficacy and privacy concerns) between the two cues of information-processing and usage intention.

Literature review

Health chatbot for consulting

Trained by massive amounts of healthcare data, chatbots that can provide medical information have exhibited potential in both academia and in practice. ²⁰ Health chatbots have been widely applied to support users’ short-term or long-term health goals, such as assisting in automatic triage, performing cancer diagnosis, providing consultations or providing self-management. ²¹ With the popularity of embedded platforms (like Facebook Messenger and Telegram) and health applications, ²² chatbots that provide health consultation help to minimize medical costs, enhance accessibility to health knowledge and reduce the health and well-being gap.^4,23 Chatbots have apparent advantages, such as being anonymous, non-judgmental and convenient.^4,24 Taking the app of “the Left-hand Doctor” in China as an example, when a user inputs a simple keyword about a disease, the interface will cause the related knowledge and medical suggestions to pop up, mimicking general practitioner consultations in reality. Users expect to obtain tailored and professional information from the health chatbots, which makes personalization and professionalism become indispensable factors for this technology.^25,26 Chatbots can give personalized and tailored responses around the user's profile by storing and retrieving user information in databases. ²¹ The AI algorithm or collaborations with prestigious health specialists and professional teams further warrant the expertise of health chatbots.^12,27

Some of the previous research provided findings on the antecedents of usage intention toward health chatbots. Many of these studies focused on users’ perceptions of the chatbot system's performance. For example, Nadarzynski et al. ⁴ noted that convenience and anonymity may drive the user acceptability of health chatbots, whereas the worries related to accuracy and security may impede user acceptance. Zhu et al. ²⁸ found that the different kinds of values (i.e. functional, emotional, epistemic and conditional) that users perceive can enhance their experience and continuance intention toward healthcare chatbots. Liu et al.'s ²⁹ study revealed that factors from the Technology Acceptance model (i.e. perceived usefulness, perceived ease of use and perceived enjoyment) are significant antecedents of the continuance intention toward healthcare chatbots. Similarly, Huang and Chueh's research ²⁷ found that perceived ease of use and perceived information quality can increase satisfaction and the behavioral intention to use. Although these studies offer valuable insights, how users’ health beliefs during the usage process affect their usage intention with regard to healthcare chatbots seems to have been neglected. As people use health chatbots primarily to solve health concerns and then decide whether to continue using this technology, it is necessary to consider theories like the HBM, which emphasizes individuals’ health beliefs, when examining the adoption by users of this technology.

Personalization and source expertise in online applications

In the health chatbot context, the personalization cue and source expertise cue are quite commonly employed to provide users with better services. As for personalization, after gathering users’ personal information either overtly or covertly, information systems can provide tailored and personalized content to specific individuals, ³⁰ which to a large extent increases the personal relevance to the user.^31,32 Although the message may be personalized in varying ways, a conceptual common ground with regard to its operationalization of personalization is the “match” between the message and the user's characteristics, needs and interests.^33,34 Previous scholars provided evidence on personalization's influences in various contexts. In research on the marketing domain, personalization has been found to lead to more positive attitudes and purchase intentions. ³⁵ In the human–computer interaction (HCI) area, scholars demonstrated that personalized content can produce favorable perceptions and a greater intention to revisit the interface. ³² Overall, past studies found that a personalized message can elicit more favorable outcomes than a non-personalized message, such as being more enjoyable, more memorable and more persuasive. ³⁶

In the health domain, personalization can also exert positive effects on individuals’ judgments and behavioral intentions. Patients perceive the personalized message to be of higher quality, which further increases their behavioral intention to follow the treatment procedures, ³⁷ thereby enhancing the effects of interventions for health promotion. ³⁸ Within the context of health chatbots, personalization has been identified in a few studies as an important feature to improve user satisfaction, engagement and dialogue quality.^28,39,40 However, only a very few studies employed experiments to directly explore the effects of personalized messages from chatbots on users’ health-related outcomes. Meanwhile, some extant research lacks a theoretical framework underpinning the personalization of health chatbots. ⁴¹

With regard to source expertise, it is a term that refers to the extent to which the individuals perceive the source to have enough competence and knowledge to provide convincing arguments. ⁴² After communicators demonstrate cues related to expertise such as claiming professionality, the recipients’ perceived credibility toward the source will increase accordingly, ⁴³ which will further influence their decision-making and behavioral intentions.^44,45 For mobile services and technological systems, expertise cues are often included in the interface to improve users’ positive evaluations.^43,46 As for the health scenarios, source expertise has been found to be closely related to information judgments as users care a lot about the source from which they receive the health-related messages. ⁴⁷ Health information from an authoritative and professional source is perceived to be more credible and persuasive than information from a normal sender.^48,49 In the web-based environment, source expertise can significantly increase the perceived quality of health information, subsequently eliciting more trust from users. ⁵⁰ However, few studies explore it in the health chatbot context. Meanwhile, previous studies draw much attention to the influences of source expertise on perceptions and judgments (e.g. perceived quality and credibility) on the messages per se, leaving individuals’ health beliefs relatively neglected.

Although previous research provides insights into the effects of the two respective cues, to the best of our knowledge, no prior research has investigated people's perceptions under the simultaneous influences of both cues. In addition, the issue of how people's health beliefs may change when the chatbot presents personalization and source expertise remains underexplored. Considering that the two important cues may co-present at the same time in the messages from the health chatbot, the HSM can provide a solid framework to help us understand the intertwined influences of different messages’ cues from the perspective of information processing.

The heuristic–systematic processing of health information

A health chatbot with AI can interact with users automatically to provide medical advice based on their symptoms. Users, in turn, inevitably need to face up to and judge the suggestions provided by the health chatbot. The HSM provides a perspective that may explain how users process and receive health advice from a chatbot. The HSM is a dual-process model that presents two distinctive processing approaches when individuals assess information. ⁵¹ Heuristic processing is identified as a faster, more intuitive processing route that requires comparatively less cognitive effort. ¹⁶ In contrast, systematic processing is a slower, more analytical processing route that requires deep-thinking and reasoning. ⁵² In the context of healthcare, systems and heuristic processing have been proven to jointly influence the individual's medical decision-making process. ¹⁵ Given that users might judge the values of different cues at the same time when using a health chatbot, we suppose that the inclusion of both processing paths in the exploration can help us better understand users’ decisions and intentions in relation to this technology.

When issues are highly relevant to individuals, ⁵¹ people are more likely to engage in a systematic way of information processing, which can be triggered by cues such as personalization. In mHealth environments, personalized recommendations that meet users’ precise needs have been shown to positively influence their usage intentions.^7,28 Since health diagnoses involve a process that refers to personalized information, this study proposes that users systematically process their health issues when they are exposed to the personalization cue. Source expertise, on the other hand, is considered to be a typical type of heuristic cue. ⁵¹ Source expertise is particularly important for health-related issues as it is widely recognized as being one of the fundamental elements driving the principles of medical professionalism that underpin public trust in the knowledge and skills of healthcare practitioners. ⁵³ Hence, this research also explores how source expertise, as a heuristic cue, influences the users’ health beliefs and usage intention.

The HSM specifies that the two types of information processing can not only co-exist but can also interact with each other. ¹⁷ In particular, the bias hypothesis of the HSM emphasizes the important influence of heuristic processing on systematic processing. Chaiken et al. ⁵⁴ propose that a message could be first evaluated by heuristic cues. If the message is considered to be significant for this person, he or she might adopt a systematic evaluation of the content. Systematic processing is not entirely objective and could be biased by the expectation of an initial heuristic evaluation. As heuristic processing is the default strategy for humans’ information processing, ¹⁶ our research assumes that the users’ heuristic processing triggered by source expertise may play a crucial role when users are perceiving the personalization message from the health chatbot, which further influences their usage intention.

The interaction of the two information processing paths also helps us further understand the subsequent changes in people's attitudes. Among them, previous studies have adopted usage intention as a significant indicator to assess people's attitudes to technology and have treated it as a downstream outcome of various manipulations. ⁵⁵ In this study, the usage intention in relation to health chatbots refers to the intention to use online conversational agents in healthcare. The usage intention as a kind of judgmental decision is determined by both systematic factors and heuristic factors. ⁵⁶ Some previous health-related studies have proved that personalization, as one of the systematic cues, has a significant positive influence on usage intention.^38,57 Nonetheless, users may not rely on a sole cue for decision-making. As discussed above, the cue of source expertise is especially important in health-related settings. ⁶ In other words, the positive effect of personalized cues on usage intention may be more reinforced under the condition of source expertise, since users may perceive a more direct and stronger sense of trust. Thus, we propose:

• H1: Personalization (vs. non-personalization) will bring about more usage intention when the health chatbot provides source expertise information compared to when it does not.

Health belief model

In addition to usage intention, the forms of individual information processing also affect people's beliefs and perceptions. ⁵⁸ With regard to health decision-making, the HBM is widely applied to investigate people's health-related behaviors. The HBM initially aimed at interpreting and predicting people's behavioral reactions to illness and provided a remarkable paradigm to remedy people's health condition. ⁵⁹ With the continuous changes in practical health interventions, the HBM subsequently developed into a comprehensive theoretical framework with six constructs to explain individuals’ general health-related behaviors. The six constructs include perceived susceptibility, perceived severity, perceived benefits, perceived barriers, cues to action and self-efficacy. ⁶⁰ The HBM has been implemented to explain health behaviors in the context of online information acquisition. In recent years, in particular, the HBM has been applied to investigate the adoption of health-related applications during COVID-19 and people's behaviors in seeking online health information.^18,61 The HBM is suitable for health chatbot scenarios as users’ actions in this context contain both information processing and subsequent health-related behaviors. ⁶² Thus, we assume that the HBM has a considerable explanatory power in our study.

Previous studies often selected certain constructs from the HBM based on the characteristics of their research contexts.^14,63 In this study, the perceived benefits and privacy concerns were included to explore people's health-related Internet engagement. It should be noted that most health chatbots inevitably collect basic and even sensitive patient information to improve efficiency and personalization, including that related to allergies, medication history and contact details. However, users are often not well informed about the ways in which chatbots ensure their privacy and data security, which may cause users to feel concerned about privacy. Such worries about data being stolen, mishandled or improperly sold to a third party may further impede users’ subsequent behavioral intentions. ⁶⁴ Past studies have conceptualized privacy concerns as the perceived barrier in the healthcare contexts such as using smart healthcare services for psychiatric disorders ⁶⁵ and adopting the COVID-19 tracing app. ¹⁸ Aligned with the status quo and previous studies, we focus on privacy concerns as the perceived barrier under the framework of HBM in the health chatbot context. In addition, as self-efficacy has been considered to be a significant factor that is closely linked to the two processing paths,^66,67 it was selected from the HBM to illustrate an individual's level of confidence in terms of his or her ability to take action. ⁶⁸

Utilizing effective heuristic cues or systematic cues may help users make decisions related to health. The forms of individual information processing have been proven to affect people's beliefs and perceptions. ⁵⁸ When personalization cues are treated as systematic processing, users may be able to mainly focus on the content of the information relevant to them, and may thus feel that the diagnosis information is beneficial or helpful. Meanwhile, since the source expertise contributes to the perception of information quality ⁵⁰ by enhancing users’ perceived creditability, it is possible that the relationship between personalization and perceived benefits will be enhanced by source expertise. Hence, we propose:

• H2: Personalization (vs. non-personalization) will bring more perceived benefits when the health chatbot provides source expertise information compared to when it does not.

In this study, self-efficacy is defined as an individual's belief in his or her capabilities that is sufficient for gaining information, diagnosing and coping with diseases by using this health chatbot. Previous research has shown that self-efficacy is associated with dual-mode information processing. ⁶⁶ For instance, Gaston et al. ⁶⁹ proved that the health self-efficacy of pregnant women can be effectively improved in central route information processing. A past study related to chatbots for childhood vaccinations also showed that users who regarded information from chatbots as useful had higher vaccination self-efficacy. ⁷⁰ Considering the features of personalized messages and professional sources, personalization and source expertise may be helpful in enhancing the individual's confidence in coping with the diseases. Thus, we propose:

• H3: Personalization (vs. non-personalization advice) will lead to more self-efficacy when the health chatbot provides source expertise information compared to when it does not.

As for privacy concerns, Awad and Krishnan ⁸ proposed the personalization-privacy paradox, which argues that there is a continuous tension between the needs of a company to personalize consumer experience and consumers’ concerns over privacy. On the one hand, the higher the level of personalization, the greater the likelihood of satisfying consumers’ needs. ⁷¹ On the other hand, the tailored content may arouse users’ perceptions that the company has some manipulative purposes, thereby giving rise to privacy concerns. ⁷² Since the source expertise can enhance the perceived trust ⁵⁰ which contributes to decreasing the privacy concerns, ⁷³ the source expertise may influence the effect of personalization on privacy concerns. Therefore, we propose:

• H4: Personalization (vs. non-personalization) will elicit fewer privacy concerns when the health chatbot provides source expertise information compared to when it does not.

The constructs from the HBM have been applied to predict health-related Internet usage.^74,75 Among them, the perceived benefit is an important antecedent when deriving the usage intention regarding chatbots. ²⁷ Moreover, self-efficacy, which is one's confidence in gaining positive outcomes through computer-mediated actions, has been confirmed to positively affect the usage intention in previous research. ⁷⁶ In terms of the relationship between the perceived barriers and usage intention, Dhagarra et al. ⁶⁴ proved that privacy concerns weaken the usage intention in the setting of health-related information technology. Combined with the personalization's impact on the perceived benefits, self-efficacy and privacy concerns noted above, we hypothesize that the constructs from the HBM may play mediating roles in the relationship between personalization and usage intention. Hence, we propose:

• H5: (a) Perceived benefits, (b) self-efficacy and (c) privacy concerns mediate the effects of personalization on usage intention.

To summarize, the research model is shown in Figure 1.

Figure 1.

The research model in this study.

Method

Experiment design

This study aims to examine how source expertise and personalization jointly affect HBM-related factors and usage intention. A 2 (personalization vs. non-personalization) × 2 (source expertise vs. non-source expertise) between-subject experiment was designed. All online participants were randomly assigned to one of the four conditions. Participants were first asked to interact with a dummy chatbot designed by chatbot solution provider Sanuker (https://sanuker.com/), which is a fully-functioning platform for chatbot design enabling developers to extensively customize the conversations, deployment channels and user interfaces. Its chatbot builder Stella provides a code-free unified interface and methods for developers to embed chatbots in a variety of instant messaging or research platforms. Next, an online research company named Acadeta (Jishuyun) helped us with the randomized experiment design.

Stimulus material

By referring to the actual cases of major health chatbots in China (e.g. the Left-hand Doctor; Ping An Good Doctor), we designed a chatbot for this study. Participants could talk directly with this health consulting assistant called “Xiaokang.” When designing the health chatbot, we not only considered the purposes of our experiment but also referred to the real health chatbot products. With the standalone web chat as the deployment channel, “Xiaokang” was located on an independent website that does not require participants to register any accounts. To provide the participants with an immersive interaction, the “Xiaokang” interface includes a green chatbot logo, the dialogue frame and the chat bubbles, which is similar to extant commercial products. By allocating distinct identifiers to the participants, “Xiaokang” can identify, store, label and read users’ messages in an automatically generated database, so that the personalization features can be realized by inserting the user's information into specific replies. To exclude confounding factors, we adopted the linear conversation flows with the form of a tree structure to ensure a fixed conversation sequence in “Xiaokang.” Meanwhile, the error handling mechanism was configured for validation. The frame of the web chat of “Xiaokang” was eventually embedded into the online survey by using HTML fragments. Only after interacting with “Xiaokang,” following our instructions that were provided before the experiment, could the participant be directed to the survey interface. Before the experiment started, developers also tested the performance and debugged “Xiaokang” referring to the backend logs to ensure the chatbot's fluency and accuracy in the dialogue.

According to Sina News, ⁷⁷ the incidence of hemorrhoids in China is about 51.1%. As hemorrhoids are generally regarded as one of the most common anorectal disorders, we chose hemorrhoids as the subject of this conversation. In addition, participants were asked to strictly follow standardized pre-set instructions by responding to the keywords from the chat scripts to the chatbot, which was aligned with prior research.^12,78 We also followed the experimental path by using keyword recognition technology, in which the chatbot could match the corresponding replies based on the participants’ input. Examples of the chatbot interface at the condition of personalization × source expertise are provided in Figure 2.

Figure 2.

Chatbot interface examples.

Procedure

Participants were first asked to read a standardized instruction for the conversation. They were told to assume that they had hemorrhoids-related symptoms and were seeking health suggestions from a health chatbot. We asked participants to strictly follow these steps sequentially. To be more precise, except for Step 2 where participants needed to enter their real information, other steps had to be answered by strictly following the standardized instructions. The experimental steps were as follows. Step 1, tell the chatbot “I have hemorrhoids.” Step 2, answer questions about personal information raised by the chatbot: gender, age, have a sedentary lifestyle or not, like to eat spicy food or not. This step requires participants to input their authentic personal information. Step 3, when the chatbot asks: “Do you have a history of anorectal problems?” participants need to select the option “No.” Step 4, when the chatbot asks: “Do you have regular bowel movements?” participants need to select the option “Not very regular.” Step 5, when the chatbot asks: “Are you experiencing hemorrhoidal bleeding?” participants need to select the button “Yes, a little bleeding.” Step 6, after reading the health suggestion given by the chatbot, participants need to input “Thank you” to end the dialogue. Next, after reading the instruction and the experimental steps, participants clicked a link to be randomly assigned to one of four scenarios in which the chatbot made either personalization × source expertise suggestions, personalization suggestions, suggestions including source expertise, or suggestions with non-personalization × non-source expertise. At the end of the conversation, the chatbot offered a link to move forward to a post-experiment survey. The flowchart of the experiment is displayed in Figure 3.

Figure 3.

Flowchart of the experiment.

In stage 3, C: chatbot; P: participants.

Manipulation materials

Stimuli were manipulated by explicit cues via text without adding other visual cues. Based on the actual cases of health chatbots in China and a previous study, ⁶⁴ we manipulated source expertise by emphasizing the authority and reliability of the source in the message content, and the source information was emphasized twice in the conversation flow. At the beginning of the conversation, the chatbot first specified: “I have achieved professionalism in medicine after thoroughly analyzing thousands of authoritative medical books and millions of worldwide medical articles. How may I help you?” Second, before giving diagnostic suggestions, the chatbot starts by stating “The following diagnosis was made according to the analysis and screening of thousands of authoritative medical books and millions of worldwide medical articles…” For the condition of non-source expertise, the sentences on source expertise were omitted.

Personalization refers to the practice whereby systems utilize information from a certain user (e.g. demographics and preferences) to offer personalized services for the user. ⁷⁹ Several approaches have been used to design online personalized content, such as the customers’ usage history for product recommendations ³⁵ and customers’ personal information for advertising recommendations. ⁸⁰ Considering that health suggestions need to be tailored based on the user's physical condition and lifestyle, we embedded a series of personal information when offering final diagnostic suggestions in the personalization condition for our manipulation, including gender, age, an eating habit and a living habit. For instance, in the scenario of personalization, the chatbot replied: “Sir, considering that you are a male aged 18∼25 with a sedentary lifestyle and a preference for spicy foods…” In the non-personalization scenario, the user's personality information was not repeated and emphasized in the final health advice. All scripts for manipulating the personalization and source expertise are displayed in Table 1.

Table 1.

Scripts for manipulating the personalization and source expertise.

	Personalization × Source Expertise	Personalization × No Source Expertise	No Personalization × Source Expertise	No Personalization × No Source Expertise
Step 1	Hi, I am Xiaokang, your healthcare consultant. I have achieved professionalism in medicine after thoroughly analyzing thousands of authoritative medical books and millions of worldwide medical articles… a	Hi, I am Xiaokang, your healthcare consultant.	Hi, I am Xiaokang, your healthcare consultant. I have achieved professionalism in medicine after thoroughly analyzing thousands of authoritative medical books and millions of worldwide medical articles…	Hi, I am Xiaokang, your healthcare consultant.
Step 2	Asking about gender and other information b	Asking about gender and other information	Asking about gender and other information	Asking about gender and other information
Step 6	The following diagnosis was made according to the analysis and screening of thousands of authoritative medical books and millions of worldwide medical articles as well as your personal conditions…	The following diagnosis was made according to your personal conditions…	The following diagnosis was made according to the analysis and screening of thousands of authoritative medical books and millions of worldwide medical articles	(NULL)
Step 6	Sir/Madam, considering that you are a male/female aged … with a sedentary lifestyle/(NULL) and a preference for spicy food/(NULL)…	Sir/Madam, considering that you are a male/female aged … with a sedentary lifestyle/(NULL) and a preference for spicy food/(NULL)…	(NULL)	(NULL)

^a Phrases that are underlined are the stimulus material for manipulating source expertise.

^b Phrases in italics are the stimulus material for manipulating personalization.

Pre-test

A pre-test was conducted to ensure the function of the chatbot and that the stimulus materials could work successfully. Through snowball sampling, we received 91 participants’ valid responses. As for the stimuli, the independent-samples t-test showed that the stimulus material for source expertise was successful, t(89) = 2.189, p < 0.05. In terms of personalization, the stimulus material also passed the independent-samples t-test, t(89) = 2.510, p < 0.05. The content of the script and the wording of the questionnaire were adjusted according to the participants’ feedback from the pre-test.

Data collection

To determine the sample size for our experiment, we conducted a power analysis using G*Power. ⁸¹ We estimated the effect size f as 0.23, which is based on previous experiment studies regarding personalization and source expertise. ³⁴ The results of the analysis indicated that 248 participants were needed in our research to achieve 95% power at an alpha-error level of 5%.

The participants in the online experiment were recruited between April and May 2021. The methods of recruitment were twofold. We posted a recruitment advertisement including the experiment's purpose and procedures on a WeChat official account platform named “marketing research group” to attract participants. Meanwhile, we also used a professional survey company called Acadeta (Jishuyun) to recruit samples. Each participant was paid 5 RMB. To prevent multiple submissions, we added a setting on the chatbot and questionnaire website to ensure that a participant with a duplicate IP address was blocked from entering the interaction interface and the questionnaire again. To ensure response quality, three attention check questions were added to the questionnaire. The first question required participants to select the disease with which the dialogue started. The second attention check further asked participants to recall whether the chatbot mentioned any information source and what the source was. Lastly, participants needed to answer whether the chatbot repeated the personal characteristics at the end of the interaction.

Formal participants

A total of 452 individuals participated in this study. Before the formal analysis, 79 participants were regarded as invalid samples since they did not follow the pre-set experiment instruction to interact with the chatbot; 92 participants were excluded due to incorrect responses to the three attention checks; 21 participants were filtered out since they finished the questionnaire in too short a time, which was far below the reasonable duration. After cleaning the data, there remained 260 valid observations. Among these participants, 176 (67.7%) were female and 84 (32.3%) were male. The 18–25 age range accounted for the largest proportion with 53.8%, followed by those aged 26–34 with 37.3%. In addition, 8.5% of the participants were in the 35–49 age group and 0.4% of them in the 50–64 age group. Regarding the education level, 68.5% indicated that they had a bachelor's degree (master's degree: 22.3%; college degree: 5.8%; high school diploma and below: 3.4%). 50.8% of participants had experienced health chatbots.

Measures

Variables for manipulation checks. To check the manipulation of the personalization cue, the perceived personalization toward a chatbot was measured using two items: “I believe that this chatbot's suggestion and diagnosis are personalized to meet my needs” and “I believe that this chatbot's suggestion and diagnosis meet my needs” ⁸² (Cronbach's α = 0.74). As for the manipulation check of the source expertise, the perceived source expertise in the context of the online chatbot was measured by using the following two items: “I believe this chatbot's suggestion and diagnosis are knowledgeable” and “I believe the suggestion and diagnosis of the chatbot reflect expertise”^83,84 (Cronbach's α = 0.81). Responses for both manipulation check variables were scored on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree).

Mediating variables. All the items of the mediating variables were assessed on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). The variable of perceived benefits was measured by five items. Three of these items were revised from Forslund ⁸⁵ and Johnson and Grayson ⁸³ : “I can obtain better knowledge about curing hemorrhoids,” “This chatbot offers a credible analysis,” and “This chatbot's careful work will not complicate things.” Two of the items were developed by the authors: “I can better know how to improve my hemorrhoid symptoms” and “It helps answer my questions about hemorrhoids” (Cronbach's α = 0.81).

Self-efficacy was measured via four items adapted from Carballo et al. ⁸⁶ and Walrave et al. ¹⁸ The items were “I will be able to conduct the diagnosis myself by using this chatbot,” “I will adjust my habits based on the advice given by this chatbot,” “I fully understand how to use the chatbot,” and “By using this chatbot, I am confident that I can cure the hemorrhoids” (Cronbach's α = 0.71).

The privacy concern scale was adapted from Xu et al. ⁸⁷ by using three items. Participants were asked to evaluate the degree of their agreement with the items: “It bothers me when this chatbot asks me for this much personal information,” “I worry that others could access my personal information from this chatbot,” and “I am concerned about providing personal information to this chatbot, because it could be used in a way I did not foresee” (Cronbach's α = 0.90).

Dependent variable. Usage intention was assessed by three items adapted from Pai and Huang ⁸⁸ on a scale ranging from 1 (strongly disagree) to 5 (strongly agree). These items included “Given a chance, I intend to adopt this conversational service,” “I am willing to use this conversational service in the future if it is necessary,” and “I plan to use this conversational service in the future if it is necessary” (Cronbach's α = 0.90).

Results

Manipulation check

Source expertise. An independent-samples t-test was applied to determine the differences in the mean values between the source expertise group (N = 131) and the non-source expertise group (N = 129). The results indicated that participants in the source expertise group (M = 3.985, SD = 0.720) perceived more authoritative traits than those in the condition of the non-source expertise group (M = 3.771, SD = 0.778), t(256) = 2.294, p < 0.05. Hence, the stimulus material for the source expertise was manipulated successfully.

Personalization. As for the mean value discrepancies, we also used the independent-samples t-test to compare the personalization group (N = 122) with the non-personalization group (N = 138). Respondents who were assigned to the personalization condition (M = 4.045, SD = 0.616) perceived the message to be more tailored, compared to those who were assigned to the non-personalization condition (M = 3.833, SD = 0.781), t(258) = 2.404 p < 0.05. Therefore, the manipulation of personalization was successful.

Hypothesis testing

Usage intention. As for H1, the ANOVA results revealed that there were no significant main effects of personalization (F[1, 256] = 0.648, p = 0.422) and source expertise (F[1, 256] = 0.327, p = 0.568) on usage intention. The interaction effect between personalization and source expertise was also not significant (F[1, 256] = 1.067, p = 0.303). The descriptive values of usage intention for the four conditions are shown in Table 2.

Table 2.

Means and standard deviations (in parentheses) for the variables based on conditions.

	Condition
	Source expertise		No source expertise
	Personalization	No personalization	Personalization	No personalization
	n = 65	n = 66	n = 57	n = 72
Usage intention	4.185 (0.799)	4.000 (0.869)	4.023 (0.791)	4.046 (0.766)
Perceived benefits	4.012 (0.633)	3.709 (0.622)	3.737 (0.620)	3.814 (0.660)
Self-efficacy	4.085 (0.648)	3.814 (0.621)	3.798 (0.622)	3.903 (0.617)
Privacy concerns	3.318 (1.231)	3.551 (1.186)	3.643 (1.263)	3.412 (1.157)

Perceived benefits. As for H2, the results of the ANOVA analysis showed that both the main effects of personalization (F[1, 256] = 2.047, p = 0.154) and source expertise (F[1, 256] = 1.165, p = 0.281) on the perceived benefits were not significant. However, the results yielded a significant interaction effect between personalization and source expertise (F[1, 256] = 5.786, p = 0.017). The simple comparisons showed that in the source expertise group, participants who were exposed to personalized information scored higher perceived benefits than those who were not (F[1, 256] = 7.466, p = 0.007). Personalization did not have a significant influence on the perceived benefits when the source expertise was absent (F[1, 256] = 0.468, p = 0.494) (see Figure 4 and Table 2).

Figure 4.

Means for perceived benefits according to conditions.

**p < 0.01.

Self-efficacy. For H3, the results of the ANOVA analysis showed that no significant main effects of personalization (F[1, 256] = 1.126, p = 0.290) and source expertise (F[1, 256] = 1.608, p = 0.206) on self-efficacy were found. Nevertheless, the personalization × source expertise interaction revealed a significant effect on the participants’ self-efficacy (F[1, 256] = 5.762, p = 0.017). Simple comparisons revealed that the participants’ self-efficacy was higher in the source expertise group with personalized information compared to that for those without personalized information (F[1, 256] = 6.079, p = 0.014). Personalization did not have an effect on self-efficacy in the non-source expertise group (F[1, 256] = 0.884, p = 0.348) (see Figure 5 and Table 2).

Figure 5.

Means for self-efficacy according to conditions.

*p < 0.05.

Privacy concerns. As for H4, the ANOVA results showed that the main effects were not significant for both personalization (F[1, 256] = 0.000, p = 0.997) and source expertise (F[1, 256] = 0.387, p = 0.534). Moreover, there was also no significant interaction effect between personalization and source expertise in relation to privacy concerns (F[1, 256] = 2.384, p = 0.124) (see Table 2).

Mediation analysis

To test H5, we conducted a moderated mediation analysis using Model 8 of the Process macro of SPSS. ⁸⁹

Figure 6 and Table 3 present the conditional direct and indirect effects that were contingent on the source expertise. The results show that perceived benefits and self-efficacy played mediating roles between personalization and usage intention in the source expertise condition. Specifically, the indirect effect of personalization on usage intention via perceived benefits was significant when the participants were exposed to the source expertise (indirect effect = 0.138, 95% BootCI [0.038, 0.263]). However, such an indirect effect of perceived benefits was not significant when the source expertise was absent (indirect effect = −0.035, 95% BootCI [−0.144, 0.069]). Likewise, the indirect effect of personalization on usage intention through self-efficacy was significant when the source expertise was shown (indirect effect = 0.161, 95% BootCI [0.029, 0.313]), and not significant when it was not shown (indirect effect = −0.062, 95% BootCI [−0.205, 0.068]). However, privacy concerns did not mediate the relationship between personalization and usage intention for two source expertise conditions. The direct effect of personalization on usage intention was not significant for any conditions, suggesting that the perceived benefits and self-efficacy fully mediated the influence of personalization on usage intention when the source expertise was shown.

Figure 6.

Statistical significance of paths in a moderated mediation model.

Table 3.

Results of conditional effects for moderated mediation (n = 260).

	Conditional direct effect of source expertise conditions
Source expertise	Value	SE	t	p
0 a	0.076	0.092	0.825	0.410
1	−0.116	0.091	−1.265	0.207
	Conditional indirect effect through perceived benefits
	Value	SE	LLCI b	ULCI
0	−0.035	0.053	−0.144	0.069
1	0.138	0.057	0.038	0.263
	Conditional indirect effect through self-efficacy
	Value	SE	LLCI	ULCI
0	−0.062	0.068	−0.205	0.068
1	0.161	0.072	0.029	0.313
	Conditional indirect effect through privacy concerns
	Value	SE	LLCI	ULCI
0	−0.001	0.009	−0.022	0.018
1	0.001	0.009	−0.019	0.021

^a 0: non-source expertise condition; 1: source expertise condition.

^b LL: lower limit; CI: confidence interval; UL: upper limit; Bootstrapped at sample size = 5000.

Discussion and implications

This study investigated how the personalization and source expertise of the information from a health chatbot influence individuals’ health-related outcomes, privacy concerns and usage intention in relation to the health chatbot. The findings of this research merit a detailed discussion.

We found that personalization can increase perceived benefits and self-efficacy only in the case where the health chatbot provides the source expertise. As noted in previous research, personalization can increase the perceived benefits^90,91 and self-efficacy ⁹² in different contexts. However, our study found that when users interact with the health chatbot, this effect has a precondition regarding the source expertise. A possible explanation is that currently the users do not fully trust the information provided by the health chatbot. For individuals who are seeking health-related information from the Internet, the assessment regarding the credibility and trustworthiness of the sources is highly important ⁹³ since information or suggestions about health may have significant effects on individuals. ⁹⁴ As a health-related service on the Internet, the health chatbot is still at the nascent stage. Many users are uncertain about the trustworthiness and quality of this service. ⁴ The concern about an incorrect diagnosis is a salient issue of the health chatbot. ⁹⁵ When users doubt the quality and the authenticity of the message from the chatbot, the effects of the information on their health beliefs may be minimal. However, the source expertise of the message can help users to build perceptions about credibility and trustworthiness, ⁹⁶ which will effectively help individuals form favorable evaluations toward the information ¹⁷ and further mitigate the uncertainties toward the message. Only when individuals believe the message is credible and trustworthy will the information further influence their health beliefs.

This finding adds discussion to the previous literature on the effects of both personalization and source expertise. With regard to personalization, there have been some contradictory findings pertaining to its influences. ⁸⁰ On the one hand, scholars provided evidence to show the favorable effects of personalization^28,97; on the other hand, a few studies found that a personalized message may not lead to significant changes.^98,99 Our result provides a new potential explanation for the previous inconsistent findings. Previous studies ⁹⁸ often explored personalization as the sole research focus. By demonstrating the moderating effect of source expertise on personalization, this study further shows that in some cases, the effects of personalization are contingent upon other important message cues. Future studies need to comprehensively consider the potential boundary conditions when exploring the influences of personalization based on the research context.

In regard to source expertise, we found no direct main effects of the expertise cue in our experiment, which is different from previous studies.^46,50 This may be because the dependent variables are users’ health beliefs in the current study rather than perceptions related to message quality, which were often explored in previous studies. The interaction effect in our findings sheds light on how source expertise plays a role in the process of establishing health-related outcomes. Specifically, in the context of the health chatbot, although source expertise per se cannot alter users’ health beliefs, users seemed to digest personalized information and change their beliefs based in part on their perceptions of expertise related to the source. Such a crossover effect indicates that source expertise serves as a necessary anchor that users employ to accept persuasive health messages.^43,50 As one of the first studies to explore source expertise in health chatbots, our research contributes to the stream of research regarding its influences in the health domain.

From the theoretical perspective of information processing, our study has provided evidence of an interaction effect between the systematic cue and the heuristic cue, which provides further evidence to deepen our understanding of the HSM. In our research, the personalization of the message is a systematic cue whereas the source expertise is a heuristic cue. Aligned with the “bias hypothesis,” heuristic processing will influence individuals’ systematic processing by leading to expectancies about the message, especially when individuals hold a high motivation for information processing and the message is meanwhile not sufficiently clear and convincing. ⁵⁴ As discussed above, users have specific motivations when using the health chatbot and the uncertainty of the health information from the Internet service makes them feel less assured about the credibility of the health message provided by the chatbot. Thus, as the “bias hypothesis” proposes, there is an interdependent interaction whereby the heuristic processing of source expertise influences the effect of the systematic processing of personalization on people's attitudes and beliefs. It adds to the discussion about the role that the heuristic cue plays in the individual's processing. Previous research has proved that in biased processing, the systematic cue is not always the determining factor. ¹⁷ In actual fact, based on the least effort principle, the heuristic cues are particularly important since individuals tend to rely on them for a convenient and quick assessment of the information. ¹⁶ Our study provides evidence that the heuristic cue may be a necessary condition for the effect of the systematic cue, which shores up the related arguments of the HSM.

From a practical point of view, it is recommended that health chatbot developers include source expertise as one of the design features. For example, the developers could include visual cues that might trigger the cognition of source expertise into the system design. It is suggested that service providers select a credible source to use as the database, for instance by collaborating with prestigious medical institutions, or recruit a team of medical experts to provide authoritative advice regarding the health information and diagnosis. Then, the chatbot could disclose their collaboration at the beginning of the conversation with users to present the well-founded expertise cue. In addition, to further help the users realize the chatbot's expertise, the service provider could notify users that there are regular updates of the chatbot's database and analytic systems. These measures would help to establish a sense of credibility and trust closer to what the patients usually receive from visiting a hospital in real life and further improve the users’ acceptance of the health chatbot.

Meanwhile, we have also found that personalization cannot directly influence the intention to use the health chatbot. Perceived benefits and self-efficacy fully mediate between personalization and the intention to use the health chatbot. The reason may be that people have specific and concrete health-related demands and motivations when they make decisions about whether they will continue to use a health chatbot. In our study, the chatbot was designed to give users instantaneous diagnoses and suggestions about hemorrhoids, so that the chatbot was a goal-oriented chatbot with clear and specific functions. These chatbots are designed to provide users with the necessary and specific information that they want in short conversations. ¹⁰⁰ The core requirement for the participants in our study is to acquire useful health-related information, specifically, diagnoses and related treatments. Under these circumstances, only when the users feel that the conversation with the health chatbot can bring benefits and improve self-efficacy will they further use the chatbot. Based on the HBM, our research is one of the early studies using the experiment to investigate how personalization influences health outcomes in a chatbot context. It contributes to the previous research^28,40 on personalization in the health context by distinctly demonstrating its effects on individuals’ health beliefs. Meanwhile, in a way that differs from the previous HCI literature ³⁴ showing that personalization can directly increase behavioral intention, our findings on the full mediation effects of health belief factors demonstrate the users’ unique psychological mechanisms in the health chatbot context.

The factors from the HBM are useful for understanding the mechanisms of the increased usage intention of the health chatbot. To be specific, perceived benefits and self-efficacy are important health beliefs that directly facilitate the users’ usage intention to use this service. These results are in accordance with previous studies that also use constructs of the HBM to explore health-related intentions and behaviors.^18,19 Our study shows that the HBM is a helpful tool in explaining why users are willing to use the health chatbot, which provides implications that further research on health-related AI services and products could refer to the HBM to investigate users’ perceptions and actions. For instance, by emphasizing the potential health-related advantages that the service would bring to the users, chatbot providers will be more likely to retain active users since the health-related benefits are the core demands of users when they use the health chatbots.

Finally, somewhat unexpectedly, we found that personalization has no effect on privacy concerns. Privacy concerns cannot influence the intention to use the health chatbot either. While it is widely recognized that the personalization is closely related to the risks and concerns about privacy, ¹⁰¹ our results indicate that privacy concerns do not constitute an important issue for usage intention in the context of the health chatbot. The first explanation may be that the benefits of personalization may overshadow privacy concerns in the case of the health chatbot. As discussed earlier, users have specific motivations when using health chatbots. In addition, privacy-related perceptions with regard to the personalization paradox may be context-dependent. ⁸¹ Previous studies indicated that personalization may be related to privacy concerns in scenarios involving social media websites, ¹⁰¹ online shopping, ¹⁰² and so on. Considering that our research context is the health chatbot providing diagnosis and therapeutic suggestions, we manipulated personalization based on the participants’ physical condition and lifestyle. Compared to these contexts, personalization may be relatively more acceptable in the health context because it can indeed help individuals to obtain health information that is more suitable for them. When users direct the most attention to health-related benefits during their interaction with the chatbot, they may ignore the possibility that their private information might be collected and tracked. This point is consistent with previous research, which indicated that Chinese people are willing to sacrifice privacy for potential benefits. Another explanation is that currently the health chatbot is a new service. In the nascent stages of a new technology, privacy concerns are not a salient issue for its users. ¹⁰³

Although it seems that users do not care much about the perception of privacy in our research setting, this does not mean that chatbot platforms do not need to pay attention to privacy. Due to the popularity of and easy access to AI devices in our daily lives, the active concept of privacy protection referred to as “privacy by design” ¹⁰⁴ should be highlighted by device designers. That is, the thought of privacy protection could be initially considered in the design of the product or service. For example, a clear purpose statement for data collection should be provided to users via the chatbot's privacy policy. Besides, users can type keywords like “forget/delete my personal information” to fulfill their right to be forgotten. By doing this, users’ privacy concerns can be alleviated at the beginning.

Limitations and future research

There are some limitations to this study. Our experimental design is based on the participants’ assumptions regarding their health situation, regardless of whether they have a certain disease or not. While requiring subjects to imagine symptoms and follow certain experimental procedures can effectively enable researchers to avoid the uncontrollable effects in online experiments,^12,78 it is inevitable that this may reduce ecological validity. In addition, the results of this study are based on data collected from Chinese users. In the context of health chatbots, Chinese participants have exhibited a lack of concern when disclosing their privacy, and such results may not be replicated in other countries or regions. Therefore, future studies could consider replicating and extending the current experimental designs of this study with different stimulus materials, contexts and other background characteristics to ensure more robust external validity. Furthermore, considering that our experiment involved 260 valid responses, we recommend that future studies regarding chatbots arrive at a larger sample size to improve the generalizability of the results.

The final limitation is that we conducted a one-time simulation test based on the HBM. More long-term intervention experiments are needed to enrich the relevant research in this field. Future research could consider using health chatbots as an intervention in the experimental group, and comparing the health beliefs perceived by users in the control group after a few months. By conducting such longitudinal studies, the effectiveness of the intervention can be examined from a long-term perspective.