Skip to main content
NCBI home page
Digital Health logoLink to Digital Health
. 2022 Mar 30;8:20552076221090031. doi: 10.1177/20552076221090031

“I am chatbot, your virtual mental health adviser.” What drives citizens’ satisfaction and continuance intention toward mental health chatbots during the COVID-19 pandemic? An empirical study in China

Yonghan Zhu 1,✉, Rui Wang 2, Chengyan Pu 3
PMCID: PMC8971968  PMID: 35381977

Abstract

Introduction

In order to address the psychological problems during the COVID-19 pandemic, mental health chatbots have been extensively used by public sectors. According to Theory of Consumption Values, this paper proposed an analytical framework to investigate the determinants behind users’ satisfaction and continuance intention toward mental health chatbots.

Methods

The empirical study was conducted through an online survey, facilitated by the use of questionnaire posted on the WeChat platform. Seven-point Likert scale and closed-ended questions were employed. Totally 371 valid samples were collected. The research data was tested via the partial least squares structural equation modeling. Gender, age, and income were included as control variables.

Results

Analysis of samples collected from 371 Chinese users suggested that personalization (functional value), enjoyment (emotional value), learning (epistemic value), and the condition of the COVID-19 pandemic (conditional value) have positive influences on user satisfaction and continuance intention, but such effects were weak. The findings also revealed that user satisfaction has weakly positive impact on continuance intention. However, voice interaction (functional value) was an insignificant predictor of user satisfaction and continuance intention.

Discussion

This study developed a critical perspective on the role of Theory of Consumption Values in the context of mental health chatbot usage, while Theory of Consumption Value has been increasingly employed to explain the use of AI-based public services. Thus, this research devotes to the enhancement of theoretical frameworks regarding the usage of mental health chatbots.

Keywords: Mental health chatbots, COVID-19 pandemic, user satisfaction, continuance intention, Theory of Consumption Values

Introduction

The ongoing coronavirus pandemic 2019 (COVID-19) has caused millions of deaths and infected hundreds of millions of people worldwide, in turn leading to a plethora of mental disorders. 1 Government agencies in many countries have used mental health chatbots to provide virtual therapies and services. Although the use of mental health chatbots is widespread, our knowledge about what influences citizens’ continuance intention to use chatbots is limited. In addition, our knowledge about the determinants of citizens’ user satisfaction with mental health chatbots is limited. Such knowledge gaps significantly prevent the implementation of mental health chatbots during the COVID-19 pandemic.

The main purposes of this research are thus twofold: (1) employing Theory of Consumption Values (TCVs) as a theoretical framework and testing whether this theory can reveal the determinants behind user satisfaction and continuance intention during the COVID-19 pandemic; and (2) examining precisely the relationship between user satisfaction and continuance intention during the COVID-19 pandemic. These objectives are noteworthy for several reasons. Theoretically speaking, with the wide implementation of AI-based chatbots, it is necessary to extend the theoretical frameworks to explain user satisfaction and continuance intention. 2 However, prior research on the adoption of AI-based chatbots has mainly relied on Technology Acceptance Model (TAM). 3 Currently, the explanatory power of TCV has been widely demonstrated in the field of business, but this theory has rarely been employed to explain user behaviors of AI-based chatbots, especially mental health chatbots. Thus, this research may offer a critical insight into TCV by demonstrating whether it can be used to explain user satisfaction and continuance intention of mental health chatbots during the pandemic. Moreover, although many studies have revealed a positive relationship between user satisfaction and continuance intention when people use information systems (ISs),4,5 there is a lack of theoretically driven hypothesis-based research on this relationship in the public health emergency context. Therefore, this research might extend the existing literature on the relationship between user satisfaction and continuance intention in the IS area. Practically speaking, although chatbots are extensively used in the digital era, people's continuance intention to use chatbots remains low. 6 Expectation-Confirmation Theory (ECT), IS Continuance Theory (ISCT), and Technology Continuance Theory (TCT) have argued that people will be reluctant to continue using ISs if they feel dissatisfied with it.7,8 Thus, a deeper understanding of the antecedents of chatbot satisfaction and continuance intention can help government agencies and chatbot engineers develop chatbot systems and ultimately improve the adoption rate of mental health chatbots.

Based on TCV, this research developed a research model to examine the determinants behind citizens’ user satisfaction and continuance intention. Furthermore, the relationship between user satisfaction and continuance intention to use chatbots was tested in this study.

Theoretical background and hypotheses development

Mental health chatbots

Chatbots, also known as virtual agents or intelligent agents, are generally defined as computer systems that use AI-based technologies to mimic humanlike behaviors and offer a task-oriented framework with evolving dialog, making it capable of participating in conversations.9–11 A conversation between humans and chatbots is triggered by user input. The contextual awareness technologies allow chatbots to provide feedback based on receiving users’ messages. 12

Mental health is considered as one of the areas that will benefit greatly from chatbot services.11,13 Currently, mental health chatbots are increasingly being used for therapeutic and training objectives, such as suicide prevention 14 and cognitive-behavioral therapy (Ail, 2018). In some cases, virtual agents, such as HARR-E and Wysa, are also tailored to certain groups. 11

Compared with manned hotlines, mental health chatbots are more accessible and flexible. 15 It is believed that these advantages can offer great support to the existing mental healthcare system during the pandemic. For instance, mental health chatbots can simultaneously provide therapies and services for many more people than a manned hotline, 16 in turn improving the effectiveness of service delivery. In addition, mental health chatbots are devoted to addressing the shortage of human resources, especially mental health advisers, during the COVID-19 pandemic. 1 Given that the mental healthcare systems could benefit greatly from mental health chatbots, government agencies in many countries such as Singapore, America, and China have deployed virtual agents to deliver 24-h psychological assistance during the COVID-19 pandemic.16,17

Many previous studies have focused on the effectiveness and acceptability of mental health chatbots. For example, Fitzpatrick et al. 18 proved that a chatbot called “Woebot” could effectively help college students with depression. Cameron et al. 19 identified some limitations of a mental health agent named “iHepr” and gave suggestions to improve its functions. However, few studies have investigated the determinant factors behind users’ continuance intention to use mental health chatbots during the COVID-19 pandemic.

During the COVID-19-related lockdown, the government agencies in China then collaborated with their AI service contractors to launch a number of mental health chatbots to offer psychological assistance. For instance, a mental health chatbot named Psybot, which can listen, analyze, and talk, was employed by the local government in Hangzhou to help people express their true feelings and ease their moods. On encountering an emergency case, it sends out early warnings and asks professional human consultants to step in. 20 Another chatbot called Xiaoi, as deployed by the local government in Hubei Province. With its emotional computing and empathy module, Xiaoi could more clearly analyze citizens’ problems and provide a warm-hearted interaction. 21 This study selected a mental health chatbot (Xiaolv) as the research target.

User satisfaction and continuance intention

User satisfaction is a central concept in behavioral research.4,22 Expectation Disconfirmation Theory is an influential theory that explains the formation of user satisfaction by comparing expectations and post-experience belief in services. 23 According to this theory, individuals often set expectations before experiencing a service. 24 After experiencing this service, a post-experience belief is generated. 25 When comparing expectations and post-experience beliefs, individuals can feel a sense of disconfirmation. 26 If post-experience belief is better than expectation, the sense of disconfirmation will be positive, and individuals will feel satisfied. 27 In contrast, if post-experience belief is worse than expected, the sense of disconfirmation will be negative, and individuals will be dissatisfied. 28 Thus, user satisfaction typically refers to the fulfillment response that results from comparing a service's perceived expectation of performance. 22 In the field of ISs, user satisfaction has been widely employed by researchers and practitioners to predict people's continuance intention. 8

The notion of continuance intention is different from the concept of acceptance intention: the former emphasizes user loyalty, while the latter highlights initial acceptance. 7 According to ECT, ISCT, and TCT, loyalty and long-term use lead to the success of IS services.8,29,30 In this sense, continuance intention should be regarded as a key factor in IS success. 31 Thus, when IS technologies were being used, researchers and practitioners tried to investigate why people continued to adopt them. This was the case for e-learning, 32 online banking, 33 online shopping, 34 and online games. 31

User satisfaction is often regarded as an important predictor that stimulates continuance intention to use IS product or service. 29 Influential theories such as ECT, ISCT, and TCT ascertain that continuance intention is determined primarily by user satisfaction, which means that people who have higher levels of satisfaction with ISs are more likely to stay loyal longer.7,35 Generally, if individuals feel satisfied after using an IS product or service, they will believe that this product or service can also fulfill their expectations next time as well, and thus have greater motivation to continuously use it. 36

Theory of Consumption Values

TCV is a theory that reveals how people judge and make decisions to adopt a product or service. 37 TCV has been widely used to explain people's decision-making behavior in economics, business, and sociology. 38 According to TCV, products or services potentially contain multiple consumption values, including functional value, emotional value, epistemic value, conditional value, and social value. 39 Consumers attach different values to different products or services, which in return affects their intention to use. 38

Specifically, functional value is defined as the value acquired through the possession of utilitarian or physical attributes. 39 It is closely linked to practical factors such as reliability, effectiveness, and price. 38 In most cases, functional value is an important driver of consumers’ decisions.37,40,41 Emotional value is the utility derived from an alternative capacity to evoke affective feelings. 42 According to Sheth et al. (1991), products or services are frequently related to emotional responses, such as esthetics and feelings. These emotions and affections often influence customers’ motivation to use a product or service. Social value is defined as the utility derived from an alternative's relationship with specific groups (Yao et al., 2016). A product or service may represent a specific social group, customers can enhance a sense of belonging to that group through the consumption of such product or service. 43 Thus, people's decisions to use a product can be influenced by social value. Epistemic value is the utility of a product or service to arouse curiosity, or satisfy a desire for knowledge. 39 A product or service may be selected because people feel curious. 42 Conditional value is the utility acquired from a specific situation involved in customer choice. 40 When people are involved in a specific event or situation, they might value the use of a product or service because it can help them solve problems. As a result, people tend to use the product or service in a specific event or situation.39,44 For instance, people usually choose to use umbrella when it is raining. Conditional value arises suddenly, and consumers benefit from it temporarily. 45 Compared with other value components in TCV, conditional value more highlights the influence of condition on people's decisions.

The original TCV is used only to explain customers’ choice motivations. Recently, this theory has presented a wider view wherein consumption values also influence user satisfaction. 26 Consumption values arise before or during the purchase process, whereas user satisfaction is created at the post-purchase stage. 38 In this sense, consumption values may be the determinants of user satisfaction.

According to TCV, the relative consumption values vary with changing context. 41 Thus, the value components in this research were contextualized based on the characteristics of mental health chatbots. According to Zhu et al., 26 personalization and voice interaction are important functions of AI-based machines. As for mental health chatbots, personalization can help offer more appropriate therapies and thus improve the effectiveness of human–chatbot interaction. Voice interaction can give chatbots more human-like factors and make the human–chatbot interaction more natural. Thus, the functional values of mental health chatbots in this study are contextualized as personalization and voice interaction. As many previous studies have argued, enjoyment is a significant element of human–chatbot interaction. 10 The effective interaction between users and mental health chatbots induces a feeling of enjoyment, which in turn helps address psychological problems such as anxiety and stress. 46 Therefore, emotional value is contextualized as enjoyment. In addition, learning is another important element of human–computer interaction.40,41 Curiosity is one of our basic attributes and we are nearly oblivious of its pervasiveness in our lives. 47 Curiosity is also a component of subjective well-being. 48 As a significant notion in positive psychology, subjective well-being emphasizes that only individuals can assess their levels of well-being in any condition, including those who live with mental health conditions. 49 Thus, people can feel curious even if they encounter mental disorders. As a new AI-based technology, mental health chatbots can easily arouse curiosity and stimulate the desire to learn. Once individuals are motived to learn, they are more likely to continuously interact with chatbots to acquire knowledge. Meanwhile, through the learning process—that is, human–chatbot interaction—individuals can satisfy their desire for knowledge about mental health chatbots. As a result, user satisfaction will improve. Thereby, we employ learning to represent the epistemic value. People found themselves suddenly involved in a special condition (e.g. the national lockdown, overtaxed manned hotlines, and serious infection) due to the outbreak of the COVID-19 pandemic. Such a condition endows mental health chatbots with great utility, making them a valuable and available approach to offer psychological assistance. 26 In other words, the value of mental health chatbots can be better expressed and perceived by people during the COVID-19 pandemic. Thus, we contextualized the conditional value as the condition of the COVID-19 pandemic. However, social value is excluded in this study because a person is free of social bonds when interacting with a mental health chatbot.

Hypotheses development

Based on TCV, the functional values of mental health chatbots in this research are contextualized as personalization and voice interaction. Personalization refers to the ability of the system to provide personalized services that match users’ unique characteristics and needs.34,50 Currently, the achievements of machine learning and data mining technology allow chatbots to deeply analyze users’ historical data. Based on historical data, chatbots can offer services that may meet users’ demands better. 51 According to TCV, functional values are essential in forming user satisfaction 26 and continuance intention. 41 This point has been supported by many empirical studies on AI-based applications.52,53 In this study, personalization as a functional value can identify specific psychological problems accurately and offering personalized therapies or services. Thus, personalization may greatly help those who are with psychological issues during the COVID-19 pandemic. Based on the argument above, we formulated the following hypotheses:

  • H1a:

    Personalization is positively related to citizens’ user satisfaction with mental health chatbots.

  • H1b:

    Personalization is positively related to citizens’ continuance intention to use mental health chatbots.

Voice interaction is another function value of AI-based chatbots. It refers to the communication between human and chatbots through voice. 26 Because voice is commonly used by a person to interact with others in daily life, when chatbots use voice to communicate with people, people tend to feel natural and friendly. 54 As a result, the overall user experience of chatbots can be improved. 55 In this sense, the important role of voice interaction is to bring a feeling of natural and friendly. As some previous studies have proved, voice interaction may be a key variable that affects people's user satisfaction 56 and continuance intention 40 when they interact with AI-driven applications. During the COVID-19 pandemic, voice interaction may help the interaction between users and chatbots become more natural, which in return improves the overall interaction quality and might ultimately enhance user satisfaction and continuance intention. Therefore, we proposed the following hypotheses:

  • H2a:

    Voice interaction is positively related to citizens’ user satisfaction with mental health chatbots.

  • H2b:

    Voice interaction is positively related to citizens’ continuance intention to use mental health chatbots.

In this research, we expected that the emotional value of mental health chatbots (enjoyment) shows positive influences on user satisfaction as well as continuance intention. According to prior studies, a feeling of enjoyment is important to user satisfaction with AI-based chatbots.2,57 As to mental health chatbots, when users interact with them, a feeling of enjoyment can contribute to addressing psychological issues such as anxiety and depression, which in return improves the overall user satisfaction. 58 Thus, we hypothesize that:

  • H3a:

    Enjoyment is positively related to citizens’ user satisfaction with mental health chatbots.

For many years, enjoyment was believed to have influence on the acceptance behavior of ISs. 59 A feeling of enjoyment usually generates positive feedback and thus encourage user adoption of ISs.60,61 Prior studies based on TCV have proved that enjoyment as emotional value can stimulate people's motivation to adopt ISs.41,62 Recently, Kasilingam 57 has found that citizens’ motivation to adopt chatbots for online shopping can be greatly affected by a feeling of enjoyment. When people interact with a mental health chatbot, they may feel enjoyable because this chatbot provides high-quality services and therapies. Such a feeling of enjoyment can help to reduce people's stress, anxiety, and depression, 46 which in return may encourage their continuance intention to use the chatbot. Thus, we hypothesize that:

  • H3b:

    Enjoyment is positively related to citizens’ continuance intention to use mental health chatbots.

As a basic element of our cognition, curiosity motivates learning and decision-making. 47 Users choose to adopt a product or service may be because they feel curious and intend to learn. 38 According to TCV, learning is an important value component that can satisfy users’ curiosity. 45 Commonly, if a product or service successfully arouses people's curiosity, people will take action to learn more about it. The learning process is usually completed through the continuous use of this product or service. 63 Thus, curiosity motivates the desire to learn, and learning drives the continuance intention to use. Moreover, if people perceive that their knowledge is enriched through the use of a product or service, they tend to feel more satisfied. 63 The research by Kim et al. 64 proved that those who improve their degree of knowledge through the use of mobile shopping applications will have a higher level of user satisfaction. In addition, Lee et al. 40 found that learning can positively affect citizens’ intention to reuse Web-based taxi services. Thus, we formulated the following hypotheses:

  • H4a:

    Learning is positively related to citizens’ user satisfaction with mental health chatbots.

  • H4b:

    Learning is positively related to citizens’ continuance intention to use mental health chatbots.

According to TCV, the condition in which a product is used drives the conditional value. 38 Although a condition or event may be temporary, it can exert a great impact on user satisfaction and continuance intention. 65 The national lockdown, overtaxed manned hotlines, and serious infections triggered the special conditions of the COVID-19 pandemic. In such a condition, people tended to highly value the use of mental health chatbots because these virtual agents were available and helpful. As a result, citizens might be more likely to feel satisfied with mental health chatbots and show greater continuance intention. The research by Zhu et al. 26 noted that the condition of the COVID-19 pandemic is positively associated with user satisfaction when using mental health chatbots. Based on the arguments above, we hypothesize that:

  • H5a:

    The condition of the COVID-19 pandemic is positively related to citizens’ user satisfaction with mental health chatbots.

  • H5b:

    The condition of the COVID-19 pandemic is positively related to citizens’ continuance intention to use mental health chatbots.

User satisfaction is considered as a key factor to form and retain continuance intention by many theories. The early theories such as ECT and ISCT hold that user satisfaction is a basic and pivotal predictor of continuance intention.7,25,66 The recent theory such as TCT also ascertains that user satisfaction plays a great role in building user loyalty. 29 Based on these theories, existing literature has proved that there is a positive link between user satisfaction and continuance intention. To list a few, Kim et al. 36 argued that continuance intention is greatly influenced by user satisfaction when people use e-commerce systems. Nascimento et al. 67 proved that satisfaction has a great impact on continuance intention to adopt smartwatch. In line with these studies, we hypothesize that:

  • H6:

    User satisfaction is positively related to citizens’ continuance intention to use mental health chatbots.

All the hypotheses and the research model are shown in Figure 1.

Figure 1.

Figure 1.

Open in a new tab

The research model in this study.

Method

Data collection

This is a cross-sectional study conducted using data collected from Wuhan, Chongqing, and Hangzhou, during the COVID-19 pandemic in China. A complete national lockdown was implemented in China from January to March 2020. 26 This serious lockdown made millions of people experience a sudden change. It also caused a shortage of living supplies and financial distress. 68 In such a condition, many people felt anxious, depress, and sleepless.

In order to help those who are with mental health problems, local governments launched an AI-based mental health chatbot called Xiaolv. People with issues of stress, anxiety, depression, and sleep could interact with Xiaolv through Wechat, a popular mobile application in China. 26 Figure 2 shows the typical mental health care supervised by Xiaolv. At the beginning, Xiaolv welcomes users via a self-introduction. Then, users can freely “talk” to Xiaolv about their worries, emotions, and feelings through both spoken and written languages. After analyzing users’ inputs, Xiaolv will organize languages to encourage and comfort users. For instance, when a user with depression input “Due to the pandemic, I have to stay at home. I am not in mod,” Xiaolv comforted this person by saying “Do not be nervous my dear, no matter what happened, I will always be with you.” Such a conversation could offer a feeling of warm to users, especially those who are in home isolation, and make them believe they find a good listener. Also, Xiaolv could provide other mental health supports to help users relax, such as game recommendations.

Figure 2.

Figure 2.

Open in a new tab

A case of mental health care supervised by Xiaolv. It will analyze users’ problems via a two-way communication and offer personalized services accordingly.

From February to April 2020, many Chinese undergraduate students worked as volunteers to fight on the front line of urban and rural communities for epidemic prevention. Community health service centers randomly assigned several families to each student volunteer. The students then provided health services for the designated families. We contacted 49 volunteer students in Wuhan, Chongqing, and Hangzhou, they helped us to send the Xiaolv chatbot's link through WeChat groups to 166 families in 43 communities. After a week, the online questionnaire was uploaded on Wechat groups and potential respondents were requested to complete it.

Before the formal survey, several experts in the area of ISs were invited to give suggestions. In addition, we also conducted a pilot test with 23 volunteers. According to their revision advice, we modified the questionnaire, making it contain three parts. In the first part, we used three questions to identify those who had adopt Xiaolv: “could you please tell us the name of the chatbot?,” “have you interacted with the chatbot?” and “can this chatbot speak to you?.” Then, we collected demographic information of the respondents. Third, we measured consumption values, user satisfaction, and continuance intention through a set of items. As a response, we totally collected 445 samples. Then, we deleted those responses with missing data and those samples who indicated that they never interact with Xiaolv. Finally, 371 responses were considered as valid. The demographic information, user frequency, and duration are shown in Table 1.

Table 1.

The information of the research samples.

Category Number Percentage
Gender Male 217 58.5%
Female 154 41.5%
Age 18–30 63 17.0%
31–40 212 57.1%
41–50 79 21.3%
Over 50 17 4.6%
Education Under high school 71 19.1%
High school 82 22.1%
Bachelor degree 202 54.5%
Master's degree and above 16 4.3%
Annual income $0–$4500 77 20.8%
$4501–$15,000 200 53.9%
$ 15,001–$45,000 81 21.8%
Over $45,001 13 3.5%
User frequency Once within a week 43 11.6%
Twice within a week 105 28.3%
3–5 times within a week 173 46.6%
More than 5 times within a week 50 13.5%
Duration Less than 5 min once 118 31.8%
5–10 min once 203 54.7%
10–20 min once 41 11.1%
Over 20 min once 9 2.4%
Open in a new tab

Measurement development

All measurement items used in this paper were adapted from relevant studies. Measurement items for personalization were adapted according to Roy et al. 69 Three items were used to measure voice interaction based on the research by Zhu et al. 26 Measurements for enjoyment were modified from Lee et al. 40 To measure learning, three items were used according to Teng. 41 Items measuring the condition of the COVID-19 pandemic were adapted from Omigie et al.. 65 User satisfaction was measured according to the research by Li and Fang. 70 Finally, continuance intention was measured based on three items from Ashfaq et al.. 2 Each item in this study was evaluated by using a seven-point Likert scale, with anchors ranging from 1 (highly disagree) to 7 (highly agree). All the measurement items were shown in Appendix. In order to make the potential respondents understand our questions, we translated the questionnaire from English to Chinese. As Brislin 71 has argued, translation is necessary in the formulation of questionnaires for cross-cultural research. In this research, we checked the back-translation according to the approach suggested by Brislin 71 and Werner and Campbell 72 : (1) simple sentences, (2) repetition of nouns rather than use of pronouns, (3) avoiding English passive tense, (4) avoiding metaphor and colloquialisms.

Control variables

Several commonly controlled variables such as age, gender, income, and education were considered as control variables in this research. Chen and Mitomo 73 believed that these specific variables might have impact on the empirical results. Thus, we included them as control variables and did not test their influences in this research.

Results

Measurement model assessment

In this study, we employed the partial least squares structural equation modeling (PLS-SEM) to examine the 11 hypotheses. According to Kasilingam, 57 PLS-SEM has the ability to compare two or more groups by specifying a permutation-based analysis of variance approach. Thus, it is widely used for hypothesis-testing analysis.

The measurement model was assessed through reliability, convergent validity, and discriminant validity. According to Table 2, the Cronbach's alpha values for all variables are higher than 0.70, suggesting that the measurement is reliable. 74 In addition, the composite reliabilities of all the cases are higher than the recommended value of 0.70. 26 Moreover, the factor loadings of all items are greater than the desirable level of 0.70 and reported as significant. 75 The average variance extracted (AVE) ranges between 0.628 and 0.728, exceeding the recommended value of 0.5. 76 Thus, the measurement model meets the conditions of reliability and convergent validity.

Table 2.

The reliability and validity of the measurement.

Constructs Items Factor loading Mean Cronbach's alphas CR AVE VIF
Personalization (P) P1 0.817 4.299 0.839 0.892 0.674 2.925
P2 0.765 4.814 2.530
P3 0.876 4.445 3.244
P4 0.822 4.774 2.684
Voice interaction (VI) VI1 0.791 4.647 0.702 0.835 0.628 1.445
VI2 0.746 4.776 1.269
VI3 0.838 4.752 1.636
Enjoyment (E) E1 0.785 4.437 0.752 0.858 0.669 1.429
E2 0.797 4.501 1.509
E3 0.870 4.685 1.722
Learning (L) L1 0.795 4.631 0.746 0.856 0.664 1.463
L2 0.792 4.679 1.435
L3 0.856 4.863 1.723
Condition (C) C1 0.804 4.892 0.784 0.875 0.701 1.678
C2 0.800 4.644 1.624
C3 0.903 4.752 2.312
User satisfaction (SA) SA1 0.831 4.655 0.805 0.886 0.721 1.845
SA2 0.810 4.749 1.612
SA3 0.904 4.580 2.365
Continuance intention (CI) CI1 0.824 4.663 0.811 0.889 0.728 1.815
CI2 0.818 4.690 1.746
CI3 0.914 4.666 2.546
Open in a new tab

In order to test discriminant validity, we used cross-factor loadings and Fornell-Larcker criterion. Table 3 shows the cross-factor loadings of all variables, proving that they loaded highest on their respective construct. 26 According to Table 4, none of the intercorrelations of the constructs are higher than the square root of the AVE for the constructs, indicating that the condition of discriminant validity is met by the measurement model. 75 Moreover, there is an adequate model fit when the value of SRMR is less than 0.08. 57 In this research, SRMR is 0.07.

Table 3.

Cross-factor loadings of all variables.

P VI E L C SA CI
P1 0.817 0.404 0.645 0.635 0.629 0.683 0.660
P2 0.765 0.404 0.508 0.516 0.588 0.510 0.640
P3 0.876 0.429 0.677 0.656 0.662 0.702 0.689
P4 0.822 0.430 0.552 0.555 0.614 0.523 0.651
VI1 0.393 0.791 0.369 0.412 0.414 0.421 0.412
VI2 0.424 0.746 0.408 0.388 0.410 0.387 0.429
VI3 0.385 0.838 0.411 0.396 0.405 0.388 0.399
E1 0.588 0.375 0.785 0.570 0.590 0.563 0.619
E2 0.551 0.410 0.797 0.550 0.582 0.573 0.592
E3 0.649 0.440 0.870 0.606 0.670 0.677 0.708
L1 0.595 0.418 0.555 0.795 0.620 0.604 0.634
L2 0.612 0.357 0.579 0.792 0.625 0.582 0.661
L3 0.562 0.458 0.585 0.856 0.611 0.601 0.672
C1 0.611 0.445 0.593 0.601 0.804 0.626 0.683
C2 0.620 0.430 0.625 0.625 0.800 0.655 0.672
C3 0.676 0.426 0.670 0.678 0.903 0.660 0.728
SA1 0.627 0.386 0.623 0.595 0.633 0.831 0.661
SA2 0.618 0.460 0.625 0.605 0.659 0.810 0.676
SA3 0.648 0.438 0.641 0.660 0.677 0.904 0.725
CI1 0.662 0.419 0.661 0.669 0.681 0.674 0.824
CI2 0.683 0.446 0.641 0.662 0.693 0.669 0.818
CI3 0.712 0.472 0.704 0.728 0.749 0.728 0.914
Open in a new tab

Table 4.

Discriminant validity.

P VI E L C SA CI
P (0.821)
VI 0.507 (0.792)
E 0.731 0.500 (0.818)
L 0.723 0.505 0.704 (0.815)
C 0.760 0.518 0.753 0.759 (0.837)
SA 0.743 0.504 0.742 0.731 0.773 (0.849)
CI 0.804 0.523 0.785 0.805 0.830 0.810 (0.853)
Open in a new tab

Multicollinearity

In order to check multicollinearity, we examined the variance inflation factor (VIF). Table 2 shows that all VIF values are lower than the recommended value of 5.0, 26 indicating that the research model is free from multicollinearity.

Structural model analysis

The proposed hypothesis and path coefficients were examined by applying bootstrapping procedure with 2000 subsamples and two-tailed test with a significance of 0.05 in Smart PLS 3.3. According to Figure 3, user satisfaction (β = 0.207, t = 2.547, p < 0.01) and continuance intention (β = 0.199, t = 2.811, p < 0.01) were positively influenced by personalization, offering support to H1a and H1b. Enjoyment is positively related to user satisfaction (β = 0.225, t = 2.471, p < 0.05) and continuance intention (β = 0.152, t = 2.232, p < 0.05), offering support to H3a and H3b. In addition, the influences of learning on user satisfaction (β = 0.185, t = 2.037, p < 0.05) and continuance intention (β = 0.227, t = 2.470, p < 0.05) offer support to H4a and H4b, respectively. Similarly, the condition of the COVID-19 pandemic is positively correlated with both user satisfaction (β = 0.281, t = 2.905, p < 0.01) and continuance intention (β = 0.235, t = 3.063, p < 0.01), supporting H5a and H5b. User satisfaction also shows positive impact on continuance intention (β = 0.197, t = 2.150, p < 0.05). Thus, H6 is supported. However, voice interaction shows insignificant effect on both user satisfaction (β = 0.048, t = 1.078, p > 0.05) and continuance intention (β = 0.011, t = 0.329, p > 0.05), H2a and H2b should be rejected. Overall, 9 out of the 11 hypotheses are supported, as shown in Table 5. According to Figure 3, there were weak betas (<0.30) but high R2 values (>0.67) in the research model (Seo & Bernsen, 2016). This may be because the determinant factors were highly interrelated (shown in Table 4). While these interrelated variables were significantly correlated with the dependent variables on an individual level, together they might compete to explain the same variance.

Figure 3.

Figure 3.

Open in a new tab

Hypotheses testing. Path significance: *p < 0.05, **p < 0.01.

Table 5.

Results of hypotheses testing.

Hypothesis Path T statistics Result
H1a P → SA 2.547 Supported
H1b P → CI 2.811 Supported
H2a VI → SA 1.078 Not supported
H2b VI → CI 0.329 Not supported
H3a E → SA 2.471 Supported
H3b E → CI 2.232 Supported
H4a L → SA 2.037 Supported
H4b L → CI 2.470 Supported
H5a C → SA 2.905 Supported
H5b C → CI 3.063 Supported
H6 SA → CI 2.150 Supported
Open in a new tab

Discussion and implications

AI-based mental health chatbots have been extensively used to address the COVID-19 outbreak-related mental disorders across the world. This study investigated the determinant factors behind citizens’ user satisfaction and continuance intention when interacting with a mental health chatbot. In addition, it further explored the relationship between user satisfaction and continuance intention.

Specifically, voice interaction failed to bring greater satisfaction and stronger continuance intention to users. This result is consistent with the research by Zhu et al.. 26 Moreover, although personalization showed positive influences on user satisfaction and continuance intention, the effects were not as strong as the research by Chen et al. 37 Similarly, enjoyment showed weakly significant influences on user satisfaction and continuance intention. This result is in line with some previous studies.40,77 Meanwhile, learning was positively related to people's user satisfaction and continuance intention, but the relationship was not strong. This result is different to some prior research26,41 that found a great impact of learning on both user satisfaction and user loyalty. The condition of the COVID-19 pandemic also had weakly significant influences on user satisfaction and continuance intention. This is consistent with the research by Zhu et al. 26 Finally, the positive relationship between user satisfaction and continuance intention was weakly supported in this research. This result is in line with the research by Ashfaq et al. 2 (weakly supported) but is different to the study by Kim et al. 5 (strongly supported).

Thus, although the positive influences of personalization, enjoyment, learning, and the condition of the COVID-19 pandemic were successfully demonstrated in this research, their effects were all rather weak. One appropriate explanation for this is that these value components of TCV frame the therapy as something to be consumed or sold to consumers, but this might be an incorrect way to consider mental health service. As mental health service in this research was free of charge public service that offered warm, personable, and reflective care, it cannot be handled from a consumer's perspective with TCV. Thereby, although the determinants of TCV could explain user satisfaction and continuance intention, they failed to exert strong influences. Another potential explanation is that we failed to find the most appropriate value components to predict user satisfaction and continuance intention toward mental health chatbots. TCV allows researchers to contextualize value components according to the specific product or service. 5 Hence, we contextualized five value components based on the features of mental health chatbots and prior literature. It is possible that some other value components may have greater influences but we failed to identify them in this research.

Theoretical implications

Although value components were not strongly related to user satisfaction and continuance intention, the findings in this research can make contribution to the strengthening of theory regarding the acceptance and use of chatbots. First, this research is one of the early attempts to investigate chatbot users’ satisfaction and continuance intention based on TCV. Currently, the wide use of mental health chatbots creates a demand to extend the theoretical frameworks to predict user satisfaction and continuance intention. While the existing studies have predominately used TAM, 3 this study provides a new perspective by employing TCV.

Second, this study also paves the way to future studies. Through the contextualized theorizing process, we identified five value components to predict user satisfaction and continuance intention toward mental health chatbots. As the results shown, voice interaction failed to play a significant role, the influences of personalization, enjoyment, learning, and the condition of the COVID-19 pandemic were weakly significant. These findings offer empirical evidences to future studies when choosing value components.

Practical implications

For practices, the findings in this research had the potential to contribute to user satisfaction and continuance intention because it developed end-user awareness on what factors should be considered for better performance of mental health chatbot services. First, our findings are straightforward: the personalized service is the predictor of user satisfaction and continuance intention. Accordingly, in order to make users keep using mental health chatbots, chatbot developers and designers should ensure that their products can offer personalized, relevant, and effective mental services. The design process should take into account the chatbot's capacity to understand user's demands and provide appropriate therapy.

Second, if people feel enjoyable when interacting with a mental health chatbot, they will tend to keep using this chatbot. Recently, with the evolutionary development of entertainment technologies, gamification techniques and features have been increasingly used in human–computer interaction to generate a feeling of enjoyment. 78 In the context of mental health chatbot usage, chatbot developers and designers can added more gamification features into mental health chatbots to make the human–chatbot interaction more pleasant and enjoyable. For example, these chatbots may recommend more funny games to users. It is believed that gamification elements have potential to offer a feeling of enjoyment and ultimately improve user satisfaction and continuance intention.

Moreover, if people can enrich their knowledge and satisfy their desire to learn when using a mental health chatbot, they will tend to feel satisfied and continuously adopt it. Thus, chatbot developers need to ensure that people can satisfy their desire to learn through the human–chatbot interaction. For instance, chatbot developers and designers might embed subject-related knowledge about mental health chatbots, such as purposes, functions, and cases, into chatbot systems so that users can read the information and enrich their knowledge about mental health chatbots. Also, chatbot developers may conduct a survey to understand what people desire to know about mental health chatbots. This can help mental health chatbots offer information to meet people's curiosity and improve their user satisfaction and continuance intention.

Research limitations

This research is not without limitations. First, as mentioned above, one of the most obvious limitations in this study is the use of TCV because the value components of TCV fail to exert strongly positive influences on user satisfaction and continuance intention. Second, there are problems regarding the generalizability of the findings, as the research samples were only collected from Wuhan, Hangzhou, and Chongqing. Future studies should conduct international investigation to validate the findings of this research. Third, we only chose a mental health chatbot as a research target, the results may be influenced by the advantages or shortages of the chatbot. Thus, researchers should consider more research targets in the future. Fourth, most of the correlations between factors were over 0.7, indicating that these determinant factors may share commonalities. This may influence the findings. Finally, this study considered demographic factors as control variables. Future studies should aim to test the potential moderating effect of demographic factors on continuance intention.

Conclusion

The wide use of mental health chatbots during the COVID-19 pandemic provides an opportunity to explore the determinant factors behind users’ satisfaction and continuance intention toward these virtual agents. This paper is one of the early attempts to investigate the usage of a mental health chatbot through the lens of TCV. Although the findings outlined the significance of personalization, enjoyment, learning, and the condition of the COVID-19 pandemic, their effects were all rather weak. Also, the positive relationship between user satisfaction and continuance intention was weakly supported. On the other hand, this research found that voice interaction is an insignificant predictor of user satisfaction and continuance intention. All these findings developed a critical perspective on the role of TCV in the context of mental health chatbot usage, while this theory has started to be increasingly used in the context of AI-based public service delivery. Based on the findings, several managerial implications for chatbot developers and designers were suggested to improve user satisfaction and continuance intention of mental health chatbots.

Although there were limitations in this research, it might make considerable contributions to both existing and future studies. First, it filled in our knowledge gaps by identifying four determinants that influenced user satisfaction and continuance intention of mental health chatbots during the COVID-19 pandemic. Moreover, it developed a critical perspective on TCV that prior studies could not acknowledge. In addition, it provided empirical evidence to enhance our understanding on the relationship between user satisfaction and continuance intention. Future studies can benefit from the findings in this research when investigating the usage of mental health chatbots.

Acknowledgments

We are grateful to the editor and anonymous reviewers for their valuable comments on this research. In particular, we should appreciate professor Theo Arvanitis and editor Kaur Simran.

Appendix

Measures and sources

Variables Measurement items Source
Personalization P1. The chatbot knows my specific moods and needs. Roy et al. 68 ;
P2. The chatbot offers personalized recommendations according to my demands.
P3. The services provided by the chatbot are customized to my needs.
P4. I can get personalized therapies that are tailored to my mental issues by using the chatbot.
Voice interaction VI1. The chatbot can communicate with me like a human. 26
VI2. The chatbots’ voice sounds like a real human.
VI3. The chatbot can fluently use human voice to talk with me.
Enjoyment E1. Interacting with the chatbot makes me feel pleasant. Lee et al. 40
E2. I feel relaxed when I interact with the chatbot.
E3. The use of the chatbot is enjoyable.
Learning L1. I can enrich my knowledge about AI through the use of the chatbot. Teng 41
L2. I can learn more about AI by interacting with the chatbot.
L3. The use of the chatbot can satisfy my desire to learn.
The condition of the COVID-19 pandemic C1. The mental health hotline is overtaxed due to the impact of the COVID-19 pandemic. Omigie et al. 65
C2. I cannot go outside to receive therapy due to the impact of the COVID-19-related lockdown.
C3. I cannot find any professionals to help me due to the impact of the COVID-19 pandemic.
User satisfaction SA1. Using this chatbot is wise choice. Li and Fang 69
SA2. This chatbot can hardly make me feel disappointed.
SA3. Overall, I feel satisfied with this chatbot.
Continuance intention CI1. I will continue to use mental health chatbots in the future. Ashfaq et al. 2
CI2. Mental health chatbots become one of my first choices when I experience psychological problems.
CI3. I will recommend mental health chatbots to others.
Open in a new tab

Footnotes

Conflict of Interest: The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Contributorship: Yonghan Zhu wrote the original draft of the manuscript, completed revisions, participated in the conceptualization of the manuscript content, and conducted data collection and data analysis. Rui Wang completed revisions, conducted data collection, and assisted in the data analysis. Chengyan Pu conducted data collection and provided revisions on the manuscript. All authors approved of the final manuscript draft.

Ethnical approval: Not applicable.

Funding: The authors received no financial support for the research, authorship, and/or publication of this article.

Guarantor: Yonghan Zhu.

References

  • 1.Ransing R, Nagendrappa S, Patil Aet al. et al. Potential role of artificial intelligence to address the COVID-19 outbreak-related mental health issues in India. Psychiatry Res 2020; 290: 113176. [DOI] [PubMed] [Google Scholar]
  • 2.Ashfaq M, Yun J, Yu Set al. et al. I, chatbot: modeling the determinants of users’ satisfaction and continuance intention of AI-powered service agents. Telemat Inform 2020; 54: 101473. [Google Scholar]
  • 3.Pillai R, Sivathanu B. Adoption of AI-based chatbots for hospitality and tourism. Int J Contemp Hosp Manage 2020; 32: 3199–3226. [Google Scholar]
  • 4.Deng L, Turner DE, Gehling Ret al. et al. User experience, satisfaction, and continual usage intention on IT. Eur J Inf Syst 2010; 19: 60–75. [Google Scholar]
  • 5.Kim M, Chang Y, Park Met al. et al. The effects of service interactivity on the satisfaction and the loyalty of smartphone users. Telemat Inform 2015; 32: 949–960. [Google Scholar]
  • 6.Forbes AR. How business can begin using chatbots the right way. 2019. Retrieved from https://www.forbes.com/sites/forbestechcouncil/2019/01/18/how-businesses-can-begin-using-chatbots-the-right-way/#1d3b68252183 (Assessed, 13 July 2020). [Google Scholar]
  • 7.Bhattacherjee A. Understanding information systems continuance: an expectation-confirmation model. MIS Q 2001: 351–370. [Google Scholar]
  • 8.Rahi S, Khan MM, Alghizzawi M. Extension of technology continuance theory (TCT) with task technology fit (TTF) in the context of Internet banking user continuance intention. Int J Quality Reliab Manage 2021; 38: 986–1004. [Google Scholar]
  • 9.Shawar BA, Atwell E. Chatbots: are they really useful? LDV-Forum 2007; 22: 29–49. [Google Scholar]
  • 10.Valtolina S, Barricelli BR, Gaetano SD. Communicability of traditional interfaces VS chatbots in healthcare and smart home domains. Behav Inf Technol 2019; 39: 108–132. [Google Scholar]
  • 11.Vaidyam AN, Wisniewski H, Halamka JDet al. et al. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. Can J Psychiatry 2019; 64: 456–464. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Pearl C. Designing voice user interfaces: principles of conversational experiences. O’Reilly Media, Inc, 2016. [Google Scholar]
  • 13.Nadarzynski T, Miles O, Cowie Aet al. et al. Acceptability of artificial intelligence (AI)-led chatbot services in healthcare: a mixed-methods study. Digit Health 2019; 5: 1–12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.WIRED. Virtual therapists help veterans open up about PTSD. 2018. https://www.wired.com/story/virtual-therapists-help-veterans-open-up-about-ptsd/. Retrieved from. [Google Scholar]
  • 15.Smith AC, Thomas E, Snoswell CLet al. et al. Telehealth for global emergencies: implications for coronavirus disease 2019 (COVID-19). J Telemed Telecare 2020; 26: 309–313. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Ahuja AS, Reddy VP, Marques O. Artificial intelligence and COVID-19: a multidisciplinary approach. Integr Med Res 2020; 9: 100434. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Liu S, Yang L, Zhang Cet al. et al. Online mental health services in China during the COVID-19 outbreak. Lancet Psychiat 2020; 7: 30077–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment Health 2017; 4: e19. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Cameron G, Cameron D, Megaw Get al. et al. Assessing the usability of a chatbot for mental health care. Int Workshop Int Sci 2019, April: 121–132. Cham: Springer. [Google Scholar]
  • 20.Chinadaily. Supporting epidemic prevention through digital technology. Government in Hangzhou recommends mental health chatbot 2020. Retrieved from http://caijing.chinadaily.com.cn/a/202004/03/WS5e86c784a3107bb6b57aa9f9.html.
  • 21.Chinanews. The chatbot Xiaoi offers several AI-based services to support epidemic prevention. 2020. Retrieved from https://m.chinanews.com/wap/detail/zw/business/2020/02–17/9094091.shtml.
  • 22.Kotler P, Keller KL. A framework for marketing management. Essex: Pearson Education Limited. 2012.
  • 23.Oliver RL. Effect of expectation and disconfirmation on postexposure product evaluations: an alternative interpretation. J Appl Psychol 1977; 62: 480–486. [Google Scholar]
  • 24.Olson JC, Dover PA. Disconfirmation of consumer expectations through product trial. J Appl Psychol 1979; 64: 179–189. [Google Scholar]
  • 25.Oliver RL. Measurement and evaluation of satisfaction processes in retail settings. J Retail 1981; 57: 25–48. [Google Scholar]
  • 26.Zhu Y, Janssen M, Wang Ret al. et al. It is me, chatbot: working to address the COVID-19 outbreak-related mental health issues in China. User experience, satisfaction, and influencing factors. Int J Hum Comput Interact 2021. [Google Scholar]
  • 27.Lankton N, McKnight DH, Thatcher JB. Incorporating trust-in-technology into expectation disconfirmation theory. J Strategic Infor Syst 2014; 23: 128–145. [Google Scholar]
  • 28.Yu M, Zhou R, Cai Zet al. et al. Unravelling the relationship between response time and user experience in mobile applications. Internet Res 2020; 30: 1353–1382. [Google Scholar]
  • 29.Liao C, Palvia P, Chen JL. Information technology adoption behavior life cycle: toward a Technology Continuance Theory (TCT). Int J Inf Manage 2009; 29: 09–320. [Google Scholar]
  • 30.Sun Y, Liu L, Peng Xet al. et al. Understanding Chinese users’ continuance intention toward online social networks: an integrative theoretical model. Electron Mark 2014; 24: 57–66. [Google Scholar]
  • 31.Wang X, Goh D, & Lim HLet al. et al. Understanding continuance intention toward crowdsourcing games: a longitudinal investigation. Int J Hum Comput Interact 2020; 36: 1168–1177. [Google Scholar]
  • 32.Lin WS. Perceived fit and satisfaction on web learning performance: IS continuance intention and task-technology fit perspectives. Int J Hum Comput Stud 2012; 70: 498–507. [Google Scholar]
  • 33.Hoehle H, Huff S, Goode S. The role of continuous trust in information systems continuance. J Comput Inf Syst 2012; 52: 1–9. [Google Scholar]
  • 34.Kang JW, Namkung Y. The role of personalization on continuance intention in food service mobile apps. Int J Contemp Hosp Manage 2019; 31: 734–752. [Google Scholar]
  • 35.Xiang L, Zheng X, Zhang KZKet al. et al. Understanding consumers’ continuance intention to contribute online reviews. Ind Manage Data Syst 2018; 118: 22–40. [Google Scholar]
  • 36.Kim DJ, Ferrin DJ, Rao HR. Trust and satisfaction, two stepping stones for successful e-commerce relationships: a longitudinal exploration. Inf Syst Res 2009; 20: 237–257. [Google Scholar]
  • 37.Chen T, Guo W, Gao Xet al. et al. AI-based self-service technology in public service delivery: user experience and influencing factors. Gov Inf Q 2021: 101520. [Google Scholar]
  • 38.Yeo BL, Mohamed RN, Muda M. A study of Malaysian customers purchase motivation of halal cosmetics retail products: examining theory of consumption value and customer satisfaction. Procedia Econ Finance 2016; 37: 176–182. [Google Scholar]
  • 39.Sheth JN, Newman BI, Gross BL. Why we buy and what we buy: a theory of consumption values. J Bus Res 1991; 22: 159–170. [Google Scholar]
  • 40.Lee S, Lee J, Kim H. A customer value theory approach to the engagement with a brand: the case of KaoKao talk plus in Korea. Asia Pacific J Infor Syst 2018; 28: 36–60. [Google Scholar]
  • 41.Teng C. Look to the future: enhancing online gamer loyalty from the perspective of the theory of consumption values. Decis Support Syst 2018; 114: 49–60. [Google Scholar]
  • 42.Sweeney JC, Soutar GN. Consumer perceived value: the development of a multiple item scale. J Retail 2001; 77: 203–220. [Google Scholar]
  • 43.Mathwick C, Wiertz C, de Ruyter K. Social capital production in a virtual P3 community. J Consum Res 2008; 34: 832–849. [Google Scholar]
  • 44.Hung CL, Hsieh CY. Searching the fit pattern between cultural dimensions and consumption values of mobile commerce in Taiwan. Asia Pacific Manage Rev 2010; 15: 147–165. [Google Scholar]
  • 45.Furukawa H, Matsumura K, Harada S. Effect of consumption values on consumer satisfaction and brand commitment: investigating functional, emotional, social, and epistemic values in the running shoes market. Int Rev Manage Market 2019; 9: 158–168. [Google Scholar]
  • 46.Cheng Y, Jiang H. AI-powered mental health chatbots: examining users’ motivations, active communicative action and engagement after mass-shooting disasters. J Contingencies Crisis Manage 2020; 28: 339–354. [Google Scholar]
  • 47.Kidd C, Hayden BY. The psychology and neuroscience of curiosity. Neuron 2015; 88: 449–460. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Macaskill A, Denovan A. Assessing psychological health: the contribution of psychological strengths. Br J Guid Counc 2014; 42: 320–337. [Google Scholar]
  • 49.Macaskill A. A feasibility study of psychological strengths and well-being assessment in individuals living with recurrent depression. J Posit Psychol 2012; 7: 372–386. [Google Scholar]
  • 50.Sheng H, Nah F, Siau K. Values of silent commerce: a study using value-focused thinking approach. Proc Eleventh Am Conf Inf Syst 2005, August: 11–14. NE. [Google Scholar]
  • 51.Shi S, Wang Y, Chen Xet al. et al. Conceptualization of omnichannel customer experience and its impact on shopping intention: a mixed-method approach. Int J Inf Manage 2020; 50: 325–336. [Google Scholar]
  • 52.Wang YS, Li HT, Li CRet al. et al. Factors affecting hotels adoption of mobile reservation systems: a technology-organization-environment framework. Tourism Manage 2016; 53: 163–172. [Google Scholar]
  • 53.Kocaballi AB, Berkovsky S, Quiroz JCet al. The personalization of conversational agents in health care: systematic review. J Med Internet Res 2019; 21: e15360. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Ni Y, Wang Y. Design of a smart storytelling toy based on voice interaction. Proceedings of the 2019 2nd World Conference on Mechanical Engineering and Intelligent Manufacturing, November 2019: pp. 229–233. IEEE. [Google Scholar]
  • 55.Mavridis N. A review of verbal and non-verbal human-robot interactive communication. Robot Auton Syst 2015; 63: 22–35. [Google Scholar]
  • 56.Kiseleva J, Williams K, Awadallah AHet al. et al. Predicting user satisfaction with intelligent assistants. Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, July 2016: pp. 45–54. ACM. [Google Scholar]
  • 57.Kasilingam DL. Understanding the attitude and intention to use smartphone chatbots for shopping. Technol Soc 2020; 62: 101280. [Google Scholar]
  • 58.Abd-alrazaq AA, Alajlani M, Alalwan AAet al. et al. An overview of the features of chatbots in mental health: a scoping review. Int J Med Inf 2019; 132: 103978. [DOI] [PubMed] [Google Scholar]
  • 59.Zhang L, Zhu J, Liu Q. A meta-analysis of mobile commerce adoption and the moderating effect of culture. Comput Human Behav 2012; 28: 1902–1911. [Google Scholar]
  • 60.Agrebi S, Jallais J. Explain the intention to use smartphones for mobile shopping. J Retail Consum Serv 2015; 22: 16–23. [Google Scholar]
  • 61.Merhi MI. Factors influencing higher education students to adopt podcast: an empirical study. Comput Educ 2015; 83: 32–43. [Google Scholar]
  • 62.Kaur P, Dhir A, Rajala Ret al. et al. Why people use online social media brand communities. A consumption value theory perspective. Online Inf Rev 2018; 42: 205–221. [Google Scholar]
  • 63.Qaoumi KE, Masson PL, Weil Bet al. et al. Testing evolutionary theory of household consumption behavior in the case of novelty – a product characteristics approach. J Evol Econ 2018; 28: 437–460. [Google Scholar]
  • 64.Kim Y, Wang Q, Roh T. Do information and service quality affect perceived privacy protection, satisfaction, and loyalty? Evidence from a Chinese O2O–based mobile shopping application. Telemat Inform 2020: 101483. [Google Scholar]
  • 65.Omigie NO, Zo H, Rho JJet al. et al. Customer pre-adoption choice behavior for M-PESA mobile financial services. Extending the theory of consumption values. Ind Manage Data Syst 2017; 117: 910–926. [Google Scholar]
  • 66.Anderson EW and Sullivan MW. The antecedents and consequences of customer satisfaction for firms. Marketing Science 1993; 12: 125–143. [Google Scholar]
  • 67.Nascimento B, Oliveira T, Tam C. Wearable technology: what explains continuance intention in smartwatches? J Retail Consum Serv 2018; 43: 157–169. [Google Scholar]
  • 68.Yuan Z, Xiao Y, Dai Zet al. et al. Modeling the effects of Wuhan's Lockdown during COVID-19, China. Bull W H O 2020; 98: 484–494. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 69.Roy SK, Balaji MS, Sadeque Set al. et al. Constituents and consequences of smart customer experience in retailing. Technol Forecast Soc Change 2017; 124: 257–270. [Google Scholar]
  • 70.Li C, Fang Y. Predicting continuance intention toward mobile branded apps through satisfaction and attachment. Telemat Inform 2019; 43: 101248. [Google Scholar]
  • 71.Brislin RW. Back-translation for cross-cultural research. J Cross Cult Psychol 1970; 1: 185–216. [Google Scholar]
  • 72.Werner O., Campbell D. T. (1970). Translating working through interpreters, and the problem of decentering. Handbook Cultural Anthropol. American Museum of Natural History, New York. [Google Scholar]
  • 73.Cheng JW, Mitomo H. The underlying factors of the perceived usefulness of using smart wearable devices for disaster applications. Telematics & Informatics 2017; 34: 528–539. [Google Scholar]
  • 74.Zhang B, Zhu Y. Comparing attitudes towards adoption of e-government between urban users and rural users: an empirical study in Chongqing municipality, China. Behav Inf Technol 2021; 40: 1154–1168. [Google Scholar]
  • 75.Ju J, Liu L, Feng Y. Public and private value in citizen participation in E-governance: evidence form a government – sponsored green commuting platform. Gov Inf Q 2019; 36: 101400. [Google Scholar]
  • 76.Fornell C, Larcker DF. Evaluating structural equation models with unobservable variables and measurement error. J Retail Consum Serv 1981; 18: 39–50. [Google Scholar]
  • 77.Melián-González S, Gutiérrez-Taño D, Bulchand-Gidumal J. Predicting the intentions to use chatbots for travel and tourism. Curr Issues Tourism 2021; 24: 192–210. [Google Scholar]
  • 78.Cabrera-Sánchez JP, Villarejo-Ramos AF, Liebana-Cabanillas Fet al. et al. Identifying relevant segments of AI applications adopters – expanding the UTAUT2's Variables. Telemat Inform 2021; 58: 101529. [Google Scholar]

Articles from Digital Health are provided here courtesy of SAGE Publications

RESOURCES