Effectiveness of chatbot-based interventions on mental well-being of the general population in Asia: protocol for a systematic review and meta-analysis of randomised controlled trials

Abstract

Introduction

In Asian countries, stigma against psychiatric disorders and shortage of manpower are the two major challenges that hinder people from receiving treatments. Chatbots can surely help people surpass the stigmatising and manpower shortage challenges. Since a comprehensive review in the Asian context is lacking, this systematic review will evaluate the effects of chatbot interventions on the mental well-being of the general population in Asia.

Methods and analysis

Four electronic databases (PubMed, CINAHL, PsycINFO and MEDLINE) will be searched until December 2024. Randomised controlled trials with English/Chinese full text available will be included. Random-effect models will be used for meta-analyses. The risk of bias (RoB) and certainty of evidence across studies will be assessed using the Cochrane RoB2 and Grading of Recommendation Assessment, Development and Evaluation tools, respectively.

Ethics and dissemination

This study will not require ethical approval. The findings will be disseminated through peer-reviewed publications.

Funding

School Research Grant of the Tung Wah College (2023-04-52-SRG230401)

PROSPERO registration number

CRD42024546316.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• It was estimated that 1 in every 8 people worldwide, equating to 970 million people, were living with a mental disorder.

• Conventional mental health services usually have limited accessibility (eg, limited opening hours) and manpower shortage (eg, 0.1–90 psychiatrists per 1 000 000 people).

• In line with the WHO Special Initiative for Mental Health (2019–2023) aimed at increasing access to quality and affordable care for mental health conditions, digital health due to its ubiquity and wide applicability can be considered an adjuvant treatment option in favour of access to quality mental health services.

• Chatbot-based interventions have broad applicability (eg, health education, panic management and physical activity promotion) across multiple settings and populations.

• Chatbot-related mental services are relatively flexible, readily accessible and 24/7 available compared with traditional mental health services.

WHAT THIS STUDY ADDS

• This systematic review will provide evidence about the effects of varying chatbot interventions on mental health outcomes among the general population in Asia.

• This study will examine the strength of the causal association between chatbot interventions and mental health outcomes among the general population in Asia.

• This study will provide evidence about the feasibility of chatbot interventions in mental health services in respect of recruitment rates, completion rates, safety and adverse effects.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• Apart from conventional mental health services, mental health chatbots can provide patients with more therapeutic options for mental illnesses.

• Given that stigma against psychiatric illnesses (eg, dangerous, aggressive and weak) is very common and serious in Asian societies, advocating the use of chatbots in mental health services can help Asian people surpass the stigmatising challenges because they are more receptive and open to talk with a humanised machine than a real person.

• An improved understanding of the chatbots’ effectiveness within the context of mental health could surely better shed light on their adjunctive roles to already available interventions in healthcare and more importantly, enhance the comprehensiveness of services by offering more treatment options that are highly acceptable and easily accessible to people having mental health service needs.

Introduction

A chatbot, also known as a virtual conversational agent, is a computational system that simulates real-life human conversations.¹ Rule-based chatbots adopt a set of predefined rules to generate responses whereas artificial intelligence (AI)-powered chatbots employ more advanced algorithms (eg, machine learning) to allow deeper, adaptable communications by comprehending, generating and manipulating human language as it is written or spoken. During the conversations, people can interact with the agent using spoken and written language and the agent can respond verbally and non-verbally via spoken, written and visual (eg, body movements and facial expressions) language.² Recent systematic reviews showed that chatbots were a feasible, effective and safe adjunct modality for improving mental health (eg, depression, anxiety, distress, stress, acrophobia, subjective psychological well-being and affect).2,4 In particular, chatbot interventions were clinically beneficial for alleviating depression (95% CI: −0.87 to −0.23).²

Since only a paucity of studies (n=2 to 4) were included for meta-analysis² and many reviews even did not quantitatively evaluate the effects of chatbot-based interventions on mental health outcomes,⁴ a comprehensive review involving more updated evidence is required to uncover the potential efficacy of chatbots on individuals who may have multifaceted needs (eg, emotional support) that are unresolved by conventional therapies. Moreover, previous reviews fell short in addressing the effects of different cultural backgrounds across countries on people’s chatbot adoption behaviour given that culture is a strong predictor for the way people choose to cope with everyday problems and adversities.^{5 6} Importantly, stigma against psychiatric illnesses (eg, dangerous, aggressive and weak) is very common and more serious in Asian societies.⁷ For instance, patients’ family members can be blamed for poor family lineage and hence their marriage and economic prospects will be adversely affected. Advocating the use of chatbots in mental health services not only provides patients with more therapeutic options but also helps them surpass stigmatising challenges because they are more receptive to share sensitive information with a humanised machine than a real person.⁴

This systematic review aims at systematically examining the effects of chatbot interventions on the mental well-being of the general population in Asia. Evidence about the feasibility of chatbot interventions in respect of recruitment, completion and safety will also be synthesised.

Methods and analysis

Reporting of the review findings will be in line with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 updated guidance and exemplars for reporting systematic reviews.⁸ The protocol was registered in the PROSPERO registry. One change has been made to the project title from ‘artificial intelligence-driven conversational agents’ to ‘chatbot-based interventions’ which generally consider all kinds of chatbots to ensure a more comprehensive analysis of their mental health impacts by comparing different chatbot-based interventions. However, all other processes including the eligibility criteria for study inclusion, data extraction and synthesis and reporting bias assessment remain unchanged.

Eligibility criteria

The Population, Intervention, Comparison, Outcomes, Study Design framework will be employed to guide the inclusion of eligible studies.⁸ The eligibility criteria are shown as follows:

• This systematic review will only include randomised controlled trials (RCTs) among Asian people of any age whom both healthy people and people at risk of mental health conditions (eg, socially disadvantaged groups including lesbian, gay, bisexual, transgender or people having chronic conditions/substance misuse) will be considered. If the study populations consist of both Asian and non-Asian people, only data derived from the Asian participants will be extracted for data analyses. Parallel, cross-over and cluster RCTs will be considered. For cross-over RCTs, only data before the cross-over period will be used for data analyses. Only RCTs with English/Chinese full text available will be included. Studies such as conference abstracts/proceedings and case studies on one or a few subjects or other than original research including grey literature, reviews, meta-analyses, study protocols, editorials and commentaries or retracted articles will be excluded.

• Standalone chatbot-related interventions using either mobile application-based or online web-based platforms have to be in at least one of the interventional arms within the trials. Studies involving chatbot interventions combined with other types of mental health services in one single experimental arm will be excluded because the therapeutic effects of the chatbot intervention may be caused by other interventions. Meanwhile, chatbots controlled by human operators will not be considered.

• The effects of the chatbot-based interventions will be compared with either no interventions including ‘usual care’, ‘no treatment’ and ‘wait-list control’ or compared with other interventions.

• The RCTs aimed at examining the effects of AI-driven chatbots on improving mental health outcomes including depression, anxiety, stress, affect, distress, sleep and subjective psychological well-being^{9 10}.

Information sources

Studies published in the databases of PubMed, CINAHL, PsycINFO and MEDLINE will be searched from their inception through December 2024. The search queries for every electronic database are summarised in online supplemental information. The studies that were conducted in Asian countries/regions will be manually identified.

Search strategy

The initial screening (titles and abstracts) and full-text assessment of the searched studies will be conducted by the first author (WL) and subsequently verified by an independent reviewer (SCL). Since subject recruitment can be conducted through internet-based methods (eg, social media) which can target populations of different countries,¹¹ identification of whether the studies were conducted in Asian populations will be carried out through the full-text assessment (except relevant information explicitly specified in the abstracts or titles). To avoid missing any potentially eligible studies, the reference lists of the searched relevant systematic reviews2,412 and all the included articles will also be screened for additional potential studies. Disagreements between the two reviewers (WL and SCL) will be resolved by discussion involving all review authors. The proposed PRISMA 2020 flow diagram for the study selection process is presented in online supplemental figure S1.

Data extraction

Information including publication details (authors, publication year and countries/regions), participant characteristics, intervention and control groups, training volume, recruitment and completion rates, safety and adverse effects and main findings will be extracted and summarised in an evidence table by the first author. The recruitment rate will be defined as the percentage of participants giving consent after being assessed as eligible while the completion rate will be defined as the percentage of participants remaining in the study until the last assessment visits.⁴ To guarantee the robustness of evidence, all authors will verify and edit every entry. The template data collection form is shown in online supplemental table S1.

Risk of bias assessment

The Cochrane Risk of Bias 2 (RoB 2) will be employed to examine the RoB of the included RCTs.²¹ The tool is structured into five domains covering all possible biases that may potentially affect the results of the included trials, namely bias arising from the randomisation process, bias due to deviations from intended interventions, bias due to missing outcome data, bias in the measurement of the outcome and bias in the selection of the reported result. An overall RoB judgement for every study (low risk, some concerns, high risk) will be drawn based on the results of each bias domain. The RoB assessments will be first conducted by the first author and independently verified by another review author. Disagreements between the two reviewers will be resolved through discussion.

Certainty of evidence assessment

The Grading of Recommendation Assessment, Development and Evaluation approach will be employed to examine the certainty of evidence (CoE) across included studies for clinical endpoints (eg, depression and anxiety) that were most reported. Evidence about the clinical endpoints will be downgraded one place if (1) the RoB is high (ie, most included trials have some concerns or high RoB), (2) there is evidence of imprecision (ie, 95% CI fell within the minimal clinically important difference (MCID) or the effect estimate is derived from only one or two small studies), (3) there is evidence of unexplained inconsistency or high heterogeneity (ie, I² >50%), (4) the directness of evidence is questionable in respect of the participants studied, intervention of interest, measurements of the outcome (eg, surrogate vs intended markers) and the method of the trial of a candidate intervention (ie, depending on whether direct comparisons are available or not), (5) publication bias is detected if asymmetry of the funnel plot is observed. Overall, the CoE for the clinical endpoints will be classified as very low, low, moderate and high. The first author will assess the CoE which will hence be verified by an independent reviewer. Disagreements between the two reviewers will be resolved through discussion.

Data synthesis

Meta-analysis will be conducted for mental health outcomes (eg, depression and anxiety)⁹ that were reported in at least two included studies.^{2 4} For controlled trials, pairwise meta-analyses of postintervention data will be performed using random-effects models.²² The overall effect will be examined using either standardised mean difference or weighted mean difference depending on whether the outcomes were assessed using different scales or the same scale, respectively. Subgroup analyses will also be conducted for potential moderators (eg, age, sex, ethnicity, medical history, types of chatbots and comparison controls). Boundaries of an MCID for the outcomes will be set as half of a SD from the mean value in the control arm(s) of the included studies at baseline.²

The degree of heterogeneity across study findings will be assessed using the Higgins I² statistics. Percentages of the I² statistics in 0–25%, 25–50% and>50% represent low, medium and high heterogeneity, respectively. For the meta-analyses that will be identified as having high heterogeneity, sensitivity analyses will be carried out by excluding studies with a high RoB. If there is missing information, we will contact the study authors directly via email. All meta-analyses in the present proposed study will be performed using Review Manager (RevMan V.5.4) software.

Assessment of publication bias

Funnel plots will be constructed to visually examine the bias if there are 10 or more trials included in the meta-analyses or subgroup analyses.⁸ Asymmetry of the funnel plots indicates possible publication bias.

Ethical and dissemination

No formal ethical approval will be required because the present review will only consider published articles and all data available among included studies should be anonymous with no concerns about participant privacy or confidentiality. The findings of this systematic review will be disseminated through peer-reviewed publications.