Conversational artificial intelligence interventions to support smoking cessation: A systematic review and meta-analysis

Abstract

Background

Conversational artificial intelligence (chatbots and dialogue systems) is an emerging tool for tobacco cessation that has the potential to emulate personalised human support and increase engagement. We aimed to determine the effect of conversational artificial intelligence interventions with or without standard tobacco cessation interventions on tobacco cessation outcomes among adults who smoke, compared to no intervention, placebo intervention or an active comparator.

Methods

A comprehensive search of six databases was completed in June 2022. Eligible studies included randomised controlled trials published since 2005. The primary outcome was sustained tobacco abstinence, self-reported and/or biochemically validated, for at least 6 months. Secondary outcomes included point-prevalence abstinence and sustained abstinence of less than 6 months. Two authors independently extracted data on cessation outcomes and completed the risk of bias assessment. Random effects meta-analysis was conducted.

Results

From 819 studies, five randomised controlled trials met inclusion criteria (combined sample size n = 58,796). All studies differed in setting, methodology, intervention, participants and end-points. Interventions included chatbots embedded in multi- and single-component smartphone apps (n = 3), a social media-based (n = 1) chatbot, and an internet-based avatar (n = 1). Random effects meta-analysis of three studies found participants in the conversational artificial intelligence enhanced intervention were significantly more likely to quit smoking at 6-month follow-up compared to control group participants (RR = 1.29, 95% CI (1.13, 1.46), p < 0.001). Loss to follow up was generally high. Risk of bias was high overall.

Conclusion

We found limited but promising evidence on the effectiveness of conversational artificial intelligence interventions for tobacco cessation. Although all studies found benefits from conversational artificial intelligence interventions, results should be interpreted with caution due to high heterogeneity. Given the rapid evolution and potential of artificial intelligence interventions, further well-designed randomised controlled trials following standardised reporting guidelines are warranted in this emerging area.

Introduction

Tobacco use persists as a leading cause of premature death and contributor to the global burden of disease.^1,2 Prevalence of tobacco use is disproportionately high in low- and middle-income countries (>80%) compared to high-income countries, thus contributing to the widening of global health burden inequalities. ³ Acceptable, accessible, and effective smoking cessation support is vital given the significant short- and long-term individual health benefits. A multifaceted approach to smoking cessation is considered best practice due to the complex interaction of physiological and behavioural factors that influence tobacco dependence, ⁴ as well as the relapsing-remitting nature of the condition. ⁵ A well-established evidence base exists for the effectiveness of various pharmacotherapies to aid smoking cessation,^6–9 and their effect is increased when used in combination with behavioural support. ¹⁰ Yet these professional interventions, while effective, are limited in their reach^11,12 and accessibility,^13,14 and as such many people quit or attempt to quit unassisted despite the lower success of unassisted quitting.^15,16

Digital platforms have the capability to expand access to behavioural support for tobacco cessation. Current digital cessation interventions have advanced from computer-generated tailored letters, ¹⁷ to internet-based resources, ¹⁸ text-messaging programs, ¹⁹ and smartphone applications. ²⁰ However, these digital platforms in isolation are limited in their ability to provide highly personalised support. According to Mohr's ‘Model of Supportive Accountability’, engagement with digital health interventions is promoted with the addition of human support by fostering a sense of personal accountability to a competent, trustworthy, and caring coach. ²¹ To address this gap between professional tailored support and scalable digital interventions, innovations in artificial intelligence (AI) are being increasingly explored to emulate human support.

Conversational AI, typically referred to as a chatbot or dialogue system, enables two-way communication with users via text and/or audio without human input by employing natural language processing and machine learning algorithms. Modern conversational AI systems commonly use a combination of two approaches. A rule-based approach relies on rules defined and programmed by expert knowledge (e.g., a conversation tree) ²² which allows for greater control over the conversation pathway and AI-generated responses. In contrast, a probabilistic approach learns how to classify answers and potentially how to generate responses through exposure to training text. ²² Advances in so-called large language models have shown this technique scales with very large training text that gives impressive predictive performance. There are, however, frequent issues with erroneous text being produced that appear legitimate but entirely fictional. There is still considerable work required to determining the ethical and legal liabilities caused by these fictional responses. Regardless of the technique, through continual use, interactions with AI systems may become more tailored and fluid. ²³ Furthermore, the large amount of novel data generated through these systems has the potential to identify patterns in individual and population behaviours, which may enhance our understanding of smoking populations and improve the delivery and outcomes of highly personalised smoking cessation advice.

The use and effectiveness of conversational AI agents in health care is an emerging area of research. Specific health applications of this technology include but are not limited to, the delivery of cognitive behavioural therapy for mental health conditions ²⁴ and substance use disorders, ²⁵ supporting people with dementia, ²⁶ triage support, ²⁷ and screening for health conditions. ²⁸ A systematic review of 31 studies of AI conversational agents in health care found overall positive or mixed effectiveness, as well as high usability and satisfaction. ²⁹ The application of conversational AI for smoking cessation is still in its infancy but holds great potential in expanding tailored support. Benefits may include greater accessibility and may overcome some barriers to seeking counselling support due to fears about potentially negative and stigmatising interactions, which are often experienced by people who smoke, including when interacting with health professionals.^30,31 A 2022 scoping review reported some benefits of chatbots on smoking cessation but noted variable outcome measures and methodological issues. ³² More recently, a systematic review and meta-analysis of conversational agents for smoking cessation found positive effects on cessation outcomes across 6 studies as well as overall high acceptability. ³³ However, this analysis included automated synchronous text messaging systems which provided little detail regarding AI paradigms and described constraints on natural language input, and smoking cessation outcomes timepoints included in the meta-analysis varied. Additionally, the relationship between frequency of engagement and cessation outcomes (i.e., a dose-response) was not evaluated. Given the rapid development of conversational AI interventions and limited evidence syntheses, this systematic review and meta-analysis aimed to evaluate the effectiveness of conversational AI interventions and the relationship between level of engagement on tobacco cessation outcomes.

Methods

A study protocol was not registered due to an oversight in handover during project coordinator staffing changes but is attached as Supplemental File 1. This review was completed as part of a project for the World Health Organization (WHO) and the protocol was reviewed and approved by a WHO advisor.

Review questions

1. What effect do conversational AI interventions have on individual tobacco cessation outcomes compared to: (a) No intervention; (b) usual care; or (c) active comparator intervention/s when delivered alone or in combination?

2. What is the relationship between frequency and duration of interaction (i.e., dose) with conversational AI interventions and tobacco cessation outcomes?

Search strategy

A systematic search strategy was developed in collaboration between authors HB, HM and a specialist medical librarian. Databases searched on 3rd June 2022 included PubMed, Embase, CINAHL Complete, APA PsycINFO, CENTRAL, Web of Science. The search was supplemented by searches of four trial registries on 6th June 2022 (ClinicalTrials.gov, WHO ICTRP, EU Clinical Trials Register, and the Australian Clinical Trials Register), and by hand-searching reference lists of included papers and other relevant reviews. MeSH terms included ‘Tobacco Use Cessation’, ‘Smoking Cessation’, ‘Smoking/prevention and control’, ‘Smoking/therapy’, and ‘Artificial Intelligence’. Other AI search terms included ‘chatbot’, ‘conversation* agent’, ‘dialogue system’, ‘virtual agent’, ‘virtual assistant’, ‘machine learning’, ‘natural language’, ‘predictive analysis’, and ‘ELIZA effect’. The full search strategies are attached as Supplemental File 2.

Inclusion/exclusion criteria

Randomised controlled trials (RCTs) that tested the effectiveness of a conversational AI intervention as a smoking cessation aid (either alone or in combination with other behavioural or pharmacological smoking cessation treatments) among people aged 15 years and over who smoked combustible tobacco upon study enrolment (i.e., seeking and/or consented to receive smoking cessation treatment) were included. We included studies that involved a conversational AI tobacco cessation intervention compared with a control group, including no intervention, usual care, and active comparators. Active comparators could include but were not limited to, non-AI tobacco cessation smartphone or tablet applications, internet-based interventions, text-messaging-based interventions, pharmacological cessation aids, telephone counselling, behavioural counselling delivered face-to-face or through non-smartphone apps. Studies with more than one intervention arm/comparator were also included.

AI interventions could be delivered through a smartphone, mobile phone, tablet, computer or webpage, but needed to allow for bidirectional (conversational) communication. Automated messaging (AM) interventions were excluded if the underlying dialogue system had not been described in detail or did not, or was unlikely to have, employed AI algorithms or allowed for unconstrained natural language input.

Evidence selection

The search and retrieval process is illustrated in Figure 1. Search results were initially imported into Endnote referencing software where duplicates were removed. The updated library was then imported into Covidence where individual studies’ title and abstract were independently screened by one of two reviewers (HB & HM). Full-text screening was independently completed by HB, HM and SL, with any conflicts resolved through discussion between the reviewers or with another senior author (CG). Where required, original study investigators were contacted via email for further information regarding the type of AI system/algorithm to determine inclusion/exclusion. Final determination of AI systems was reached by consensus between an experienced computer scientist who specialises in AI chatbot development (DI) and all other co-authors.

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram.

Data extraction

Data were extracted independently and agreed upon by two reviewers (HB and HM) using a form created in Covidence software. Extracted data included data on trial design and characteristics, participant characteristics, intervention and comparator, outcome measures, loss to follow-up, narrative/qualitative findings (if applicable), and authors’ conclusions and declarations. Discrepancies unable to be resolved between the two reviewers were referred to a third reviewer (SL).

Risk of bias appraisal

Quality assessment of included individual studies was completed using Version 2 of the Cochrane risk of bias tool for randomised trials (RoB 2). ³⁴ The tool provides an overall risk of bias (low, some concerns, high) after assessment across five domains: (a) Bias arising from the randomisation process; (b) bias due to deviations from intended interventions; (c) bias due to missing outcome data; (d) bias in measurement of the outcome; and (e) bias in selection of the reported result. Appraisal of each study was completed by one study team member (HB) for review by HM, and determination was reached by consensus.

Data synthesis and meta-analysis

Results of included studies were summarised by narrative synthesis to describe the effectiveness of conversational AI interventions, and the relationship between frequency and duration of engagement on cessation outcomes. Random effects meta-analysis of intention-to-treat (ITT) results was conducted to estimate the effect of conversational AI interventions on smoking abstinence at 6-month follow-up. This outcome deviated slightly from our protocol outcome of continuous abstinence of at least 6-months due to differences between study designs, but consistency in follow-up timepoints. Three studies were deemed similar enough to be included in this analysis. Abstinence rates were collected and reported as risk ratios. To test if the estimates were sensitive to the inclusion of any single study, leave-1-out analysis was conducted. To test the potential impact of publication bias, trim-and-fill analysis was conducted. All analyses were conducted in R with the metafor package.

Results

Among the 819 unique records identified through database searches (n = 813) and snowballing (n = 6), five articles were included in the analysis (Figure 1). Two articles^35,36 were included after full-text review, and three additional articles were identified from a recent scoping review,^37,38 and from a trial protocol ³⁹ found in the initial search. Extracted data are summarised in Table 1 and Supplemental File 3. Study investigators of one included paper were contacted via email for further information regarding RCT details, however, we received no response and could therefore only analyse the limited published information.

Table 1.

Study characteristics and findings.

Author, year country	Study design and size	Participant characteristics	Intervention	Comparator	Outcome data
Perski 2019 Online (no country/ies specified)	Parallel RCT (automated) Allocated Intervention: 5339 Control: 51 875 LTFU Intervention: 4278 (80.1%) Control: 46825 (90.2%)	Smoke daily or non-daily CPD, mean (SD) Intervention: 16 (11.4) Control: 14.7 (8.9) Time to first cigarette <5mins Intervention: 17.2% Control: 19.3% Setting Online Recruitment Via commercial App Store. People who purchased the “Pro” version of the Smoke Free app.	Smoke Free Pro app with chatbot Intervention Dose 2x daily check in for 4 weeks AI system details AI-driven chatbot	Smoke Free Pro app only	Self-reported continuous abstinence at 4-week follow-up Intervention: 844/5339 (15.8%) Control: 3704/51,875 (7.1%) OR 2.44 (2.25–2.64, p < .001) Frequency of engagement with app, median (IQR) Intervention: 16 (65.5) Control: 5 (22) IRR 2.07 (1.97–2.17, p < .001)
Olano-Espinosa 2022 Spain	Pragmatic, multicenter RCT Allocated Intervention: 242 Control: 271 LTFU Intervention: 138 (57.0%) Control: 143 (52.8%)	Smoke consecutively (≥1 cigarette in past month) Age, mean (SD) Intervention: 49.01 (11.22) Control: 50.66 (10.42) Female Intervention: 146 (60.3%) Control: 158 (58.3%) HSI, mean (SD) Intervention: 2.71 (1.59) Control: 2.65 (1.68) Setting Online Recruitment Via Family practitioners and nurses.	Chatbot delivered intervention (based on 5As: behavioural ± pharmacological treatment) Intervention Dose Daily interactive contacts, but patient could contact chatbot at any time and decide on frequency and duration of contact. AI system details Probabilistic expert system based on Bayes’ theorem	Standard face-to-face intervention (based on 5As: behavioural ± pharmacological treatment)	Biochemically validated continuous abstinence at 6 months Intervention: 63/242 (26.0%) Control: 51/271 (18.8%) OR 1.50, 95% CI 1.00–2.31; P = .05. 6 month abstinence: Intensive contact (>4 contacts lasting >30 min throughout 6 month period) vs non-intensive Intervention Intensive: 68.6% Non-intensive: 40.9% P = .02 Control Intensive: 47.6% Non-intensive: 35.4% P = .30
Karekla 2020 Cyprus	Pilot Parallel RCT Allocated Intervention: 49 Control: 35 LTFU Intervention: 22 (44.8%) Control: 7 (20%)	Smoke daily Age, mean (years) Intervention: 22.5 Control group: 22 Female Intervention: 64% Control: 65% FTND, mean (SD) Intervention: 3.19 (2.09) Control: 2.91 (2.02) Setting Three universities Recruitment Flyers posted in university cafeterias and classroom announcements.	Internet-based, digital avatar led ACT program. Intervention Dose 6 ACT smoking cessation sessions (∼25 min each). Frequency of contact spaced out over 3–30 days (I.e. time to complete the entire program and post-assessment varied). AI system details Rule-based. Different continuation of dialogue based on user response.	Wait-list control group (questionnaires only)	Self-reported 7-day PPA at post-treatment time-point (after 6 sessions completed) Intervention: 14/49 (28.57%) Control: 4/35 (11.43%) OR 3.10, 95% CI 0.92–10.41, p = 0.067; Mean treatment completion: 44 days
Wang 2018 Hong Kong	Parallel RCT Allocated Intervention: 205 Control: 196 LTFU Intervention: 71 (34.6%) Control: 43 (21.9%)	Past 7-day smoking Age, mean (years) Intervention: 32.8 Control: 33.1 Female Intervention: 78 (39.8%) Control: 84 (41%) Nicotine dependence Not reported Setting Not reported Recruitment Not reported	WeChat groups with conversational agent. AI system details Rule based, context aware dialogue management with natural language processing module.	Smoking cessation information and tips without social support or interactions with other participants	Verified abstinence at 6-month post-intervention follow-up a Intervention: 27/205 (13.2%) Control: 19/196 (9.7%) Participation in conversations vs smoking outcomes Mean number of WeChat posts amongst bioverified quitters was 22.8 compared to 9.1 for participants who did not have bioverified quitting. Participants who had not smoked in the past week were involved in a significantly higher number of conversations than participants who had smoked (p = 0.02).
Masaki 2020 Japan	Parallel, multi-center, randomised, controlled, open-label trial. Allocated Intervention: 285 Control: 287 LTFU Intervention: 40 (14%) Control: 42 (14.6%)	Currently smoke (smoking history of pack-years ≥10) Age, mean (SD) Intervention:47 (11) Control: 45 (12) Female Intervention: 69 (24.2%) Control: 77 (36.7%) FTND, mean (SD) Intervention: 5.2 (2.0) Control: 5.3 (2.1) Setting 31 smoking cessation clinics Recruitment Via physicians.	CASC system app (24 weeks) +12 week standard smoking cessation treatment (concurrent) The CASC system included interactive counselling with a personalised chatbot and mobile CO checker. Standard smoking cessation treatment: 5 on-site examinations and in-person counselling by a primary physician at each outpatient clinic within 3 months. AI system details Personalised automated guidance system	Control app (24 weeks) +12 week standard smoking cessation treatment (concurrent). Control app provided basic functions of CASC app, not including a chatbot and not accompanied by a mobile CO checker	Biochemically verified continuous abstinence from weeks 9–24 Intervention: 182/285 (63.9%) Control: 145/287 (50.5%) OR 1.73, 95% CI 1.24–2.42, p = 0.001

RCT: randomised controlled trial; LTFU: loss to follow-up; CI: confidence Interval; SD: standard deviation; OR: odds ratio; IRR: incidence rate ratio; HSI: heaviness of smoking index; IQR: interquartile range; CPD: cigarettes per day; FTND: Fagerström test for nicotine dependence; ACT: acceptance and commitment therapy; PPA: point prevalence abstinence; CASC; CureApp smoking cessation; AI: artificial intelligence.

^a Metric and measurement method not reported. Intention to treat results calculated by reviewers.

Participant and study characteristics

Five RCTs conducted in Spain, ³⁹ Cyprus, ³⁷ Hong Kong, ³⁶ Japan ³⁸ and online, ³⁵ were published between 2018 and 2022 (Table 1). The total combined randomised sample size across the studies was 58,796 participants (range 84–57,214). The minimum age of inclusion for all studies was 18 years. Mean ages of participants and percentage of females ranged from 22 years to 50.7 years, and 24% to 65%, respectively. Participant loss to follow-up ranged from 14% to 80.1% for intervention groups, and 12.2% to 90.2% for comparison groups. Baseline measures of tobacco dependence were reported in four studies and included cigarettes per day (CPD),^35,37–39 heaviness of smoking index (HSI), ³⁹ Fagerstrom Test for Nicotine Dependence (FTND),^37,38 and time to first cigarette. ³⁵ Mean levels of nicotine dependence ranged from low to moderate across studies reporting FTND (range 2.91–5.3) and HSI (range 2.65–2.71) measures. Significant differences in baseline CPD and time to first cigarette were reported in one study but adjusted for in sensitivity analyses. ³⁵

Conversational AI interventions included chatbots embedded in a multicomponent smoking cessation app,^35,38 a chatbot smoking cessation app, ³⁹ a conversational agent embedded in social media groups, ³⁶ and an internet-based avatar. ³⁷ Three studies compared conversational AI interventions to standard/usual care including in-person visits, ³⁹ questionnaires, ³⁷ and smoking cessation information without social/professional support. ³⁶ Two studies used a control/standard version of the intervention app without the chatbot component.^35,38 Standard smoking cessation treatment (behavioural and pharmacological) was delivered to both intervention and comparator groups in two studies,^38,39 with the behavioural component delivered by chatbot in the intervention group in one study. ³⁹

Smoking cessation outcomes were biochemically verified in two studies using exhaled carbon monoxide test (threshold <10 ppm),^38,39 self-reported in two studies,^35,37 and one study was unclear as to how “verified abstinence” was ascertained. ³⁶ Primary smoking cessation outcomes varied, including self-reported continuous abstinence at 4-week follow-up, ³⁵ self-reported 7-day point-prevalence abstinence (PPA) at the posttreatment timepoint, ³⁷ biochemically verified continuous abstinence from weeks 9–24, ³⁸ and biochemically verified continuous abstinence at 6 months. ³⁹ The metric of measuring abstinence at 6 months was not described in the remaining study. ³⁶ Secondary smoking cessation outcomes included weekly ³⁶ 7-day PPA and at 4 weeks, 8 weeks, 12 weeks, ³⁸ 6 months, ³⁷ 12 months, ³⁸ and biochemically verified continuous abstinence from weeks 9 to 52 and weeks 9 to 12. ³⁸ Additional outcomes included changes in quality of life, ³⁹ cigarettes smoked per day, ³⁷ nicotine dependence, ³⁷ intention to quit smoking, ³⁷ self-efficacy, ³⁷ withdrawal symptoms, cravings, and misperceptions of smoking, ³⁸ and time to first lapse after the quit date. ³⁸

Effectiveness as a smoking cessation aid

Study results are presented in Table 1. While Wang et al. ³⁶ reported per-protocol results only, the data reported in the paper also allowed for an ITT analysis. Of the studies evaluating chatbot smoking cessation apps, only one compared the intervention to face-to-face support. Olano-Espinosa et al. ³⁹ found that participants who received behavioural treatment through a chatbot app with pharmacotherapy were significantly more likely to remain abstinent at 6 months compared to those who received standard behavioural and pharmacological treatment from a doctor or nurse (26.0% vs 18.8%; OR 1.50, 95% CI 1.00–2.31, P = 0.05). This result lost statistical significance, but the estimate remained relatively unchanged after adjustment for baseline CO-oximetry and bupropion intake (OR 1.52, 95% CI 0.99–2.33, p = 0.53), and 6-month abstinence was correlated with receiving the chatbot intervention (OR 1.52, 95% CI 0.99–2.33; p = 0.053) and bupropion prescription (OR 2.81, 95% CI 1.49–5.32; p = 0.001). ³⁹ Similar results were reported by Masaki et al. when comparing the combination of standard in-person counselling alongside either a multicomponent app (CASC) which includes a personalised chatbot, or a control app with basic functions. ³⁸ Those who received the CASC intervention were significantly more likely to remain abstinent from weeks 9 to 24 of the trial (63.9% vs 50.5%; OR 1.73 95% CI 1.24–2.42, p < 0.001) and achieve 7-day PPA at week 24 (72.3% vs 58.2%; OR 1.88, 95% CI 1.33–2.68, p < 0.001), with results remaining significant at week 52. ³⁸

Perski et al. compared two different versions of the same smoking cessation app, with the intervention version containing a proactive and on-demand chatbot. ³⁵ When restricting analysis to participants with an elected quit date window within two days before or 14 days after downloading the app (not a pre-specified primary outcome), participants were significantly more likely to report being abstinent after 4 weeks (15.8% vs 7.1%; OR 2.44 (2.25–2.64, p = <0.001). ³⁵ When the analysis was broadened to include participants without a quit date or a quit date outside this window, the ITT analysis outcomes remained positive and significant (ORadj 1.60, 95%CI 1.51–1.69, p < 0.001) but per-protocol results were null (ORadj 1.02, 95%CI 0.92–1.13, p < 0.71). ³⁵

Karekla et al. ³⁷ investigated the effect of an internet-based digital avatar among young adults (university students). Students were randomised to the Avatar-delivered Acceptance and Commitment Therapy (ACT) behavioural support program or to the waitlist-control group who received questionnaires only. ³⁷ Those in the intervention group were more likely to report 7-day PPA at the post-treatment timepoint, although this was not significant (28.6% vs 11.4%; OR 3.10, 95% CI 0.92–10.41, p = 0.067) which was likely due to the small sample size. ³⁷ In the per-protocol analysis the intervention group were 6 times more likely to report abstinence and this was statistically significant (51.9% vs 14.3%; OR 6.46, 95% CI 1.76–23.71, p = 0.005). ³⁷

Finally, Wang et al. ³⁶ briefly reported findings comparing a conversational agent embedded in WeChat Groups to smoking cessation information/tips only without social/professional support. The authors stated that details on the RCT were deliberately withheld. In the reported per protocol analysis, a higher proportion of those in the intervention group was verified abstinent at 6 month follow-up (27/134, 20.1%) compared to the control group (19/153, 12.4%), however, this difference decreased when using an ITT approach (27/205, 13.2% vs 19/196, 9.7%). ³⁶ Effect estimates were not provided for this outcome.

Meta-analysis of smoking cessation outcomes

The random effects meta-analysis (Figure 2) of abstinence outcomes at 6-month follow-up (n = 3) found that participants in the conversational AI intervention were significantly more likely to quit smoking compared to participants in control conditions (RR = 1.29, 95% CI (1.13, 1.46), p < 0.001). Overall, there was no significant heterogeneity (Q(2) = 0.29, p = 0.867, I² = 0%), yet this should be interpreted with caution given the small number of studies. The test of funnel plot symmetry did not provide any evidence of asymmetry (p = 0.646) (Supplemental File 4). Trim-and-fill suggested that there was one potential study missing due to publication bias. Imputing effect size from this potential missing study did not substantially change the effect size (RR = 1.26, 95% (1.12, 1.42), p < .001). Leave-1-out sensitivity analysis demonstrated that the estimated effect size may be sensitive to excluding a study. Excluding Masaki, et al. increased the effect size slightly (RR = 1.38, 95% CI (1.04, 1.82)). All effect sizes remain highly statistically significant (p ranged from <0.001 to 0.026)

Figure 2.

Forest plot of 6-month follow-up abstinence outcomes.

Frequency and duration of engagement

Outcomes related to frequency of engagement or usage rate were reported in four studies,^35,36,38,39 Masaki et al. ³⁸ did not specifically report on interactions with the chatbot component within a multicomponent app, but found that daily use of app functions (e.g., chatbot, diary, video tutorials) was higher among participants who had quit compared to those who had not. Olano-Espinosa et al. ³⁹ reported significantly higher abstinence rates among those in the intervention group who interacted with the chatbot intensively (>4 contacts lasting >30 min throughout the 6-month period) compared to those who did not (68.6% vs 40.9%, p = 0.02). A higher quit rate was also observed in the control group among those who received intensive contact with health professionals compared with those who received less intense support, but the effect was not statistically significant (47.6% vs 35.4% respectively, p = 0.30). ³⁹ Wang et al. ³⁶ also noted a significant relationship between greater participation in conversational agent enhanced social-media conversations and smoking abstinence (p = 0.02). Perski et al. ³⁵ found the frequency of engagement with the smoking cessation app was significantly higher for the chatbot version compared to the standard app version, with and without a quit date criterion applied. However, this result was not analysed in relation to smoking cessation outcomes.

Adverse events

One study reported findings regarding treatment emergent adverse events (AEs). AEs were reported by 64.7% and 63.8% of intervention and control group participants, respectively, and related to the study pharmacotherapy or nicotine withdrawal. ³⁹ No specific AEs were related to the intervention app. ³⁹

Quality of the evidence

Overall, the risk of bias was high across the studies (Figure 3). As mentioned previously, one study purposively withheld RCT methodological details ³⁶ and was therefore deemed a high risk of bias across all domains. For the remaining studies, randomisation was adequate in three studies,^37–39 while some concerns were raised in the remaining study due to baseline differences between groups because of errors in randomisation coding. ³⁵ Low risk of bias was consistent across the four studies for deviations from intended interventions and measurement of outcome data.^35,37–39 Two of the four studies were deemed high risk of bias due to missing outcome data as there was no evidence of appropriate sensitivity analyses ³⁷ or statistical significance was lost when assessing results on a complete-case basis. ³⁹ Finally, high risk of bias was found in two studies for selection of the reported result due to inconsistencies in outcome measurements between trial registry information and published results, ³⁷ and, in the second, for multiple analyses allowing for selection of a significant result for a criteria defined outcome not outlined in the research questions. ³⁵

Figure 3.

Risk of bias (RoB2 tool) quality assessment.

Discussion

This systematic review and meta-analysis sought to evaluate the effect of conversational AI interventions on tobacco cessation outcomes, and the relationship between frequency of engagement with these interventions on these outcomes. Our analysis of three studies demonstrated significant positive effects of conversational AI interventions for smoking cessation, however, these results should be interpreted with caution due to the limited number of studies and slight variation in abstinence outcomes. These findings are consistent with a previous systematic review of conversational agents for smoking cessation (n = 6) which reported favourable abstinence outcomes compared to control groups (OR 1.66, 95% CI = 1.33–2.07, p < 0.001), but also highlighted variation in study design and quality. ³³ Generalisability of results may also be limited given that available measures of nicotine dependence ranged from low to moderate across included studies, warranting further research in cohorts with high levels of nicotine dependence. Additionally, there is limited but promising evidence supporting the ability of conversational AI to increase engagement with mobile health (mHealth) interventions, including a ‘dose-response’ relationship which warrants further comprehensive investigation.

Loss to follow-up (LTFU) was considerable in most studies,^35–37,39 and greater than 20% in all but one included study. LTFU was markedly higher when an automated RCT approach was used ³⁵ compared to trials that involved some level of contact with the study team or clinicians.^38,39 In contrast to the included mHealth studies, a Cochrane review of 52 studies of combined pharmacotherapy and behavioural interventions found most trials (n = 48) reported a LTFU rate of less than 20%. ⁴⁰ Maintaining participant engagement and motivation is often a challenge for automated mHealth trials, especially for longer duration trials ⁴¹ which are required for appropriate evaluation of the effectiveness of tobacco cessation interventions. Regular or semi-regular contact with study team members or clinicians through intervention delivery or follow-up (i.e., a hybrid approach ⁴¹ ) as implemented by Masaki et al. ³⁸ and Olano-Espinosa et al. ³⁹ may help to improve retention rates within mHealth trials. However, for trials of this nature, the advantage of high retention rates must be weighed against the advantages of large and generalisable samples with minimal human resource requirements achievable through automated trials, where appropriate sensitivity analyses can account for high LTFU rates.

A key challenge in determining inclusion of studies in this review was defining “artificial intelligence”. The definition of AI is broad, often subjective, and differs between experts in the field to such a degree that there is no agreed definition of AI.^42,43 Our selected studies slightly contrasted with a recent systematic review that assessed the effectiveness of conversational agents for smoking cessation ³³ as we excluded AM systems. While AM systems can be incorporated into conversational AI systems, AM itself does not inherently require or employ AI. We only evaluated bidirectional conversational AI systems, as opposed to solely AM, through consultation and consensus with a senior computer scientist; an approach to eligibility screening which has been employed in previous systematic reviews of AI in health care. ⁴⁴ Key terms beyond AI, such as “chatbot”, “rule-based”, “natural language processing”, “probabilistic”, were assumed to be relevant for this review when considered alongside the intervention context, platform and description. Additionally, the true conversational ability of AM systems was assessed and we excluded those that were limited to only one-word utterances by users. Nevertheless, the lack of a clear, objective definition of AI may pose problems in future reviews of efficacy as this technology for smoking cessation further advances.

We observed a difference in how AI intervention trials were reported in health- versus computer science-based journals. Studies published in health-based journals often lacked technical information regarding the AI intervention,^35,37,38 whereas the study published in a computer science journal lacked detailed trial information and health outcome data necessary to evaluate efficacy and quality. ³⁶ While this may be indicative of differing aims and priorities between the fields in terms of research communication, standardised reporting of trials is critical to understanding the potential and efficacy of digital health interventions. A lack of standardised reporting was also noted in a previous systematic review of conversational agents for smoking cessation. ³³ The CONSORT-AI extension was developed via expert and stakeholder consultation in 2019, to provide minimum reporting guidelines for trials evaluating interventions with an AI component. ⁴⁵ It adds 14 unique items to the existing CONSORT 2010 reporting guidelines ⁴⁶ including, but not limited to, clear descriptions of the AI intervention, how the intervention handles inputs and outputs, and the interaction between humans and AI. ⁴⁵ Future trials of AI enhanced smoking cessation interventions, and AI health interventions more broadly, should ensure these guidelines are considered during the planning phase, and followed when reporting results to strengthen future evidence syntheses and individual quality assessments.

Trials of digital health interventions are also not immune from perceived or actual conflicts of interest. Author declarations from three included studies stated connections to the parent company (with or without financial interest) or intellectual property ownership rights to the intervention.^35,38,39 While the former is well established as a common conflict of interest (COI), previous qualitative research of trial investigators found considerable variability in what was considered and reported as a COI, and highlighted the difficulty in detecting and reporting intellectual COIs. ⁴⁷ Trial researchers who have been involved in the design and development of novel or established digital interventions should declare their intellectual COIs given the potential for bias. Furthermore, independent design and evaluation of digital health trials must be ensured and managed in the same way as pharmaceutical trials. While there was some evidence of strategies to improve impartiality in data analysis by using researchers without any IP rights to the intervention, ³⁹ the majority of the included studies did not describe management strategies to minimise the influence of potential COIs.

Finally, while early trial results are encouraging, the value of AI-enhanced interventions for health behaviour change must be balanced by acknowledging and addressing potential limitations. Important ethical and legal considerations of AI include: Trust and transparency; data use and privacy; patient safety; bias and health equity; cybersecurity; research and development; governance, testing and evaluation; scope; content decisions; licensing; and third-party involvement. ⁴⁸ Conversational AI may prove to be an important tool in overcoming equity issues by providing scalable personalised smoking cessation support in which individual access/engagement is not limited by felt stigma. However, we must be cognizant of the risk of digital health interventions increasing disparities between those who do and do not have the skills and access to them, including geographic location, given that many social determinants of health (e.g., healthcare, education, community and social support) rely on digital access and literacies. ⁴⁹ Given the disparities in smoking prevalence between demographic groups and variation in tobacco control legislation and smoking cessation services between countries, digital inclusion of priority populations on a national and international scale is vital for the success of AI interventions for smoking cessation. While we cannot surmise this potential impact from the current data, given that smoking cessation is multifaceted, we foresee the role of conversational AI is to complement current standards of care (i.e., pharmacotherapy and behavioural support) not to replace them entirely. Therefore, a lack of smoking cessation services, or health services more broadly, due to geographical location could influence the uptake and/or effectiveness of this technology. Future programs of research should seek to promote digital inclusion by; understanding user needs and ensuring the intervention is of benefit, low cost and importance to users; building trust, and ensuring transparency and security of personal data use and storage; multidisciplinary collaboration; and evaluating wider outcomes. ⁵⁰

Conclusion

Conversational AI for tobacco cessation is a rapidly evolving field of research. At present, there is limited but promising evidence supporting the use of conversational AI to assist tobacco cessation. The ability of the technology to increase engagement with an intervention may be a key driver for positive cessation outcomes. A high level of heterogeneity between all studies was identified. Future trials should seek to employ more standardised methods and measures to RCT designs, alongside standardised reporting guidelines, to improve our ability to assess conversational AI intervention effectiveness, study quality and the AI systems used.