Development and Usability Testing of a Chatbot to Promote Mental Health Services Use Among Individuals with Eating Disorders Following Screening

Abstract

Objective:

A significant gap exists between those who need and those who receive care for eating disorders (EDs). Novel solutions are needed to encourage service use and address treatment barriers. This study developed and evaluated the usability of a chatbot designed for pairing with online ED screening. The tool aimed to promote mental health service utilization by improving motivation for treatment and self-efficacy among individuals with EDs.

Method:

A chatbot prototype, Alex, was designed using decision trees and theoretically-informed components: psychoeducation, motivational interviewing, personalized recommendations, and repeated administration. Usability testing was conducted over four iterative cycles, with user feedback informing refinements to the next iteration. Post-testing, participants (N=21) completed the System Usability Scale (SUS), the Usefulness, Satisfaction, and Ease of Use Questionnaire (USE), and a semi-structured interview.

Results:

Interview feedback detailed chatbot aspects participants enjoyed and aspects necessitating improvement. Feedback converged on four themes: user experience, chatbot qualities, chatbot content, and ease of use. Following refinements, users described Alex as humanlike, supportive, and encouraging. Content was perceived as novel and personally relevant. USE scores across domains were generally above average (~5 out of 7), and SUS scores indicated “good” to “excellent” usability across cycles, with the final iteration receiving the highest average score.

Discussion:

Overall, participants generally reflected positively on interactions with Alex, including the initial version. Refinements between cycles further improved user experiences. This study provides preliminary evidence of the feasibility and acceptance of a chatbot designed to promote motivation for and use of services among individuals with EDs.

Introduction

Eating disorders (EDs) are complex mental illnesses characterized by debilitating, pervasive psychological and physiological consequences when left untreated. Given that lack of or delay in treatment initiation results in poorer prognosis and higher relapse risk (American Psychiatric Association, 2006), novel solutions are needed to facilitate treatment engagement. Such solutions must address the most prominent barriers patients face in seeking and receiving prompt treatment, including stereotypes surrounding EDs, denial of illness severity, compromised motivation for treatment, and knowledge deficits regarding treatment resources (Ali et al., 2017; Ali et al., 2020).

We partnered with the National Eating Disorders Association (NEDA) in the U.S. to disseminate an evidence-based online EDs screen, which garners about 200,000 respondents annually. Our prior work showed that while most respondents (86%) screened positive for an ED, few had previously received treatment (14%) or were currently receiving treatment (3%) (Fitzsimmons-Craft et al., 2019), which is consistent with other work suggesting less than 20% of those with EDs ever receive treatment (Eisenberg, Nicklett, Roeder, & Kirz, 2011; Kazdin, Fitzsimmons-Craft, & Wilfley, 2017). Additional investigation suggested that only 16% of NEDA screen completers who screened positive for an ED and were not in treatment had initiated care following screen completion, with this figure likely an overestimate given its basis on only those who provided follow-up data (Fitzsimmons-Craft et al., 2020). Thus, additional tools are needed to increase motivation for ED treatment and service utilization.

Digital tools offer a user-friendly, cost-effective, and accessible solution to tackle patient concerns and to broadly disseminate mental health-related interventions. Such tools include chatbots, which are computer programs that simulate human conversations (Torous et al. 2021). Despite having awareness of conversating with a robot, humans still respond to chatbots as they would to humans, as evidence supports (Gratzer et al., 2021; Montenegro, da Costa, & da Rosa Righi, 2019), which reflects the success chatbots have shown in establishing digital therapeutic alliances, relationships which may mediate the effectiveness of digital mental health tools by increasing user trust, fidelity, and engagement (Torous & Haim, 2018). In addition, chatbots may improve mental health symptoms like body image (Car et al., 2020; Fitzsimmons-Craft et al., 2022) and promote positive health-related behavior change (Car et al., 2020; Pereira & Díaz, 2019; Zhang, Oh, Lange, Yu, & Fukuoka, 2020).

These findings and others suggest that a chatbot may be appropriate to facilitate motivation for and utilization of services in individuals with EDs. As such, this study developed and evaluated the usability of a chatbot designed for that purpose and to be used following online screening. We detail the process of developing this chatbot using user-centered design and multiple iterative cycles of usability testing (Graham et al., 2019; Lyon & Bruns, 2019).

Method

Participants and Recruitment

Participants were recruited via digital ads on social media platforms, physical flyers posted throughout St. Louis, Missouri, and an email listserv of individuals interested in research. Advertisements directed participants to Qualtrics, which administered an online eligibility questionnaire and collected demographic information. Eligibility criteria required that participants were at least 18 years old, had screened positive for a clinical or subclinical ED based on the Stanford-Washington University ED Screen (SWED) (Graham, Trockel et al., 2019), reported not currently being in treatment for an ED, and owned a mobile phone. Eligible participants were asked to provide contact information and were subsequently invited to participate by a member of our research team.

Chatbot Design

The chatbot, named Alex, was developed using a user-centered design approach, which includes an iterative, six phase process (investigate, ideate, prototype, evaluate, refine, develop) aimed to produce technology that integrates feedback from stakeholders and members of the target population throughout development (Graham, Wildes et al., 2019). This study is part one of a three-study series following the Multiphase Optimization Strategy (MOST) framework, which ensures that interventions are optimized before they are tested in an RCT (Collins, 2018), and which includes three phases: 1) preparation; 2) optimization; and 3) evaluation. The final version of Alex from this study will be tested in study two (the optimization phase) to assess the effectiveness of chatbot components on behavior change and help-seeking (NCT04806165). Study three will follow, which will include an RCT using the optimized bot. All procedures were approved by the institutional review board.

Alex was hosted by a private mental health chatbot company, X2AI, and featured a fully automated, synchronous conversation delivered via SMS or Facebook Messenger. Alex was developed using decision trees, as part of a rule-based approach that is more time- and cost-efficient, revisable, and controllable than an artificial-intelligence approach (Boucher et al., 2021). Decision trees use “if-then” statements to navigate conversations via pre-programmed logic, including conditional statements and branching patterns (Figure 1), and recognize specific keywords, simple responses, and common phrases (e.g., “yes” or “I don’t know”) in users’ messages (Buntine, 1992). Given that users in past rule-based chatbot studies commonly perceived the tools as “robotic” (Abd-Alrazaq et al., 2021; Denecke, Abd-Alrazaq, & Househ, 2021), several guiding principles informed Alex’s design, each intended to provide users with a human-like, amiable experience: 1) chatbot responses were kept short in length and included colloquial, empathetic, and congenial rhetoric, and 2) chatbot responses were limited to four consecutive messages.

Fig. 1.

Screens illustrate user conversations with Alex for each component tested. From left to right: psychoeducation, motivational interviewing, personalized recommendations, and repeated administration.

Chatbot Components

Alex started with a brief introduction covering what to expect, protocol for crisis, and limitations of the tool. Four theoretically-informed components (i.e., psychoeducation (PE), motivational interviewing (MI), personalized recommendations for services (PR), and repeated administration (RA)) followed (Figure 2).

Fig. 2.

Four components were tested: psychoeducation, personalized recommendations for intervention, motivational interviewing, and repeated administration. Each component targets theoretically-informed mechanisms for increasing motivation for treatment and self-efficacy. Modules ultimately aimed to improve uptake of mental health services.

The PE component was based on the mental health literacy model, which has been shown to improve patients’ help-seeking attitudes and treatment need perceptions (Xu et al., 2018). Following the Academy for Eating Disorders’ “Nine Truths about EDs” (Schaumberg et al., 2017) this component refuted common ED stereotypes and myths, informed users of ED health consequences, provided tailored information for each specific ED psychopathology (e.g., restriction, compensatory behaviors, etc.), and emphasized the seriousness of user concerns.

The MI component maps closely onto cognitive dissonance theory (Festinger, 1962) and self-perception theories (Bem, 1967), and was guided by Miller and Rollnick’s (2012) core elements of MI. MI highlights discrepancies between individuals’ unhealthy behaviors and their healthy goals (Lundahl & Burke, 2009), which has been shown to improve outcomes and uptake of care by improving self-efficacy and motivation to change (Copeland, McNamara, Kelson, & Simpson, 2015). A review of meta-analytic findings showed that MI is significantly more effective than no treatment (ds up to .57) for a range of problems and for increasing treatment engagement (Burke, Arkowitz, & Menchola, 2003; Lundahl & Burke, 2009).

The PR component provided personalized recommendations for seeking intervention. The module was based on the Elaboration Likelihood Model (ELM) (Petty & Cacioppo, 1981), whereby individuals are more likely to process information in an active and thoughtful manner when they find messages personally relevant. Messages that are processed in this way are retained longer and tend to improve individuals’ motivation and self-efficacy (Kreuter & Wray, 2003). In this component, participants responded to questions regarding treatment preferences (in-person or online) (Figure 3), and Alex followed-up with tailored resources recommended from the NEDA website (in contrast to a lengthy list of options often presented at the end of online screens). Findings have indicated that tailored messages have a greater impact on health behavior versus a control group (r=.074) (Noar, Benac, & Harris, 2007), and that text message-based health promotion messages have an effect of d=.33, with greater efficacy for those using personalization (p=.001) (Head, Noar, Iannarino, & Harrington, 2013).

Fig. 3.

The personalized recommendations component offered users with tailored resource and service recommendations based on items assessing treatment preferences. This figure outlines the branching logic used by Alex to determine which resources and services to provide users.

* =Iterative refinements added pathways for users to see both in-person/telehealth and online options, and multiple options from each

**=Additional online resource options were added between cycles, including references to online support groups and an online peer-to-peer support community

The RA component provided up to three interactive check-ins which reminded users of resources and services available to them and promoted reflection on overcoming barriers. Three days after completing the main conversation, users received their first check-in. Those who endorsed seeking help stopped receiving RA check ins, while those users who did not continued to receive up to two more check-ins each spaced three days apart until they reported services seeking or utilization. Reminders have been shown to increase adherence to health-related interventions and treatment (Fenerty, West, Davis, Kaplan, & Feldman, 2012). For example, meta-analytic findings have indicated that reminder systems improve medication adherence in chronic disease (d=.41) (Thakkar et al., 2016) and other issues (Misono et al., 2010).

The introduction, PE, MI, and PR components were designed for completion in 15-20 minutes. RA check-ins took ~3 minutes to complete.

Procedure

In-person and remote usability testing was conducted with four sequential testing groups, as deemed optimal when developing mHealth interventions using user-centered design (Molina-Recio et al., 2020). The first two cycles were conducted in-person and were moderated by a research team member, who supervised the participants during testing to catch technical glitches, provide instruction when needed, and capture detailed feedback (Wozney et al., 2016). This ensured that necessary revisions were made in order for users in consequent remote, unmoderated cycles to be well-equipped to independently navigate the tool.

Cycles 1 & 2.

Eligible participants near St. Louis, Missouri were invited to our lab where informed consent was obtained and testing sessions were conducted with a team member. Via SMS or Facebook Messenger, participants first assessed the PE, MI, and PR components of the chatbot using the think-aloud protocol, which provided instantaneous reactions, thoughts, and opinions of participants during testing (Jaspers, Steen, van den Bos, & Geenen, 2004). The RA component was not tested in this cycle. A 60-minute semi-structured interview immediately followed. Participants were audio-recorded throughout the testing session and interview. Post-interview, participants completed an online questionnaire, including the measures described below, as well as the Eating Disorder Examination-Questionnaire (EDE-Q) (Fairburn & Beglin, 1994) to assess ED psychopathology. Participants were compensated with a $20 Amazon electronic gift card.

Cycles 3 & 4.

Remote, unsupervised usability testing in Cycles 3 and 4 was conducted over two weeks. Eligible participants throughout the United States were invited to participate via a Qualtrics survey link, which hosted an informed consent form and a baseline survey collecting demographic information and EDE-Q results. After completing these items, users initiated the SMS conversation by entering the unique start code IDs assigned by Qualtrics’ random number generator. These cycles completed each component tested by Cycles 1 and 2, in addition to the RA component. After two weeks, participants scheduled a ~60 minute semi-structured interview call and completed questionnaires, which included the quantitative measures below. Interviews were conducted by a research team member and were audio recorded. Participants were compensated with a $30 Amazon electronic gift card.

Measures

Quantitative Data

The System Usability Scale (SUS) was administered post-engagement to assess usability (Brooke, 1995). The scale consists of 10-items scored on a 5-point Likert scale, with response options ranging from strongly disagree (1) to strongly agree (5). SUS scores range from 0-100, with an evaluation of 500 studies delineating a score over 68 illustrating “above average” usability and demonstrable user acceptance (Sauro, 2011). The scale is validated for use in small sample sizes (Bangor, Kortum, & Miller, 2008; Lewis & Sauro, 2009).

Users also completed The Usefulness, Satisfaction, and Ease of Use (USE) Questionnaire Short-Form, which measures four dimensions: usefulness, ease of use, ease of learning, and satisfaction of users (Lund, 2001). Items are scored along a 7-point Likert scale, with response options ranging from strongly disagree (1) to strongly agree (7). Subscale scores are generated by averaging item scores independently on each dimension.

Qualitative Feedback

Semi-structured interview questions gathered user perceptions of each chatbot component (PE, MI, PR, RA), design of the chatbot, usefulness, and overall positive and negative experiences. See the full interview guide in the Online Supplement.

Analytic Strategy

Descriptive Analysis

Descriptive statistics were calculated for each cycle using the Statistical Package for Social Sciences (SPSS) version 17.

Qualitative Analysis

Iterative Development.

The initial version of Alex was tested by Cycle 1 participants. Interviews were transcribed and qualitative feedback was assessed by frequency and feasibility to inform refinements to Alex between testing cycles. Some suggestions, including improving the chatbot’s recognition of complex language and enabling users to schedule reminders in the RA component, were not feasible given Alex’s rule-based design.

Thematic Analysis.

Qualitative data was classified following the six-phase thematic analysis framework outlined by Braun and Clarke (2006), which strikes balance between flexibility and rigor, and provides opportunity to evaluate latent themes. Audio recordings were transcribed independently by one team member (J.S.) and later checked independently by team member (B.D.). Interview responses were categorized by component or as overall experience feedback. Using methods by Hannah and Lautsch (2011), J.S. and B.D. privately generated preliminary codes and met at multiple points to compare, evaluate, and discuss similarities and differences. Once members reached a consensus, codes were categorized as positive or negative feedback and then sorted into themes. The final codebook presents positive and negative user feedback, including changes in user sentiment toward the chatbot following iterative refinements.

Results

Participants

Qualitative measures require a sample size above nine to achieve coding saturation and between 16-24 to achieve meaning saturation (Hennink, Kaiser, & Marconi, 2017; Creswell & Creswell, 2017). Therefore, a total of 21 participants were enrolled (n=5, Cycles 1-3; n=6, Cycle 4) (Table 1), and no additional participants were considered. Participants largely identified as white (non-Hispanic) (n = 17, 81%) and female (n=20, 95%), with one participant of male sex included in Cycle 4. Represented ED diagnoses based on SWED criteria included unspecified feeding or eating disorder (UFED) (n=8, 38%), clinical or subclinical bulimia nervosa (BN) (n = 6, 29%), clinical or subclinical binge eating disorder (BED) (n = 5, 24%), anorexia nervosa (AN) (n = 1, 5%), and purging disorder (PD) (n = 1, 5%). The mean EDE-Q global score amongst participants (4.48, SD=1.29) was above the clinical cutoff (4) (Lund, 2001), confirming high ED psychopathology among users. See Table 1 for full details on study participants.

Table 1.

Table of Participant Characteristics by Usability Testing Cycle

	Participant #	Race and Ethnicity	Sex	SWED-Diagnosis	EDE-Q Global Score (Mean ± SD)
Cycle 1 (n=5)					3.59±1.02
	P1	White	Female	UFED
	P2	White, Hispanic	Female	UFED
	P3	White	Female	UFED
	P4	White	Female	UFED
	P5	Asian	Female	UFED
Cycle 2 (n=5)					3.94±1.03
	P6	Asian, White	Female	UFED
	P7	White	Female	UFED
	P8	White	Female	Subclinical Binge Eating Disorder
	P9	White	Female	Subclinical Bulimia Nervosa
	P10	Asian	Female	Subclinical Binge Eating Disorder
Cycle 3 (n=5)					4.88±0.42
	P11	Hispanic	Female	Anorexia Nervosa
	P12	White, Hispanic	Female	Purging Disorder
	P13	White	Female	Binge Eating Disorder
	P14	Black or African American, White	Female	Subclinical Bulimia Nervosa
	P15	White	Female	Bulimia Nervosa
Cycle 4 (n=6)					3.51±1.47
	P16	Black or African American, White	Female	Bulimia Nervosa
	P17	White	Female	Bulimia Nervosa
	P18	White	Female	Subclinical Binge Eating Disorder
	P19	White	Female	Bulimia Nervosa
	P20	White	Male	UFED
	P21	Black or African American	Female	Subclinical Binge Eating Disorder
Total (n=21)					4.48 ±1.29

Note. UFED = unspecified feeding or eating disorder, EDE-Q= Eating Disorder Examination-Questionnaire, SWED= Stanford-Washington University ED Screen

Scale Scores

The average SUS score from Cycle 1 (83.0) indicated the initial intervention had above average usability and demonstrable user acceptance even prior to any refinements (Table 2). Average SUS scores across cycles ranged from 75.0-85.8. Though there was not a sizable change in average SUS scores between the first (83.0) and final (85.8) iteration, scores over all cycles were comparable or higher than scores obtained in other mental health chatbot usability studies (Cameron et al., 2019, Islam et al., 2021, Oh et al., 2021). Overall, Alex was rated positively based on USE scores, which were above 4 (out of 7) on most domains. High USE scores for the first iteration may explain why refinements did not drastically improve USE scores (Table 2).

Table 2.

Descriptive Statistics for SUS and USE Questionnaires by Usability Testing Cycle

	Cycle	Descriptive
SUS Total		Mean	SD	Min.	Max.
	1	83.0	13.2	62.5	97.5
	2	77.0	16.7	50.0	90.0
	3	75.0	19.8	50.0	95.0
	4	85.8	10.2	70.0	97.5
USE Usefulness
	1	5.13	0.95	3.83	6.33
	2	4.67	0.89	3.33	5.67
	3	4.07	1.60	1.50	5.50
	4	5.00	1.06	3.33	6.17
USE Ease of Use
	1	6.12	0.66	5.40	7.00
	2	5.84	0.33	5.40	6.20
	3	4.68	2.09	1.60	6.80
	4	5.73	1.26	3.40	7.00
USE Ease of Learning
	1	6.33	0.62	5.67	7.00
	2	5.80	0.96	4.67	7.00
	3	5.53	2.06	2.00	7.00
	4	6.39	0.57	5.67	7.00
USE Satisfaction
	1	5.32	1.27	3.40	6.80
	2	4.52	1.60	1.80	5.80
	3	3.84	1.35	1.60	5.20
	4	4.80	1.46	3.00	6.60

Note. SUS= System Usability Scale (SUS), USE=Usefulness, Satisfaction, and Ease of Use Questionnaire

Thematic Analysis

Feedback from participants was labeled as “positive” or “negative”, with the latter identifying suggestions to improve the chatbot and which informed modifications between iterations (Table 4). Initial codes converged under the following general themes: user experience (subjective user perception and feelings), chatbot qualities (traits and characteristics of the bot as perceived by users, such as human-like or automated), chatbot content (overall and module-specific user reactions to information and resources), and ease of use (navigation and technical components) (Table 3).

Table 3.

Themes, Descriptions, and Feedback from Users across Usability Testing Cycles

Themes	Description:	Illustrative Quotes (P#)
Overall Experience
Positive
Experience	Confidential Comfortable	“I like that you can be more honest with your feelings because you're not face to face with somebody.” (P4) “It was forgiving. There were affirmations and it backs up and restates what you tell her, so it was very spot-on. It was able to follow and help make connections. It was not as impersonal as you would think it would be.” (P7) “I felt really comfortable. It was like talking to a person but in a more anonymous way.” (P18)
Chatbot Qualities	Quick Responses Realistic Engaging Impartial	“It was kind of relaxed because, you know, it's not judgmental, but then also kind of anxious because, you know, in your head you're still working through all this. So with something like this, everything and everybody is judgmental. I mean, the entire world out there is judgmental, particularly with social media and crap. So it was kind of nice to be able to get it out.” (P7) “It was really easy. It replied really fast, which was nice.” (P9) “It was better [than] in other apps that I've used, and it felt closer to when I've talked about things with people in real life…” (P10) I felt it wasn’t as robotic I felt it would be. I really felt at a point there could have been times where it could have carried on a very lengthy conversation and to some people who are very down because of their weight or ED, even though this isn’t like a psychologist or counselor, maybe this could shoot some helpful information…” (P15) “It didn’t give you too much to read through or anything and the messages were very succinct. It also wasn’t super clinical, and it used emojis and things like that to make the experience more comfortable and to feel more normal.” (P19)
Content	N/A	N/A
Ease of Use	Easy navigation Easy to respond Self-instructive	“I liked it! I think it’s really cute. I think that it was really encouraging…”. (P11) “It was interesting and gave you some things to think about on your own. It was personalized, so it was helpful.” (3) “It was really easy and you don’t need instructions. If you can chat with someone, you can do this.” (P16)
Negative
Experience	No personal salience	“This is not for me. I feel uncomfortable that I took this. Like, I feel worse about myself now because it just like made me feel like I was taking things that I shouldn't be taking.” (P5) “…towards the end, this chatbot really made me feel like this was not the thing that I should be using because it was directed to people. It is directed towards people who have real problems versus like my just general like mental blah about this versus like actually doing dangerous things.” (P5)
Chatbot Qualities	Robotic/Automated Limited Response Options	“At first, I had a little bit of a language barrier because he would say to answer a question and I would answer the question but it would be like a “yes” or “no” so I would have to go back because maybe I elaborated too much or had answered in a way he couldn’t understand.” (P15) “The only negative I have is that some of the questions where I had an answer that was in between yes and no, I had to pick one or the other. But I don’t really know how you would get around that.” (P19) “The one thing I didn’t like is that it bottlenecked you which I understand that it’s a bot and can’t recognize all sentences completely. So sometimes It couldn’t understand what I was talking about. You needed to use really basic responses.” (P20)
Content	N/A	N/A
Ease of Use	N/A	N/A
Psychoeducation
Positive
Experience	Comfortable Nonjudgmental	“… I spend all my time researching all these things. To try to learn about this stuff, I’m coming from zero. And then when I go to see the doctor…it feels like they don’t understand or hear me.” (P8) “I did like this part because it made it easier to start the conversation. I like the idea of giving a couple of options because in the start it can be hard to open up and have that conversation.” (P11)
Chatbot Qualities	N/A	N/A
Content	Offers affirmation Promotes reflection Addresses stereotypes Validates personal concerns	“I did like that. Yeah. I think because of the general thought is people think that like eating disorder can be someone that's like really, really skinny, like sick looking, or it could be to the other extreme of overly overweight or obese. So I think I like it addressed that. You know, it could be normal people that are healthy and stuff, but they are going through that disorder.” (P2) “It was helpful because like I said, you know, you… when you think eating disorder, you think of people who are generally more skinny that are throwing up or know something like doing something drastic. You don't necessarily think of being overweight as an eating disorder, but it can be because you have the binge eating and all that stuff.” (P3) “It had a lot more information in fact than I even thought was out there. I had no idea there were providers who helped you besides your primary.” (P12) “I liked it. It just felt easier to answer questions like that not in person.” (P13)
Ease of Use	N/A	N/A
Negative
Experience	No personal salience	“All the resources offered [to] me were just like things that are like real serious. And I don't think I'm that bad. In this sense, not that that I'm bad, but like. So there's nothing there for me. I'm just going to walk around hating my body, you know?” (P5) “I think it was incomplete because I'm struggling with several things right now and I just kind of had to pick one.” (P9) “It was really going for purging, bulimic, anorexia, and you know there are other disorders involved, like I will sit there and overeat and then I feel like I need to purge but I can’t get myself to do it but I can’t because I don’t want to get this horrible disease. Mine is an emotional overeating so there should be some sort of category where you give everybody an opportunity to feel included in that because I didn’t really know what category I fit into.” (P15)
Chatbot Qualities	Limited response options	“It was pretty well versed, but it had pretty vague responses. Like they weren’t very detailed, but I don’t know how you would change that because it is a chatbot. (P20) “It’s like those phone menu things where it’s like ‘okay if you need this, press 0.’ And then you have to answer other questions. Some of this stuff, I don’t know how you would change it. So that was annoying.” (P18)
Content	Lacking content Diverse coverage of disorders Novel information	“Maybe they could have filtered it into some of the other stuff, too…like, say statistics of people getting help, you know, or something.” (P1) “…you kind of-- kind of already know it [the psychoeducation]. It's kind of inherent that, you know, some of what was being said back in terms of like, you know, ‘it's not a typical person’ in a way anybody would think that most people probably know that. And maybe some people don't know…” (P4)
Ease of Use	N/A	N/A
Motivational Interviewing
Positive
Experience	Supportive Comfortable Encouraging	“The chatbot sort of motivated me to maybe, you know, potentially look for treatment or help.” (P8) “I think it encouraged me to be more honest and get a little more open instead of like minimizing.” (P10) “I liked it. It enabled me to think through things easier and answer as if though it was how I was feeling at that time, and how I was feeling.” (P13)
Chatbot Qualities	Depth of Conversation Detailed Informative Increased motivation	“Yeah, I mean, it kind of puts you more in the mind of thinking about it again and again. Just like trying to do better and then having something say, well, you can do it. I think it said at one point that you can do something uplifting. You know, it's going to make you want to try harder than, I mean, this is outside the circle, but I tried that [name of weight-loss program] app. Yeah, and I just thought it was very condescending. So having something that's like this that's more supportive and not really condescending was good. The attitude of the conversation is at a good point.” (P3) “It made me think about it more and made me more willing to make changes…” (P7)
Content	Scales Promotes reflection Concretized concerns Introspective	“The scales make you self-aware, like why don’t you want to seek treatment? It helps you learn more about your confidence level in that aspect.” (P1) “I really liked that part because, like, I don't think I've thought about what I can do to make my motivation or confidence higher. I just keep it where it is. But I liked that it asked about just like making it go one point higher.” (P10) “I did because it made me accountable at that time because I may say and think it in my head, but when someone asked me a question and I saw the number in black and white in front of me, it made me accountable. Saying this to the robot made me accountable and I had to take responsibility for that. It was basically my “aha” moment.” (P15)
Ease of Use	N/A	N/A
Negative
Experience	Not motivating	“I don’t think it moved me to do anything. Maybe if they determine that therapy was the right direction and they pushed me towards this side of this list or whatever that they could say, you know, of the X amount of people who struggle with this X percent say they received this amount of benefit or, you know, overwhelming benefit in talk therapy or something like that. Some, like statistics.” (P8)
Chatbot Qualities	N/A	N/A
Content	Scales Confusing, challenging Limited to behavioral focus	“I didn’t mind doing the ranking but the follow up questions where it asks why you pick a number, I didn’t know what would make my answer go from a 7 to an 8. Like am I supposed to know what would do that, because most people won’t. Isn’t it common sense that you would already do that?” (P4) “I didn't really like that. I thought the rating stuff was helpful in the sense that I really obviously didn’t want to think about that stuff. But then I had to justify myself and like really feel out why I was feeling where I was feeling about that. So I think that was helpful. But just like the rest of it just felt very external versus like I feel like my personal thing is like a very internal thing.” (P5)
Ease of Use	N/A	N/A
Personalized Recommendations
Positive
Experience	Autonomy No pressure Independent decision-making	“Like I said, I liked that it was not pushy but was just a friendly reminder.” (P2) “No, I liked it because some people thrive on having that one-on-one interaction and some people would rather be able to look up things on their own and then just touch base with a chatbot so that they don’t have to hear things they don’t want to hear. So yeah, I liked it. It lets you allow the dose in which you take it in instead of having a person push and push and push.” (P3) “I loved it because I think this is the new normal, and I think we see more and more people live longer and they’re getting into electives and technology because their kids are teaching their parents how to use technology so the older people have technology. They’re ready.” (P15) “I liked that section for sure, it was nice to have those options laid out. So if there’s only one or two options, it laid out things I maybe hadn’t thought of before. Then, it was really easy to get the information on each of those. And I can sit down and make my own decisions on what I need rather than someone telling me what I need.” (P19)
Chatbot Qualities	N/A	N/A
Content	Resources Novel Organized options	“I liked that because like I mentioned, it included why I thought that, the actual link, and then some user review of it. And then it asked me what I thought of that. I kind of wonder what would happen If I said I didn’t need that and if it would suggest another type of help maybe.” (P1) “I liked that those were available. That made me feel like I didn't just do this for nothing. I went through this chatbot to help direct me towards things that are more specific to my needs, then--as opposed to just going through all the Web pages myself and trying to find things.” (P9) “I liked the resources it gave me the most, a lot of things I didn’t even know existed.” (P18)
Ease of Use	N/A	N/A
Negative
Experience	Not motivating	“I think it could be good, but the delivery was a little off for me because I think that's a space where it could really be like it's not just like this one picture. It's like eating disorders and the fact that it's like saying this like neutrally disordered eating your body image problems, I think. And I think the problem is, is that like it sort of sort of like negated what they're saying in the beginning with how it was talking to me in the end.” (P5)
Chatbot Qualities	N/A	N/A
Content	Not novel Offers no accountability component	“Well, I listed both and they listed NEDA. But they didn’t go into other lists other than the NEDA and non-NEDA providers. It didn’t even say what NEDA stood for. Losing weight isn’t always about talking to somebody. There are weight management doctors, surgeries, wiring my mouth shut. I don’t know.” (P8) “I didn't like that part because, I mean, it sent me to NEDA's Web site…I think some people would also just like need a referral to like a regular doctor, and like, how do you even find a doctor? So that was a little more difficult.” (P10) “I think I remember that. So I went into this whole thing on the fence of whether I has a problem, and after talking to the bot, I thought, ‘Wow, there are actually people that need help with these issues.’ Talking to the bot made me feel I didn’t need help, and if there were things I needed help with I could talk to my family about it.” (P20)
Ease of Use	N/A	N/A
Repeated Administration
Positive
Experience	Motivating Beneficial	“I do like this option because it helped take the first step when you have online options, so that’s great.” (P11) “I liked that it gave me that little nudge to actually get online and check things out and think things through, like a to-do list.” (P13) “…I was really impressed by this. I think you guys really covered it all, and I thought it was really informative and I thought it was really good.” (P17) “I like this aspect. I only had one set of follow ups because I had reached out for help at that point.” (P18) “I did like the check in, I think that it was like three days after the initial conversation? Yeah, that was nice.” (P20)
Chatbot Qualities	N/A
Content	Resources Personalized	“…I thought the recommendations were tailored for my responses. It did send me a reminder, which again I then did go back to those resources.” (P21)
Ease of Use	N/A
Negative
Experience	N/A
Chatbot Qualities	Conversation Flow Check-in scheduling	“… they need to remind during business hours.” (P13) I disliked…when it replies. I think mine replied right after a weekend when most offices closed so there wasn’t much time to make an appointment.” (P14) “…you want to put on there that if something is going on in your life and you need me to check back in a few days, maybe having a simple little reply like ‘reply with a 2.’ So if someone is sick with the flu or something they won’t get reminders and you don’t lose them when they get upset.” (P15) “I’ve made the appointment but I haven’t actually gone to it…and then the chatbot checks in…well nothing is changed and I can’t really give updates on what has changed because nothing has in a good or in a bad way.” (P19) “…maybe give more time for people to respond before giving out more reminders after the first one. It might take people time to take out or check their phones because it gave me two reminders within a thirty second window.” (P20)
Content	N/A
Ease of Use	N/A

Overall Experience

Users largely reported having enjoyable, gratifying experiences across cycles. Most users affirmed that chatbot content reinforced help-seeking, personal reflection, and behavior change. Participants did not report issues navigating the tool, reporting that it required little-to-no instruction: “…If you can chat with someone, you can do this” (P16). Of significance is how human-like most users described Alex as, using terms like “engaging”, “impartial”, and “realistic”. These qualities were attributed to Alex’s empathetic and comforting language: a user (P19) reported, “It also wasn’t super clinical, and it used emojis and things like that to make the experience more comfortable and to feel more normal.” Another user (P7) reported, “There were affirmations, and it backs up and restates what you tell her, so it was very spot-on. It was able to follow and help make connections.” Some users even found that removing human interaction compelled them to respond more honestly than if they were seeking in-person help: “I like that you can be more honest with your feelings because you're not face to face with somebody” (P4).

Some users in early cycles did report concerns and negative feedback, which informed refinements and are detailed below and in Table 4.

Psychoeducation Component

The PE component addressed stereotypes surrounding the behaviors, symptoms, and “appearance” of EDs. It aimed to validate users’ concerns and encourage participants to reflect on their personal eating and body image struggles. Users generally offered positive feedback on content addressing ED myths, as expressed by P3 in Cycle 1, “It was helpful…when you think [of] eating disorders, you think of people who are generally more skinny, that are throwing up or…something drastic.” Participants with BED responded most positively to this component, both pre- and post-refinements, as they felt the ED stereotype content normalized and validated their concerns. A common user suggestion included adding content about more complex ED presentations. A user in Cycle 2 (P9) reported, “I think it was incomplete because I’m struggling with several things right now and I just kind of had to pick one.” Other users suggested increasing personal relevance by enabling users to select multiple concerns (i.e., binge eating, restricting food, purging, and preoccupation or anxiety around food and body image) to learn more about.

To address suggestions, initial language refinements between iterations added more supportive and encouraging messages, improved conversation flow, and reduced self-blame. Content was revised to promote user reflection on information learned and to enable users to learn about multiple endorsed concerns.

In cycles following refinements, most users found the content adequately addressed a wide range of possible ED-related behaviors and challenges. One user in Cycle 3 (P14) noted, “…when you think of disordered [images] in the media it’s a lot of exercising and excessive purging and weight loss which isn’t really what I’m affected by. I'm affected by bingeing. It’s good to clarify misconceptions there too because that’s something you don’t see a lot anywhere else” (P18). Some users in later cycles still felt limited in their answer choices: “It’s like those phone menu things where it’s like ‘okay if you need this, press 0’… then you have to answer other questions” (P19). This highlights limitations inherent in a rule-based chatbot approach.

Motivational Interviewing Component

The MI component encouraged users to evaluate on a scale from 1-10 how important they felt it was to address their EDs and their confidence in making changes addressing related behaviors. Users were then asked to reflect on their responses. Participants generally reported these quantitative items fostered introspection on their actual confidence and motivation to change. P15 explained that the scales “…made me accountable at that time because I may say and think [about something] in my head, but when someone asked me a question and I saw the number in black and white in front of me, it made me accountable…It was basically my ‘aha’ moment.” Conversely, most negative feedback from users expressed that follow-up scale items (Table 4a) were confusing and difficult to respond to: “I didn’t mind doing the ranking but the follow-up questions where it asks why you pick a number, I didn’t know what would make my answer go from a 7 to an 8” (P4). Such follow-up scale items were revised between iterations, among other content and language changes, to promote greater self-reflection among users. Additional language revisions aimed to improve users’ motivation to pursue treatment and validate users with prior help-seeking challenges.

Participants responded well to the above refinements. They increasingly found that language in the module promoted honest disclosure and reflection, which augmented their motivation to work on their ED-related challenges. P10 found that the refined language “…encouraged me to be more honest and get a little more open instead of…minimizing.” Users also reflected more positively on revised follow-up questions (Table 4b-d), as opposed to early participants who received aforementioned scaled-follow-up items, and identified them as the primary driver of their increased motivation to seek treatment post-engagement.

Personalized Recommendation Componente

In the PR component, Alex prompted users to select seeing in-person or online help options before navigating users through options for each based on the decision trees illustrated in Figure 3. Users found the component easy to use, pragmatic, and introspective. Most positive feedback outlined that talking to Alex, versus a human, provided users with a pressure-free environment and decision-making autonomy. P19 explained, “…it was nice to have those options laid out…I can sit down and make my own decisions on what I need rather than someone telling me what I need.” While there were no concerns regarding navigation or ease of use in this component, users provided suggestions to improve their perceptions of provided recommendations. Participants wanted the choice to select both in-person and online resources, to decline help, and to see more resource options in the online help pathway. One user wanted to see more motivational language used to promote resource utilization. These suggestions informed refinements between iterations, which included adding validating language for users who faced prior barriers to seeking treatment and for users not yet ready for help. Content modifications added options for users to see both in-person and online resources and to receive multiple recommendations from each, to decline help, and to receive online self-help recommendations (Figure 3).

Following refinements, fewer users in Cycles 3 and 4 (2 of 11, 19%) provided suggestions for improvement than did users in Cycles 1 and 2 (6 of 10, 60%), and their feedback reinforced that content refinements improved personal salience of tailored recommendations to perceived user needs. Regarding the telehealth resources provided, P15 commented, “I loved it because I think this is the new normal…”. P17 explained, “Just knowing that there are more avenues besides…keeping this to yourself [or going] to one of those treatment centers. It's like, yeah, I can get on this whole online community. And, you know, that's a safe first step.”

Repeated Administration Component

The RA component was tested by users in Cycles 3 and 4, who received repeated follow-up check-ins over the two weeks following their initial conversation with Alex. Users generally found that the resource option reminders offered during these check-ins reinforced help-seeking behaviors. For example, P12 in Cycle 3 expressed, “It reminds you and you can’t just drop it. You get busy with things and forget to take care of the issue, but the chatbot reminded me.” Other users thought the check-ins solidified their motivation to seek treatment and provided reassurance that their disordered behaviors were “worthy” of care.

The primary concern users expressed in this component was the inability to schedule when check-in messages were received, including the day, time of day, and time between follow-up messages. The platform did not allow for a customized check-in schedule to be implemented.

Minor changes were made between Cycles 3 and 4, including content additions and language refinements to motivate help-seeking prior to the user’s next check-in.

Discussion

This paper describes the process of developing a user-informed chatbot, Alex, to increase motivation for treatment and mental health services use among a sample (N=21) screening positive for EDs and not yet in treatment. Feedback over four testing cycles guided modifications to each of four theoretically-informed components.

Feedback across cycles converged on four broad themes (user experience, chatbot qualities, chatbot content, and ease of use), which mirrored common themes in a recent review of 37 mental health chatbots exploring user experiences with conversational interventions (Abd-Alrazaq et al., 2021). Alex showed excellent usability and acceptance among participants even prior to refinements. Both in-person and remote participants reported no technical challenges or difficulty navigating the chatbot. Rather, suggestions for improvement addressed Alex’s content and language. This feedback was the basis of minor and major content refinements to the PE, MI, and PR components. Language refinements were included throughout all four components and enhanced Alex’s empathetic, supportive, and encouraging qualities. Over cycles of testing, fewer users reported finding Alex “robotic” or unaccommodating to their unique ED-related behaviors and resource needs. SUS scores demonstrated that perceived usability improved, albeit marginally, between iterations. However, usability did not improve linearly across cycles, likely attributable to the intervention being highly regarded from the first iteration.

Findings from other chatbot usability and acceptance studies suggest that perceived chatbot qualities (i.e., robotic, supportive, empathetic) are a likely mediator of general user acceptance and success of the intervention at improving mental health outcomes (Eisenstadt, Liverpool, Infanti, Ciuvat, & Carlsson, 2021). Similarly, users in our study largely expressed that Alex’s empathetic, personable, and human-like qualities augmented their honesty during conversations, trust in the resources and information provided, and confidence and willingness to take the next steps to address their ED and body image concerns. This phenomenon is known as the “digital therapeutic alliance”, which previous work deems critical to optimize chatbot utilization and user perceptions (Darcy, Daniels, Salinger, Wicks, & Robinson, 2021), and may explain the robust, positive feedback and usability scores received by Alex’s users across both in-person and remote testing environments.

Growing mental health resource demand has recently contributed to the development of more interventions using digital interfaces, yet very few integrate evidence-based content with user-informed refinements or test the applicability of these tools in remote settings (Torous et al, 2021). Other chatbot studies report users feeling inundated with resource options and feeling content was irrelevant to their mental health needs, leading to user disengagement and compromised intervention outcomes (Beilharz, Sukunesan, Rossell, Kulkarni, & Sharp, 2021). Alex was designed to bypass the aforementioned issues by 1) using theory-driven content iteratively refined based on users’ perceived needs and expectations; 2) providing users with a tailored experience, including individual specific resources and content; and 3) being optimized for use in clinical and remote settings.

To date, only three chatbots relating to EDs have emerged. The chatbot Tessa featured an established CBT-based EDs prevention program and aimed to improve body image and prevent EDs (Chan et al., 2022; Fitzsimmons-Craft et al., 2021). Results indicated that Tessa successfully reduced women’s concerns about weight and shape through 6-month follow-up and showed potential to reduce ED onset (Fitzsimmons-Craft et al., 2022). Another chatbot, KIT, was designed to support people with body image and eating concerns. KIT provided psychoeducation, information on help seeking, and coping skills to patients and their caregivers (Beilharz, Sukunesan, Rossell, Kulkarni, & Sharp, 2021). A chatbot similar to KIT is currently being developed and tested among adolescents in Brazil and aims to improve body image (Matheson et al., 2021). To our knowledge, Alex is the first chatbot designed for pairing with an EDs or other mental health-related online screen, with the goal of ultimately increasing service utilization. Uniquely within this population, perceived and internalized barriers, even above treatment access, may preclude patients from receiving or seeking treatment (Ali et al., 2017; Hamilton et al., 2022). As such, more interventions like Alex are needed that refer individuals to the diverse range of ED care options available (e.g., self-help apps) at little to no cost to patients and that can be accessed immediately. Greater development and demand of such tools that promote self-efficacy may grow awareness of hospitals and ED programs of innovative means to expand service options.

This study presents several limitations. Alex is a rule-based chatbot, which may introduce a comprehension barrier between itself and users. For example, some participants found the conversation robotic at times when Alex could not understand their more complex responses. Further development of Alex’s interface may include transforming it into an artificially intelligent chatbot, which uses natural language processing and is perceived as more “human-like” than rule-based chatbots (Boucher et al., 2021). Moreover, results may not be generalizable to some subgroups, given that study recruitment methods attracted individuals already receptive to new service models and motivated for care, and provided scarce representation of restrictive ED diagnoses, males, and non-white participants. Of note, a select few users felt their ED challenges were not severe enough to use Alex and expressed feelings of guilt for using the tool. Denial of illness severity remains one of the largest barriers to ED treatment, and these results suggest that digital interventions like Alex must be further adapted to combat such barriers. Future investigation is needed to 1) determine Alex’s efficacy at immediately and longitudinally improving motivation, self-efficacy, and help-seeking among users, including based on predictors like ED psychopathology and severity, and 2) ascertain which components of the chatbot are responsible for driving these improvements. A factorial study is currently underway to test these questions (Washington University School of Medicine, 2021).

Overall, this study provides preliminary evidence of the chatbot’s acceptability as a tool to improve motivation and help-seeking behaviors among individuals identified by an online screen as having an ED but not yet in treatment. If demonstrated efficacious for increasing services utilization among this population, Alex could be implemented broadly to increase uptake of treatment following screening for EDs, as well as adapted for pairing with screening for other mental health disorders.

Public Significance Statement:

• Low rates of service utilization and treatment have been observed among individuals following online eating disorder screening. Tools are needed, including scalable, digital tools that can be easily paired with screening, to improve motivation for addressing eating disorders in order to promote service utilization.