User Experience Evaluation of a Smoking Cessation App in People With Serious Mental Illness

Abstract

Introduction:

Smoking rates among people with serious mental illness are 3 to 4 times higher than the general population, yet currently there are no smoking cessation apps specifically designed to address this need. We report the results of a User Experience (UX) evaluation of a National Cancer Institute smoking cessation app, QuitPal, and provide user centered design data that can be used to tailor smoking cessation apps for this population.

Methods:

Two hundred forty hours of field experience with QuitPal, 10 hours of recorded interviews and task performances, usage logs and a self-reported usability scale, informed the results of our study. Participants were five individuals recruited from a community mental health clinic with a reported serious mental illness history. Performance, self-reports, usage logs and interview data were triangulated to identify critical usability errors and UX themes emerging from this population.

Results:

Data suggests QuitPal has below average levels of usability, elevated time on task performances and required considerable amounts of guidance. UX themes provided critical information to tailor smoking cessation apps for this population, such as the importance of breaking down “cessation” into smaller steps and use of a reward system.

Conclusions:

This is the first study to examine the UX of a smoking cessation app among people with serious mental illness. Data from this study will inform future research efforts to expand the effectiveness and reach of smoking cessation apps for this highly nicotine dependent yet under-served population.

Implications:

Data from this study will inform future research efforts to expand the effectiveness and reach of smoking cessation apps for people with serious mental illness, a highly nicotine dependent yet under-served population.

Introduction

The smoking rate among adults with serious mental illness, such as schizophrenia spectrum, bipolar, and recurring depressive disorders, is an alarming 60%–88%—3 to 4 times the rate of the general population.^1,2 People with psychotic disorders are heavier smokers,³ are more dependent on nicotine,⁴ and on average spend 27% of their income on cigarettes.⁵ Since face-to-face interventions are not likely to reach the majority of this population,⁶ there is an important need to develop effective and wider reaching behavior change tools for smokers with mental health symptoms.⁷

A potential solution to deliver wider reaching smoking cessation treatment is the use mobile health technology. People with serious mental illness are increasingly adopting mobile technologies to their daily lives. A recent survey shows that up to 72% of these individuals own or use a mobile device, 33% of which with internet connectivity, suggesting they might be using smartphones.⁸ The opportunity for dissemination of evidence-based smoking cessation treatment in this population is therefore at hands reach. Further, the number of smoking cessation apps is rapidly proliferating. A recent report indicates that there are more than 400 of these apps available for download for the public,⁹ with a few already tested in randomized controlled trials.^10,11

QuitPal is a free smoking cessation app developed by the National Cancer Institute. This app provides tools based on US Clinical Practice Guidelines, which have received extensive empirical support and are considered the gold standard in smoking cessation. QuitPal was iteratively developed using user centered design principles,¹² and according to developers it has been downloaded about 10 000 times since it launched in 2012 (NCI, personal communication on January 2015). Despite the potential wide reach of QuitPal, and the empirical support the US Clinical Practice Guidelines have received over the years, no study to date has examined the design requirements of smoking cessation apps for people with serious mental illness, who may have special needs.

Proper User Experience (UX) design is a critical benchmark of software systems, especially for those targeting public health, and its importance is widely embraced by the software industry.¹³ UX research is rooted in the psychology of human factors and ergonomics, evolving in the emerging interdisciplinary field of User Centered Design,¹³ and has developed a rich range of tools and methods to evaluate and design the use of systems involving human computer interaction.¹⁴ Failure to develop software systems that are user centered can lead to increased user errors, reduced effectiveness, low levels of user acceptance and the need for increase training and support.¹³

Evaluating both the usability and experience of health behavior apps is thus a critical step towards the development of more effective and engaging smoking cessation software tools. Accordingly, the purpose of the current study is to conduct a UX evaluation of QuitPal amongst people with serious mental illness. This study will examine two critical questions in UX research, which separately address both the user’s performance and their psychological experience while using a specific technology: (1) How usable is QuitPal amongst people with serious mental illness? and (2) What is their psychological experience while using it? Results from this study will inform future research efforts to develop more engaging and effective smoking cessation interventions that might provide lower cost and higher reaching smoking cessation interventions for this population.

Methods

Participants and Procedures

Inclusion criteria were: (1) smoking five or more cigarettes per day, (2) receiving outpatient services at a community mental health center, (3) being 18 years or older, (4) being able to give informed consent, (5) being able speak and write in English, (6) taking psychiatric medication as prescribed by their care provider. We excluded individuals who (1) reported problematic drug or alcohol use in the past 30 days, or (2) had severe cognitive impairment or active psychosis interfering with cognition.

We recruited participants through fliers put up in a community mental health clinic. Providers in the clinic also offered study handouts to potential participants. Interested individuals called study staff and were screened for initial eligibility over the phone (Figure 1). Those who met study criteria were scheduled for an initial meeting at a private office near the clinic, and were evaluated for the presence of significant cognitive impairments or active psychosis. All research activities were conducted by the first and second authors.

Figure 1.

Study procedures.

The course of participation was 3 days. Participants received and iPod touch (Black/Silver Model A1509), and an iPod cable and charger for each device. Each device had the app QuitPal downloaded, and wireless internet capabilities were disabled. During day 1, we introduced participants to QuitPal with a brief hands-on demonstration and explained its different functions (Figure 2). Participants completed a series of predefined tasks and asked to use a “think aloud” procedure. The tasks consisted in the execution of normal app operations, such as tracking cigarettes or setting up a quitdate. We informed participants they would be audio-recorded, and following UX testing standards,¹⁵ research staff observed and took notes. During days 2 to 3, participants field tested QuitPal and interacted with the app in order to gain a more in depth UX. They filled out a daily diary sheet with questions about app usage. Finally, after a minimum of 2 days of experience with the app, participants returned to our lab to conduct an in depth semi-structured interview and complete additional measures.

Figure 2.

QuitPal’s mainscreen (panel A) and tracking feature (panel B).

Interviews were recorded using PearNote (www.usefulfruit.com/pearnote/), which allowed synchronized audio-recording and note taking. We compensated participants with a total of $20 in gift-cards. In order to achieve our recruitment target, and consistent with usability testing standards (which suggest a $50 to $100 stipend per user)¹⁶ and the fact the study did not pose potential clinical benefit, we increased compensation to $75, made valid only for the last participant. All procedures were approved by the University of Washington’s Human Subjects Division.

Measures

System Usability Scale

The System Usability Scale (SUS),¹⁷ is a valid and reliable 10-item usability questionnaire¹⁸ widely used by UX researchers. This scale has 10 items with response options on a 5-point Likert scale (1 for “strongly disagree”) and scores ranging from 0 to 100 (Table 2). Higher scores indicate higher levels of usability. Scores below 70 indicate “below average” usability.¹⁹

Table 2.

Participant Outcomes on the System Usability Scale and Critical QuitPal Tasks

			Setting up quit date		Tracking cigarettes
	SUS	Diagnosis	Time on task	Guidance prompts	Time on task	Guidance prompts
P1	50	Schizophrenia	8′5″	18	6′57″	12
P2	85	Schizophrenia	1′20″	0	2′32″	3
P3	93	Schizoaffective	4′	1	2′43″	0
P4	73	Depression, Recurrent	3′3″	2	1′42″	0
P5	66	Bipolar	6′28″	18	5′50″	14
Avg.	65.5		4′30″	7.8	3′36″	5.8

Guidance prompts = number of guidance prompts provided by interviewer to help participant complete each required task; SUS = total score on the Systems Usability Scale; Time = time on task measured in minutes and seconds.

Time on Task

We measured time on task by subtracting the time at which a task was completed minus the time at which an instruction was given. Since “think aloud” procedures interfere with normal task efficiency,²⁰ time on task measurements need to be interpreted in the context of other measures.

Guidance Prompts

We coded participant’s performance based on the amount of verbal prompts they needed to complete a task. Examples of guidance prompts included statements such as “What do you think this icon is for?” or “Did you save the information?” We defined “completed without guidance” as being able to complete the task without any errors and without the need for guidance.

Daily Diary

Paper diaries consisted of a single sheet of paper with two blocks of four questions (one for each day), and space for writing. Each block asked open-ended questions that covered the following areas: app content that participant might have liked the most or least and why, specific problems encountered during the use of the app, and places and times of the day where they might have enjoyed the most using the app.

Semi-Structured Interview

We wrote interview questions a priori and covered the following areas: (1) navigation and design of the app features, (2) interest in using the app to quit smoking, (3) utility of specific app features, (4) experienced barriers to using the app, (5) situations in which using the app might have been pleasant or engaging, and (6) overall suggestions to make the app more useful or engaging.

App Entry Logs

Participants kept logs of smoked cigarette during the 2-day field experience using the QuitPal cigarette tracking function.

Data Analytic Strategy

We recruited five individuals to participate in the study, as research suggests 85% of usability problems are detected with the first five users.²¹ To interpret and integrate data from different sources we triangulated data from task performance, field experience, app logs, paper diary, semi-structured interviews and questionnaire.²² Triangulation ensures that the design failures of a system are captured with a range of methods and experiences. For example, while many UX studies are exclusively conducted in the laboratory, these studies cannot identify design flaws that might have only arisen after a prolonged and ecologically valid experience with a system (eg, in the course of user’s daily life). Therefore, our study design balanced direct observation and ecological validity. Finally, we analyzed interviews using affinity diagrams,²³ a grounded field theory approach²⁴ that inductively extracts design themes at the individual level, and later on clusters them across individuals based on similarity, dependence or proximity. This dynamic and visual method allows the identification and contrast of emerging themes and reveals opportunities for innovation.²²

Results

Recruitment and Compliance With Procedures

Between November 2014 and January 2015 we received a total of 10 calls from study candidates. Three individuals did not meet eligibility criteria, one individual scheduled but did not attend the appointment, and one individual called the office but did not provide contact information. Our five target users (Table 1) had an average age of 51 (SD = 4.3) and had received mental health services for at least 3 years (M = 18.6; SD = 14.8). We recruited all participants from a mental health clinic located in a public, safety net medical center dedicated to serve individuals with serious mental illness. Participants P1 and P2 reported a diagnosis of schizophrenia, P3 schizoaffective disorder, P4 recurrent major depression without psychotic features, and P5 bipolar disorder. The presence of “extreme” users in our sample is a highly desired quality in UX research.²⁰ Consistent with this standard, participants ranged between high and low levels of literacy, high and low levels of fine motor skills and both mild and more severe mental health symptoms.

Table 1.

Baseline Characteristics of Participants Testing QuitPal (n = 5)

Demographics
Age, mean (SD)	51.2 (4.27)
Male	5 (100%)
High school or less education	2 (40%)
Caucasian	1 (20%)
Smoking behavior
Light smoking (5–10 cigarettes per day)	4 (80%)
Heavy smoking (≥11 cigarettes per day)	1 (20%)
Years smoking, mean (SD)	28.8 (11.08)
Mental health
Number of years in treatment, mean (SD)	18.6 (14.84)
Diagnosed with a thought disorder (eg, schizophrenia)	2 (40%)
Diagnosed with a mood disorder (eg, bipolar)	3 (60%)

Participants returned all iPod devices with their accessories, their compliance with study procedures was generally positive (eg, all participants used the app during the 2-day field experience), and none of the devices were damaged. However, two participants did not fill out the diary cards, and most participants, except for one, demonstrated low compliance with the “think aloud” procedure.

How Usable is QuitPal Among People With Serious Mental Illness?

Objective Measures

System Usability Scale

Average score was 65.5 (SD = 18.6), five points below the industry standard. These scores however had a wide range (Table 2).

Time on Task

Average time on task for setting up a quitdate and tracking their cigarettes were 4.5 minutes (SD = 2.69), and 3.6 minutes (SD = 2.1), respectively.

Guidance Prompts

Average guidance prompts for setting up a quitdate was 7.8 (SD = 9.3) whereas tracking cigarettes required an average of 5.8 (SD = 6.7).

App Usage Logs

All participants tracked their cigarettes on a daily basis during the 2-day field experience. Their smoking levels were consistent with self-report. In on case (P1), we observed data entry errors were observed and confirmed this during the interview. This participant logged 272 cigarettes in 1 day, which was consistent with this individual’s difficulty to use touchscreen technology.

Task Performance and Think-Aloud Themes

Entering Information in the App

A common critical error observed was failure to enter information in QuitPal. Users had trouble identifying the “plus” button in several of its features (Figure 3). All participants required direct guidance to find this button in several of the app screens. The participant most familiar with software and mobile devices (P4) commented, “that red plus thing it wasn’t much intuitive…even for me.” One participant (P2) complained about the design of the “I was smoke free today” button (Figure 3) since “you have to wait a whole day to press it.” Other problems with entering information in the app included (1) not being able to “pull up the keypad” (P5), (2) not being able to see the different logging options to track cigarettes (eg, mood at the time of smoking a cigarette), (3) pressing the different interface buttons (P2: “[keypad buttons are] too close together”).

Saving Information in the App

QuitPal requires saving each data entry (similar to Microsoft Word processors). This caused several critical errors. During completion of one of the tasks, one participant (P2) entered how many cigarettes he smoked twice because he did not click the “save” button. He also struggled to identify a method to save his “saving goal,” one of the app features. Another participant (P3), failed at saving a customized reminder he created. In addition, the “save” button was not consistently present in all of the app features (“Savings goal”), which was a source of confusion and led to additional complains and errors (P1 and P4).

Lost in Layers

Difficulties with the QuitPal’s layer structure was a frequent noncritical error that we observed across all participants. One participant needed multiple prompts and guidance to return to the mainscreen interface (P1: “it took me a long time to get back to that menu frame”) and another participant (P5) directly stated that the number of layers and steps to accomplish tasks was excessive.

Levels of Familiarity With the Use of Technology

Three participants were familiar with mobile technology (owned Android or iOS smartphones), and all of them owned or had owned a mobile device in the past. Lack of familiarity with mobile technology was an important barrier for one participant (P5) as he expressed concerns about familiarity with touchscreen technology: “I don’t have one of those touchstone phones,” and “I’m not used to the skills of touching [screens].”

Tremor and Fine Motor Skills

We observed hand tremors in two participants (P2 and P5). This aspect had some influence on their ability to complete tasks, although one participant (P2) compensated by his high levels of familiarity with smartphone technology. One participant (P5) suggested a solution to this problem: “Better to have a stylus, have a stylus that goes with it […] my fingers are so clumsy.” In fact, another one (P3) pulled up an electronic pen to complete the task.

How Do People With Serious Mental Illness Experience the Use of QuitPal?

The following themes emerged after completion of the 2-days field experience and during semi-structured interviews (Table 3).

Table 3.

List of Emerging Themes From User Experience (UX) Testing With QuitPal

Performance themes	Field experience themes
1. Entering information in the app	1. Incremental rewards and emphasis on small gains
2. Saving information in the app	2. Only a few app features were used
3. Lost in layers	3. QuitPal served as an awareness tool
4. Familiarity with technology	4. Lack of more elaborate skills to quit smoking
5. Tremor and fine motor skills	5. Lack of finer-grained cigarette tracking
	6. Notifications and reminders
	7. The importance of coaching and assistance
	8. The need for interactive and motivating features

Incremental Rewards and More Focus on the Process of Cutting Down Rather Than Quitting

All participants mentioned monetary incentives as being very motivating for them (P5: “if you [were] smoking a pack a day…you saved $50 or $60 [per week]…, and that’s quite a bit of money…”). However, they preferred a more incremental approach to incentives and some felt that the “savings goal” feature seemed to be tailored for larger rather than smaller financial goals (P2: “the piggy bank…doesn’t give me much incentive”). This brought up issues with how the app seemed to be very focused on “quitting,” rather than helping people cut down (P5: “just to quit smoking is unsurmountable… better if you can break down the habit”), and complained they were only able to see their progress once they had past their quit date. They also suggested nonmonetary rewards, like a “reward myself” system that would allow people to treat themselves with food or activities they like (P2 and P4).

Only a Few App Features Were Used

Our interviews indicated that the tracking feature was by large the most used across participants (P1: “I could keep up with the count of how many cigarettes I’m smoking”) and participants used few other features during the field experience. When asked if other features were important, one participant stated that “you don’t need the rest, […] yeah if you don’t use them… what’s the sense in being there?” (P1). The key issue for participants was that the app did not activate many features until one passed the quitdate (P2, P3). For example, graphs and visual features were inactive prior to quitting: “they are good […], but there’s no reference point to utilize them […]” (P3).

QuitPal Served as an Awareness Tool

For some participants the act of turning on the mobile device reminded them of their quitting goals (P2: “just by having the phone makes you think a little bit more, more aware”) and added that this awareness could be used to cut down on a daily basis. One participant started the interview by stating the app made him more mindful (P3), and another one emphasized it was important for him to keep track of his mood and context while smoking (P5: “you got to be aware which triggers you are going to go with and which ones [not]”).

Lack of More Elaborate Skills to Quit Smoking

One participant emphasized the app lacked more elaborate skills for quitting (P4: “a short guided meditation to help you deal with the craving… […] I think that’s pretty good tool for dealing with addictions”), and most participants indicated the importance of psychological skills to break down the habit (P4: “you wanna smoke a cigarette, you are stressed out, […] you don’t know what to do here… there’s a CBT skill…”).

The Need for Finer-Grained Cigarette Tracking

Participants emphasized the current tracking feature did not fit their smoking habits (P3: “I listed one cigarette because I took a couple of drags […] I couldn’t list half a cigarette.”) and therefore lacked sensitivity (P4: “[I] smoke about half a cigarette at a time and then put them out, save the other half for later”).

Notifications and Reminders

It seemed like most participants had a basic positive attitude towards the reminder function, but saw different ways of using it for their own purposes. One participant (P1) emphasized being able to have a visible quitdate as a reminder and other participants attempted to create personalized reminder messages for themselves (P3: “when I get to the store don’t buy a pack of cigarettes,” and P5: “delay cigarette”). However, the notifications feature created different reactions. One participant (P4) indicated the reminders felt like “nagging,” whereas another would have preferred enhanced notifications (P2: “[Read] the fact of the day”).

The Importance of Coaching and Assistance

One participant (P1) indicated learning to use the app required additional face-to-face guidance: “At least a day trying to show the person how to work with, not just a few minutes…” and added having someone to call would have been helpful. This element was made very obvious by another participant (P5), who left a voicemail to study staff the day after the first meeting requesting assistance (P5: “It’d be better […] to come in and be able to go through the very steps until I got whatever I was trying to do”).

The Need for Interactive and Motivating Features

Most participants saw the need to increase the motivational design of the app in order to serve as an encouragement tool (P1: “you are doing good, you are not smoking now.”) When suggested the possibility of a feature that provided points one participant (P2) stated “yeah that would be good like a medal […] like in a game, you have first place on this level,” but added that his preference would be a system that combines both “play” and “seriousness” since “if I treat it as a game […] I won’t be too serious about it.” When we inquired whether a more “playful” component would dissuade other people with serious mental illness use the system, another participant (P4) stated “I don’t think people care how old you are for interactive stuff… we all like that stuff,” and added “it seems like all my friends [whose age range from] twenty something all the way up to…60s and 70s, […] play that game [candy crush saga] … it’s really addictive.” Finally, another participant (P5) showed preference for an app that would combine both the gaming component as well as tools (“you can think of it as a tool or you can think of it as play thing…, as a recreational thing…, and it is not really recreational it is really like a tool”).

Discussion

The aim of this study was to evaluate the usability and likability of a smoking cessation app among people with serious mental illness. Results indicate QuitPal had below average levels of usability, elevated time on task performances, and required face-to-face guidance. Observations of individuals’ performance while using the app indicated the app’s design was conducive to a number of critical and noncritical errors, such as entering information in the app and saving it for later. Although one of the usability metrics (System Usability Scale) was notably high for some participants, our in depth study design was able to show that participants’ actual performance required guidance and considerable amount of time.

In-depth interviews and observational data suggested a number of important themes relevant to the psychological experience of QuitPal in this population. Amongst them, the most important were the need for an app that breaks down the process of smoking cessation into smaller steps and targets, and the relevance of an interface that highlights the use of a reward system (including monetary incentives). Most participants considered only a few app features were essential (ie, tracking), and their engagement with other app features was minimal. In addition, participants considered QuitPal’s tracking feature functioned as an awareness tool. Less common themes included the importance of coaching and assistance, the need for fine-grained cigarette tracking (ie, half a cigarette at a time) and differences in opinion about the use of reminders (eg, “nagging” vs. useful).

Results of this study are consistent with previous research on the development of e-Health and m-Health interventions among people with serious mental illness. For example, several authors have recommended the implementation of minimal steps in order to access content,^25–27 interfaces with large buttons,²⁵ avoiding of scrolling options,²⁷ and maximizing design consistency,²⁷ all of which was supported by our findings. Some authors also emphasized the use of reminders.²⁵ Our study generally supported their use, with the caveat that some individuals perceived it as intrusive.

The study also shed light to specific themes recently identified in a UX research of smoking cessation apps by Paay et al.²⁸ but never reported in people with serious mental illness. These include great interest in tracking nicotine use, the appeal of successive approximations to quitting, the importance of monetary incentives, the need for new approaches to quitting (eg, cognitive behavioral and mindfulness skills to quit), preference for an app that incorporated interactive and motivating features (eg, gaming) and ambivalence about the use of reminders. Unlike previous studies, our study emphasized a series of design elements that are unique to mobile systems and to smokers with serious mental illness. For example, our study emphasized the importance of coaching and assistance (Paay et al.²⁸ users preferred the “solo struggle”) and identified usability issues and opportunities for innovation that are unique to our population, such as problems with data input and navigation, preference for tracking “half” cigarettes, and how to the address the problem of tremors, a common side effect of certain psychiatric medications (ie, stylus).

Limitations

Our study had a number of limitations. First, despite the fact that the use of five individuals per test is a standard and highly cost-effective practice in UX research,^21,29 the sample was very small and therefore our results might not generalize to this population. Further, some authors have suggested that usability results can be improved with larger samples.³⁰ This limitation was counterbalanced by the fact that our sample came from a diverse group of individuals with different degrees of functional outcome, different types of mental disorder and different smoking histories, which enriched our findings and ensured we gathered data from a broad spectrum of individuals within this population.²⁰ Second, we did not screen our sample using diagnostic interviews, which could have provided a more reliable and detailed picture of the characteristics of our sample and how their psychiatric status might have influenced their experience with QuitPal. Likewise, a measure of cognitive functioning could have improved interpretation of participant’s task performance and interview data. Third, participants used the app during a very limited period of time. Using the app for a longer period could have provided a different and more nuanced level of participant input. In addition, our think-aloud procedure failed to work across participants. Most participants struggled to coordinate their speech while conducting a task, which suggests that this UX method might not be well suited for people with serious mental illness. Data triangulation across UX metrics and a 2-day field experience with QuitPal addressed these methodological problems and enriched our data considerably. UX evaluation commonly relies on laboratory interviews, therefore, our study design provided deeper observational data and insights into the use of this smoking cessation app.²³ Finally, as is typical from UX research and other forms of ethnographic research, interviewer’s personal characteristics (eg, gender, investigator role) could have influenced study procedures and interview results. Further, the fact interviews were recorded might have elicited certain anxiety and influenced their responses to questions. Data triangulation might have reduced this influence, but it is important to note this potential source of bias.

Conclusions

Our study is the first UX evaluation of a smoking cessation app amongst people with serious mental illness. Results of this study may inform future research efforts to design user-centered apps for people with serious mental illness, and therefore contribute to expanding the effectiveness and reach of smoking cessation interventions in this population.

Funding

Support for the writing of this document was provided by a grant from the National Institute on Drug Abuse to RV (1K99DA037276-01) and from the National Institute on Alcohol Abuse and Alcoholism to MGM (R01AA020248-01).

Declaration of Interests

None declared.