User Experience Affects Dropout from Internet-Delivered Dialectical Behavior Therapy

Abstract

Background: The emergence of computerized treatment may help reduce the gap between mental health treatment needs and accessibility, but unfortunately, dropout from these interventions is often high.

Introduction: To increase the effectiveness of computerized interventions and reduce dropout, particularly among high-risk and clinically complex populations, better understanding of how usable and acceptable (i.e., user experience) these interventions are, informed by human computer interaction research, is needed. This study examines user experience of internet-delivered dialectical behavior therapy (iDBT). The major aim is to explore whether treatment dropout was affected by the complexity of population and/or user experience.

Methods: Secondary analyses were conducted using data from a randomized controlled trial that evaluated iDBT in a sample of 59 suicidal and heavy episodic drinkers. Multivariate logistic regression and chi-square tests were performed to examine the roles of clinical characteristics and user experience in differentiating dropouts and nondropouts.

Results: The only significant pretreatment predictor of dropout was the presence of a barrier, with technological and unknown barriers being most strongly associated with dropping. No clinical characteristics emerged as significant predictors of dropout.

Discussion: The current results highlight technological problems as a possible barrier to adherence to computerized interventions. Future research would profit from increased integration of human–computer interaction to identify and solve user experience problems.

Introduction

Considerable interest exists regarding the implementation of technology to deliver psychological intervention such as computerized treatments (CTs). Despite the proliferation of randomized controlled trials (RCTs) evaluating CTs, the bulk of these interventions are evaluated among populations with little clinical complexity,^1–3 which may not be representative of those receiving treatment outside the research laboratory.⁴ Furthermore, CTs have high dropout rates (e.g., 30–60% dropping),^2,5,6 and patient complexity has been found to be associated with treatment dropout in previous trials of CT.^7,8 As such, understanding of the user experience of complex patient populations is needed.

Unfortunately, user experience is often ignored in CT outcome research. Within the field of computer science, human–computer interaction (HCI) research is designed to better understand the interaction between humans and computing technology. A review examining both HCI and psychotherapy outcomes in CT found that HCI outcomes have rarely been explored.⁹ Given high-dropout rates that are common across CTs, increased attention to HCI may help reduce dropout by improving intervention user experience.

The goal of this study is to examine user experience outcomes in the context of one CT: internet-delivered dialectical behavior therapy (iDBT) for suicidal and heavy episodic drinkers.¹⁰ In this study, we examine whether treatment dropout was affected by clinical complexity and/or user experience by conducting a secondary analysis of an RCT of iDBT.¹⁰

Methods

An in-depth explanation of the study participants has been published elsewhere.¹⁰ Participants were 59 adults recruited for a pilot RCT evaluating iDBT for suicidal and heavy episodic drinkers (Table 1).

Table 1.

Participant Demographics and Clinical Characteristics at Baseline

DEMOGRAPHIC CHARACTERISTIC
n = 56a	n (%)	M	SD
Age		38.0	10.4
Gender
Male	18 (30.5)
Female	41 (69.5)
Race
Caucasian	46 (82.1)
African American	8 (14.3)
Asian	2 (3.6)
Education
Some high school	2 (3.6)
High school/equivalent	10 (17.9)
Some college or training beyond high school	23 (41.1)
College graduate	17 (30.4)
Beyond college	4 (7.1)
Income year
<$15,000/year	23 (41.07)
>$15,000/year	33 (58.93)
Marital status
Married	9 (16.1)
Not marriedb	47 (83.9)

^aA total of 56 out of 59 participants provided demographic information.

^bSingle, divorced, separated, or widowed.

Measures

Sociodemographic and clinical characteristics

Participants' demographic information was obtained using Demographic Data Schedule—Short Version (DDS [Linehan M. Demographic data schedule (DDS), (Unpublished manuscript). Seattle, Washington: University of Washington, 1982.]). Suicidal ideation was assessed using the scale for suicidal ideation.¹¹ The severity of alcohol consumption was assessed using the Alcohol Use Disorders Identification Test.¹²

User experience

Participants responded to two single items measuring their urge to quit and perceived usefulness of session on a 5-point Likert scale at the end of each iDBT session. Higher scores indicate higher urges to quit and usefulness, respectively. If a participant missed a session and/or rated an elevated urge to quit, he or she was contacted by phone and assessed on factors associated with discontinuing the intervention (e.g., lack of interest, technical, and access problems). Participants who were unreachable or did not report were coded as having an “unknown barrier.”

Statistical Analyses

A participant was considered a dropout if he or she missed two sessions in a row and did not recommence iDBT. We conducted multivariate logistic regression with treatment dropout (yes/no) as the dependent variable and clinical (alcohol use and suicide ideation) and user experience variables (barriers, urge to quit treatment, and perceived usefulness) as the independent variables using R Project. To increase the reliability of p-values in the logistic regression, statistical significance was corrected using permutation testing (1,000 resampling) through the function permTest(). Betas from logistic regression were exponentiated and are presented as odds ratios.

Results

Thirty-two (54.24%) individuals were considered dropouts. In general, participants reported high usefulness and low urges to quit throughout the sessions (Table 2). Multivariate logistic regression revealed that alcohol use severity, suicide ideation, and urge to quit at session 1 were not associated with dropping from treatment (Table 3). For each unit decrease in the rating of session 1 usefulness, there was a 22% increase in the likelihood of dropout (OR = 0.88, permutated p = 0.052). Having a barrier was the most robust predictor of dropout (OR = 4.63, permutated p < 0.001). A chi-square test was conducted to examine which type of barrier affected dropout. Technological problems (χ² = 5.38, p = 0.02) and an unknown barrier (χ² = 9.74, p = 0.002) had the strongest association with dropout status, whereas difficulty accessing the intervention (χ² = 2.29, p = 0.13) and a lack of interest (χ² = 1.75, p = 0.79) did not emerge as significant factors.

Table 2.

Number and Percentage of Individuals Attempting Sessions and Endorsing Homework Completion

SESSION NUMBER	ATTEMPTED, n (%)a	COMPLETED, n (%)a	USEFULNESS, M (SD)a,b	URGE TO QUIT, M (SD)a,b
Session 1	53 (89.83)	50 (84.47)	3.69 (1.18)	1.78 (1.06)
Session 2	44 (74.58)	37 (69.71)	3.69 (1.14)	1.92 (1.34)
Session 3	38 (64.41)	29 (49.15)	4.08 (1.23)	1.97 (1.49)
Session 4	35 (59.32)	25 (42.37)	4.09 (0.90)	1.97 (1.56)
Session 5	28 (47.46)	23 (38.98)	4.00 (1.21)	2.00 (1.50)
Session 6	26 (49.15)	23 (38.98)	4.00 (1.18)	1.80 (1.54)
Session 7	22 (37.29)	19 (32.20)	3.76 (1.03)	1.42 (1.44)
Session 8	18 (30.51)	18 (30.51)	3.94 (1.20)	1.50 (1.00)

^aDenominator represents the number of people who attempted session.

^bScale is from 0 (low) to 5 (high).

Table 3.

Associations of Clinical Complexity and User Experience Factors by Dropout Status

FACTOR	NONDROPOUTS (N = 27)		DROPOUTS (N = 32)
	M (SD)	n (%)	M (SD)	n (%)	OR	PERMUTATED p-VALUE
Clinical complexity
AUDIT total score	21.43 (8.06)		23.28 (8.69)		0.99	0.92
SSI	13.12 (8.58)		14.79 (6.88)		1.05	0.45
User experience factors
Barrier		5 (18.5)		31 (96.9)	0.37	0.004
Session 1 urge to quita	1.85 (1.35)		1.75 (1.18)		0.77	0.29
Session 1 usefulnessa	3.78 (1.01)		3.00 (1.86)		0.82	0.052

^aScale is from 0 (low) to 5 (high).

AUDIT, Alcohol Use Disorders Identification Test; SSI, Scale for Suicide Ideation.

Discussion

There were two main findings from this study. First, the analyses revealed that individuals who dropped from iDBT prematurely did not differ clinically from individuals who did not drop. Second, the only pretreatment predictor of treatment dropout was the presence of technological and unknown barriers. In addition, individuals who dropped from treatment found the first session slightly less useful than individuals who did not drop.

Our results suggested that participants generally found iDBT useful and endorsed low urges to quit throughout the sessions, with some indication that the initial ratings of usefulness may drive retention rates. In previous research, dropout within CT has been found to be associated with clinical complexity.^7,8 However, we did not see a similar pattern in current results. Specifically, technological barriers were the largest contributors to dropout, highlighting an important factor to consider in this field.

These results need to be interpreted within the context of several limitations. For one, our sample size was small and with variable (i.e., complex) clinical presentations. In fact, the impetuous for the original RCT was to show that internet-delivered interventions can be conducted safely and feasibly with individuals with complex behavioral dysfunction.¹⁰ In addition, one of the largest predictors of dropout that emerged from our analyses was an “unknown barrier,” which levies more questions than answers.

The use of technology to increase access to high-quality interventions represents an exciting innovation in mental health treatment research. However, as researchers continue to explore technology-delivered treatment, many implementation challenges will need to be concurrently addressed to increase likelihood of clinical effectiveness. For example, previous research has shown that manipulating engagement strategies (e.g., levels of personalization in push notifications) was associated with faster improvement in depression and anxiety.¹³ Researchers in this area should consider user experience as a potential mediator when they develop and evaluate technology-delivered interventions. To improve retention, it would behoove researchers in m-health and e-health to borrow from the field of HCI to identify and solve user experience problems within these platforms that may be contributing to treatment dropout.

Disclosure Statement

No competing financial interests exist.

Funding Information

This study was supported by National Research Service Awards from the National Institute on Alcohol Abuse and Alcoholism (F31 AA 24658-01; PI: C.R.W.) and a pilot grant from the Alcohol Drug Abuse Institute (ADAI-0315-2, PI: C.R.W.).