A Mixed-methods Evaluation of the Feasibility, Acceptability and Preliminary Efficacy of a Mobile Intervention for Methadone Maintenance Clients

Abstract

Despite the recent explosion of behavioral health interventions delivered on mobile devices, little is known about factors that make such applications practical, engaging and useful to their target audience. This study reports on the feasibility, acceptability and preliminary efficacy of a prototype of a novel, interactive mobile psychosocial intervention to reduce problematic drug use among clients in methadone maintenance treatment (MMT). A mixed-methods pilot study with new MMT clients (n=25) indicated that the mobile intervention approach was feasible, and that participants found the intervention highly acceptable and useful. On 100-point visual analog scale (VAS) items, participants reported high levels of liking the program (M=75.6), and endorsed it as useful (M=77.5), easy to use (M=80.7), and containing a significant amount of new information (M=74.8). When compared with 25 study participants who received standard MMT alone, pilot participants rated their treatment significantly higher in interestingness and usefulness, and were significantly more satisfied with their treatment. In qualitative interviews, participants reported using the mobile intervention in a range of settings, including during times of heightened risk for substance use, and finding it helpful in managing drug cravings. Additionally, pilot participants showed evidence of increased treatment retention and abstinence from illicit opioids (in terms of effect size) over a 3-month period relative to those in standard MMT, suggesting the application’s potential to enhance treatment outcomes. These promising findings suggest that an evidence-based mobile therapeutic tool addressing substance use may appeal to drug treatment clients and have clinical utility as an adjunct to formal treatment.

Approximately 681,000 Americans reported using heroin in 2013, according to the latest National Survey on Drug Use and Health, a 69% increase from the 2002 rate (SAMHSA, 2014a), while more than 11 million people misused prescription opioids in 2013 (SAMHSA, 2014b). Given the recent increase in the number of Americans reporting nonmedical use of prescription opioids (SAMHSA, 2014a) – and emerging evidence that a growing proportion of nonmedical prescription opioid users are transitioning to heroin use (Frank, Mateu-Gelabert, Guarino, Bennett, Wendel, & Teper, 2014; Jones, 2013; Mars, Bourgois, Karandinos, Montero, & Ciccarone, 2013; Mateu-Gelabert, Guarino, Jessell, & Teper, A., 2014) – the demand for effective treatment for opioid use disorders is high and likely to intensify in the near future. Yet the vast majority of those who could benefit from treatment do not receive it. This may be due to the limited availability of existing treatment programs and to a range of societal and individual-level barriers, including the stigma associated with identifying oneself as a substance user and difficulty attending a treatment program on a regular basis (Campbell, Muench & Nunes, in press; Cunningham, Sobel & Chow, 1993; Rapp, Xu, Carr, Lane, Wang, & Carlson, 2006). In addition, drug treatment programs face numerous barriers to providing comprehensive, evidence-based psychosocial treatment to their clients, including limited financial and staffing resources, high staff turnover and high-volume clinician-patient caseloads (D’Aunno & Vaughn, 1995; Kraft, Rothbard, Hadley, McLellan, & Asch, 1997). Although methadone maintenance treatment (MMT) has been demonstrated to be one of the most efficacious treatment modalities for opioid-dependence, MMT programs offer the lowest amount of psychosocial treatment and have the highest client-to-staff ratio relative to other types of drug treatment, such as outpatient counseling, inpatient rehabilitation and residential programs (Etheridge, Craddock, Dunteman, & Hubbard, 1995).

Technology-based interventions have strong potential to help address these barriers and to expand access to evidence-based psychosocial treatment both within MMT and in drug treatment more generally. Relative to traditional, face-to-face treatment modalities, technology-based interventions can offer greater privacy, convenience, reach and cost-effectiveness Cucciare, Weingardt, Greene, & Hoffman, 2012; Cucciare, Weingardt & Humphreys, 2009; (Marsch, Carroll & Kiluk, 2014). These benefits may be especially relevant in MMT, where patient attendance burden is high and staff and financial resources are strained. In addition, technology-based therapeutic tools, especially those utilizing mobile technologies, have the potential to deliver personalized, on-demand treatment to individuals with problematic substance use, providing support in real-world contexts, at times and in situations when it may be most needed (Marsch, 2012).

There is a nascent, but growing, evidence base demonstrating the efficacy of technology-based interventions for substance use disorders (SUDs), especially when they are used as adjuncts to traditional, face-to-face treatment modalities. Meta-analyses and systematic reviews of technology-based interventions for SUDs have consistently found that such tools are associated with significantly better outcomes than minimal or no-treatment controls and may be equivalent in efficacy to other evidence-based psychosocial treatments, such as clinician-delivered Cognitive Behavioral Therapy and Motivation Enhancement Therapy (Campbell, Muench & Nunes, in press; Litvin, Abrantes & Brown, 2013; Moore, Fazzino, Garnet, Cutter, & Barry, 2011; Quanbeck, Chih, Isham, Johnson, & Gustafson, 2014). However, the preponderance of research in this area has focused on web-delivered interventions for problem drinking and smoking cessation, while fewer technology-based interventions have been developed and evaluated for other SUDs, including opioid dependence. The research literature is also sparser for web-based tools that are accessible on mobile devices as compared to desktop-based web applications (Moore et al., 2011).

Multiple studies and reviews of technology-based tools for a range of behavioral health conditions, including SUDs, have generally found significant correlations between use of, or “engagement” with, an intervention and positive outcomes, with individuals who display higher rates of intervention use being most likely to evince successful treatment outcomes (Bennett & Glasgow, 2009; Freyne, Saunders, Brindal, Berkovsky, & Smith, 2012; Gainsbury & Blaszczynski, 2011; Glasgow, Christiansen, Kurz, King, Woolley, Faber et al., 2011; Gustafson, McTavish, Chih, Atwood, Johnson, Boyle et al., 2014; Litvin, Abrantes, & Brown, 2013; Strecher, McClure, Alexander, Chakraborty, Nair, Konkel et al., 2008). However, effectively motivating members of a target population to engage with technology-based tools, and particularly to sustain engagement over time, remains a challenge. Similar to standard face-to-face treatments, user attrition has been a salient problem in research evaluations of technology-based interventions, with a 50% attrition rate in the active intervention period a common finding (Donkin & Glozier, 2012; Freyne et al., 2012; Kelders, Kok, Ossebaard, & Van Gemert-Pijnen, 2012; Litvin, Abrantes & Brown, 2013). Study retention and intervention adherence rates have generally been worse in purely technology-based trials that include no face-to-face interaction with research or clinical staff. While this may indicate that factors beyond the acceptability of the intervention itself may be involved, it does suggest the important role of human/provider contact in the effective implementation of technology-based interventions. Together, these findings underscore the importance of systematically examining the acceptability and appeal of an intervention from the end user’s perspective, as individuals are unlikely to use tools they do not like (Kay-Lambkin, Baker, Lewin, & Carr, 2011). In-depth investigations of users’ experiences with technology-based behavioral health tools may help identify program features that reliably promote sustained engagement, thus facilitating the development of tools that are maximally appealing, engaging and effective.

Initial research in this area has identified certain features of technology-based behavioral health tools and their implementation that are associated with increased engagement, including individual-level personalization of content, “push reminders” (e.g., automated text messages or emails) to promote use of a tool, provision of incentives for use of a tool, and integration of human/social support and encouragement (Mohr, Cuijpers & Lehman, 2011; Strecher et al., 2008). However, extant process research on participants’ experiences with technology-based interventions for SUDs is quite limited, and studies applying qualitative or mixed methods to this issue are particularly scarce. A qualitative or mixed-methods approach to process research is especially well-suited for gaining insight into participants’ experiences and perspectives, understanding how local contexts influence these experiences, and uncovering salient themes that may be unanticipated at the start of an investigation. More research of this type is needed to deepen our understanding of the specific intervention features and components that best promote acceptability and sustained engagement with technology-based interventions for SUDs.

The current study was conducted within the context of a larger randomized, controlled trial to evaluate the effectiveness of a web-based psychosocial program called the Therapeutic Education System (TES; Bickel, Marsch, Buchalter, & Badger, 2008) when delivered to opioid-dependent individuals in MMT. TES is an interactive, customized program that is theoretically grounded in the evidence-based behavior therapy program for substance abuse known as the Community Reinforcement Approach. This approach aims to train individuals in skills and behaviors to help reduce problematic substance use and HIV/HCV risk behaviors, improve employment status and family/social relations, and increase non-substance-related recreational activities (Budney & Higgins, 1998). TES has been demonstrated to significantly improve treatment outcomes when delivered to individuals in a range of community-based substance abuse treatment settings, including MMT (Bickel, Marsch, Buchalter & Badger, 2008; Brooks, Ryder, Carise, & Kirby, 2010; Marsch, Grabinski, Bickel, Desrosiers, Guarino, Muehlbach et al., 2011; Marsch, Guarino, Acosta, Aponte-Melendez, Cleland, Grabinski et al., 2014; Campbell, Nunes, Matthews, Stitzer, Miele, Polsky et al., 2015).

As a supplement to this parent trial (Marsch et al., 2014), our team developed a prototype version of an interactive, mobile phone-based psychosocial intervention, based on key elements of the Community Reinforcement Approach and Cognitive Behavior Therapy, to promote skills acquisition and reduce illicit drug use among MMT clients. Designed to be theoretically compatible with TES in its focus on practical skills-acquisition and self-management of drug use, this mobile application, called the Check-In Program, was conceptualized as a “treatment extender”, enabling provision of flexible, on-demand therapeutic support to participants in settings and at times of their choosing.

The present mixed-methods pilot study reports on the feasibility, acceptability and preliminary efficacy of this novel application when delivered to a sample of participants in conjunction with web-based TES. We hoped to learn whether MMT clients would be interested in and readily able to use the mobile application, whether they would be able to retain their study phones in working condition for the duration of the intervention period and whether they would find the tool helpful in reducing or abstaining from illicit opioid use. To evaluate the application’s potential to enhance treatment outcomes, we also assessed participants’ opioid use (via urine toxicology) and treatment retention relative to a group receiving standard MMT alone.

The Check-In Program

The prototype version of the Check-In Program evaluated in this study consisted of two skills-based modules, a Functional Analysis module which aimed to help individuals identify their patterns of problematic substance use and the specific triggers to substance use they face in their day-to-day lives, and a Self-Management module which assisted individuals in developing a plan to manage the triggers to substance use identified in the Functional Analysis module. Each module contained explanatory text defining key concepts (e.g., “triggers”, “self-management plan”), followed by an interactive exercise. A standardized, daily text message prompt (“Don’t forget to use the Check-In Program to help you stay on track.”) was sent to each program user’s mobile phone to encourage use of the program.

Upon first accessing the program, users were required to complete both modules in their entirety at least once. Progress through the application at each subsequent use of the program was tailored based on responses to recurring queries presented at each login asking if the individual had used any opioids (besides his/her daily methadone dose) or been exposed to any high-risk situations that could lead to opioid use since his/her last login. Affirmative responses led the user into the exercise portion of the Functional Analysis or Self-Management module, respectively, while negative responses allowed the user to either log out of the program or electively complete either module’s exercise. At any time, users could elect to review the introductory didactic modular content.

In the Functional Analysis exercise, participants were asked to recall their most recent episode of non-MMT opioid use and then to identify their most salient triggers, or emotional, cognitive or environmental cues that increase the likelihood of engaging in problematic opioid use, including both internal triggers (e.g., certain thoughts or feelings) and external triggers (e.g., certain people, places or situations). Participants were then asked to identify the short-term and long-term consequences of their opioid use, both positive and negative. For all interactive items within both modules, users had the option of selecting a response from a checklist provided or selecting “other” and typing in their own response using the phone’s touchscreen keyboard.

In the Self-Management exercise, participants were asked to select one of their self-identified triggers and were guided through successive steps to develop a plan for managing this trigger in the future without resorting to opioid use. Self-management plans could involve strategies such as re-arranging one’s daily routine or environment to avoid the trigger or identifying alternative responses to the trigger that do not involve using opioids. In order to help motivate behavior change and prepare participants for potential challenges in managing triggers, participants were then asked to identify the likely positive and negative consequences of implementing their chosen plan. Finally, participants were asked to rate the difficulty of their plan, using a scrollbar to select a difficulty rating ranging from 1 (least difficult) to 10 (most difficult).

Method

Parent Study Procedures

In the parent trial, opioid-dependent adults (n=160) within their first 30 days of entering outpatient MMT at the study site, a large New York City MMT program, were randomized to receive either 12 months of 1) standard treatment or 2) a reduced schedule of standard treatment plus biweekly sessions using TES. Eligible participants had to be at least 18 years of age, meet DSM criteria for opioid dependence (required for entry into MMT), have sufficient English language ability to understand the study assessments and intervention content, and be enrolled during the first month of their current MMT episode. Details of the parent trial have been published elsewhere (Marsch et al., 2014; Acosta, Marsch, Xie, Guarino, & Aponte-Melendez, 2013).

Standard treatment included daily administration of a therapeutic dose of methadone (typically ranging from 80-120 mg/day). MMT was initiated immediately at treatment entry and usually required clinic attendance six days per week. Standard treatment also included the routine substance abuse counseling provided by the MMT program, i.e., meeting with one’s assigned counselor for 30-60-minute individual sessions once weekly during the first month of treatment and once monthly thereafter. Participants randomized to the reduced-standard treatment-plus-TES condition received the same methadone treatment plus standard counseling as those in the standard condition, except that one half of each counseling session was spent with their counselor, while the other half was spent using TES in an on-site computer lab.

Pilot Study Procedures

At the conclusion of participant enrollment in the parent trial, an additional sample of MMT clients (n=25) meeting the above eligibility criteria were enrolled and received reduced-standard treatment-plus-TES, as well as access to the mobile Check-In program, for the 12-week pilot study intervention period.

As a supplement to their biweekly computer-based TES sessions, each pilot study participant was provided with a Nokia feature phone preloaded with the Check-In application for use throughout the intervention period. Participants were given an initial, one-on-one tutorial (approximately 30 minutes long) in which research staff explained how to use key features of the Nokia phone and how to access, securely login to and complete the Check-In program.

Participants were encouraged to use the Check-In program at any time, as frequently as they wished, particularly in settings outside of their treatment program. Because the prototype Check-In program was designed as a native (not web-based) application, no wireless connection was necessary to use the mobile intervention. After the initial tutorial, participants were asked to meet with research staff on a weekly basis. During these sessions (about 15 minutes long, on average): Check-In program usage data were downloaded from the phones onto a central server (as there was no viable way to remotely extract data from the native application); prepaid minutes were added to mobile phone accounts; technical issues were resolved; and booster tutorials were provided, as needed. Research staff members were also readily available, by phone and in-person, to provide ongoing technical support and troubleshooting assistance to participants outside of regular weekly meetings, and detailed logs of these contacts were maintained.

Participants were offered a $50 incentive for returning their original study phones at the end of the study period. (This allowed research staff to reset the returned phones and distribute them to newly enrolled participants, thereby helping to contain costs.) Because an aim of the pilot study was to assess the feasibility of the mobile phone-based intervention delivery approach, a lenient phone replacement policy was adopted such that participants were not penalized for lost, stolen or broken phones, and missing phones would be replaced – multiple times, if necessary – as determined on a case-by-case basis.

All study procedures, for both the parent trial and the pilot study, were approved by the appropriate IRB, and all participants provided written informed consent prior to study enrollment.

Quantitative and Qualitative Data Collection

Participants’ demographics, as well as their drug use and drug treatment histories, were assessed at baseline with the Addiction Severity Index, a reliable, valid and widely-used measure of substance misuse and associated domains (ASI; McLellan, Luborsky, Cacciola, Griffith, Evans, Barr et al., 1985). An assessment battery including a structured Feedback Survey was administered at monthly follow-up time points for the duration of the 12-week study period (for a total of 3 Feedback Surveys per participant). Participants were compensated $50 for completing each assessment battery. Check-In program usage data were automatically saved in the mobile application’s back-end tracking system and manually downloaded onto the server during weekly research visits.

The Feedback Survey included 11 visual analog scale (VAS) items using a 0-100 scale in which 0 represented “not at all” and 100 represented “a great deal”. This measure was developed by our team from a version used in multiple prior studies evaluating technology-based interventions with substance-using populations. Six items asked both pilot study and parent trial participants how much they thought their assigned intervention: 1) was interesting; 2) was useful; 3) conveyed new information; 4) clarified any misunderstandings; 5) was easy to understand; and 6) was satisfactory. Five items asked pilot study participants only to rate the mobile Check-in program in terms of its: 1) ease of use; 2) likeability; 3) potential usefulness if more content were included; 4) helpfulness in reducing drug cravings; and 5) helpfulness in reducing the likelihood they would use drugs. (All items are summarized in Table 2). In addition, three open-ended items asked participants for their general comments on the Check-In program, suggestions for improvement and additional content areas for future expansions of the application.

Table 2.

Participants’ Treatment Ratings by Study Group

VAS Item	TES + Check-In Program Group (n=25) M (SD)	Standard Treatment Group (n=25) M (SD)
Items common to both groups
Interesting	71.7a (17.6)	49.4 (28.3)
Useful	77.5a (14.4)	52.3 (30.4)
New Information	74.8a (18.4)	46.2 (29.7)
Clarified misunderstandings	65a (20.9)	45.4 (25.7)
Easy to understand	58.5 (31.4)	61.8 (30.1)
Satisfied	77.2b (17.8)	60.5 (24.1)
Items unique to TES + Check-In group
Easy to use	80.7 (16.6)	--
Like	75.6 (18.6)	--
Useful if offered more content	82.3 (14.5)	--
Helped reduce drug cravings	63.2 (27.8)	--
Helped reduce likelihood you’d use	61.8 (27.6)	--

Note: All VAS items use a 0-100-point scale in which 0=not at all and 100=a great deal.

^aDifference between mean VAS scores is significant at the p<.0001 level.

^bDifference between mean VAS scores is significant at the p=.0004 level.

Participants provided weekly urine samples to research staff, with sample collection randomly observed by a research associate of the same sex. Using point-of-care qualitative urine test cups, each urine sample was screened for: THC, cocaine; barbiturates; benzodiazepines; methamphetamine; opiates; methadone; propoxyphene; and oxycodone (Drug Check Drug Test Cup, Drug Test Systems, Dover, NH). Only the urine toxicology results for opioids were used for the primary outcome analyses reported here. Participants were compensated $10 for each urine sample provided.

Qualitative feedback was systematically elicited from pilot study participants at each of the three monthly follow-up time points and during weekly study visits. These brief, semi-structured interviews consisted of open-ended probes designed to elicit participants’ experiences using the application, including: general impressions of the program’s usefulness and appeal; the settings in which they had used the program; barriers and facilitators to use of the program; technical difficulties encountered; program features they liked the most and least; the extent to which they found the program helpful in managing their substance use; and suggestions for future versions of the program. Detailed notes summarizing this feedback, including, to the extent possible, direct quotations, were recorded by research staff during and/or immediately following interviews.

Data Analysis

Descriptive statistics summarizing participants’ demographic, drug use and drug treatment characteristics, program usage metrics and responses to individual Feedback Survey VAS items (aggregated across all three time points) were prepared in Excel and SPSS. Because the Feedback Survey is not a validated and scored scale, each item was examined individually.

Key outcomes from the 25 pilot study participants were compared to the first 12 weeks of outcomes (a comparable time window) from the last 25 participants randomized to the standard condition of the parent trial (who were enrolled closest in time to pilot participants). For the 6 VAS items that were administered to both parent trial and pilot study participants, differences in mean VAS scores across conditions were assessed with t-tests conducted in SAS using the Mixed Procedure. In SPSS, the percent of participants retained in the study for twelve weeks was compared across the two conditions with the chi square technique, while a t-test was used to assess the difference in mean number of weeks of opioid abstinence. An opioid abstinent toxicology result was defined as a result negative for opiates, propoxyphene and oxycodone. For this analysis, missed urine tests were coded as opioid-positive. Since the study was underpowered for significance testing, as is typical of pilot studies, an effect size index (d) was also calculated for the opioid abstinence results, using the approach outlined by Cohen (1988), and a medium or large effect size was considered to indicate the potential efficacy of the combined computer TES-plus-mobile Check-In intervention.

Data from open-ended Feedback Survey items and field notes from brief semi-structured interviews were inductively coded for key themes, using an integrated code list to merge the two qualitative data sets. To help ensure the validity of the analysis, two team members with expertise in qualitative research methods (HG and YAM) developed the code list in collaboration, and then independently coded the entire dataset; the few discrepancies that emerged were resolved in consensus sessions. The number of times each theme was articulated by a different participant was calculated, allowing the themes to be arranged in rank order according to frequency.

Results

Participants

Pilot study participants (n=25) ranged in age from 21 to 61 (M=42.24, SD=12.54), and most were male (80%). They identified their race as Black/African American (44%), White (40%), or Other (16%), with 20% endorsing Hispanic/Latino ethnicity. Over a third (36%) reported having hepatitis C and 8% reported being positive for HIV.

The demographic characteristics of the last 25 participants assigned to the standard (MMT-only) condition of the parent trial were similar. Standard condition participants were 40.64 years old on average (R=22-63, SD=9.82) and 84% male. Forty percent identified their race as White, 36% as Black/African American, and 24% as Other, while 26% endorsed Hispanic/Latino ethnicity. Twenty percent reported being antibody-positive for hepatitis C, while 4% reported being HIV-positive.

Pilot study participants reported a mean of 17.28 years (range = 1-41; SD = 12.47) of regular use of their primary opioid, which, for the vast majority (92%), was heroin. (N.B. One year of “regular” use is defined, per the ASI, as use at least 3 times/week for at least 6 months). Poly-substance use was prevalent. In the 30 days prior to baseline, 44% reported using marijuana, 40% cocaine or crack and 36% sedatives, hypnotics or tranquilizers (largely benzodiazepines), while 16% reported using alcohol to intoxication. Participants reported a mean of 11.6 prior episodes of substance abuse treatment (range = 1-113; SD = 21.97), including all formal treatment modalities from inpatient detoxification and outpatient counseling to residential and medically-assisted treatment (MAT). All participants reported at least one previous episode of MMT, and 4 (16%) had previously been treated with buprenorphine. The mean number of prior MAT episodes was 7.92 (range = 1-61; SD = 11.9).

Participants varied in their baseline level of proficiency with mobile technologies. Virtually all participants owned mobile phones, but in many cases, these were traditional phones, not feature phones or smart phones similar to those used in this trial. Based on qualitative interview data, 9 participants had no prior experience with feature/smart phones, while 7 participants (generally those under age 35) were highly proficient in their use.

Feasibility of the Mobile Phone-based Intervention Approach

Findings suggest that delivering intervention content to MMT clients via a mobile phone is a feasible approach. Almost all participants, including those of older age and those with limited technological experience, were interested in and willing to try the mobile intervention. Ninety-two-percent (23/25) used the Check-In program at least once, with the majority using the program far more frequently (as detailed below). Of the two participants who did not use the Check-In program, one, whose work schedule made it difficult for him to meet with study staff, never received a study mobile phone and was discharged from MMT less than one month after enrolling in the study; the second spent much of the study period in inpatient drug treatment.

Nine participants (36%) required additional technical assistance (beyond the standard initial tutorial and brief weekly data download/check-in meetings) in order to successfully use the Nokia phone and/or the Check-In program. These 9 participants engaged with staff for supplementary coaching purposes for an average of 4.6 contacts (range = 2-10) over the 12-week study (the vast majority of these contacts were in-person, given participants’ daily clinic attendance). These contacts were generally brief; over the life of the trial, this subset of participants utilized an average of 45 minutes (range = 10-90) of staff time for technical support. The total staff time required for provision of technical support (beyond initial instruction and weekly maintenance activities) in this trial was 6 hours and 45 minutes (across two staff members).

At the end of the study period, 75% of participants returned the mobile phones that were provided to them for use during the study, with 44% returning their original study phone and the remainder returning a replacement phone. Despite our liberal phone replacement policy, the majority of participants (88%, 21/24) received only one or two study phones, providing further evidence of the feasibility of a mobile phone-based delivery method in an MMT population. It should be noted, however, that two participants had considerable difficulty retaining their study phones in operational condition and received multiple replacements for phones that had been lost, stolen or broken. Two additional participants who were living in “three-quarter” houses for individuals in substance abuse treatment reported that theft of personal property was a routine occurrence in these facilities which made it challenging to use the Check-In program at their residence without attracting unwanted attention.

Participants’ Use of the Check-In Program

Usage metrics automatically tracked by the mobile intervention indicate that participants used the Check-In application fairly frequently, averaging 29.8 total module completions (across the two program modules combined) over the 12-week study, with an average of 8.8 completions for the Functional Analysis module and 21.1 completions for the Self-Management module (see Table 1). The fact that participants used the Self-management module more frequently than the Functional Analysis module is, at least in part, an artifact of the application’s navigational structure such that, if participants responded negatively to the initial query asking if they had used opioids since their last login, but indicated that they had faced high-risk situations that might increase their likelihood of using opioids, they were automatically directed to the Self-Management module, bypassing the Functional Analysis module. Rates of program use were relatively stable during the three months of the study, averaging 8.9 (SD=6.9) total module completions in month one, 11.8 (SD=13.7) in month two and 9.2 (SD=14.1) in month three.

Table 1.

Participants’ Use of Check-In Program (n=25)

	Mean (SD)	Range	Median
Total Completions, Functional Analysis Module	8.8 (12.8)	0 – 52	3
Total Completions, Self-Management Module	21.1 (20.1)	0 – 67	15
Total Module Completions	29.8 (28.1)	0 – 109	20

Acceptability of the Mobile Check-In Program – Quantitative Findings

As presented in Table 2, pilot participants’ responses to Feedback Survey VAS items indicate that the mobile intervention was acceptable to, and perceived as useful by, participants. Participants reported high levels of satisfaction with (M=77.2; SD=17.8) and liking (M=75.6, SD=18.6) the program, and rated it positively on measures of usefulness (M=77.5, SD=14.4) and ease of use (M=80.7; SD=16.6). Participants also reported that the intervention contained a significant amount of new information (M=74.8; SD=18.4) and helped clarify misconceptions they had had about topics addressed in the program (M=65; SD=20.9). The program received moderate, but somewhat positive, scores for ease of understanding (M=58.5; SD=31.4). Moreover, participants felt that using the Check-In program helped reduce both their drug cravings (M=63.2; SD=27.8) and the likelihood they would use drugs (M=61.8; SD=27.6), and strongly agreed that the program would be useful if it offered additional topic areas of skills-building and support (M=82.3; SD=14.5).

Comparing pilot participants’ VAS ratings with those of the last 25 participants enrolled in the standard condition of the parent trial reveals that participants rated the reduced-standard treatment-plus-TES-and-Check-In program condition significantly better than standard MMT counseling on a number of indicators (see Table 2). Specifically, pilot participants’ VAS ratings were significantly higher for interestingness (t=4.48; p<.0001), usefulness (t=4.48; p<.0001), the amount of new information learned (t=5.58; p<.0001) and the extent to which the treatment clarified any prior misconceptions (t=4.01; p<.0001). Participants who were provided with access to the Check-In application along with web-based TES were also significantly more satisfied with their treatment than those in standard treatment alone (t=3.62; p=.0004). The only item which did not differ significantly between participants in the intervention vs. standard conditions was ease of understanding (t=.55; p=.58).

Acceptability of the Mobile Check-In Program – Qualitative Findings

The key themes that emerged from participants’ qualitative feedback (including both open-ended survey items and brief, semi-structured interviews) clustered into three broad categories: benefits of using the Check-in program; difficulties experienced with and barriers to use of the program; and suggestions for improvements to future iterations of the program. A summary of these themes is presented below; within each of the three primary categories, sub-themes are listed in rank order, with the most frequently endorsed themes presented first.

Benefits of the Check-In Program

Participants’ response to the mobile intervention was generally quite enthusiastic, with comments characterizing it as “helpful” and “a great program” among the most frequently expressed sentiments. Participants most appreciated the Check-In program’s utility as a tool to help manage drug-use cravings and “keep you on track”. In an interview, one participant referred to the program as his “buddy” whom he called on for support when experiencing the urge to use heroin. Another explained that “sometimes the program help[ed]” him avoid using heroin because it encouraged him to “stop and think” instead of acting impulsively. Several participants noted that they used the program during (or in anticipation of) times of heightened risk for drug use, such as when experiencing withdrawal symptoms. One man explained that he often got the urge to use heroin early in the morning when he woke up experiencing withdrawal symptoms, but he was able to quell this desire by using the Check-In program. As he stated, “I was feeling anxious and wanted to use, but I used the program instead and that killed it [urge to use heroin].” Although the program explicitly targeted problematic opioid use, some participants found the program helpful in managing cravings to use other drugs. One woman reported that she was often tempted to use cocaine while spending the weekend with drug-using friends in another area of the city, so she made a point to complete the Check-In program frequently, both before and throughout these visits, to help reinforce her commitment to avoiding cocaine.

Participants reported using the Check-In program in a range of settings encountered in their daily lives, and the tool’s flexibility and ease of use were the next most widely appreciated benefits. As one participant commented, “I can use [the Check-In Program] all the time, on the train, at home.” Others were pleased that “everything is simple” and “fast”.

Some participants highlighted the program’s usefulness as a cognitive tool – either to facilitate gaining insight into one’s own patterns of drug use or to interrupt counter-productive thought patterns. For example, one man noted that completing the program “makes me think about things before doing them, especially about going to specific places” that are triggers for him, while another stated that using the application “keeps my mind busy and stops me from thinking negatively. I learn a lot.”

Difficulties with the Check-In Program/Barriers to Use of the Program

When asked about negative aspects of the mobile intervention or their experience with it, participants pointed to technical issues related to the use of the feature phone or the Check-In application much more than the intervention content. Technical difficulties, including difficulty remembering how to log into the program, trouble using the on-screen keyboard to enter text responses, and finding the Nokia phone “too complicated”, were cited by a number of participants as barriers to using the program, although the extent to which participants were impeded by these issues varied widely depending on their technical facility, age and prior mobile phone experience. It is important to note that a minority of participants (n=4) made frequent use of the technical assistance provided by research staff to resolve questions related to use of the phone or the mobile application; however, with this assistance, all interested participants, including those who had extremely limited experience with computer and mobile technologies and had never used a feature phone or smart phone before, were able to successfully master the essential operations necessary to use the Check-In program on an independent basis.

The next most frequently cited complaint concerned the small size of both the mobile phone’s touchscreen and the text within the Check-In program; these were particularly salient issues for those with vision impairments. Most of the middle-aged participants had to wear reading glasses in order to view the program text, yet 5 of these participants reported that they were not able to afford glasses.

Not surprisingly, given that the Check-In program was a limited prototype application consisting of only two brief modules, some participants became “bored” with viewing the same content and completing the same two exercises repeatedly over three months. This reaction is exemplified by the following statement: “I feel that the program should have more options and find the program to be repetitive.” Finally, two participants commented that the intervention’s content was not relevant to them as they were not currently using illicit drugs; as one stated, “Because I’m not using any drugs except methadone, the mobile phone program is not helping me that much.”

Suggestions for Improvements to the Check-In Program

Participants expressed a strong interest in expanded content in the Check-In program. Requests for a “longer”, “more intensive” application with “more topics” were by far the most common suggestions for how to improve a future version of the mobile intervention. Several participants commented that the program should address multiple drugs of abuse beyond opioids – cocaine and crack, in particular. Other topics suggested by participants included “anger management, self-esteem and reminders of the negative effects of getting high,” as well as steps for coping with withdrawal symptoms, anxiety and other physiological or emotional states that can trigger the desire to use drugs.

Participants also offered a number of suggestions for additional features to incorporate into future iterations of the application – most commonly, internet links to outside resources, such as local 12-step/recovery support group meetings, or a 24-hour helpline that would enable individuals who were feeling particularly “anxious” to “hear a person’s voice”. Other suggested features included text messages with personalized content and a voiceover narration to improve usability for individuals with vision or reading difficulties.

Preliminary Efficacy of the Mobile Intervention – Treatment Retention

Participants who received access to the mobile application in combination with computer TES were significantly more likely to be retained for 12 weeks (84% retained for the entire 12-week study period) than were participants who received standard MMT alone (56% retained for the first 12 weeks of the parent trial; chi-square = 4.7, p = .031). Since all participants who dropped out of the pilot study did so because they either left or were discharged from treatment at the MMT program study site, these findings also bear on participants’ relative retention in MMT.

Preliminary Efficacy of the Mobile Intervention – Opioid Abstinence

Pilot participants were documented, via urine toxicology, to be opioid abstinent for a greater number of total study weeks than those in standard MMT by a margin that closely approaches statistical significance (mean = 4.88 [SD=4.18] opioid abstinent weeks in the TES-plus-Check-In Program condition vs. mean = 2.72 [SD=3.57] opioid abstinent weeks in the standard MMT condition; t(48) = −1.97; p = .055). These results produced an effect size index of d =.56 for opioid abstinence.

Discussion

Preliminary evidence from this pilot study suggests that mobile delivery of intervention content to MMT clients is a feasible approach that is highly acceptable to members of its target audience. Results also suggest that providing MMT clients with flexible access to evidence-based psychosocial support outside of their formal treatment setting may have the potential to enhance retention in treatment and reductions in illicit opioid use.

The great majority of participants positively engaged with the Check-In Program with most using it repeatedly, at various times and in a range of settings outside of the MMT program. The fact that most participants were able to retain their study phone (or a single replacement phone) for the duration of the three-month study supports the feasibility of mobile-delivered interventions for MMT clients. It should be noted, however, that the cost of providing mobile phones to participants and replacing lost or damaged phones may limit the feasibility of our approach in this population. To maximize cost-effectiveness, future research may seek to develop and evaluate multi-platform applications that could be installed on participants’ own mobile phones, an approach that will likely become more feasible as smart phone ownership continues to expand.

Feedback from participants indicates that the Check-In application was perceived as appealing and helpful. The program garnered especially high ratings on measures of satisfaction and usefulness, with pilot study participants rating their treatment significantly more positively on five of six indicators than a comparable group of participants assigned to standard MMT alone. Of particular note are participant reports that using the Check-In program helped reduce drug cravings, as well as the likelihood that they would use drugs. The potential utility of the mobile tool is reinforced by results indicating that pilot participants were retained in treatment for a significantly longer duration than a sub-sample of standard MMT participants and showed greater evidence of objectively-verified opioid abstinence. Although the opioid abstinence results fell just short of finding a statistically significant benefit for the enhanced intervention relative to standard MMT alone, this is not unexpected for a small pilot study, and the finding of a medium effect size (Cohen, 1988) for the combined computer TES-plus-mobile intervention demonstrates the potential value of this approach. Together, these findings suggest that the mobile intervention, when implemented in conjunction with standard MMT and computer-based TES, may be effective in enhancing MMT clients’ engagement with treatment and helping them manage problematic opioid use.

These results are consistent with extant literature showing that technology-based interventions for SUDs and other behavioral health conditions are particularly effective when used as enhancements to traditional treatment (Carroll, Ball, Martino, Nich, Babuscio, Gordon et al., 2008; Copeland & Martin, 2004), and suggest that adding a mobile intervention component to routine MMT may help increase clients’ engagement in and commitment to treatment, thereby potentially boosting the treatment’s effectiveness. Based on these promising preliminary results, our team is currently conducting a randomized controlled trial to evaluate the efficacy of an expanded mobile tool in enhancing MMT clients’ treatment retention and reductions in objectively-confirmed drug use (ClinicalTrials.gov protocol registry #NCT01632982). In direct response to pilot participants’ feedback, this 5-module version of the Check-In program (designed for the Android platform) includes a web-linked resources page, customizable text messages, an interactive calendar to encourage non-drug-related prosocial activities, and a graph of participants’ self-reported opioid and other drug or alcohol use over time. The expanded focus on poly-substance use represents a particularly important advance in our second-generation application suggested by these pilot findings, given the prevalence of poly-substance use in our target MMT population.

Another notable finding is that MMT clients in this study were able to master the technical skills needed to use and derive perceived benefit from the mobile tool, provided a moderate amount of initial instruction and ongoing coaching was made available to them. Indeed, the program’s most highly rated quality was “ease of use” which, coupled with the other feedback, suggests that these learning demands did not substantially reduce the convenience and utility of the tool or dampen participants’ enthusiasm for it. This suggests that drug treatment clients who vary widely in age and level of familiarity and skill with mobile technologies, as did the MMT clients in this study, can not only learn to use a mobile-delivered, self-management-focused intervention, but may find it a highly useful addition to their treatment.

Results of this pilot study may provide guidance for developers of technology-based interventions for SUDs regarding program features that are most appealing to target users and appear to best foster participant engagement. First, consistent with research that has investigated factors associated with persistent engagement with technology-based behavioral health tools (e.g., Donkin & Glozier, 2012; Nijland, van Gemert-Pijnen, Kelders, Brandenburg, & Seydel, 2011; Ritterband, Thorndike, Cox, Kovatchev, & Gonder-Frederick, 2009), individual-level personalization of content and interactivity (specifically, the interactive exercises and customized summaries) were highly valued features that appeared to be critical in sustaining participants’ interest in using the tool. Thus, mobile phone-based interventions that incorporate self-monitoring and behavior-tracking tools may be ideal accompaniments for treatment approaches grounded in self-management principles and techniques. Second, participant self-reports revealed that the Check-In program’s convenience, portability and immediacy were widely appreciated benefits that participants took ample advantage of, accessing the program in a range of locations, and at times when support to refrain from using drugs was most needed. This finding not only highlights the specific advantages of mobile interventions for individuals in drug treatment – namely, the ability to provide interventions “on demand” to individuals in their natural environment – but also demonstrates how a mobile application can complement and add value to existing treatment modalities, including medication-assisted treatment for opioid dependence, an important finding given the dearth of technology-based therapeutic tools that have been developed and evaluated for opioid-dependent populations to date (Gainsbury & Blaszczynski, 2011). Third, a program feature that was not included in the prototype the Check-In program, but which was frequently mentioned by participants as a desirable addition to subsequent iterations of the tool, is a social support or human interaction component. This supports and extends prior research showing that technology-based interventions that incorporate some interaction with a trained clinician have lower dropout rates and better evidence of efficacy relative to stand-alone, user-driven interventions and may enhance users’ motivation to complete an intervention (Donkin & Glozier, 2012; Gainsbury & Blaszczynski, 2011).

Limitations

These results should be interpreted with caution in light of some limitations. Most saliently, because this was a pilot study with a small sample size, we were not able to conduct statistical analyses to examine the extent to which the mobile application’s acceptability and efficacy may have been related to participants’ level of engagement with the tool, as measured by rates of program usage. This is an important question that should be addressed in future evaluations of technology-based interventions, as we intend to do in our current trial evaluating the second-generation Check-In program. Also, because this was a preliminary study, we tested only a prototype version of the Check-In program with limited content and interactive features. As revealed in participants’ qualitative feedback, this caused some user fatigue/boredom with modular content over the three-month intervention period and may have reduced usage rates, especially in the second and third months of the intervention period. Nevertheless, participants still reported high levels of satisfaction with the program and continued to use it at a fairly steady rate throughout the study period. The study design comparing the 25 pilot study participants with the last 25 participants randomized to the standard condition of the parent trial (as opposed to comparing concurrently randomized groups) could also be considered a limitation. However, we believe this design is appropriate for the aims of this pilot study. Moreover, the potential time-effect confound may be mitigated by: the relative closeness in time of enrollment of these two groups (within approximately one year); the consistency in research study procedures and staff, as well as MMT clinic operations, during this period; and the lack of significant differences in participant characteristics between the two groups.

Additionally, standardized measures were not used to assess participants’ baseline levels of mobile/feature phone experience or self-perceived proficiency with these technologies; these factors may have influenced participants’ experiences with the intervention, including the extent to which they accessed the application and their satisfaction with it, which in turn could have impacted the program’s efficacy in enhancing opioid abstinence and treatment retention. However, these topics were discussed during semi-structured interviews with participants and relevant qualitative data were collected on these topics, as summarized above. Another potential limitation is the possibility that the weekly data download meetings required of all pilot study participants may have affected participants’ experience of the intervention to an unknown extent. Finally, while our approach to qualitative data collection was structured and systematic, we did not conduct in-depth, audio-recorded interviews with users of the Check-In program which may have added nuance and detail to our qualitative analyses.

Despite these limitations, the promising findings from this study support the feasibility, acceptability and potential utility of a mobile psychosocial intervention in an MMT context. These results, together with the emerging body of research on technology-based interventions for SUDs, may have important implications for the drug treatment system in the U.S. Due to numerous structural, institutional and individual-level barriers, demand for treatment for opioid use and other SUDs far exceeds availability, and this unmet need is likely to grow in the near future, as national data indicate that misuse of opioids is increasing throughout the country. Using new technologies to deliver intervention content may help expand the reach of evidence-based psychosocial treatment within MMT and in substance abuse treatment more broadly, potentially making new forms of treatment available to those who might not otherwise access treatment and enhancing traditional forms of treatment (by, for example, increasing the intensity of treatment without increasing clients’ attendance burden and enabling treatment facilities with limited financial and human resources to provide evidence-based psychosocial treatment).

The potential of mobile tools, in particular, in substance abuse treatment is underscored by the fact that mobile phones are so widely used in the U.S. The “digital divide” that exists with regard to access to personal computers with high-speed internet across socioeconomic status and racial/ethnic groups is not apparent with mobile phone technology (Campbell, Muench, & Nunes, in press; Pew, 2013). Yet one of the most common barriers to use of the Check-In program experienced by participants in the present sample, which included a significant proportion of lower-income, middle-aged individuals and racial/ethnic minorities as is typical in MMT settings, was a lack of facility and previous experience with feature/smart phone technology. This finding raises an important caveat for developers of technology-based interventions for SUDs and suggests that, while a broad range of socioeconomic, age and racial/ethnic groups in the U.S. may have access to mobile technology, the specific kinds of mobile phones (i.e., feature/smart phones vs. traditional mobile phones) and/or the types of activities they engage in with these devices may vary in systematic ways at the present time (with poorer groups and older adults owning older, less advanced devices and/or using these devices for a narrower range of tasks). Skills-based technical barriers of this nature may reduce the convenience and appeal of technology-based interventions, including mobile tools, for some users and may in turn negatively impact engagement and outcomes if not adequately addressed. Indeed, research on engagement with technology-based tools for behavioral health has found that individuals’ technical limitations can function as significant barriers to persistent engagement with these tools (Donkin & Glozier, 2012). Our experience suggests that such barriers can be surmounted, although the learning curve may be rather steep for a minority of individuals who may require initial training and ongoing coaching for remediation. For the full promise of technology-based behavioral health interventions, including mobile applications – and especially those designed to address SUDs – to be realized, a proportion of the target audience may need ongoing technical support to effectively use these tools. Nevertheless, data from this pilot study suggest that mobile psychosocial interventions, when implemented with adequate support, are a promising means to increase the reach and potency of evidence-based interventions for a broad range of individuals with SUDs.