Using Smartphones to Improve Treatment Retention Among Impoverished Substance-Using Appalachian Women: A Naturalistic Study

Abstract

Objectives:

Longer retention in treatment is associated with positive outcomes. For women, who suffer worse drug-related problems than men, social technologies, which are more readily adopted by women, may offer promise. This naturalistic study examined whether a smartphone-based relapse-prevention system, A-CHESS (Addiction-Comprehensive Health Enhancement Support System), could improve retention for women with substance use disorders in an impoverished rural setting.

Methods:

A total of 98 women, age 18 to 40, in southeastern Kentucky and mandated to treatment, received A-CHESS with intensive outpatient treatment for 6 months. For comparison, data were obtained for a similar but non-equivalent group of 100 same-age women also mandated to treatment in the same clinics during the period. Electronic medical record data on length-of-stay and treatment service use for both groups were analyzed, with A-CHESS use data, to determine whether those using A-CHESS showed better retention than those without.

Results:

Women with A-CHESS averaged 780 service units compared with 343 for the comparison group. For those with discharge dates prior to the study’s end, A-CHESS patients stayed in treatment a mean of 410 vs 262 days for the comparison group.

Conclusions:

Given associations between retention and positive outcomes, mobile health technology such as A-CHESS may help improve outcomes among women, especially in settings where access to in-person services is difficult. The findings, based on a non-equivalent comparison, suggest the need for further exploration with rigorous experimental designs to determine whether and to what degree access to a smartphone with A-CHESS may extend and support recovery for women.

Introduction

Alcohol, opioid, and other substance use disorders (SUDs) are chronic conditions characterized by frequent relapse.^1-4 Evidence shows that continuing care for SUDs is associated with better outcomes,^5-7 and that retention in treatment is a key predictor of successful recovery.^8-10 Nonetheless, patients are not typically offered continuing care with ongoing monitoring,^1,11,12 in part because of the overstressed infrastructure for delivering SUD treatment.¹³

Mobile health (mHealth) technology offers a way to provide continuing care by making recovery support, information, and monitoring available to patients virtually anytime and anywhere^4,14 without further stressing the infrastructure. This method of delivering care could be especially important in rural areas, where access can be difficult, even when patients are motivated to attend scheduled appointments, group sessions, and 12-step meetings.

In addition, mHealth systems may be especially effective for women, who have been shown to adopt social technologies more readily than men, even though they are less confident of their technology skills in certain contexts.^15-20 At the same time, women with SUDs, in comparison to men, exhibit worse drug, health, emotional, and employment problems,²¹ and young mothers with SUDs may fare especially poorly.²² As such, gender-based solutions are worth seeking, and technology social features could help answer the call.

This article describes a naturalistic, non-equivalent control group quasi-experiment conducted to assess the potential value of an mHealth system to women in a setting with significant access challenges. Specifically, our goal was to evaluate whether an evidence-based relapse-prevention smartphone system known as A-CHESS (Addiction-Comprehensive Health Enhancement Support System^3,23) system could increase retention in treatment among women with SUDs in an isolated, impoverished rural setting. The naturalistic design enabled observation of the women as they would actually behave in these conditions, informing questions for more controlled study and suggesting directions for treatment. To our knowledge, this is one of the first studies to evaluate the impact of mHealth on retention in addiction treatment for women.²⁴

Methods

Participants and procedures

A-CHESS group

Participants were women with SUDs served by Kentucky River Community Care (KRCC) in southeast Kentucky’s mountainous coalfields. All came from the agency’s eight-county Appalachian area and were enrolled in an intensive outpatient program named “Solutions,” specifically for women mandated to treatment, offered in four KRCC clinics. Patients were age 18 to 40 years, reported having children or being pregnant, and were referred from child welfare agencies, drug courts, or other criminal justice agencies. Mothers had lost custody of their children or were likely to without treatment. Most reported co-occurring disorders, mainly depression and anxiety.

The Solutions program provided individual and group counseling services, behavioral health education (eg, fundamentals of relapse prevention), and assistance establishing a support network with peers, providers, and the community. Patients and families received family counseling, during which they addressed relationship conflicts and strains due to addiction and, for those separated from children, prepared to live together again. Family members received awareness training about addiction, including warning signs and recommended responses to various problems. The clinics also offered psychiatric, psychological, case management, therapeutic rehabilitation, developmental, housing, and recovery coaching services.

The KRCC Institutional Review Board (IRB) approved a project in which patients would receive A-CHESS for a period of 6 months in addition to Solutions. IRB and informed consent procedures were required by KRCC. Patients signed consent for treatment and research procedures and accepted in writing the terms and participant-protection conditions (eg, absence of coercion, ability to withdraw at any time) of the Substance Abuse and Mental Health Services Administration (SAMHSA) program that funded the project. SAMHSA required participants to sit for intake and follow-up interviews known as the GPRA (Government Performance and Results Act of 1993); SAMHSA’s Center for Substance Abuse Treatment’s CSAT GPRA Client Outcome Measures for Discretionary Programs²⁵ was used. Consenting to the A-CHESS study was not a condition of SUD treatment.

PJ, a clinic staff media specialist and assistant to the chief information officer, was in charge of recruiting, intake, and training of all women who joined the study; she also collected and uploaded GPRA data to the federal database. During the data collection period (February 2012 to August 2014), PJ recruited and completed the intake GPRA with 180 women. GPRA responses at intake are reported in Table 1 to help establish a picture of the participants.

Table 1.

Characteristics of A-CHESS patients (N = 180), as reported on the GPRA at intake to the study.

GPRA measure	Total N responding	n	Percent (n/N)
Race
White	180	176	97.8
Hispanic	180	2	1.1
African American	180	0	0
Asian	180	1	0.55
Native American	180	1	0.55
Employment status
Unemployed	174	162	93.1
Income from any source	179	22	12.3
Education
11th grade or less	180	64	35.6
7th to 9th		30	16.7
10th to 11th		34	18.9
High school diploma or equivalent	180	57	31.7
Some college (without 4-year degree)	180	58	32.2
Housing
Own/rent	180	98	54.4
Living in someone else’s home	180	77	42.8
Other housing	180	3	1.7
Homeless	180	2	1.1
Parental and children’s status
Participants with children	180	177	98.3
Participants’ children living elsewhere by protective order	176	127	72.2
Crime and criminal justice status
On parole or probation	179	30	16.8
Awaiting charges, trial, or sentencing	179	20	11.2
Physical health in last 30 days
Hospitalized for physical ailment	180	6	3.3
Outpatient treatment for physical condition	180	57	31.7
Emergency room visit	180	18	10.0
Overall health excellent or very good	178	30	16.9
Mental health in last 30 days
Drug use caused emotional problems	180	14	7.8
Daily anxiety/tension (not drug-related)	180	54	30.0
Cognitive problems on 2 or more daysa	180	64	35.6
Hallucinations	180	2	1.1
Problems controlling violent temper	180	11	6.1
Attempted suicide	180	1	0.55
Support seeking outside of treatment
Self-help meetings, non-religious (eg, 12-step)	179	70	38.9
Self-help meetings, faith-based	179	12	6.7
Family members when feeling troubled	179	127	70.6
No family or friend support for recovery	179	12	6.7
Violence and trauma
Experienced violence or trauma in lifetime	180	127	70.1

Abbreviations: A-CHESS, Addiction-Comprehensive Health Enhancement Support System; GPRA, Government Performance and Results Act.

Ns vary from 180 because a patient could decline to answer a given question. Percentages may not add up to 100 because of rounding, multiple possible responses, or no applicable response.

^aPhrased as “problems concentrating, understanding, or remembering.”

As evidence of sincerity, patients were to return after the GPRA intake for their phone and training. Of the 180, a total of 98 (54.4%) returned to receive a phone and remained in treatment during their intervention period. The other 82 could have left treatment, been incarcerated, or disappeared to avoid incarceration, or they simply decided not to participate. Many expressed privacy concerns about using a KRCC phone, which may have been a factor for those deciding not to enter the study.

Because of poor credit and inability to afford phones and access, most of the women (the exact number is unknown) had never owned or even used a smartphone and needed to be trained in use of the device (eg, texting, taking pictures) as well as the A-CHESS app. Participants received basic phone and A-CHESS training from PJ during group and individual sessions and via videos produced by the app’s developers. PJ also gave phones with A-CHESS to all counselors, enabling counselors to monitor patients and patients to contact counselors at any time.

Study phones were serviced through a regional wireless company that had the greatest number of towers in the area and partnered with a national network for roaming. However, Internet access was a challenge. The women generally did not have WiFi at home and would go to public places for free access. Moreover, phone service could be unreliable, necessitating finding other locations where service was better. “That was hard,” said PJ. “Most of the women needed public transportation.” As testimony to the perceived value of the app, 47% of the women trained to use A-CHESS were still using it at the end of the 6-month study, despite these challenges.

Phones were in limited supply. After each participant’s 6-month intervention period, she would return the phone to KRCC so that it could be given to a new participant. KRCC tracked each participant’s clinical service use from admission date until she left treatment. Leaving treatment was defined as not having received any clinic service for 3 months.

Information about the primary substance bringing each patient to treatment was not available to the research team. However, at intake to the study, patients reported their primary substance used in the last 30 days on the GPRA interview. Of specific drugs reported, opioids were the most frequently named (11.7%); alcohol was named by 3.3% (see Table 2). The number of women reporting no substance use (43.3%) was much higher than one might expect from a group in SUD treatment, but since the GPRA reflects substance use after treatment was already under way, patients may have been truthful in denying recent use. Also, as referrals from drug court and child welfare, patients were being drug tested regularly and likely feared losing child custody or being incarcerated.⁴ Because the GPRA reports do not reflect substance use on entering treatment, they are not reliable for analysis, but they suggest a general picture of the types of substances used and for this reason are reported.

Table 2.

Primary substance used by A-CHESS patients (N = 180) in the past 30 days, as reported on the GPRA at intake to the study.

Primary substance	n	%
Unspecified illegal drugs	41	22.8
Opioidsa	21	11.7
Marijuana/hashish	18	10.0
Benzodiazepines	12	6.7
Alcohol	6	3.3
Barbiturates	3	1.7
Amphetamines	1	0.6
No substance use	78	43.3
Total	180	100

Abbreviations: A-CHESS, Addiction-Comprehensive Health Enhancement Support System; GPRA, Government Performance and Results Act.

Percentages are rounded to the nearest one-tenth.

^aOpioids were Percocet (10, or 5.6%), OxyContin/oxycodone (5, or 2.8%), morphine (2, or 1.1%), codeine (1, or 0.6%), Dilaudid (1, or 0.6%), heroin (1, or 0.6%), and non-prescription methadone (1, or 0.6%).

Comparison group

At the conclusion of the study, KRCC developed a data set of 231 patients meeting inclusion criteria. Each (1) was female, (2) had a SUD diagnosis, and (3) received treatment in KRCC outpatient programs (4) during the same period that the study took place. Patients were (5) of the same age as the A-CHESS group (range = 18-40; Ms = 27.89 and 27.93, respectively) and (6) from the same eight-county area. Like the intervention group, (7) all were mandated to treatment, suggesting a similar degree of SUD severity. From this data set, 100 were randomly selected, using SPSS, to form a post hoc comparison group.

Intervention

Participants received at no cost to them a large-screen Android smartphone with unlimited voice and texting capacity, 5 GB monthly data, and A-CHESS, which has been shown in a randomized clinical trial to improve adherence to treatment.²³ A-CHESS offered more than a dozen services supporting addiction recovery in these ways: (1) communication with addiction experts and peer support groups, including confidential chat rooms and private messaging; (2) timely monitoring, feedback-related assessment, and links to interventions addressing relapse risk; (3) reminders and alerts to encourage adherence to therapeutic goals; (4) one-touch communication with the patient’s counselor and/or case manager; (5) addiction-related educational materials and tools; and (6) customizable location-based resources (eg, alerts that would display if a patient neared a high-risk location such as a bar or supplier she used to frequent). Screen shots of A-CHESS are available at http://chess.wisc.edu/achess-archive/.

Each week, A-CHESS checked in with a brief survey on the phone’s screen. This survey collected patient data on recent substance use as well as status on five protective and five risk factors from the Brief Alcohol Monitor.²⁶ A-CHESS used the check-in information for triage and feedback. Case managers and counselors received a summary of each patient’s check-in data any time they wished, the day before a scheduled appointment, and whenever a patient reported a lapse.

System data revealed 94 unique A-CHESS users, indicating that 4 of the 98 participants who received smartphones never logged in to the app, although they likely used the phone for other purposes. These 4 women would have been among the first to enter the study. “All [98 participants] were trained on their own phone,” according to PJ. “But in the very beginning, we hadn’t fully set up the process, and we quickly realized we needed to show them how to use the app at the same time they got the phone. The first few didn’t have that and might have never logged in.”

Data collection and analysis

A-CHESS use

For each A-CHESS patient, de-identified data on logins and page views were collected by computer at the Center for Health Enhancement Systems Studies, University of Wisconsin–Madison, where the app was developed.

GPRA reports

Responses to A-CHESS patients’ intake and follow-up GPRA surveys were entered into the federal database by KRCC staff (PJ) and made available to author WDM. The comparison patients comprise a post hoc data set; they were not study participants and did not sit for the GPRA. GPRA interview responses are therefore used only for descriptive insight and not for analyses.

Outpatient treatment data from medical records

For each patient in both the A-CHESS and comparison groups, all treatment data came from the KRCC electronic medical record (EMR) database and were de-identified prior to delivery from the agency to the research team. Data consisted of dates of birth, admission, and discharge; diagnosis; types of clinical services received; and number of service units received. Service types were group sessions, individual sessions, peer support, case management sessions, and drug screens, reported as 15-min units. Clinical service types and units were gathered for both groups post hoc.

EMR data enabled the calculation of length of stay in treatment, number of units received of each clinical service, and total number of units received. Levene’s test²⁷ was used to assess the equality of variances for retention-in-service variable distributions. This indicated inequality between the two groups and suggested non-parametric analysis. Thus, Mann-Whitney U and chi-square tests were used to analyze between-group effects.

Outcome measures

It was hypothesized that patients who used A-CHESS would show better retention in treatment than women without it, as indicated by the length of time between admission and discharge dates in the EMR. It was also hypothesized that these patients would access more care, as indicated both by the number of units of clinical service according to type and by the sum of all service units.

Results

EMR data and outcome measures

The Mann-Whitney U test revealed significant differences that favored the A-CHESS group (P < .001; Bonferroni adjusted alpha = .007) for length of stay; for case management, group sessions, individual sessions, and drug screens; and for total units of service. The difference between groups for peer support was nonsignificant (P = .067). All effect sizes are in the moderate range (.47-.64).

EMR records revealed that A-CHESS patients stayed in treatment a mean of 410 days, vs 262 days for the comparison group (Table 3). These length-of-stay results include only the 68 A-CHESS and 84 comparison patients with discharge dates prior to the end of the study. Eventual discharge dates for the remaining 30 (30.6%) A-CHESS women and 16 (16.0%) comparison women are unknown; their length-of-stay, when calculated as of the final date of the study, shows a similar trend: 584 days vs 360 days. For all participants, when length of stay is calculated by either discharge or end-of-study date, as applicable, M = 463 days for the 98 A-CHESS patients and M = 277 days for the 100 comparison patients.

Table 3.

Length of stay in treatment and clinical services used for A-CHESS and comparison groups.

Outcome	Group	N	Mean	SD	95% Confidence interval	P	Effect size (Cohen’s d)
Length of stay						.000	0.47
	A-CHESS	68a	410.0	240.82	351.74-468.32
	Comparison	84a	261.7	212.74	213.94-306.27
Case mgmnt. sessions						.000	0.57
	A-CHESS	74b	8.36	5.30	7.14-9.59
	Comparison	47b	4.68	3.67	3.60-5.76
Group sessions						.000	0.54
	A-CHESS	95b	668.52	461.24	574.56-762.47
	Comparison	88b	332.56	412.16	245.23-419.89
Individual sessions						.000	0.64
	A-CHESS	97b	120.16	105.02	99.00-141.33
	Comparison	96b	46.69	44.26	37.72-55.66
Drug screens						.000	0.50
	A-CHESS	93b	30.94	20.31	26.75-35.12
	Comparison	80b	17.00	19.03	12.76-21.24
Peer support						.067	0.51
	A-CHESS	21b	31.48	34.60	15.73-47.23
	Comparison	11b	13.45	16.32	2.49-24.42
All services						.000	0.57
	A-CHESS	98b	779.59	547.18	669.89-889.30
	Comparison	100b	342.77	435.05	256.45-429.90

Abbreviation: A-CHESS, Addiction-Comprehensive Health Enhancement Support System.

Length of stay is measured by number of days in treatment from admission date to discharge date; all other outcomes (services) are measured as 15-min increments of use. A Bonferroni adjustment was made to account for multiple comparisons, alpha = .007.

^aClients with treatment discharge dates only.

^bNumber of records showing service use out of 98 possible for A-CHESS group and 100 possible for comparison group.

For clinical services used, EMRs revealed that the 98 A-CHESS patients received an average of 779.6 total units of service (measured in 15-min increments) vs 342.8 total units for the 100 comparison patients. Table 3 shows average use of each service for the two groups, with A-CHESS patients logging roughly twice as many units for every service.

A-CHESS use

Table 4 shows the monthly use patterns of A-CHESS participants logging in throughout their intervention period. While app use decreased over time, almost half of the patients (46.8%) were still logging in during their final month of the study, a rate that is significantly better than the average for even the most frequently downloaded mHealth apps on the market, which typically show rapid drop-off in use.^28-30 In one study, 25% of users were found to abandon mHealth apps after just one use³¹; another found that the average retention rate is 20% after 90 days.³²

Table 4.

A-CHESS use among intervention group participants using the app (n = 94).

A-CHESS use measure	Month 1	Month 2	Month 3	Month 4	Month 5	Month 6
Logins to A-CHESS
No. (%) of users	94 (100)	82 (87.2)	77 (81.9)	67 (71.3)	60 (63.8)	44 (46.8)
Total logins	6476	4269	3511	2751	2415	1219
Avg. logins per user	68.89	52.06	45.60	41.06	40.25	27.70
Weekly survey check-in
No. (%) of users	89 (94.7)	63 (67.0)	58 (61.7)	53 (56.4)	42 (44.7)	28 (29.8)
Total pages viewed	3634	1846	1512	1216	842	402
Avg. pages per user	40.83	29.30	26.07	22.94	20.05	14.36
Discussion groups
No. (%) of users	80 (85.1)	65 (69.1)	55 (58.5)	50 (53.2)	39 (41.5)	27 (28.7)
Total pages viewed	24 330	12 372	7448	6085	4419	1340
Avg. pages per user	304.13	190.34	135.42	121.70	113.31	49.63
Private messaging
No. (%) of users	93 (98.9)	74 (78.7)	64 (68.1)	47 (50.0)	45 (47.9)	25 (26.6)
Total pages viewed	10 747	5584	3970	3365	2319	1319
Avg. pages per user	115.55	75.45	62.03	71.60	51.53	52.76
Profiles (self and other members)
No. (%) of users	93 (98.9)	50 (53.2)	31 (33.0)	30 (31.9)	18 (19.2)	9 (9.6)
Total pages viewed	11 795	2626	1384	1771	1356	531
Avg. pages per user	126.83	52.52	44.65	59.03	75.33	59.00
Events and announcements
No. (%) of users	78 (83.0)	31 (33.0)	23 (24.5)	17 (18.1)	9 (9.6)	8 (8.5)
Total pages viewed	629	83	63	65	27	29
Avg. pages per user	8.06	2.68	2.74	3.82	3.00	3.63
Information tools
No. (%) of users	79 (84.0)	33 (35.1)	27 (28.7)	20 (21.3)	16 (17.0)	6 (6.4)
Total pages viewed	830	128	61	66	33	7
Avg. pages per user	10.51	3.88	2.26	3.30	2.06	1.17
Audio interviews of people in recovery
No. (%) of users	69 (73.4)	18 (19.1)	13 (13.8)	10 (10.6)	4 (4.3)	5 (5.3)
Total pages viewed	348	40	44	13	4	7
Avg. pages per user	5.04	2.22	3.38	1.30	1.00	1.40
Personal motivations
No. (%) of users	28 (29.8)	5 (5.3)	7 (7.4)	1 (1.1)	1 (1.1)	3 (3.2)
Total pages viewed	326	18	56	10	6	11
Avg. pages per user	11.64	3.60	8.00	10.00	6.00	3.67
Audio relaxation and meditations
No. (%) of users	58 (61.7)	12 (12.8)	5 (5.3)	5 (5.3)	7 (7.4)	2 (2.1)
Total pages viewed	207	17	17	9	16	3
Avg. pages per user	3.57	1.42	3.40	1.80	2.29	1.50
Recovery podcasts
No. (%) of users	45 (47.9)	14 (14.9)	7 (7.4)	7 (7.4)	6 (6.4)	2 (2.1)
Total pages viewed	219	24	18	21	22	2
Avg. pages per user	4.87	1.71	2.57	3.00	3.67	1.00

Abbreviation: A-CHESS, Addiction-Comprehensive Health Enhancement Support System.

In month 3, average A-CHESS logins per user numbered 45.6, or an average of 1.5 times per day; in month 6, average logins numbered 27.7, or an average of about 1 time per day. The most visited features were the Weekly Survey check-in, which provided feedback to both the patient and the clinic counselor; Discussion Groups, which function like chat rooms for the closed community of the participants; and Private Messages, which function like email between participants. Between 27% and 30% of all A-CHESS participants were using each of these three features at 6 months.

The remaining features were viewed by fewer than 10% of participants at 6 months. These included announcements of events and meetings, informational resources such as news and links to relevant websites, inspirational recordings of people in recovery telling their stories, participants’ own photos and phrases for motivation, audio relaxation and meditation modules, and podcasts of recovery meeting keynote addresses. Profiles were visited less over time, but this might be expected as relationships developed.

GPRA intake and follow-up interviews

For reasons described above, GPRA reports of substance use do not indicate change between study intake and follow-up. Generally for the GPRA, in the absence of control or comparison group responses, no pre-post differences can be attributed to A-CHESS, but one change may be worth noting: While unemployment was extremely high (93.1% at intake, 88% at follow-up), it appears that efforts to improve were being made, based on reported rates of enrollment in educational or vocational programming, which increased from 6.7% to 13.3% at follow-up.

Discussion

A-CHESS patients with known discharge dates stayed in treatment 156.7% as long as the patients without it (410 days vs 262 days). In addition, all patients with A-CHESS received more than twice (227.4%) as many service units (780 units vs 343 units). Given the associations between retention in treatment and positive outcomes, these results suggest that a smartphone containing A-CHESS, used with intensive outpatient treatment, has potential to improve long recovery rates for women with an SUD in an impoverished and/or isolated area.

There are several possible reasons why A-CHESS might have improved retention. First, A-CHESS data revealed strong and sustained use of the app’s social and communication features, which enabled patients to communicate with their counselors and other group members whenever and as often as needed. The Weekly Survey prompted contact between patient and counselor by regularly informing the counselor of the patient’s status on both negative and positive variables, laying groundwork for treatment sessions, and opening the door for communication via private messages or phone calls. Patients also made considerable use of Discussion Groups and Private Messages, the two main peer support features. The ability to communicate with one another anywhere, anytime may have created a camaraderie that increased the willingness to attend group sessions and other treatment services.³³ As one KRCC counselor (SS) put it, “A-CHESS made a major difference for every woman. They always had access to someone to help.” With these features, A-CHESS may have served as an intensifier of treatment.

Other possible reasons, which should be monitored in future studies, reflect the key principles of self-determination theory,^23,34 which holds that a person’s adaptive functioning improves when three needs are met: (1) feeling competent, (2) feeling connected to others, and (3) feeling internally motivated rather than coerced. A-CHESS could have helped patients feel competent merely by being a tool in their hands that they had the choice to use. More specifically, certain features may have helped them cope with daily stressors, increasing perceptions of competence. For example, feedback from the Weekly Survey—which was used more than any other feature—pointed out when cravings seemed to be highest and then prompted patients to features designed to help them cope with those cravings. The coping features, such as informational resources, meeting announcements, and relaxation recordings, were not popular over time, though, possibly because there was not enough content to sustain interest.

In terms of feeling connected to others, A-CHESS was most used for its social features, as described above, which is consistent with previous findings that women, more than men, use social media, even when they are not confident with technology.^15-20 The frequent use of these features, as indicated by the use data, may well have fostered a sense of support and connection among the participants. Furthermore, a feeling of belonging in a closed group of women who were all recovery-oriented may have helped the women maintain their commitment to succeed.

Finally, intrinsic motivation to stay with treatment might have been boosted with a combination of A-CHESS features. A new inspirational message appeared each day on the home screen when a user logged in. Women could also upload personally meaningful photos and words to remind themselves of their reasons to stay clean, and they could listen to interviews with people in recovery as well as podcasts of speeches from 12-step conferences and other recovery events. However, other than the new inspirational message that every user saw each day she opened the app, use of these features was low after the first month. As with informational resources, motivational content was limited in terms of quantity or refreshment, which might explain the lack of continuing interest. Subsequent to this study, A-CHESS developments have included new motivational and informational tools, in terms of both type and number, as well as navigational and display enhancements for ease of use and visual appeal. In future studies, outcomes based on self-determination theory should be monitored.

Limitations

A number of limitations prevent generalizing from the results, but the results raise interesting questions and contribute to the literature on those questions. First, our quasi-experimental posttest-only design employed a similar but non-equivalent comparison group that was not created through random assignment. The project’s goal was to evaluate A-CHESS in a natural setting, where the study would not interfere with typical treatment, a condition that made random assignment and a pretest virtually impossible.

There are benefits to this kind of naturalistic study. Data collections do not interfere with treatment as usual; this is a natural sample involving patients in an intensive treatment program instead of a highly restricted sampling. Unlike many randomized trials where outcomes are measured by qualitative surveys (which in mandated treatment conditions are positively related to recovery-oriented responses),⁴ data used in this study are based solely on actual behaviors and not self-report. The study also provides information about how many services would be delivered and how long the patient would stay in treatment in the absence of A-CHESS, an important consideration given that treatment retention is among the most important predictors of sustained recovery.³⁵

There are costs as well. Namely, the value of information provided by a comparison group depends on how similar the group is to the treatment group.³⁶ As with any situation where a comparison group is used in a quasi-experimental design, our non-equivalent control group had selection biases³⁷; even random selection of a subset from the pool who met criteria may have introduced bias. The absence of a true control group created through random selection of study participants makes it difficult to determine whether treatment outcomes are due to treatment or to differences between groups. We controlled for many sources of dissimilarity, and given the homogeneity of the counties in terms of minorities and economic conditions, the groups were also likely to be similar demographically. Still, we could not control for commitment to recovery or actual substance use during the study. Furthermore, even though patients were mandated to treatment, implying at least a moderate addiction severity for all, the severities may have been different. As a result, we cannot say the results are definitive, only provocative.

Second, any effects of A-CHESS cannot be separated from access to a smartphone, which was a first for most women and which on its own may have improved the social support and information the women received, providing them with means and motivation to change. However, a study conducted several years ago suggests that A-CHESS rather than the phone was responsible.³⁸ In that randomized clinical trial, breast cancer patients were assigned to (1) a control group that received books and audio resources about cancer, (2) a group that received a laptop plus Internet access but not CHESS (the predecessor to A-CHESS), or (3) a group that received a laptop with CHESS for breast cancer. In that 9-month study (the final 3 months were follow-up after removing the computers), the Internet group was never superior to the no-laptop control group, while the CHESS group was superior to both other groups in quality of life and social support outcomes. In that case, it was demonstrably CHESS that made a difference. Given that most women did not have their own computers when that study took place, the computer-based interventions may have offered similar novelty as the smartphones for the A-CHESS women in Kentucky.

In addition, a separate short survey for the A-CHESS patients after participating in the current study suggests that A-CHESS itself helped treatment, with 84.2% responding it helped “quite a lot” (52.6%) or “somewhat” (31.6%); no participant responded that it helped “not at all.” But only two of the four clinics returned the survey, so while responses were positive, they are not reliable.

Third, this study examined retention in treatment, not recovery outcomes. We do not know that patients with A-CHESS had improved recovery outcomes, such as fewer days of substance use or greater number of days of continuous abstinence. However, given that retention is key to successful outcomes, we can speculate that they were more likely to have better recovery outcomes than the comparison group, whose retention indicators were markedly lower. A final limitation is that because A-CHESS was used by counselors and case managers along with the intensive outpatient treatment to maintain connections with study participants, we do not know whether A-CHESS alone would constitute effective continuing care.

Conclusions

The results of this quasi-experiment merit further exploration with more rigorous experimental designs.³⁹ Questions to pursue include: Do patients in SUD treatment have better retention when provided with A-CHESS than do patients without the technology? (Some data on this have begun to appear.)²⁴ Do women have better retention and improved outcomes in comparison with men when provided with A-CHESS? Would a smartphone alone vs a smartphone with A-CHESS function as effective continuing care? Does the cost-effectiveness of A-CHESS justify the expense of providing a smartphone for continuing care for SUDs? The results of the project, combined with earlier evidence,³⁸ suggest that the technology might improve both access to and retention in continuing care, especially for women in isolated or impoverished rural areas.