Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Sep 1.
Published in final edited form as: Psychiatry Res. 2020 Jul 5;292:113282. doi: 10.1016/j.psychres.2020.113282

The predictive validity of the Progress Assessment, a clinician administered instrument for use in measurement-based care for substance use disorders

Deborah H A Van Horn a,*,1, Jessica Goodman a,b, Kevin G Lynch a, Marcel O Bonn-Miller b,c,d, Tyrone Thomas a, AC Del Re d, Kimberly Babson c, James R McKay a,b
PMCID: PMC9434595  NIHMSID: NIHMS1616764  PMID: 32711168

Abstract

We tested the predictive validity of the Progress Assessment (PA), a brief counselor administered tool for use in measurement-based care for substance use disorders. The PA includes 5 items assessing relapse risk and 5 items assessing factors protective against relapse. Data were drawn from a completed study of continuing care for cocaine dependence (McKay et al., 2013) and includes 12 months of follow-up on158 participants (76% male) who received brief telephone or face-to-face sessions. Each session began with the administration of the PA, followed by cognitive-behavioral counseling tied to the results of the PA and anticipated risky situations. Outcome was assessed via urine toxicology every 3 months. As administered in an effectiveness trial, average PA risk and protective scales within each 3-month segment of the study predicted urine toxicology results at the end of that period, with higher risk scores and lower protective scores predicting greater rates of cocaine positive urine drug screens. PA scores did not predict dropout from continuing care participation. The 10-item PA shows promise as a pragmatic clinical tool for ongoing monitoring during continuing care for substance dependence.

Keywords: cocaine dependence, continuing care, progress monitoring, distance counseling

1. Introduction

There is growing evidence that progress monitoring can improve adherence and outcomes for substance use disorder (SUD) treatment (Crits-Christoph et al., 2012; Raes et al., 2011; Russell et al., 2018). These findings mirror strong evidence that monitoring progress and providing feedback to clinicians and patients results in better outcomes in general mental health treatment (Connolly Gibbons et al., 2015; Goodman et al., 2013; Lambert et al., 2018; Miller et al., 2006; Shimokawa et al., 2010). Professional associations, third-party payers, and accrediting agencies are increasingly calling for systematic progress monitoring throughout behavioral health care (American Psychological Association, 2009; The Joint Commission, 2018; Valenstein et al., 2009). Such data, if obtained in a timely manner, could be provided to both therapists and patients and used to inform modifications to treatment to reduce the likelihood of relapse and dropout (McKay, 2009).

With increased calls for implementation of progress monitoring comes a need for measures with adequate predictive validity that also meet clinician and patient needs for brevity, flexibility, and clinical relevance (Boswell et al., 2015; Solstad et al., 2017). An essential component of progress monitoring in SUD treatment is, of course, assessing current substance use. However, from a clinical standpoint, it can be at least as useful to identify other factors that predict future substance use. Such information could be used to flag patients who are at risk and to highlight potential areas to address within treatment to shore up recovery and prevent relapse. We developed a brief multidimensional assessment tool, the Progress Assessment (PA), to monitor progress in continuing care for substance use disorders. The PA includes five items assessing risk factors for relapse (i.e., poor medical or mental health treatment adherence, depression, craving, being in high risk situations, low abstinence self-efficacy), five items assessing positive factors that promote recovery (i.e., active coping, attendance at self/mutual help meetings, spending time with a sponsor, other abstinence-oriented social and recreational activities, pursuing recovery oriented personal goals), and several items assessing recent substance use. It is intended to be administered at the beginning of continuing care contacts, with the information it yields used to inform the focus of the session that day, and to track progress across sessions.

The structure of the measure in terms of relapse risks and protective factors was guided in part by a cognitive-behavioral treatment model emphasizing anticipation of and planning for relapse risks (Carroll, 1998; McKay et al., 2013a), with specific item selection informed by prior research into circumstances and behaviors associated with SUD relapse and improved abstinence outcomes. Indeed, research has shown that exposure to risky situations, such as those directly associated with substance use or those that give rise to the unpleasant emotional states, appears to precipitate relapse (Cacciola et al., 2005; Freedman et al., 2004; Schiffman et al., 1996; Zywiak et al., 1996); whereas employing a variety of active coping strategies to address risky situations and relapse triggers can help individuals achieve better abstinence outcomes (Farabee et al., 2013; Maisto et al., 2000; McKay et al., 2013b; Moggi et al., 1999).

Reviews have noted that across populations and methodologies, negative affect states, craving, cognitive factors such as self-efficacy and motivation, interpersonal problems, and limited use of coping skills placed individuals with SUD at greater risk of relapse whereas participation in treatment and self-help were associated with higher rates of abstinence (McKay, 1999; McKay et al., 2006). Further research has borne out the importance of the constructs selected: for example, whereas general ratings of psychiatric severity have consistently predicted alcohol use outcomes (Adamson et al., 2009), additional research has highlighted the importance of depressive symptoms (Gamble et al., 2010; Martin et al., 2011; McKay et al., 2013b), and extended those findings to drug use (McKay et al. 2013). Self-efficacy for abstinence continues to be a robust predictor of substance use outcomes across addictive behaviors (Adamson et al., 2009; Dolan et al., 2008; McKay et al., 2013b), with even a single question about abstinence self-efficacy strongly predicting outcome (Ludwig et al., 2013). Craving, too, has been found to be a strong predictor of cocaine and alcohol use (Gordon et al., 2006; Schneekloth et al., 2012; Weiss et al., 2003) and its inclusion in the DSM-5 confirms the importance of cravings in assessing and treating addiction (American Psychiatric Association, 2013).

When considering “protective” behaviors other than active coping directly with risky situations and relapse triggers, we were informed by research highlighting the importance of social support in reducing the likelihood of relapse. For example, spending time with those who are abstinent is associated with better outcomes among alcohol and drug-dependent individuals (Farabee et al., 2013; Litt et al., 2007; McCrady, 2004), and self-help participation has consistently emerged as one of the strongest predictors of abstinence. Numerous studies have found that attendance at 12-step meetings such as Alcoholics Anonymous or Narcotics Anonymous predicts short- and long-term substance use outcomes across a wide range of populations and settings (Bonn-Miller et al., 2011; Cacciola et al., 2005; McKay et al., 2001; McKay et al., 2013b; Pagano et al., 2013; Staines et al., 2003). Importantly, having a sponsor in AA has been shown to be associated with higher abstinence rates, even controlling for attendance at AA meetings (Tonigan and Rice, 2010; Witbrodt et al., 2013).

Several of this paper’s authors (DVH, TT, JRM) and other counselors experienced with a similar continuing care model incorporating progress monitoring, feedback, and counseling (McKay et al., 2010b), were involved in discussion of how best to operationalize the 10 areas to be assessed with one item each. Originally conceptualized as a “risk” assessment, the measure was reframed as a “progress” assessment, consistent with a strengths-based counseling approach. For example, participants were asked about their days of abstinence and their confidence in remaining drug-free, rather than their days of use and concern about relapse, and an item assessing progress toward goals was added to better align the measure with an emphasis on achieving patients’ own personal goals for recovery. Several items, such as those assessing depressed mood and craving were rephrased for clarity and ease of incorporation into a brief continuing care contact. When implemented in a study of the effectiveness of extended continuing care for cocaine dependence, the PA was administered at the start of each contact; the treatment manual included specific adaptations to session content and frequency in response to concerns raised by specific items or by overall shifts in assessed risk and protection (McKay et al., 2013a). For example, depressed mood was with relapse prevention counseling regarding coping with unpleasant feelings; simple CBT-informed advice for managing depressed mood; or, referral to community resources for additional treatment beyond that which could be provided within a brief counseling session (McKay et al., 2010a).

While the PA targets factors that have consistently been associated with substance use outcomes in prior work, the predictive validity of the measure has not been explored. In this study we used data from a completed treatment effectiveness study (McKay et al., 2013a) to determine the ability of the PA scales, excluding the items directly assessing substance use, to predict substance use outcomes over a 12-month period. Specifically, we hypothesized that higher risk scale scores would predict greater likelihood of substance use, whereas higher protective scale scores would predict reduced substance use. Given that sustained treatment engagement is associated with more successful outcome, we also hypothesized that the PA could be used to predict continued engagement in or dropout from continuing care.

2. Material and Methods

2.1. Participants

We present data on 158 participants in a completed study of telephone continuing care for cocaine dependence (McKay et al., 2013a). Participants in the present analysis were randomly assigned to receive one of two continuing care treatment conditions and initiated study treatment prior to the first outcome follow-up assessment. An additional 108 participants who were assigned to treatment as usual, 47 participants who never began study treatment, and 8 participants whose treatment entry was delayed past the first follow-up assessment are not included. All participants were adults between the ages of 18–65 who were enrolled in two publicly funded IOPs in Philadelphia and who met criteria for lifetime DSM-IV cocaine dependence with use within the 6 months prior to entering treatment. The other criteria for eligibility were completion of two weeks of IOP; willingness to participate in research and be randomly assigned to a treatment condition; no psychiatric or medical condition that precluded outpatient treatment (e.g., severe dementia, current hallucinations); no regular IV heroin use (i.e., 3 or more times per week) within the past 12 months; ability to read at approximately the 4th grade level; and minimal housing stability (i.e., not living on the street). To facilitate follow-up, participants had to be able to provide the names, addresses, and telephone numbers of at least three contacts.

Participants in the present analysis were on average 43.8 (SD= 7.0) years old and had 11.5 (SD= 1.7) years of education. Most participants were male (79%) and African American (90%). Most met criteria for current cocaine dependence (84%), and nearly half had current alcohol dependence (43%). Participants used cocaine on an average of 43.4% (SD=30.1) of the days in the six months prior to baseline, and drank alcohol on 34.5% (SD= 32.6) of the days. Rates of current/prior lifetime dependence on other drugs were lower (cannabis—14.6%/ 27.9%, opiates—1.9%/8.2%, and sedatives—1.3%/3.8%). Participants averaged 4.1 (SD= 5.1) prior treatments for drug problems.

2.2. Treatment and Therapists

2.2.1. Intensive outpatient treatment (IOP).

The IOP programs provided approximately 9 hours of group-based treatment per week, and patients could attend for up to 3–4 months (McKay et al., 2013a; McKay et al., 2010b). Patients who completed the IOP were typically offered 2–3 months of standard outpatient treatment, i.e., one group counseling session per week, for a total of up to 6 months of “treatment as usual.”

2.2.2. Continuing care treatment.

Participants randomly assigned to receive continuing care treatment received brief telephone calls or face-to-face clinic sessions for up to 24 months, starting in weeks 3–4 of IOP. This treatment was provided in addition to treatment as usual received by all participants. These 20-minute sessions were offered weekly for the first 8 weeks, every other week for the next 44 weeks, once per month for 6 months, and every other month for the final 6 months. Therefore, the total number of possible scheduled sessions in the protocol was 39. Each session began with the administration of the PA. The CBT-based counseling was linked to the results of the PA and also addressed any anticipated risky situations. Participants identified potential coping strategies and behaviors, with help from the counselor as needed, and these were briefly rehearsed during the remainder of the session (McKay et al., 2013a). Participants receiving continuing care were randomized to two conditions, one of which provided $10 gift coupons for each continuing care session completed; incentives increased participation in continuing care but did not affect substance use outcomes (McKay et al., 2013a; Van Horn et al., 2011). Seven therapists, all of whom had prior experience providing outpatient treatment for substance use disorders, delivered both continuing care conditions. All sessions were audiotaped to facilitate supervision and monitor adherence to the protocol as described in the manuals, and any deviations from the treatment protocol were immediately addressed by the clinical coordinator.

2.3. Procedures

2.3.1. Recruitment.

Potential participants were screened and informed consent was obtained for those who appeared eligible after completing 2 weeks of IOP. The study was conducted in compliance with the policies of the Institutional Review Boards of the University of Pennsylvania and the City of Philadelphia. For a detailed description of procedures, including CONSORT diagram, please see McKay et al. (2013).

2.3.2. Assessments.

Comprehensive baseline assessments were completed in week 3 of IOP, and participants deemed eligible were then randomly assigned to receive treatment as usual or continuing care as described above. Follow-up assessments of substance use were completed at 3, 6, 9, 12, 18, and 24 months post baseline. These assessments were conducted by experienced research personnel who were blind to the study hypotheses, but not to treatment condition. The PA was administered at the start of each continuing care session by the study therapist.

2.4. Measures

2.4.1. Progress Assessment (PA).

The PA included questions about cocaine and other substance use since the last continuing care session, five questions about current risk factors for relapse, and five questions about current protective or pro-recovery activities. Since the PA was a very brief clinical tool used to guide the selection of content for continuing care sessions, the aim was to assess a wide range of risk and protective factors, with one item representing each factor. Thus, the guiding principle for item selection was breadth, rather than depth. Each item was rated 0, 1, or 2, representing not present, present to some degree, or fully present. Therefore, total risk and protective scale scores ranged from 0–10, with higher scores on the risk scale representing greater relapse risk and higher scores on the protective scale representing greater protection against relapse. Each item was anchored by frequency counts and/or severity of the particular behavior or internal experience. For example, the item assessing depression is scored 0 for no depression, 1 for depressed less than half the time or lessening of depressed mood over the interval since the last contact, and 2 for depressed more than half the time or worsening of depressed mood during the interval since the last contact (see Appendix 1).

Eight of the 10 risk and protective items were selected on the basis of reviews of the literature and constituted factors that predict or are associated with the onset of relapse, as outlined in the introduction. One additional item on the risk scale assessed whether the patient had attended scheduled medical and/or psychiatric appointments since the last contact, and taken medications as prescribed. One additional item on the protective scale assessed the patients efforts and progress toward specified personal goals during the interval since the last contact. Missing scheduled appointments and taking prescribed medications improperly was seen as a failure in self-care, which was addressed in the continuing care sessions; failure to follow up on mental health appointments or take psychiatric medications was also seen as a potential precursor to the psychiatric instability that can present a risk for relapse. Identification of personal goals that could increase non-substance-related rewards was included in the continuing care intervention as a potential protective factor against relapse. These items were included because they concerned issues that were addressed in the continuing care intervention and are widely believed to be important factors in recovery, although there has been relatively little research on whether they predict outcome. The specific content of several items, including medical/psychiatric adherence, high-risk situations, abstinence-oriented social/leisure activities, and personal goals, were collaboratively tailored to each patient and reviewed at least once every 3 months to ensure that they were relevant to the patient’s current situation.

2.4.2. Urine toxicology.

Urine samples were obtained at each follow-up point to provide an objective measure of cocaine and other drug use (amphetamines, opiates, barbiturates, benzodiazepines, and THC). The samples were tested at the Philadelphia Veterans Affairs Medical Center laboratory with a homogenous enzyme immunoassay method, using established cutoffs for drug positive results. Only the presence or absence of cocaine was included in the present analyses.

2.5. Data Analytic Plan

Both the planned frequency of continuing care contacts – referred to as “calls” as most were completed by phone -- and observed treatment participation dropped substantially in the second year of eligibility, and the follow-up interval changed from every 3 months to every 6 months. Of the 3304 calls made by included participants in the 24 months after randomization, 2666 (81%) were made within the first 12 months. Therefore, to have adequate call data for the analyses, as well as to maintain a consistent follow-up interval, the present analyses include only the first 12 months of participation.

We first conducted a set of preliminary analyses including obtaining descriptive statistics on calls completed in each follow-up period and risk and protective scores obtained. Prior analyses indicated that participants in the incentivized treatment group completed more calls overall (McKay et al., 2013a; Van Horn et al., 2011), and our preliminary analyses indicated the incentivized group made more calls within each 3-month follow-up period. Here, we present descriptive statistics on calls per follow-up period in each group to provide context for our subsequent analyses. Given the variability in number of calls completed, we wished to address whether call completion was related to PA scores. We used generalized estimating equation (GEE; Diggle et al 2012) poisson regression models, with identity link, for the continuously distributed risk and protective scores; to predict mean risk and protective score from number of calls per period, we included the number of calls as time-varying covariates in these models. These models included a binary factor for group, a four-category discrete factor for period, and their interaction.

Our main analyses included prediction of substance use outcomes and retention in continuing care. To predict substance use outcomes, we used GEE logistic regression models with logit link for the binary urine drug screen (UDS) test results, assuming ignorable missingness, and then assuming missing UDS were positive. In all GEE models, we used an exchangeable working correlation structure, and based inference on robust standard errors and confidence intervals. To predict time to drop out from the phone calls from condition and risk scores within each period, we used a stratified Cox regression model, using periods as strata. We used all 158 participants in analyses with numbers of calls made as a response, and the 150 participants who made at least one call for analyses involving the risk and protective scores collected during the calls.

3. Results

3.1. Treatment group comparison on PA completion and scores per follow-up period

Descriptive statistics on call completion and PA risk and protective scores by treatment condition and quarterly follow-up period are presented in Table 1. Overall, the mean risk score was 1.84 (SD = 1.76), and the mean protective score was 7.00 (SD = 2.15) on a scale of 0–10. There were no significant effects of period or condition for the risk score (condition by period: χ2 (3)=3.66, p=0.30, period: χ2 (3)=6.28, p=0.10; condition: χ2 (1)=0.01, p=0.92) or for the protective score (condition by period: χ2 (3)=4.27, p=0.23; period: χ2 (3)=1.29, p=0.73; condition: χ2 (1)=0.89, p=0.34;). There was no association between the number of calls per period and the mean risk score (χ2 (1)=0.16, p=0.69), but a greater number of calls was associated with a lower mean protective score (χ2 (1)=6.40, p=0.01, beta=−0.11, 95% CI [−0.19, 0.03]).

Table 1.

Descriptive statistics on call completion, risk and protective scores, and urine drug screen outcomes for non-incentivized and incentivized participants.

Non-incentivized Incentivized
Mean (SD) calls completed
Q1 4.80 (3.48) 7.80 (3.22)
Q2 2.91 (3.16) 4.60 (3.34)
Q3 2.14 (2.66) 4.02 (3.10)
Q4 1.41 (2.26) 4.07 (3.24)
Mean (SD) risk score
Q1 1.68 (1.41) 1.63 (1.31)
Q2 1.82 (1.52) 1.97 (1.91)
Q3 1.67 (1.76) 1.79 (1.72)
Q4 1.90 (2.07) 1.80 (1.86)
Mean (SD) protective score
Q1 7.08 (2.04) 6.98 (1.97)
Q2 6.75 (2.18) 7.28 (2.19)
Q3 6.48 (2.11) 7.26 (1.98)
Q4 6.88 (1.96) 7.11 (2.42)
Percent positive urine drug screen (missing data ignored)
Q1 22.95 20.00
Q2 22.41 27.14
Q3 27.78 30.88
Q4 31.48 26.87
Percent positive urine drug screen (missing data = positive)
Q1 36.49 28.57
Q2 39.19 39.29
Q3 47.30 44.05
Q4 50.00 41.67
Open in a new tab

3.2. Urine toxicology results

We obtained urine samples on 86%, 81%, 77% and 77% of the participants at the 3-, 6-, 9- and 12-month visits, respectively: there was no significant condition effect on percentage of samples obtained (χ2 (1)=1.24, p=0.27), while the period effect was significant (χ2 (3)=11.78, p=0.01). There was no significant association (p=0.52) between average risk score and having a missing UDS (OR=1.10, 95% CI [0.82, 1.48]), while higher average protective scores were significantly (p=0.002) associated with increased odds of having a missing UDS (OR = 1.41, 95% CI [1.11,1.74]).

Urine drug screen results by treatment condition and follow-up period are presented in Table 1. 21% of urine samples collected were positive for cocaine at 3 months, 25% at 6 months, 30% at 9 months, and 31% at 12 months. When missing UDS were ignored, neither treatment condition (χ2 (1)=0.06, p=0.81) nor follow-up period (χ2 (3)=4.83, p=0.18) effects were significant. Neither the risk score (p=0.32) nor the protective score (p=0.86) showed significant interactions with period, or with condition (p=0.44 for the risk score and p=0.59 for the protective score). Higher average risk scores predicted higher rates of cocaine positive UDS (χ2(1)=5.02, p=0.03) with an odds ratio of 1.25 (95% CI [1.04,1.51]) associated with one unit increase in average risk score. Higher average protective scores predicted lower rates of cocaine positive UDS (χ2 (1)=8.78, p=0.003), with an odds ratio of .78[95% CI = (0.67, 0.90)] associated with a one unit increase in the protective score.

When missing UDS were treated as cocaine-positive, the condition effect remained nonsignificant (p=0.25), while the risk score effect remained significant (p=0.01, OR = 1.24, 95% CI [1.06, 1.46]). The period effect was significant, since the rates of missing UDS were significantly different across periods (χ2 (3)=12.16, p=0.01). The effect of the protective score was nonsignificant (p=0.18, OR = 0.91, 95% CI [0.80, 1.04]), reflecting its separate effects of being associated with higher rates of missing UDS, and lower rates of positive UDS among available UDS.

3.3. Dropout from continuing care participation

For within-period analyses, neither treatment condition nor average protective score showed significant association with time to dropout from the telephone continuing care: estimated hazard rates for incentivized versus non-incentivized treatment condition varied between 0.68 and 1.42, and between 0.96 and 1.10 for average protective score (p > 0.05 in all cases). For the risk score, the estimated hazard rates for the four periods were 0.85 (p=0.04), 1.10 (p=0.18), 1.01 (p=0.86) and 1.08 (p=0.29), respectively. For the protective score, the estimated hazard rates for the four periods were 0.96 (p=0.41), 1.10 (p=0.18), 1.01 (p=0.86) and 1.08 (p=0.29), respectively. For treatment condition, the estimated hazard rates for the four periods were 0.68 (p=0.05), 0.82 (p=0.40), 1.42 (p=0.20) and 1.04 (p=0.89), respectively. For the stratified Cox model using all periods, there were no significant effects, with estimated hazard rates of 0.94 (p=0.57), 0.98 (p=0.56) and 1.03 (p=0.35) for condition, risk score, and protective score, respectively.

4. Discussion

The goal of this study was to examine the predictive validity of 5-item risk and protective factor scales from a brief assessment instrument administered at the beginning of continuing care sessions, in a sample of cocaine-dependent IOP patients. Our main finding was that average scores on both the risk and protective scales within calls completed during a 3-month interval were predictive of cocaine use outcomes as measured by urine drug screens obtained the end of each interval. These results were consistent across the 4 consecutive quarterly follow-up periods, with no difference between incentivized and non-incentivized participant groups. Risk scores remained predictive of outcome when missing urine drug screen results were treated as positive. Protective scores did not predict outcome when missing data were considered positive, because protective scores were associated with higher rates of missing outcome data but lower rates of use among available outcome data. Remarkably, the measure showed validity in real-world conditions that included variability in modality (telephone versus in-person sessions), frequency and timing of administration relative to outcome assessment, and number of assessments per patient. Furthermore, several items on the measure were collaboratively tailored to each patient and periodically updated to reflect changes in their circumstances and goals.

While risk and protective scores were associated with subsequent substance use, the effects were relatively small. This may reflect the fact that the PA was administered in a clinical context and the scores were used to guide active, manualized intervention geared toward reducing risk factors and increasing protective factors. Therefore, treatment may have attenuated the relationship between items measured on the PA and subsequent substance use. The fact that scores on the PA still predicted substance use status when used in the context of an adaptive treatment intervention highlights the need for further work to better incorporate measurement-based approaches in addiction treatment and improve our ability to respond successfully to ameliorate risk and amplify protective factors during treatment.

Lower average protective scores on the PA were associated with higher rates of continuing care call completion, suggesting that those with lower protective factors, such as failure to attend IOP, diminished participation in self-help activities, and infrequent contact with community supports, may have self-selected higher intensity continuing care involvement to provide additional recovery support. There was no evidence that PA scores predicted subsequent dropout from continuing care participation. This is not surprising, given that the measure was designed primarily with substance use, rather than treatment participation, in mind. However, the finding suggests that different factors may be predictive of substance use and treatment dropout.

This study had a number of strengths, including a relatively large sample size, use of urine toxicology as the outcome measure, well-delivered treatments with good adherence provided over an extended period, and a 12-month follow-up. Moreover, the design of the study allowed us to examine a clinician-administered assessment in actual clinical work with adequate adherence monitoring to be sure that the PA was being implemented with fidelity.

However, the study also had some significant weaknesses. Our sample was overwhelmingly lower-income, African American, and male, and excluded patients with significant opioid use. Further research is needed to determine the measure’s performance with more diverse patient populations. In addition, the aim of progress monitoring is not merely to assess patients, but to use assessment data to inform treatment and improve outcomes. We have demonstrated that our measure was able to predict subsequent substance use, but the design of the study precludes making any assertions about whether use of the PA actually improves outcomes. Such conclusions would require an experimental design, in which participants were randomized to the same treatment interventions with and without the PA.

The PA was designed to meet clinician and patient needs for brevity, flexibility, and clinical relevance (Boswell et al., 2015; Solstad et al., 2017). Its 10 items include assessment of constructs commonly associated with relapse risk and abstinence outcomes, with manual-guided tailoring of several items to patients’ unique circumstances. For example, on the risk scale, all patients are asked about depression, self-efficacy, and craving, whereas the item assessing time spent in risky situations specifically references situations and experiences chosen by the patient for monitoring. Similarly, on the protective scale, all patients are asked about coping skills, self-help attendance and contact with a sponsor, whereas the items addressing additional social supports and personal goals require personalization. Therapists successfully implemented the PA within 20-minute continuing care sessions, and used the results to guide each session’s content and suggest modifications to session frequency when indicated by significant shifts in overall scores. Therefore, it showed promise from a clinical perspective, and the present study presents initial evidence that the measure also demonstrates adequate predictive validity for progress monitoring.

5. Conclusions

We developed a brief multidimensional progress monitoring measure for use as a component in adaptive continuing care for substance dependence, with scales reflecting relapse risk as well as factors protecting against relapse. As administered in an effectiveness trial, with several items tailored to participants’ unique circumstances, the PA risk and protective scales predicted substance use status. The 10-item PA shows promise as a pragmatic clinical tool for ongoing monitoring in continuing care for substance dependence.

Supplementary Material

1

Highlights:

  • We tested a brief clinician-administered progress monitoring tool for continuing care

  • Items were chosen for breadth of coverage of factors known to affect relapse risk

  • Scores on 5-item “risk” and “protective” scales predicted urine toxicology outcomes

Funding Source:

This work was supported by the National Institute on Drug Abuse under Grants R01 DA020623 and K24 DA029062; and the Center of Excellence in Substance Abuse Treatment and Education of the Department of Veterans Affairs. The funding sources had no involvement in study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the article for publication.

Footnotes

CRediT author statement:

Deborah H. A. Van Horn: Conceptualization; writing – original draft and review/editing; investigation; project administration

Jessica Goodman: Writing - original draft

Kevin G. Lynch: Methodology; formal analysis; writing - original draft

Marcel O. Bonn-Miller: Methodology; Formal analysis; writing- original draft

Tyrone Thomas: investigation

Kimberly Babson: Formal analysis

James R. McKay: Conceptualization; writing – review and editing; supervision; funding acquisition

Declaration of interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Adamson SJ, Sellman DJ, Frampton CMA, 2009. Patient predictors of alcohol treatment outcome: A systematic review. J. Subst. Abuse Treat 36 (1), 75–86. 10.1016/j.jsat.2008.05.007. [DOI] [PubMed] [Google Scholar]
  2. American Psychiatric Association, 2013. Diagnostic and statistical manual of mental disorders, 5 ed. American Psychiatric Association, Washington, DC. [Google Scholar]
  3. American Psychological Association, 2009. Criteria for the evaluation of quality improvement programs and the use of quality improvement data. Am. Psychol 64 (6), 551–557. 10.1037/a0016744. [DOI] [PubMed] [Google Scholar]
  4. Bonn-Miller MO, Zvolensky MJ, Moos RH, 2011. 12-Step self-help group participation as a predictor of marijuana abstinence. Addiction Research and Theory 19, 76–84 [Google Scholar]
  5. Boswell JF, Kraus DR, Miller SD, Lambert MJ, 2015. Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions. Psychotherapy Research 25 (1), 6–19. 10.1080/10503307.2013.817696. [DOI] [PubMed] [Google Scholar]
  6. Cacciola JS, Dugosh K, Foltz C, Leahy P, Stevens R, 2005. Treatment outcomes: First time versus treatment-experienced clients. J. Subst. Abuse Treat 28 (2, Supplement), S13–S22. 10.1016/j.jsat.2004.09.005. [DOI] [PubMed] [Google Scholar]
  7. Carroll KM, 1998. A cognitive-behavioral approach: Treating cocaine addiction. National Institute on Drug Abuse, Rockville, MD. [Google Scholar]
  8. Connolly Gibbons MB, Kurtz JE, Thompson DL, Mack RA, Lee JK, Rothbard A, Eisen SV, Gallop R, Crits-Christoph P, 2015. The effectiveness of clinican feedback in the treatment of depression in the community mental health system. J. Consult. Clin. Psychol 83 (4), 748–759. 10.1037/a0039302. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Crits-Christoph P, Ring-Kurtz S, Hamilton JL, Lambert MJ, Gallop R, McClure B, Kulaga A, Rotrosen J, 2012. A preliminary study of the effects of individual patient-level feedback in outpatient substance abuse treatment programs. J. Subst. Abuse Treat 42 (3), 301–309. 10.1016/j.jsat.2011.09.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Dolan SL, Martin RA, Rohsenow DJ, 2008. Self-efficacy for cocaine abstinence: Pretreatment correlates and relationship to outcomes. Addict. Behav 33 (5), 675–688. 10.1016/j.addbeh.2007.12.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Farabee D, McCann M, Brecht ML, Cousins SJ, Antonini VP, Lee AB, Hemberg J, Karno M, Rawson RA, 2013. An analysis of relapse prevention factors and their ability to predict sustained abstinence following treatment completion. The American Journal on Addictions 22 (3), 206–211. 10.1111/j.1521-0391.2012.00328.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Freedman MJ, Lester KM, Roth D, McNamara C, Milby JB, 2004. Affective and situational antecedents of crack cocaine relapse risk assessed by Ecological Momentary Assessment, College on Problems of Drug Dependence, San Juan, Puerto Rico. [Google Scholar]
  13. Gamble SA, Conner KR, Talbot NL, Yu Q, Tu XM, Connors GJ, 2010. Effects of pretreatment and posttreatment depressive symptoms on alcohol consumption following treatment in project MATCH. Journal of Studies on Alcohol and Drugs 71 (1), 71–77 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Goodman JD, McKay JR, DePhillipis D, 2013. Progress monitoring in mental health and addiction treatment: A means of improving care. Professional Psychology: Research and Practice 44 (4), 231–246. 10.1037/a0032605. [DOI] [Google Scholar]
  15. Gordon SM, Sterling R, Siatkowski C, Raively K, Weinstein S, Hill PC, 2006. Inpatient desire to drink as a predictor of relapse to alcohol use following treatment. The American Journal on Addictions 15 (3), 242–245 [DOI] [PubMed] [Google Scholar]
  16. Lambert MJ, Whipple JL, Kleinstauber M, 2018. Collecting and delivering progress feedback: A meta-analysis of routine outcome monitoring. Psychotherapy 55 (4), 520–537. 10.1037/pst0000167. [DOI] [PubMed] [Google Scholar]
  17. Litt MD, Kadden RM, Kabela-Cormier E, Petry N, 2007. Changing network support for drinking: Initial findings from the Network Support Project. J. Consult. Clin. Psychol 75 (4), 542–555. 10.1037/0022-006X.75.4.542. [DOI] [PubMed] [Google Scholar]
  18. Ludwig F, Tadayon-Manssuri E, Strik W, Moggi F, 2013. Self-efficacy as a predictor of outcome after residential treatment programs for alcohol dependence: Simply ask the patient one question! Alcoholism: Clinical and Experimental Research 37 (4), 663–667. 10.1111/acer.12007. [DOI] [PubMed] [Google Scholar]
  19. Maisto SA, Connors GJ, Zywiak WH, 2000. Alcohol treatment, changes in coping skills, self-efficacy, and levels of alcohol use and related problems 1 year following treatment initiation. Psychol. Addict. Behav 14 (3), 257–266. 10.1037//0893-164x.14.3.257. [DOI] [PubMed] [Google Scholar]
  20. Martin RA, MacKinnon SM, Johnson JE, Myers MG, Cook TAR, Rohsenow DJ, 2011. The Alcohol Relapse Situation Appraisal Questionnaire: Development and Validation. Drug Alcohol Depend. 116 (1–3), 45–51. 10.1016/j.drugalcdep.2010.11.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. McCrady BS, 2004. To have but one true friend: Implications for practice of research on alcohol use disorders and social networks. Psychol. Addict. Behav 18 (2), 113–121. 10.1037/0893-164X.18.2.113. [DOI] [PubMed] [Google Scholar]
  22. McKay JR, 1999. Studies of factors in relapse to alcohol, drug, and nicotine use: A critical review of methodologies and findings. J. Stud. Alcohol 60 (4), 566–576. 10.15288/jsa.1999.60.566. [DOI] [PubMed] [Google Scholar]
  23. McKay JR, 2009. Continuing care research: What we have learned and where we are going. J. Subst. Abuse Treat 36 (2), 131–145. 10.1016/j.jsat.2008.10.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. McKay JR, Franklin TR, Patapis N, Lynch KG, 2006. Conceptual, methodological, and analytical issues in the study of relapse. Clin. Psychol. Rev 26, 109–127. 10.1016/j.cpr.2005.11.002. [DOI] [PubMed] [Google Scholar]
  25. McKay JR, Merikle E, Mulvaney FD, Weiss RD, Koppenhaver JM, 2001. Factors accounting for cocaine use two years following initiation of continuing care. Addiction 96 (2), 213–225. 10.1080/09652140020020946. [DOI] [PubMed] [Google Scholar]
  26. McKay JR, Van Horn DHA, Lynch KG, Ivey M, Cary MS, Drapkin ML, Coviello DM, Plebani JG, 2013a. An adaptive approach for identifying cocaine dependent patients who benefit from extended continuing care. J. Consult. Clin. Psychol. 81 (6), 1063–1073 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. McKay JR, Van Horn DHA, Morrison R, 2010a. Telephone Continuing Care Therapy for Adults. Hazelden, Center City, MN. [Google Scholar]
  28. McKay JR, Van Horn DHA, Oslin DW, Lynch KG, Ivey M, Ward K, Drapkin ML, Becher JR, Coviello DM, 2010b. A randomized trial of extended telephone-based continuing care for alcohol dependence: Within-treatment substance use outcomes. J. Consult. Clin. Psychol 78 (6), 912–923. 10.1037/a0020700. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. McKay JR, Van Horn DHA, Rennert L, Drapkin ML, Ivey M, Koppenhaver J, 2013b. Factors in sustained recovery from cocaine dependence. J. Subst. Abuse Treat 45 (2), 163–172. 10.1016/j.jsat.2013.02.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Miller SD, Duncan BL, Brown J, Sorrell R, Chalk MB, 2006. Using formal client feedback to improve retention and outcome: Making ongoing, real-time assessment feasible. Journal of Brief Therapy 5 (1), 5–22 [Google Scholar]
  31. Moggi F, Oimette PC, Moos RH, Finney JW, 1999. Dual diagnosis patients in substance abuse treatment: Relationships among general coping and substance-specific coping and one-year outcomes. Addiction 94 (12), 1806–1816 [DOI] [PubMed] [Google Scholar]
  32. Pagano ME, White WL, Kelly JF, Stout RL, Tonigan JS, 2013. The 10-Year course of Alcoholics Anonymous participation and long-term outcomes: A follow-up study of outpatient subjects in Project MATCH. Substance Abuse 34 (1), 51–59. 10.1080/08897077.2012.691450. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Raes V, De Jong CAJ, De Bacquer D, Broekaert E, De Maeseneer J, 2011. The effect of using assessment instruments on substance abuse outpatients’ adherence to treatment: A multi-centre randomised controlled trial. BMC Health Serv. Res 11, 123–131. 10.1186/1472-6963-11-123. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Russell KC, Gillis HL, Law L, Couillard J, 2018. A pilot study examining outcomes associated with the implementation of progress monitoring at a substance use disorder treatment program for adolescents. Child and Youth Forum 47 (3), 403–419. 10.1007/s10566-018-9437-2. [DOI] [Google Scholar]
  35. Schiffman S, Paty JA, Gnys M, Kassel JA, Kickcox M, 1996. First lapses to smoking: Within-subjects analysis of real-time reports. J. Consult. Clin. Psychol 64, 366–379 [DOI] [PubMed] [Google Scholar]
  36. Schneekloth TS, Biernacka JM, Fall-Flavin DK, Karpyak VM, Frye MA, Loukianova LL, Stevens SR, Drews MS, Geske JR, Mrazek DA, 2012. Alcohol craving as a predictor of relapse. The American Journal on Addictions 21 (Supplement s1), S20–S26. 10.1111/j.1521-0391.2012.00297.x. [DOI] [PubMed] [Google Scholar]
  37. Shimokawa K, Lambert MJ, Smart DW, 2010. Enhancing treatment outcome of patients at risk of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system. J. Consult. Clin. Psychol 78 (3), 298–311. 10.1037/a0019247. [DOI] [PubMed] [Google Scholar]
  38. Solstad SM, Castonguay LG, Moltu C, 2017. Patients’ experiences with routine outcome monitoring and clinical feedback systems: A systematic review and synthesis of qualitative empirical literature. Psychotherapy Research 19, 1–14. 10.1080/10503307.2017.1326645. [DOI] [PubMed] [Google Scholar]
  39. Staines G, Magura S, Rosenblum A, Fong C, Kosanke N, Foote J, Deluca A, 2003. Predictors of drinking outcomes among alcoholics. The American Journal of Drug and Alcohol Abuse 29 (1), 203–218. 10.1081/ADA-120018847. [DOI] [PubMed] [Google Scholar]
  40. The Joint Commission, 2018. 2018 Standards for Behavioral Health Care. Joint Commission Resources, Oak Brook, IL. [Google Scholar]
  41. Tonigan JS, Rice SL, 2010. Is it beneficial to have an alcoholics anonymous sponsor? Psychol. Addict. Behav 24 (3), 397–403. 10.1037/a0019013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Valenstein M, Adler DA, Berlant J, Dixon LB, Dulit RA, Goldman B, Hackman A, Oslin DW, Siris SG, Sonis WA, 2009. Implementing standardized assessments in clinical care: Now’s the time. Psychiatr. Serv 60 (10), 1372–1375 [DOI] [PubMed] [Google Scholar]
  43. Van Horn DHA, Drapkin M, Ivey M, Thomas T, Domis SW, Abdalla O, Herd D, McKay JR, 2011. Voucher incentives increase treatment participation in telephone-based continuing care for cocaine dependence. Drug Alcohol Depend. 114, 225–228. 10.1016/j.drugalcdep.2010.09.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Weiss RD, Griffin ML, Mazurik C, Berkman B, Gastfriend DR, Frank A, Barber JP, Blaine J, Salloum I, Moras K, 2003. The relationship between cocaine craving, psychosocial treatment, and subsequent cocaine use. The American Journal of Psychiatry 160 (7), 1320–1325. 10.1176/appi.ajp.160.7.1320. [DOI] [PubMed] [Google Scholar]
  45. Witbrodt J, Kaskutas L, Bond J, Delucci K, 2013. Does sponsorship improve outcomes above Alcoholics Anonymous attendance? A latent class growth curve analysis. Addiction 107 (2), 301–311. 10.1111/j.1360-0443.2011.03570.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Zywiak WH, Connors GJ, Maisto SA, Westerberg VS, 1996. Relapse research and the Reasons for Drinking Questionnaire: A factor analysis of Marlatt’s relapse taxonomy. Addiction 91 (12s1), S121–S130 [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1
NIHMS1616764-supplement-1.doc (31.5KB, doc)

RESOURCES