Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2019 Sep 1.
Published in final edited form as: J Community Psychol. 2018 May 4;46(7):844–855. doi: 10.1002/jcop.21976

Process Evaluation of a Community Outpatient Program Treating Substance Use Disorders

James L Sorensen 1, Jasmin D Llamas 2
PMCID: PMC6124892  NIHMSID: NIHMS949278  PMID: 30197457

Abstract

Addiction treatment can improve its impact by providing evidence-based care for the variety of problems that accompany substance use disorders. We conducted a retrospective evaluation of a new treatment program in California that aimed at providing multifaceted services through affiliated licensed and certified outpatient providers. The process evaluation used a logic model, focusing on program inputs, activities, and outputs, to understand the services received by the initial 18 clients who entered treatment. Outcomes for these patients were not assessed. Results indicated that clients received a variety of services: On average clients contracted for 118 treatment sessions and received 143 sessions. Among the many types of services provided, the most frequently received were integrative healthcare (averaging 42 sessions), group therapy (32 sessions), and individual therapy (32 sessions). This logic-model process evaluation indicated that a range of services were provided. The comprehensive approach may have promise for extending addiction treatment beyond its usual boundaries.

Keywords: Addiction, evaluation, treatment models, logic model

Community psychologists have long been involved in developing and evaluating treatments for people troubled by addiction (Rappaport & Seidman, 2012). Part of the challenge is that addiction involves interconnected issues that need attention – from a coordinated range of professionals – to address medical, psychological, social, and other co-occurring problems. In 2012 the Center on Substance Abuse and Addiction at Columbia University (CASA, 2012) issued a report that challenged the field to view addiction treatment in the context of a wide caring network that includes medical treatment and a wide range of services.

In a more recent development, the state of Vermont has been piloting a “hub-and-spoke” model for providing medication assisted treatment of opioid use disorder (OUD) (Brooklyn & Sigmon, 2017; Mohlman et al, 2016; Saunders & McGovern, 2016; Simpatico, 2015). In Vermont, regional designated specialty addictions treatment centers (the hubs) provide specialty health, home, and medication assisted treatment services, while teams of health care professionals (the spokes) provide prescribe medication assisted drug treatment. Sigmon & Brooklyn (2017) show that the program had a major impact in Vermont, increasing the state’s OUD treatment capacity, shortening waiting lists, and increasing the number of physicians with waivers prescribe buprenorphine.

In the United States the hub-and-spoke model first came into public consciousness with the deregulation of the airline industry in the early 1980s, using a central hub for services with less comprehensive services in distant sites. In the computing industry this kind of system is known as a “star network”, a local area network (LAN) where all nodes (such as PCs) are directly connected to a central computer. Brooklyn & Sigmon (2017) point out that, while the Vermont approach is a pioneer in attempting use of the hub-and-spoke model to address OUDs, it is not unlike networks used to manage other diseases. A similar approach is Project Echo (Extension for Community Healthcare Outcomes) based in New Mexico (University of New Mexico, 2007), which has used telecommunication as a tool to extend the reach of treatment for a variety of health and human service programs. Among their approaches they hold weekly 2-hour Integrated Addictions and Psychiatry (IAP) “TeleECHO” Clinics focused on supporting primary care providers in evaluating and treating SUDs and behavioral health disorders (Komaromy et al, 2016). The hub-and-spoke model has been gaining traction as a model for addiction treatment, (Knopf, 2016) but there has been little evaluation of the approach in an intensive outpatient model.

Need for Addiction Treatment

While there have been great advances in understanding the causes and correlates of addiction, there remains a lack of specialty drug abuse treatment. In the US, in 2016 7.5% (20.1 million) of people aged 12 or older had a substance use disorder (SUD): One in 3 (37%) struggled with illicit drugs, 3 in 4 (75%) with alcohol use, and 1 in 9 (12%) with illicit drugs and alcohol. Of those 20.1 million people with a SUD, 93.1% received no care at a substance use specialty treatment facility. Despite the individual, social, and financial costs of addiction, there is a profound gap between the science of addiction and current treatment. The infrastructures of many treatment programs are fragile and unstable, making them unable to implement evidence-based care (McLellan & Meyers, 2004).

“Evidence-based” treatment refers to a treatment that has been scientifically tested, subjected to clinical judgment, and determined to be appropriate for the treatment of a given individual, population, or problem area (Sorensen, Hettema, & Larios, 2009). A variety of pharmacotherapies have been attempted as treatment of addiction, and many of these medications have a strong evidence base in treatment of opioid dependence (methadone, buprenorphine, and naltrexone), tobacco dependence (nicotine replacement therapy, bupropion, and varenicline), and alcohol dependence (naltrexone, acamprosate, disulfiram, and topiramate) (National Institute on Drug Abuse, 2009). Likewise, regardless of the efficacy of pharmacotherapy for SUD, there is strong empirical support for many psychosocial interventions (National Institute on Drug Abuse, 2009), including cognitive-behavioral therapy, community reinforcement, contingency management (often providing vouchers to reinforce abstinence), and motivational enhancement. The evidence base for acupuncture as a SUD treatment is mixed at best, indicating little effect on illicit drug use (Margolin, et al 2002; Jordan, 2006; Grant, et al, 2016) and somewhat impact for some types of pain (Asher et al, 2010; Lee & Ernst, 2011).

Most addiction treatment programs have not been subject to scientific evaluation, and evidence-based treatment has not been well integrated into many existing treatment programs. Studies indicate that treatment providers have interest in using approaches that have strong evidence of effectiveness, although providers are more likely to endorse some than other practices. For example, in a survey of 331 treatment providers, Herbeck, Hser, & Teruya (2008) found that most reported using Motivational Enhancement Therapy, Community Reinforcement Approach, and Supportive Expressive Psychotherapy. Similarly Benishek et al (2010) interviewed 136 treatment providers regarding five evidence-supported treatments (ESTs) with varied empirical support: Contingency management (CM), motivational interviewing (MI), relapse prevention (RP), 12-step approaches (TSA), and verbal confrontation (VC). Overall, providers reported positive beliefs about ESTs, although their support of CM was less positive than other treatments; baseline beliefs about empirical support for each intervention were inflated relative to that of expert raters, except for CM; and their ratings of CM continued to be relatively low even after reading information about its effectiveness. In a review of 21 studies that examined the implementation of evidence-based psychosocial SUD interventions into front-line clinical settings. Manuel, Hagedorn, and Finney (2011) found that motivational interviewing or motivational enhancement therapy and contingency management were the most prevalently implemented evidence-based practices.

There is great diversity in the organization and management of SUD programs. Private and publicly funded programs differ in the importance of funding flow, versus demands to maintain the flow of clients to have funding. In an analysis of over 700 substance abuse treatment programs in the United States, Paul Roman and colleagues found that private programs were more likely than publicly funded programs to use pharmacotherapy, while public programs were more like to use voucher-based motivational incentives (Roman, Ducharme, & Knudsen, 2006). Given that few people with addictions receive effective evidence-based treatment (Miller, Sorensen, Selzer, & Brigham, 2006), the need for comprehensive evidence-based treatment is clear.

Understanding the logic of why change can occur is a key to designing and evaluating treatment. A logic model is a systematic, visual depiction of a program’s resources, activities, and changes hoped to be achieved (W.K. Kellogg Foundation, 2004). Use of the model has become a staple for transforming general concepts into specific measurable activities. Logic models have been used to map goals used by addiction treatment programs (e.g., Conrad, et al, 1999; Edwards et al, 1995; Julian, Jones, & Deyo, 1995; Faw, Hogue, & Liddle, 2005). Orwin (2000) suggests developing logic models as a way to assess fidelity in substance abuse programs, which was done in this project.

CHI Recovery Program

CHI Recovery (CHI) is a substance abuse treatment program focused on integrative health care that admitted its first client in June 2013. Located in Northern California, the program attempts to incorporate all of the comprehensive treatments recommended by CASA (2012), including such services as physician oversight and medical care, customized treatment plans, case management, holistic care, psychotherapy, and family services. The CHI program is customized at the beginning and throughout treatment to meet the changing needs of individual clients. CHI is a private program that is competitively priced. At the time of the evaluation the cost was approximately $10,000 per month for the first three months of the intensive outpatient program, with program fees that dropped significantly after the first three months. The program was designed to last for six months.

One can view CHI as serving as a treatment “hub” providing services in house, while also referring some services to affiliated CHI providers. CHI staff and affiliated CHI providers (who can be viewed as ”spokes”) complete all treatment services, using shared protocols and treatment plans, all under the direction of the CHI Clinical Care Director/Program Manager who directs the treatment goals and makes determinations for appropriate care. CHI conducts all intakes and develops individualized treatment plans for each client, including determining the number and type of treatment sessions. CHI provides group therapy, case management, and educational interventions, while affiliated providers conducted all other services. Figure 1 differentiates services provided by CHI staff and affiliated providers.

Figure 1.

Figure 1

Open in a new tab

Services Provided by CHI Staff and Affiliated Providers

Affiliated providers include medical and health professionals from a variety of disciplines and specialties, including a physician, an acupuncturist, and mental health therapists. Affiliated providers were selected as seasoned professionals with appropriate licensure or certification in their specialty areas, and the providers meet regularly in interdisciplinary case reviews to coordinate their approaches for each client. The physician provides all medical care for CHI clients, including general health issues, pain management, blood work, medications, and medical detoxification. Medication assisted treatment is provided to patients as needed. For example, opiate users are commonly prescribed naltrexone, and suboxone is used if the client is unsuccessful without it. An acupuncturist provides acupuncture and nutrition counseling. Mental health therapists conduct the individual and family therapy.

Purpose of Current Evaluation

The aim of the present study was to understand the varied services received by clients who had entered CHI. The intent was to evaluate the degree that clients received the types of services that the program intended to provide within its treatment model, indicating the program’s ability to deliver a broad scope of treatment and to measure the participant’s ability to engage in an extensive outpatient program. This was a process evaluation aimed at examining program implementation and operations (Harachi et al., 1999; Spalding, 2008); client outcomes were not the focus. The evaluation used a logic model to address whether program activities with clients were implemented as intended. Because the program was designed to implement the expanded services advocated by CASA (2012) in a model similar to the hub-and-spoke approach (Sigmon & Brooklyn, 2017), the study results may have implications for the development of new models for SUD outpatient treatment.

Methods

Overview of Tasks and Program Records

A retrospective review was conducted of client treatment delivered by CHI and its affiliated providers. CHI staff provided deidentified records to the evaluators. Regarding human subjects protections, before beginning data collection the investigators self-certified the study following recommendations of the UCSF Institutional Review Board (IRB). The evaluators were acting as consultants to CHI and endorsed statements recognizing the confidentiality of client information. The UCSF IRB indicated this was not human subjects research because the information was deidentified and that self-certification would be appropriate but was optional.

The program’s records were gathered for the treatment of all 18 clients who had entered the program since its inception and were discharged at least six months before the data collection began in January 2016. Selecting all of the initial clients’ records allowed a complete description of program services in its early stages. The rationale for including only patients who had been discharged six months prior to the evaluation was that the lag would be sufficiently distant that the records would be as complete as possible.

The evaluation proceeded in a series of steps. With CHI staff, the evaluators examined the program’s intake and client tracking instruments and clarified the program’s standards for providing treatment, created a pilot evaluation instrument, obtained feedback from CHI staff, revised, and pilot-tested the data collection instrument. With CHI staff the evaluators completed the evaluation instrument to use with each of the 18 clients. The evaluators created a database and, clarified ambiguities with CHI staff, checked consistency of records, and ran preliminary data summaries. A draft report was created, shared with CHI staff for comments and suggestions, and a final report and oral presentation were delivered to CHI staff.

The evaluation tapped the records of CHI and its affiliated providers going back to the start of the program. CHI maintained a master account spreadsheet of services planned and delivered for each client, as well as clinical notes entered on Cliniko practice management software (https://www.cliniko.com/) and hard copies of assessment instruments. Affiliated providers recorded services with various handwritten and computer-generated records. Working with CHI staff, the evaluators examined the records and clarified each record’s purpose.

Application of a Logic Model

CHI’s program approach lent itself to application of a logic model. Figure 2 shows the logic model developed, which guided the evaluation. The logic model visually represents the program’s core components and the general goals each component aims to achieve. Logic models are generally comprised of four major components (Kaplan & Garrett, 2005; W. K. Kellogg Foundation, 2001). 1. Inputs refer to the human, financial, and organizational resources for the project (e.g., case managers, therapists). 2. Activities are what the program will do with the resources. The activities are the interventions intended to bring about the changes or the results. 3. Outputs are the direct products of activities (e.g. assessments, therapy sessions). 4. Outcomes are the specific changes expected. Outcomes are organized to denote the short, medium, and long-term goals expected to result from the program activities. Given this was the initial evaluation stage, a process evaluation was used (Spaulding, 2008), focused primarily on activities, with some attention to outputs, and short-term outcomes of the logic model, aiming to monitor progress and promote mid-course improvements if needed.

Figure 2.

Figure 2

Open in a new tab

Logic Model Guiding Process Evaluation of the Program.

In this project there were the four primary activities: Inquiry and Intake, Assessment and Placement, Physiological Health Care, and Behavioral Health Care. Inquiry and Intake refers to activities that occurred before and during clients’ enrollment into treatment, such as providing information about program orientation and fees, assessing the client problems in various domains, and making treatment plans. Treatment plans included the types of services to be provided and estimated number of sessions to be provided by staff and affiliates. Assessment and Placement refers to activities that occurred at the front end of treatment, such as a physical examination and assessment by a physician, detoxification, and referral to a sober living environment (SLE). As part of the assessment, the physician or therapist specified any co-occurring disorders. Physiological Health Care refers to activities that occur to treat the client’s health problems, such as periodic medical evaluation, prescribing medications, and acupuncture for pain management. Behavioral Health Care refers to activities that occur to address addiction and mental health problem, such as individual and group therapy.

Measures Selection, Verification, and Analytic Strategy

To use the logic model the evaluators winnowed the records by interviewing the staff of CHI and associated providers about services that they considered essential and reviewing samples of data collection forms. The program was new, and as it developed the staff tried and discarded several assessment measures, including the NIDA Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) (National Institute on Drug Abuse, 2012); WHO Disability Assessment Schedule 2.0 (Garin, et al., 2010); Altman Self-rating Mania Scale (SARM (Altman, Hedeker, Peterson, Davis, 1997); Florida Obsessive-Compulsive Inventory (FOCI) (Storch, et al, 2007), and the Physical Symptoms Questionnaire (PHQ-15) (Kroenke, Spitzer, & Williams, 2002). CHI also devised its own Multi-Scale Assessment. These measures were not selected for use in the evaluation because they had been collected on a few clients or were collected early in the program’s life and then discarded after being deemed not useful, or they were used inconsistently. Variables that remained were placed in the logic model, and a draft client tracking database was created. The draft was piloted with two clients, then five, ten, and then all clients remaining, each time sharpening the measurement to focus on variables that reflected the logic model. Outputs (such as session notes) included a total of 133 variables. Not all variables were used, as some were not evaluation indicators or were overly detailed (for example, date service was given). The product was a dataset measuring the 17 outputs listed in the logic model in Figure 2.

Once the indicators were selected and pretested, CHI staff reviewed each clinical file and entered data into the tracking database. Ten percent of records were audited to verify the accuracy of data provided by CHI staff: Client Characteristics plus each of the four types of service (Inquiry and Intake, Assessment and Placement, Physiological Health Care, and Behavioral Health Care), two client charts were randomly selected. Clinic staff located the variable in the client record and provided the count of each desired data point, and the evaluators indicated whether or not the count was an accurate report of what had been entered in the database. This method allowed clinical staff to view identified data, while the evaluation staff were not exposed to identifying information, which was not in the deidentified dataset. When differences were noted the clinicians examined the records to consider the source of the discrepancy. When a discrepancy was noted the evaluators and clinicians discussed the source of error and decided whether to revise the tally or view the information as inaccurate.

Outputs for each activity were entered into an SPSS file for further analysis. Due to the limited sample size, the analysis provided frequencies and descriptive statistics, not inferential statistics; specifically, percentages of endorsement, means, variance or ranges on items as specified in the logic model. Qualitative variables providing descriptive information were not included in analyses but were interwoven into the report to provide context and description as appropriate (e.g. types of abuse and/or traumas).

Results

Client Characteristics

The 18 clients were a varied group with 6 females, 11 males and 1 transgender. Age range was 18-58 (M=37.3; SD=12.8). Only one client reported currently having minor children, 83% lived in California, and 89% reported prior addiction treatments. They were experienced substance users, with self-reported use of alcohol (80%), marijuana (56%), methamphetamines (39%), cocaine or crack (33%), heroin (33%), and MDMA (“ecstasy”) (28%). Median age of first use was 14.5 with a mean of 22.8 years of substance use.

Verification

The verification process showed that the database accurately reflected the program’s record, with a total accuracy rate of 96%, comprised of Client Characteristics (92%), Inquiry and Intake (100%), Assessment and Placement (100%), Physiological Health Care (100%), and Behavioral Health Care (90%). The most frequent discrepancies were not due to inaccuracy of records, but rather to reporting data from the wrong place in the record.

Activities

Inquiry and Intake

During the intake process, clients completed an early evaluation form, intake form, and a program plan including number of contracted sessions, which were determined by client severity, based on responses from the intake forms. All clients contracted for a six-month program, and CHI staff or CHI affiliated providers completed all contracted sessions. Only two clients were court-involved. Most clients (72%) had a parent as a payor, and 28% of clients were self-paying for their treatment. Clients contracted for a mean of 118 sessions, averaging 5.0 medical, 26.5 integrative health, 26.6 individual therapy, 27.3 group therapy, 5.5 relational therapy, 1.8 multifamily therapy, 18.1 case management, and 10.1 educational sessions. Contracted sessions during intake provided a preliminary plan, which could be revised to meet changing client needs and transferred to different areas as treatment progressed, for example, fewer group and more individual therapy sessions than originally planned.

Assessment and Placement

During the assessment process all but three clients completed a form summarizing their alcohol/drug history. Clients reported a mean of 3.1 traumas in their lifetimes. Traumas spanned a breadth of experiences, including childhood neglect, emotional, physical, and sexual abuse, and family deaths. The mean number of co-occurring diagnoses in addition to substance use was 3.4, with 89% of clients having at least one additional co-occurring diagnosis, including 44% with PTSD, 33% with anxiety or depression, and 22% with insomnia. Detoxification was needed for 72% of clients, with 33% having a slow (over 2 weeks) and 39% having a rapid (2 weeks or less) detoxification. Based on initial assessments half of the clients were placed in a SLE. Most clients were drug tested by CHI (72%) and the others by their SLE. All clients completed a medical evaluation by the physician over the course of two appointments.

Physiological Health Care

Clients contracted for physiological health activities that included medical and integrative health sessions. Table 1 displays the contracted and client-received sessions. Medical evaluations occurred with all clients in two sessions, and medical follow-ups averaged 3.4, varying widely from 0 to 14 depending on clients’ medical needs during treatment. All clients completed their initial medical evaluations, and 39% of clients completed the five contracted sessions or more. The mean integrative health sessions received (41.5) exceeded the contracted services. All clients received acupuncture and nutritional consultations, 50% received massage, and 39% utilized pain management.

Table 1.

Contracted Activities and Services

Activity Type of Session Sessions contracted for mean, range) Sessions received by clients (mean, range) Mean proportion of services (delivered/contracted for) Proportion receiving all or more of contracted sessions
Physiological Health Care
Medical (evaluation and follow up) 5.00 (5-5) 5.39 (2-16) 107.8% 39.0%
Integrative health (acupuncture, massage, pain management, nutritional consultation) 26.50 (26-27) 42.00, 25-84 158.5% 88.9%
Behavioral Health Care
Individual therapy 26.56 (24-28) 31.83 (15-48) 119.8% 77.8%
Group therapy 27.33 (24-28) 31.83 (12-56) 116.5% 50.0%
Relational therapy 5.50 (4-6) 1.89 (0-5) 34.4% 11.1%
Multifamily therapy 1.83 (0-6) 1.22 (0-4) 66.7% 61.1%
Case management 18.11 (12-22) 12.72 (1-45) 70.2% 5.6%
Educational 10.11 (0-14) 9.93 (0-19) 98.2% 38.9%
TOTAL All contracted sessions 118.00 (79-130) 143.50 (86-201) 121.6% 66.7%
Open in a new tab

Behavioral Health Care

Clients contracted for behavioral health sessions that included several types of psychotherapy (individual, group, relational, and multifamily), case management, and educational sessions. As noted in Table 1, 78.8% of clients received or exceeded the contracted amount, with a mean of 26.6 sessions received. Half of all clients received at least the 27 group sessions contracted for, with a mean of 27.3 sessions received. Only 11% of clients received the five relational therapy sessions contracted for, and 61% received the two multifamily therapy sessions as contracted for. Only one of the 18 of clients received the 18 case management sessions contracted for, with a mean of 12.7 sessions received. While clients were contracted for 18 case management sessions, the program was designed so that half of these sessions (9) were to be used during the program and half were to be used at follow-up. At the time of this report, follow-up sessions had not been tallied and could not be examined. Only 39% of clients received all the 10 educational sessions contracted for. While not contracted for (so not in Table 1), crisis intervention sessions occurred when needed. Sixty-one percent of clients utilized at least one crisis intervention session with an average of 5.7 sessions (range: 0-24).

Discussion

This evaluation examined the services received by the first clients to enter CHI, an innovative program offering outpatient care using both core staff and affiliated professionals from multiple disciplines. A logic model (Figure 2) was devised to represent the program’s operation, measuring program inputs, activities, and outputs that reflected the core program model, with types of service that illustrated Inquiry and Intake, Assessment and Placement, Physiological Health Care, and Behavioral Health Care. Regarding Inquiry and Intake, CHI made plans for an active treatment with each client. On average, clients contracted for over 120 sessions of care over six months, including medical; integrative health; individual, group, relational, and multi-family psychotherapy; case management; and educational sessions. Regarding Assessment and Placement, clients indicated multiple traumatic events in their lifetime as well as multiple co-occurring diagnoses in addition to SUDs. Regarding Physiological Health Care, clients met with a physician at least twice for medical evaluations and on average engaged in five face-to-face medical sessions during treatment. Integrative health care was a large part of services provided, averaging 42 sessions per client, most frequently acupuncture and massage. Regarding Behavioral Health Care, psychotherapy was also a mainstay of services provided: The average number of sessions received was 70, most frequently individual or group therapy. Regarding Contracted and Delivered Services, CHI, its clients, and their payers contracted for an average of 120 sessions of Intake, Assessment and Placement, Physiological Health Care, and Behavioral Health Care. Program records indicated that the CHI program provided this level of service or more.

The results of the study add value to several developments in the field. First, the 21st Century Cures Act provides substantial support to increase the use of medication-assisted treatment in US states with unmet needs for opioid treatment (Knopf, 2017). The Cures Act funded $485 million grants to the states in May 2017, and 23 states included the Echo or hub-and-spoke approaches in their plans. The present evaluation is a small exploration of a similar treatment model that provided intensive outpatient services in a single community. It provides another reference point for understanding more about the hub-and-spoke model’s application to SUD treatment. Second, we note that studying the process of treatment is important, even when outcomes of services are unknown. Studying feasibility is a critical step in development of more effective treatment models, even when the treatments are not evidence-based. Programs vary in their ability to meet stated process goals. For example, in one of the few evaluations of a single drug abuse treatment program using a logic model Faw and colleagues (2005) found that in a residential program only about half of the planned services were delivered. In contrast, in the present evaluation the level of services delivered for the most part matched or exceeded the services that were contracted for, which is a positive sign that the program’s model was feasible in this setting. Third, Julian and colleagues (1995) pointed out that a program’s desired impacts sometimes far exceed the capacity of staff to achieve them. For example, the goal “reduce community substance abuse” is unlikely to be achieved without concerted community effort. But there are program outcomes that staff can achieve by concentrating on achieving short-term objectives, which was the approach used in the present evaluation. Most importantly, there is a need for comprehensive treatment programs for the multiple problems that accompany addiction. The CHI approach is ambitious in its pursuit of addressing these multiple problems in the context of outpatient treatment. Compared with inpatient settings, outpatient programs have less control over the daily experience of their clients, yet the outpatient approach has considerable flexibility, in that it can link clients to the evidence-based treatments and settings available in the community. This is a capacity worth further review and development.

The evaluation findings produced some recommendations for the CHI program as well. These recommendations may have utility for other programs attempting to apply this treatment model to their treatment programs. It is important for programs to work to ensure that their desired program model reflects actual program implementation. At CHI the inquiry and intake process would benefit from using consistently a set of validated measures. In this new program, more structured assessment would seem necessary to evaluate whether appropriate and evidence based interventions were delivered. Process evaluations, such as this, may help to align the treatment program with its goals. For example, providing multi-family therapy was a CHI program goal, but at its initial stages the program had too few family-connected clients to offer much multifamily therapy. Additionally, it can be beneficial for the program to be aware of service utilization data to adjust the mix of services to fit the clients’ needs expressed at intake. Services could be adjusted (up or down) to aid in program planning as well as in tailoring treatment for clients. A further suggestion is improve the tools for gathering demographic and background information. As with most programs, having a comprehensive electronic health record can aid in streamlining program data, which is especially useful for programs that depend on both staff and affiliated professionals to provide clinical records. Although the collaborative nature of the services provided makes for a naturally diverse set of treatment records, having more standard definitions of terms and operational issues would be of substantial benefit for the program.

Limitations and Future Research

This evaluation has a number of boundary conditions that can be expanded in further study. First, the evaluation did not address the quality of the treatment, and the evaluation did not focus on outcome indicators. The logic model did not distinguish between services that were versus were not evidence based: Our methodology counted all services regardless of their efficacy. We agree with the suggestion that the field would be improved if there was a usable scale that evaluated the evidence base of practices and provided greater weight to those with proven efficacy. Building such a scale was beyond present evaluation’s resources. Second, CHI was a new program that evolved based on early experiences. Assessing the records of the program’s first 18 clients is sufficient for providing general participant descriptions, but far too low to make comparisons across clients, types of services provided, or services delivered by program staff versus affiliated staff. The medications prescribed by physicians were not included in the CHI records. Different affiliated providers used different record-keeping systems to document their activities, which made it difficult to compare across providers on anything more than that a session occurred and on what date. Few indicators were available to reflect the organization’s performance on the treatment areas in its program model, and standardized, validated measures were not available. Although using a logic model–as had been done in earlier research–provided a beneficial structure on which to build, the evaluation would have more utility for the field if it had measures developed in earlier research. The state of Vermont’s hub-and-spoke program (Brooklynn & Sigmon, 2017) has developed some of its own measures including a Treatment Needs Questionnaire, and further psychometric development of these measures can be a boon to the field. Although drug screens were performed for clinical purposes, the results of the screens were not part of the research reported here. Some clients were tested at residences (such as SLEs) where results were not part of the CHI records, and the timing of screens did not correspond to when clients entered the CHI program (for example tests may have been conducted to verify suspected drug use rather than monthly from admission). Finally, the program provided a number of additional support services—such as accompanying clients to appointments—that were not in the clinical records, and these services were also potentially important interventions. Future evaluations can expand on these study limitations as well as make comparisons to other approaches to care. Lack of a comparator clinic is clear methodological limitation.

Although the CHI program strove to and did offer many evidence-based services, not all aspects of care had a strong base of scientific evidence. The logic model did not distinguish between services that were evidence-based versus not evidence-based, and our methodology counted all services received regardless of their evidence of efficacy.

We also note that providing multifaceted services in an intensive outpatient model is a challenging mission. The current treatment systems may be designed so that intensive treatments are primarily residential. For example, licensing or certification seems necessary if the program is to be a referral site for courts and reimbursed through third-party payers. A key problem has been that the licensing requirements are replete with standards made for safe facility operation, such as smoke and fire alarms. The outpatient certification issues are similar, since CHI is not a brick and mortar facility, but the regulations assume that all services will occur under one roof. CHI’s client services occur in the private offices of licensed clinicians throughout the community. For smoother operation of this kind of facility state requirements for licensing or certification may need to change, and that is a long process. At the time this report is being completed, CHI is again attempting certification by the State of California and is centralizing more of their administrative services but challenges remain, given the outsourced model used in the delivery of care.

Acknowledgments

This research was supported in part by a contract with CHI Recovery, Sebastopol, CA. Additional support was from NIH/NIDA UG1DA015815. The authors are grateful for the participation of the Sarin Pakhdikian for assistance in gathering data as well as the staff and patients of CHI Recovery. The opinions expressed are those of the author and not necessarily the funders.

Contributor Information

James L. Sorensen, Department of Psychiatry, University of California, San Francisco

Jasmin D. Llamas, Department of Counseling Psychology, Santa Clara University

References

  1. Altman EG, Hedeker D, Peterson JL, Davis JM. The Altman Self-Rating Mania Scale. Biological Psychiatry. 1997;42(10):948–955. doi: 10.1016/S0006-3223(96)00548-3. [DOI] [PubMed] [Google Scholar]
  2. Asher GN, Jonas DE, Coeytaux RR, Reilly AC, Loh YL, Motsinger-Reig AA, Winham SJ. Auriculotherapy for pain management: A systematic review and meta-analysis of randomized controlled trials. Journal of Alternative and Complementary Medicine. 2010;16(10):1097–1108. doi: 10.1089/acm.2009.0451. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Benishek KC, Kirby KC, Dugosh KL, Padovano A. Beliefs about the empirical support of drug abuse treatment interventions: A survey of outpatient treatment providers. Drug and Alcohol Dependence. 2010;107(203):202–208. doi: 10.1016/j.drugalcdep.2009.10.013. [DOI] [PubMed] [Google Scholar]
  4. Brooklyn JR, Sigmon SC. SC. Vermont hub-and-spoke model of care for opioid use disorder: Development, Implementation, and Impact. Journal of Addiction Medicine. 2017;11(4):286–292. doi: 10.1097/ADM.0000000000000310. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Center for Behavioral Health Statistics and Quality. (HHS Publication No. SMA 16-4984, NSDUH Series H-51).Key substance use and mental health indicators in the United States: Results from the 2016 National Survey on Drug Use and Health. 2017 Retrieved from http://www.samhsa.gov/data Accessed September 14, 2017.
  6. Center on Addiction and Substance Abuse. Addiction medicine: Closing the gap between Science and practice. CASA: The National Center on Addiction and Substance Abuse; New York, NY: 2012. p. 573. http://www.centeronaddiction.org/addiction-research/reports/addiction-medicine. [Google Scholar]
  7. Conrad KJ, Matters MD, Hanrahan P, Luchins DJ. Homeless prevention in the treatment of substance abuse and mental illness: Logic models and implementation of eight American projects. Haworth Press; New York, NY: 1999. p. 137. [Google Scholar]
  8. Edwards ED, Seaman JR, Drews J, Edwards ME. A community approach for Native American drug and alcohol prevention programs: A logic model framework. Alcoholism Treatment Quarterly. 1995;32(2):43–62. [Google Scholar]
  9. Faw L, Hogue A, Liddle HA. Multidimensional implementation evaluation of a residential treatment program for adolescent substance abuse. American Journal of Evaluation. 2005;26(1):77–93. [Google Scholar]
  10. Garin O, Ayuso-Mateos JL, Almansa J, Nieto M, Chatterji S, Vilagut G, Ferrer M. Validation of the “World Health Organization Disability Assessment Schedule, WHODAS-2” in patients with chronic diseases. Health and Quality of Life Outcomes. 2010;8:51. doi: 10.1186/1477-7525-8-51. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Grant S, Kandrack R, Motala A, Shanman R, Booth M, Miles J, Sorbero M, Hempel S. Acupuncture for substance use disorders: A systematic review and meta-analysis. Drug and Alcohol Dependence. 2016;163(1):1–15. doi: 10.1016/j.drugalcdep.2016.02.034. [DOI] [PubMed] [Google Scholar]
  12. Harachi TW, Abbott RD, Catalano RF, Haggerty KP, Fleming CB. Opening the black box: Using process evaluation measures to assess implementation and theory building. American Journal of Community Psychology. 1999;27(5):711–731. doi: 10.1023/A:1022194005511. [DOI] [PubMed] [Google Scholar]
  13. Herbeck DM, Hser Y-I, Teruya C. Empirically supported substance abuse treatment approaches: A survey of treatment providers’ perspectives and practices. Addictive Behaviors. 2008;35(5):699–712. doi: 10.1016/j.addbeh.2007.12.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Jordan JB. Acupuncture treatment for opiate addiction: A systematic review. Journal of Substance Abuse Treatment. 2006;30(4):309–314. doi: 10.1016/j.jsat.2006.02.005. [DOI] [PubMed] [Google Scholar]
  15. Julian DA, Jones A, Deyo D. Open systems evaluation and the logic model: Program planning and evaluation tools. Evaluation and Program Planning. 1995;18(4):333–341. [Google Scholar]
  16. Kaplan SA, Garrett KE. The use of logic models by community-based initiatives. Evaluation and Program Planning. 2005;28(2):167–172. [Google Scholar]
  17. Knopf A. Vermont’s hub-and-spoke system could be model for OTPs. Addiction Treatment Forum. 2016 Oct 11; http://atforum.com/2016/10/vermonts-hub-spoke-system-model-otps/, Accessed February 20, 2017.
  18. Knopf A. $1 billion for opioids from Cures Act, but ACA repeal could undercut the benefits. Addiction Treatment Forum. 2017 Feb 13; http://atforum.com/?s=21st+century+cures+act, Accessed March 29, 2017.
  19. Komaromy M, Duhigg D, Metcalf A, Carlson C, Kalishman S, Hayes L, Arora S. Project ECHO (Extension for Community Healthcare Outcomes): A new model for educating primary care providers about treatment of substance use disorders. Substance Abuse. 2016;37(1):20–24. doi: 10.1080/08897077.2015.1129388. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Kroenke K, Spitzer RL, Williams JB. The PHQ-15: validity of a new measure for evaluating the severity of somatic symptoms. Psychosomatic Medicine. 2002;64(2):258–266. doi: 10.1097/00006842-200203000-00008. [DOI] [PubMed] [Google Scholar]
  21. Lee MS, Ernst E. Acupuncture for pain: An overview of Cochrane reviews. Chinese Journal of Integrative Medicine. 2011;17(3):187–189. doi: 10.1007/s11655-011-0665-7. [DOI] [PubMed] [Google Scholar]
  22. Manuel JK, Hagedorn HJ, Finney JW. Implementing evidence-based psychosocial treatment in specialty substance use disorder care. Psychology of Addictive Behaviors. 2011;25(2):225–237. doi: 10.1037/a0022398. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Margolin A, Kleber HD, Avants SK, et al. Acupuncture for the treatment of cocaine addiction: A randomized controlled trial. JAMA. 2002;287:55–63. doi: 10.1001/jama.287.1.55. [DOI] [PubMed] [Google Scholar]
  24. McLellan AT, Meyers K. Contemporary addiction treatment: A review of systems and problems for adults and adolescents. Biological Psychiatry. 2004;56:764–770. doi: 10.1016/j.biopsych.2004.06.018. [DOI] [PubMed] [Google Scholar]
  25. Miller WR, Sorensen JL, Selzer JA, Brigham GS. Disseminating evidence-based practices in substance abuse treatment: A review with suggestions. Journal of Substance Abuse Treatment. 2006;31(1):25–39. doi: 10.1016/j.jsat.2006.03.005. [DOI] [PubMed] [Google Scholar]
  26. Mohlman MK, Tanzman B, Finison K, Pinette M, Jones C. Impact of medication-assisted treatment for opioid addiction on Medicaid expenditures and health services utilization rates in Vermont. Journal of Substance Abuse Treatment. 2016;7:9–14. doi: 10.1016/j.jsat.2016.05.002. [DOI] [PubMed] [Google Scholar]
  27. National Institute on Drug Abuse. Principles of drug addiction treatment: A research-based guide. 2nd. Washington, DC: National Institutes of Health; 2009. (Publication No. 09-4180). [Google Scholar]
  28. National Institute on Drug Abuse. Resource guide: Screening for drug use in general medical settings. 2012 Mar 1; Retrieved from https://www.drugabuse.gov/publications/resource-guide-screening-drug-use-in-general-medical-settings. Accessed October 1, 2017.
  29. Orwin RG. Methodological challenges in study design and implementation. Addiction. 2000;95:309–27. [Google Scholar]
  30. Rappaport J, Seidman E. Handbook of community psychology. New York: Kluwer Academic/Plenum; 2012. [Google Scholar]
  31. Roman PM, Ducharme LJ, Knudsen HK. Patterns of organization and management in private and public substance abuse treatment programs. Journal of Substance Abuse Treatment. 2006;31:235–243. doi: 10.1016/j.jsat.2006.06.017. [DOI] [PubMed] [Google Scholar]
  32. Saunders EC, McGovern MP. Responding to the Opioid Epidemic: Vermont’s Hub and Spoke Model as a System of Care. The Addiction Newsletter. 2016 Spring;2016:32–34. [Google Scholar]
  33. Simpatico TA. Vermont responds to its opioid crisis. Preventive Medicine. 2015;80:10–11. doi: 10.1016/j.ypmed.2015.04.002. [DOI] [PubMed] [Google Scholar]
  34. Sorensen JL, Hettema JE, Larios S. What is evidence-based treatment? In: Miller P, editor. Evidenced-based addiction treatment. San Diego, CA: Elsevier; 2009. pp. 3–20. [Google Scholar]
  35. Storch EA, Bagner D, Merlo LJ, Shapira NA, Geffken GR, Murphy TK, Goodman WK. Florida Obsessive-Compulsive Inventory: Development, reliability, and validity. Journal of Clinical Psychology. 2007;63:851–859. doi: 10.1002/jclp.20382. [DOI] [PubMed] [Google Scholar]
  36. Spaulding DT. Program evaluation in practice: Core concepts and examples for discussion and analysis. San Francisco: Jossey-Bass; 2008. [Google Scholar]
  37. University of New Mexico. Project Echo Project ECHO (2017) 2017 Retrieved from https://echo.unm.edu/. Accessed October 6, 2017.
  38. W. K. Kellogg Foundation. W.K. Kellogg Foundation logic model development guide. W.K. Kellogg Foundation; Battle Creek, MI: 2006. Retrieved from https://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide. Accessed February 20, 2017. [Google Scholar]

RESOURCES