Assessing Adolescent Substance Abuse Programs with Updated Quality Indicators: The Development of a Consumer Guide for Adolescent Treatment

Abstract

When adolescent substance abuse requires treatment, few parents know which treatment features are important and which treatment programs are effective. There are few resources to help them select appropriate care. We describe early work on an evaluation method and comparative treatment guide for parents based upon the premise that the quality of a program and its potential effectiveness is a function of the number and frequency of evidence-based treatment practices (EBPs) delivered. Thus, we describe the development of and measurement approach for a set of EBPs toward the goal of developing a Consumer Guide to Adolescent Substance Abuse Treatment.

INTRODUCTION

Alcohol and drug abuse among adolescents is a daunting problem worldwide (Johnston, O’Malley, Bachman, & Schulenberg, 2008; Physician Leadership on National Drug Policy, 2002). Beyond the direct negative effects of substance abuse, adolescents who abuse alcohol and other drugs are also more likely to experience school drop-out, unwanted pregnancy, violence, accidents and other significant health and social problems (Delaney, Broome, Flynn, & Fletcher, 2001; Dennis, Dawud-Noursi, Muck, & McDermeit, 2003; Diamond et al., 2002). Prevention and treatment of adolescent substance use disorders is thus a significant public health and safety issue for the country.

Fortunately there are now federally supported projects designed to encourage school nurses and primary care physicians to screen, intervene, and when necessary refer adolescents to treatment – the Screening Brief Intervention and Referral to Treatment (SBIRT) projects (Madras et al., 2009). The question that has perplexed parents, school personnel, and the criminal justice system is how to identify those substance abuse treatment programs that are appropriate and effective for adolescents once need for treatment is established. In fact, when adolescent alcohol and drug abuse requires treatment, few parents (or insurers) know which features of treatment are important and there are few resources to help them select appropriate care. This is not true when a parent wants to make a decision about which college education, health plan or even television to buy for their adolescent. In each of those decisions, consumer-oriented publications offer accurate, comparative information on important features associated with quality and value.

This kind of consumer information actually serves two critical purposes. The most immediate is to inform and direct an individual consumer’s purchase. But a second, longer range purpose is to improve the service marketplace: informed consumers are an essential force for improving availability, quality and costs of services and products – particularly healthcare services (Hibbard, Stockard, & Tusler, 2005; Ippolito, 1992; Hirth, 1999).

The need for consumer information is even more necessary given the relative dearth of adolescent-specific treatment programs throughout the country and the questionable quality of those that do exist. For example, work byKnudsen et al. (2007) analyzing data from the 2003 National Survey of Substance Abuse Treatment Services (N-SSATS) reported that of the roughly 13,600 addiction treatment programs in the country, only 52% of facilities admitted adolescent clients, and only 32% offered “programs or groups” specially designed for adolescents. This situation is even worse today in that less than 30% of addiction treatment programs in this country now offer special programming for adolescents (Mericle et al., Under Review). And even if programs offer special programming,Mark et al. (2006) report that only about 18% of those could reasonably be considered adolescent specialty care programs. Compounding this paucity of programs is the fact that despite many significant advances in substance abuse treatment (e.g., Institute of Medicine, 2005; National Quality Forum, 2007), very few of these scientific advances are in place within most organizations, but particularly within adolescent treatment programs (See McLellan & Meyers, 2004; Meyers & McLellan, 2005; Knudsen et al., 2007; Roman & Johnson, 2002; Mark et al., 2006; Young et al., 2007; Kaminer, Burleson, & Burke., 2008).

The quality problems in adolescent addiction treatment have generated broad concern [See National Institute on Drug Abuse (NIDA), 1999; Institute of Medicine (IOM), 2005; 2006; National Registry of Evidence-based Programs and Practices (NREPP), 2008]. One unified response from both researchers and policy makers has been to urge programs to increase the number of “evidence-based practices (EBPs)” they provide during treatment (e.g., comprehensive intake assessments, family-based care). In order to elucidate the status of services in US adolescent programs, Drug Strategies directed the first major study focusing on the content and quality of real-world adolescent care (See Brannigan, Schackman, Falco, & Millman, 2004; Schackman, Rojas, Gans, Falco, & Millman, 2007). In that study nine “Key Elements of Effectiveness” were derived from literature reviews available at the time and expert panel consensus. Within each of these Key Elements (KEs), five specific empirically and/or expert-recommended practices or “Components” (Cs), operationalized each KE and formed the core of the program rating system. A program director interview was then used to collect simple counts of the Cs among 144 “highly regarded” (via nominations from state authorities, relevant national organizations and recognized experts in the field) adolescent treatment programs throughout the country. All Cs were coded as present or not (yes/no) from the information provided in the Director’s interview. A total score (i.e., number of Cs present overall) and a score for each KE were calculated for each program.

All programs (100%) contacted for that study agreed to participate. Given that these programs were highly regarded, it was surprising that they reported offering only about half of the 45 Cs associated with quality treatment (mean 23, mode 22) (See Brannigan et al., 2004); and there was essentially no change one-year later (Schackman et al., 2007). This research was summarized both conceptually and by program in a user-friendly guide for parents, “Treating Teens: A Guide to Adolescent Drug Programs” (Drug Strategies, 2003), which is still available¹ and in demand.

Since the publication of those findings other researchers have used the Drug Strategies measurement approach to evaluate EBPs within other samples of adult and adolescent programs (See Knudsen, 2009; Henderson, Taxman & Young, 2008; Friedmann, Taxman, & Henderson., 2007; Henderson et al, 2007; Knudsen et al., 2007; Mark et al., 2006; Young et al., 2007). Some have applied the method via interview; some have used similar KEs and Cs to code program reports available from public databases (e.g., NSSATS) that list services offered within programs.

These different methods applied in different samples have all shown very similar results: modest levels of quality program components (See Knudsen, 2009; Mark et al., 2006; Henderson et al., 2007). For example, in one of the more comprehensive studies of adolescent treatment to date,Mark et al. (2006) showed wide variability in the availability of commonly accepted components of effective adolescent treatment such as a comprehensive admission assessment, individualized treatment plans and engagement of parents in treatment and discharge planning. While the availability of aftercare and relapse prevention were reported by more than 80% of the Program Directors surveyed, mental health evaluation, medical evaluation, and HIV counseling and testing were offered by less than 50% of those programs.

In an important methodological study,Henderson et al. (2007) used item response theory (IRT) methods to successfully sum the Drug Strategies measures into a single scoring dimension (Henderson et al., 2007) which suggests that the underlying quality dimension can be captured through a simple, unweighted sum of the individual EBPs (i.e., components) available within the program. Further, these researchers were the first to show suggestive evidence of concurrent validity for this measure of quality. Specifically, the summary score was positively related to independently collected measures of organizational function, which had previously been related to treatment outcome.

One unified response from both researchers and policy makers has been to urge programs to increase the number of EBPs they provide. Many research groups have used program counts of EBPs as indicators of or proxies for treatment quality (See Knudsen, 2009; Henderson, Taxman & Young, 2008; Friedmann et al., 2007; Henderson et al, 2007; Knudsen et al., 2007; Mark et al., 2006; Young et al., 2007; Miller, Zweben & Johnson, 2005). At the federal level, organizations such as NIDA, NIAAA and CSAT have all initiated efforts to increase the number of EBPs provided within community programs; again suggesting these are a direct measure of treatment quality. Many states now require programs to use evidence-based quality features as a condition of Block Grant funding allocations (Mark & Coffey, 2004). This promotion of EBPs by federal and state agencies is a clear indication of the acceptance of these measures as indicators of program quality and effectiveness.

This paper describes research being conducted as part of the Parents Translational Research Center (PTRC) that addresses: 1) the key research assumption underlying national efforts to improve the quality of addiction treatment [i.e., addiction treatments containing more research-derived practices should show better program performance (e.g., retention rates) and patient performance (e.g., reductions in use)]; and 2) the lack of accessible, objective, comparative information on available treatment programs for adolescents. The goals of this research are to update, improve and test the original Drug Strategies measurement approach, and produce and evaluate a web-based Consumer Guide for Adolescent Addiction Treatment for parents whose adolescents need treatment in the Philadelphia metropolitan area. Final products of the work will be a tested comparative evaluation protocol, an engaging, useful presentation format of the web-based Consumer Guide, and a dissemination plan for a Consumer Guide protocol suitable for use in other locales around the United States.

UPDATING KEY ELEMENTS / COMPONENTS (KEs/Cs)

Given that the Drug Strategies list of Key Elements/Components (KEs/Cs) is 10 years old, our first task was to systematically update that original list. To this end, two approaches were undertaken with professionals: a systematic literature review; and consultation with the original researchers and other national experts including convening an expert panel to derive consensus based revision. Also, two approaches were undertaken with parents of substance abusing teens; eliciting feedback from the PTRC’s Parent Advisory Board on this topic and asking parents of youth receiving substance abuse treatment about what is important to them regarding treatment for their youth.

Professional Approaches

Systematic Literature Review

To update and refine the original KEs/Cs, we searched PsycINFO using four exploded terms: ‘treatment’, ‘substance abuse’, ‘empirically-based’, and ‘review’ in combination with ‘adolescent’ as an age category between the years 2002 and June, 2009. This search yielded 176 records. Using abstracts, two doctoral-level psychologists coded the first 30 records and conferred to reach inclusion consensus following completion of every 10 records. Records were eliminated based on two criteria: 1) a focus primarily on prevention, tobacco use, or an issue other than AOD use (e.g., a review of treating violent behavior where substance use is discussed as a risk factor); and 2) not a formal literature review (e.g., case studies, individual clinical studies or surveys, book reviews, handbooks, and commentaries/opinion pieces). Using these criteria, the coders identified 46 records that were deemed formal reviews of empirically-based adolescent substance abuse treatment. Parallel searches were conducted using PubMed Plus and Embase and yielded no new relevant citations.

As the review articles typically focused on therapies [e.g., individual/group cognitive behavioral therapy (CBT), motivational enhancement therapy (MET), multidimensional family therapy (MDFT)], to clarify intervention principles and practices, we reviewed 42 supplemental articles. Special attention was given to the primary research on the prevalence, assessment and integrated treatment of co-occurring disorders. Some principles/practices (e.g., developmentally and culturally-informed treatment, trauma assessment and treatment) also warranted review of primary studies and literature reviews within the more general adolescent behavioral healthcare literature as there were few studies and no reviews with adolescent substance-abusing populations. Thus a total of 88 articles were read, and coded using a qualitative narrative when they provided relevant information to determine support for the original 9 KEs of effective care, suggest any modification of existing KEs/C, and identify new candidate KEs/Cs. The literature review was enhanced with other sources such as state-determinations of evidence-based treatments, National Registry of Evidence-based Practices and Programs (NREPP), Principles of Drug Addiction Treatment (PODAT), PODAT - Criminal Justice, and National Quality Forum (NQF). The results of this combined work culminated in a draft revision of the KEs/Cs wherein guidance was provided iteratively from a variety of experts in person, over the phone and electronically prior to a meeting of the Scientific Advisory Board.

Scientific Advisory Board Involvement

Eight content experts (Arria, Cavanaugh, Dakof, Dennis, Mason, Meyers, McLellan, Winters), three Drug Strategies researchers (Falco, Flynn, Schackman) and four internal experts (Alterman, Cacciola, Pechura, Rosenwasser) provided consultation on the: 1) scope and results of the literature review, and 2) revisions to the original KEs/Cs. Consultation from the experts proceeded in an iterative manner where, prior to a full day meeting, some experts were called upon to review, provide feedback on, and craft items for the revised KEs/Cs. Also, before the full day consensus meeting, expert panel members individually reviewed the updated KEs/Cs and proposed scoring schema. Participants attended the consensus meeting with written feedback prepared for discussion. During the consensus meeting, the first author reviewed the process of the revision and its results which were then discussed. The expert panel’s written feedback and recommendations were also discussed. During the meeting the group accepted, modified, added, and deleted items from the list. Additionally, information was shared about further operationalizing and appropriately describing the KEs/Cs and evaluating their integrity and fidelity. After the consensus meeting, feedback was synthesized, revisions made, and the KEs/Cs distributed for final feedback from expert panel members.

Parent Approaches

Input from The PTRC’s Parent Advisory Board

Our review revealed a salient omission in the published work, the absence of parent input regarding quality of care for their children. As parents typically serve as both payers and key supports for adolescent substance abuse treatment, their perspective is important to consider in the measurement of treatment quality. Parents of substance abusing adolescents comprise the PTRC’s Parent Advisory Board and meet yearly to provide feedback on PTRC activities. They were asked during the annual meeting to review and provide feedback on the updated KEs/Cs. They recommended that we address three additional issues. First, while they lauded the inclusion of mental health as a key element in its own right, they urged us to address the need for rapid provision of mental health services. Parents were unanimous that receiving a mental health assessment could take weeks from the point of referral and the receipt of mental health services even longer. Second, they articulated the intricacies of confidentiality laws emphasizing how such laws frequently preclude a treatment-parent partnership. They recommended that the inclusion of parents in decisions regarding treatment and recovery plans for their child be considered as a component. Third, parents spoke of the need for linkages to post-treatment support for them prior to their child’s discharge which would empower them to play a more effective role in their child’s ongoing recovery.

Parent Informational Interviews

Since parents of adolescent substance users are one of the major anticipated users of the web-based Consumer Guide, it was essential that we heard directly from them about what was important regarding treatment for their child as a validity check of the KEs/Cs. To this end, 22 parents of substance abusing adolescents recruited from programs in the metropolitan Philadelphia area that were participating in our pilot work were interviewed using a 15-item interview that contained primarily open-ended questions. Parent needs and opinions supported various program informational features to be collected as well as a number of the KEs.

When making decisions about where to send their son/daughter, parents were unanimous that “success rates”, average length of stay, and requirements for program completion be made available. Many reported that it was important to be informed about general operations (e.g., hours) and program goals. Also, after their child’s assessment, parents want a discussion of all treatment options with a program staff member so that they can make an informed decision about the best care for their child. When asked what knowledge would be useful during the course of treatment, parents wanted programs to provide information on symptoms of use, course of the disease, and what exactly could and could not be shared with them and other agencies such as the courts and schools. Aspects of treatment that they considered important included engagement approaches, individual sessions, linkages with positive free time activities, aftercare, support systems and reinforcement to balance punitive responses to behavior, coordination with the justice system, anger management, and a support system for them. Finally, when asked what type of communication was desired from a program, parents stated that in-person meetings were important as well as progress reports that included: 1) attitudinal progress, not just urine results; 2) reports of victimization; 3) program content information (e.g., working on anger this week); and 4) specific suggestions and strategies about what they, as parents, can do to help their son/daughter.

While parent feedback touched upon items contained in the program descriptor questionnaire, the specific and nuanced information they provided allowed us to develop more precise items. Also, parent feedback supported aspects of each KE. Clearly, the parent involvement we have had during the early stages of this work illustrate that the KEs have face validity, and that the planned Guide has potential practical value by addressing stated needs of the consumer group most important to the project. Lastly, parent feedback was considered in finalizing the KEs/Cs and their measurement.

Results: Updated Set of KEs/Cs

The synthesis of the information that followed from these four approaches confirmed the importance of the original 9 KEs (with some wording revisions to enhance clarity) and indicated that an additional KE be added: Attention to Mental Health. More extensive revisions to the components however were warranted to include additional items within the KE that had empirical support, to better operationalize the KE, and to incorporate parent recommendations. This process resulted in the following: we retained 2 Components, made generally minor revisions to 27, combined 11 into 5, deleted 5, and added 28. Consequently, there are now 10 KEs with 67 corresponding Cs: Assessment (5 Cs), Attention to Mental Health (6 Cs), Comprehensive Integrated Treatment (8 Cs), Family Involvement in Treatment (6 Cs), Developmentally Informed Programming (6 Cs), Engage and Retain Adolescents in Treatment (5 Cs), Staff Qualifications and Training (8 Cs), Person-First (Culturally Competent) Treatment (6 Cs), Continuing Care and Recovery Supports (7 Cs), Program Evaluation (5 Cs). Table 1 presents the updated set of KEs and Cs of Adolescent Substance Abuse Treatment.

Table 1.

Original and Revised Key Elements and Components of Effective Adolescent Substance Abuse Treatment

ORIGINAL DRUGORIGINAL DRUG STRATEGIES STRATEGIES	REVISED
KE1: Assessment & Matching	KE1: Assessment
1. In its screening and assessment process, does the program use either a standardized substance abuse instrument or a structured clinical interview?	1. In its assessment process, does the program use either a standardized substance abuse instrument or a structured clinical interview?
2. In its screening and assessment process, does the program use a standardized mental health instrument?	2. Does the program have criteria to determine treatment eligibility and level of care?
3. Beyond routinely updating the treatment plan, does the program reassess clients at some point during treatment?	3. Does the program conduct a comprehensive initial assessment that identifies problems as well as assets, interests and resources using either a standardized assessment tool or a structured clinical interview?
4. Is the client’s physical health addressed in the assessment?	4. Does the program have procedures to ensure rapid service provision?
5. Does the program specify that the treatment plan addresses mental health issues?	5. Does the program reassess clients throughout the course of treatment to monitor progress and guide treatment?
Not in Original Treating Teens	KE 2: Attention to Mental Health
KE1 - C2. In its screening and assessment process, does the program use a standardized mental health instrument?	6. In its assessment process, does the program use either a standardized mental health instrument(s) or a structured clinical interview that covers symptoms and behaviors of common co-occurring disorders?
KE1 - C5. Does the program specify that the treatment plan addresses mental health issues?	7. Does the program provide mental health diagnostic evaluation onsite or through referral if indicated?
KE2 - C6. Does the program either provide mental health services for clients onsite or coordinate their care with community mental health providers?	8. Does the program have procedures to ensure rapid mental health service provision?
	9. Does the program specify that the treatment plan addresses mental health issues when indicated?
	10. Does the program provide clients with mental health services (including medication) onsite or coordinates such care with community mental health providers?
	11. Does the program reassess clients’ mental health status and treatment compliance throughout the course of treatment to monitor progress and guide treatment?
KE 2: Comprehensive, Integrated Approach	KE 3: Comprehensive Integrated Treatment
6. Does the program either provide mental health services for clients onsite or coordinate their care with community mental health providers?	12. Does the program address physical health issues by providing medical services either onsite or by referral?
7. Does the program address physical health issues by providing medical services either onsite or by referral?	13. Does the program provide testing, counseling, and education for infectious diseases and sexual health either onsite or by referral?
8. Does the program provide sexual health services, such as testing for STDs, either onsite or by referral?	14. Does the program address educational/vocational needs of in-school and out-of-school youth by coordinating care with the client’s home school system and providing educational/vocational services either onsite or by referral?
9. Does the program maintain communication with the client’s home school system regarding academic issues?	15. For clients involved with the juvenile justice system, does the program maintain contact and coordinate care with juvenile justice officials and have policies in place to protect the rights of clients?
10. Does the program maintain contact with juvenile justice officials regarding clients who have been referred by the juvenile justice system?	16. Does the program facilitate connections with prosocial, recovery oriented community organizations, mentors, activities and alternative peer groups during treatment?
	17. Does the program address ‘other’ addictive behaviors (e.g., gambling, sex/pornography, gaming) including tobacco use?
	18. Does the program assess and continues to monitor clients for trauma and other serious stressors (e.g., family/residential instability, victimization, crime, grief and loss) and provide services either on-site or through referral?
	19. Does the program go beyond problem-focused services by also identifying and building on the client's strengths and protective factors to promote resiliency?
KE 3: Family Involvement	KE 4: Family Involvement in Treatment
11. Does the program assess the client’s family beyond simply reviewing family history or perceptions?	20. Does the program conduct an assessment of family functioning?
12. Does the program provide the client’s family with individual and/or multifamily therapy sessions?	21. Does the program refer parents and household members with alcohol or other drug problems, serious mental health problems, or domestic violence issues to treatment?
13. Does the program maintain contact with the family for the duration of the client’s treatment?	22. Does the program provide family therapy?
14. Does the program specify that family or caregiver involvement is mandatory?	23. Does the program provide and support opportunities for the family to obtain information about and have input in decisions regarding the treatment, recovery, and resiliency plans for their child?
15. Does the program specify that it will refer parents who are abusing substances to treatment?	24. Does the program have procedures in place to maintain contact with families and provide educational and multi-family support groups so as to keep families engaged in their child's treatment?
	25. Does the program involve family members of adolescent substance abusers in programming or planning (e.g., through Board of Director Involvement, Family Advisory Panel, Consumer Satisfaction Surveys)?
KE 4: Developmental Appropriateness	KE 5: Developmentally Informed Programming
16. Does the program vary activities and/or curricula based on the developmental level of the client?	26. Are adolescent clients treated only with other adolescents, as opposed to being integrated with adult clients?
17. Does the program utilize a curriculum designed specifically for adolescents?	27. Does the program vary the way in which information is presented, skills are taught, and therapy is conducted (e.g., concrete content, role-plays) given the ages, maturity and developmental levels of clients?
18. Does the program incorporate experiential or hands-on activities into treatment?	28. Does the program include adolescent-specific courses, recreational programming, or other features of particular interest to adolescents?
19. Does the program specify that its protocol is tailored to the concrete thinking needs of adolescents, rather than abstract thinking?	29. Does the program provide and support opportunities for clients to have input in decisions regarding their treatment, recovery and personal goals?
20. Are adolescent clients typically treated only with other adolescents, as opposed to being integrated with adult clients?	30. Does the program place a client's "disruptive" behavior in a developmental context (e.g., limit testing, moodiness, rebelliousness, impulsivity are common) when determining how to address the behavior and help a client learn from mistakes?
	31. Does the program acknowledge and address the developmental tasks of adolescence (e.g., peer group influences, identity formation, autonomy)?
KE 5: Engage & Retain	KE 6: Engage and Retain Adolescents in Treatment
21. Does the program emphasize building a therapeutic alliance between staff and clients to engage and retain the client?	32. Does the program have procedures to reduce barriers to attendance?
22. Does the program utilize motivational enhancement techniques, such as Motivational Interviewing?	33. Does the program emphasize building a therapeutic alliance between staff and clients to engage and retain the client?
23. Does the program incorporate positive reinforcements, such as increasing responsibilities and/or privileges, in order to provide incentives for client participation?	34. Does the program utilize motivational enhancement techniques initially and throughout the course of treatment to engage and retain clients?
24. Does the program utilize special recreational programming (such as wilderness programming or art therapy) and/or offer courses of particular interest to adolescents (such as sexual health or life skills training) to engage and retain clients?	35. Does the program incorporate contingent positive reinforcement or other incentives to engage adolescents so that they attend and participate in treatment?
25. Does the program make special efforts to incorporate the family in order to keep the client engaged?
	36. Does the program have outreach and reengagement procedures for missed treatment sessions and poor attendance?
KE 6: Qualified Staff	KE 7: Staff Qualifications and Training
26. Does at least one direct service staff member have training in adolescent development?	37. Does the clinical staff have training in adolescent development?
27. Does at least one direct service staff member have training in co-occurring disorders?	38. Does at least one clinical supervisor possess a minimum of a master’s degree in a relevant field?
28. Is at least one member of the direct service staff a trained family therapist?	39. Does the program provide direct service staff with ongoing supervision, feedback and evaluation regarding their clinical skills?
29. Does the program provide ongoing training for direct service staff?	40. Does the program provide ongoing in-service training, and reimbursement or paid leave for direct service staff and supervisors to obtain training?
30. Do all clinical supervisors possess at least a master’s degree?	41. Does the program train counselors in case management or have at least one designated case manager?
	42. Does the program have at least one master’s or higher degreed clinical staff trained in mental health or co-occurring disorders?
	43. Does the program have at least one master’s degreed clinical staff trained in family therapy?
	44. Does the program have a medical professional onsite (i.e., physician, registered nurse, nurse practitioner, or physician assistant)?
KE 7: Gender & Cultural Competence	KE 8: Person-First (Culturally Competent) Treatment
31. Is the program designed to meet the needs of minority youth?	45. Does the program consider the values, worldviews and practices of the client's culture, gender, and sexual orientation when implementing the treatment plan?
32. Does the program provide clients with gender-specific group sessions?
33. Does the content of the program’s individual and group sessions differ according to the distinct needs of males and females?	46. Does the program provide clients with separate gender-specific group sessions and curricula for some topics?
34. Is the program designed to meet the needs of gay and lesbian youth?	47. Is the program designed to meet the needs of lesbian, gay, bisexual, transgendered, and questioning youth (LGBTQ)?
35. Does the program have safety measures in place to ensure boundaries between co-ed patients and staff?	48. Does the program facilitate connections to community groups that align with clients' and families' culture, gender, and sexual orientation?
	49. Does the program have policies and procedures to ensure the emotional and physical safety of youth, to promote respect of difference, and to prevent and address bullying, victimization, and boundary violations from other clients and staff?
	50. Does the program provide cultural competency, sexual harassment, and patient/therapist boundary training to their staff?
KE 8: Continuing Care	KE 9: Continuing Care and Recovery Supports
36. Does the program create a continuing care plan for the client beyond simply referring clients to outside services?	51. Does the program provide relapse prevention services?
37. Does the program specify that the client’s continuing care plan lasts for at least 6 months?	52. Does the program educate clients and their families about continuing care and recovery supports and focus on them throughout the course of treatment?
38. Does the program link clients with relevant community services upon discharge?	53. Does the program provide an individualized transition period of tapered treatment to support recovery?
39. Does the program address relapse prevention?	54. Does the program create a comprehensive continuing care and recovery support plan covering an extended period of time after treatment is completed?
40. Does the program monitor clients with checkups following discharge?	55. Does the program link clients with relevant community services (e.g., adolescent 12 Step meetings, alternative peer groups, mentoring resources) prior to discharge to promote post-treatment service engagement and ongoing recovery?
	56. Does the program link families with relevant community services (e.g., parent support group) prior to discharge to promote ongoing support and recovery for their child?
	57. When treatment is completed, does the program monitor clients with periodic clinical checkups and maintain an ongoing connection with clients to support recovery, service referral, and re-engagement in treatment when indicated?
KE 9: Treatment Outcomes	KE 10: Program Evaluation
41. Does the program collect its own data related to client outcomes (e.g., results of post-discharge follow-up surveys) and/or provide such data to the state?	58. Does the program have a comprehensive electronic medical record?
42. Does the program analyze its internally gathered data in an effort to measure the effectiveness of its treatment services?	59. Does the program analyze its internal program performance indicators (e.g., time in treatment, type of discharge, during treatment substance use, client satisfaction) to measure the effectiveness of its treatment services?
43. Has the program conducted its own formal evaluation of the program's effects on client outcomes?	60. Does the program collect and analyze its own data related to client effectiveness or outcome (e.g., post-discharge outcomes for internal or external reports)?
44. Has an independently conducted formal evaluation of the program's effects on client outcomes been performed?	61. Does the program have others independently conduct formal treatment effectiveness or outcomes evaluations?
45. Did the formal evaluation of the program's effects on client outcomes utilize a scientifically rigorous research design (i.e., either random assignment to conditions or carefully matched treatment and comparison groups)?	62. Has the program used program performance or outcomes data to improve treatment delivery?

There were four KEs within which the Cs underwent relatively, substantial revisions: Comprehensive Integrated Treatment, Developmentally Informed Programming, Staff Qualifications and Training, and Continuing Care and Recovery Supports. Within Comprehensive Integrated Treatment, the original Cs addressing physical and sexual health, educational status, and juvenile justice involvement were retained given the continued importance of these areas for intervention and recovery. Since the time of Drug Strategies’ seminal work, however, four other areas have been increasingly identified as important for intervention and recovery: trauma (DeBellis, 2002; Dennis & Stevens, 2003; Simpson & Miller, 2002), other addictive behaviors (Winters, Arthur, Leitte, & Botzet, 2005), exposure to and interaction with alternative peer groups (Godley et al., 2001), and strengths and resiliency programming (Godley et al., 2001). Consequently, these four areas were operationalized as Cs and used to expand the Comprehensive Integrated Treatment KE.

While addressing mental health was included within Cs under both the Assessment and Comprehensive Treatment KEs in the original Drug Strategies work, the prevalence of co-morbidity among adolescent substance users (Armstrong & Costello, 2002; Meyers & McLellan, 2005) necessitated that this life area have its own KE. Hence, Attention to Mental Health is the new KE. It addresses mental health symptom and diagnostic assessments, mental health inclusion within the treatment plan, monitoring mental health issues, and timely provision of mental health treatment services via six Cs.

As its name change reflects, Developmentally Informed Programming (formerly Developmental Appropriateness) now more specifically recognizes that relative to adults adolescents are in a continuous state of biological, social, cognitive and psychological development and as such have much more difficulty regulating their moods, behavior and impulses (Deas, Riggs, Langenbucher, Goldman, & Brown, 2000; Feldman, & Elliott, 1990). It further recognizes that attention needs to be paid to the developmental tasks of adolescence and queries the program as to how its services were selected, developed and implemented with this developmental period in mind. For example, one C asks whether programs place a client's "disruptive" behavior (e.g., limit testing, moodiness) in a developmental context when determining how to deal with such behavior; another asks how developmental tasks (e.g., autonomy) are acknowledged and addressed.

Insofar as the vast majority of evidence-based treatments for adolescent substance abuse are family based (e.g., Multidimensional Family Therapy, Functional Family Therapy) (Waldron & Turner, 2008), adolescent substance abuse is characterized by extensive heterogeneity in life domains (Meyers & McLellan, 2005), and mental health problems frequently co-occur (Armstrong & Costello, 2002), certain types of education and training can support the effective delivery of treatment to adolescents. While the original Drug Strategies KE Qualified Staff generally acknowledged this, the new KE Staff Qualifications and Training was expanded by adding Cs that specifically address whether adolescent program staff possess advanced degrees in family therapy and/or mental health, and whether staff have training in case management services.

There is evidence that the provision of Continuing Care and Recovery Supports is effective in helping adolescents maintain their treatment gains following an index treatment episode of care. This is particularly true when these supports are initiated early and when responsibility of continuing care contact is shifted from the patient to the provider in order to facilitate rapid linkage (see Godley & Godley, 2011). Consequently, the number of Cs within this KE have increased to include whether programs actually focus on the need for continuing care with youth and parents throughout index treatment, offer tapered treatment, and actively link families with post-treatment support services.

DESIGNING THE MEASUREMENT BATTERY

Updating the Directors’ Interview

The updated list of KEs/Cs necessitated an update of the Drug Strategies Director’s Interview. Our research team organized the data into specific questions to query for the updated listing of KEs/Cs and we relied on members of the Scientific Advisory Board to provide feedback on the interview in various stages of its development. The Treatment Components Inventory for Directors (TCI-D) is comprised of a two-part assessment process: a questionnaire and an interview that can be completed in-person or over the phone. The questionnaire was designed to collect an array of objective, descriptive information about the program and is based upon the Addiction Treatment Inventory (ATI; Carise, McLellan & Gifford, 2000) and the National Survey of Substance Abuse Treatment Services (NSSATS, SAMHSA, 2008). In addition to structural and operating characteristics, the questionnaire part of the TCI-D assesses referral sources, licensing and accreditation status, funding and fee structure, staff mix by responsibility, and the degree to which manualized treatment interventions (e.g., Multidimensional Family Therapy) are used. The interview portion is designed to assess the degree to which programs offer the Components of high quality treatment. Because the provision of these Components typically requires significant training and continued supervision, information about these aspects of delivery adds precision to the measurement procedure. As such we are developing a more fine grained coding strategy that will inform both availability and quality (e.g., 3-point responses; 0 – component not present, 1 – component present w/low fidelity, 2 – component present w/good fidelity). The original Drug Strategies coding system was dichotomous, presence/absence. The revised coding system will allow for more precision while at the same time allowing us to replicate (and evaluate) the original response format. Retaining the original format is important because it has already been used effectively by several researchers and because that format allows the items to be more easily scored from standard government databases such as NSSATS (e.g., Mark et al., 2006).

Once completed, we piloted the draft version of the TCI-D in five intensive-outpatient programs. We used cognitive interviewing methods to elicit information regarding the meaning/understandability of the items, the appropriateness of the responses and overall comments and suggestions on the content and complexity of the questionnaire (Sudman, Bradburn, Schwarz, 1996; Schwarz & Sudman, 1996). We probed for information on what types of difficulties respondents experienced while completing the items and the basis for the response (Willis, 2005). Due to the length of the draft TCI-D, we used modified debriefing techniques, addressing items which appeared troublesome and responses that seemed inappropriate or inadequate at the time, and upon completion of the entire Inventory. This process coupled with test-retest reliability testing of five program directors yielded valuable information in the refinement of the TCI-D currently being used in the validity study (described below). The final TCI-D takes about 2 hours to complete; 30–45 minutes for the questionnaire, and 60–90 minutes for the interview.

An important methodological note is that director reports may actually be optimistic estimates. For example, Roman & Johnson (2002) interviewed 450 Directors of adult treatment programs and found that 44% reported the availability of naltrexone (the authors called this “adoption”). However, less than 12% of patients in those programs actually received the medication (the authors called this “implementation”). As measures of quality components become important purchasing features for parents and other payers, there will be increased pressure on Program Directors to exaggerate the availability of these quality features. For these reasons, future studies should measure both director and patient perspectives (e.g., Roman & Johnson; 2002; Knudsen 2009; McLellan, McKay, Forman, Cacciola, & Kemp, 2005; Etheridge, Smith, Rounds-Bryant, & Hubbard, 2001).

Thus we have developed a parallel, corroborative, patient interview to measure the components patients actually receive in treatment (i.e., Treatment Components Interview – Adolescent). During the process of item development and piloting, we found that several of the Key Elements contained multiple Components for which adolescents could provide little or no information (e.g., Staff Qualifications and Training, Person-First (Culturally Competent) Treatment). We piloted the Treatment Components Interview – Patient (TCI-P) with 17 substance-abusing adolescents in IOP/OP treatment and used cognitive interviews to the refine the instrument. The process with the adolescent interview was however, more iterative as items were revised and retested as difficulties were identified.

Finally, we are developing a third measurement approach of KEs/Cs; a Treatment Components Inventory ‘audit’ procedure (TCI-A) whereby program activity/treatment schedules, policy/procedures manuals, assessments, patient clinical charts, jobs descriptions, clinical databases etc. are obtained and coded. Again, we expect that only some Components will be able to be addressed via this mechanism. We believe that this tripartite measurement strategy (TCI-D, TCI-P, TCI-A) will maximize the validity of the overall assessment of program quality.

TESTING THE MEASUREMENT BATTERY FOR RELIABILITY AND VALIDITY

The last part of developmental work is to formally examine the test-retest reliability of both interviews. This will occur concurrently with validity studies. This work will be conducted within 20 adolescent outpatient programs (IOP/OP) in the Philadelphia metropolitan area.

Test-Retest Reliability of The Treatment Components Inventory – Director

Directors of all participating programs will be expected to participate in two assessments approximately one month apart. At the completion of the second, answers to the first and second assessments will be compared. When discrepant responses are identified, directors will be asked to explain the reason for discrepancy using a modified discrepancy interview protocol (Cottler et al., 1994). Because this is a revision of the Drug Strategies Directors interview and cognitive interviewing (as described) was used during the initial item development work, we expect high retest reliability (>80% exact agreement on items; ICC >.70 on dimension and total scores).

Test-Retest Reliability of The TCI-P

Fifty adolescents from the first three enrolled programs will be recruited for this portion of the study. Adolescents will participate in two interviews approximately three days apart. They will be asked to report on the services that were received and other experiences within the program during their first month of treatment. At the completion of the second interview, answers to the first and second interview questions will be compared. As above using a modified discrepancy interview, when TCI-P answers are discrepant, adolescents will be asked to explain the reason for discrepancy. Because cognitive testing (as described) and an iterative process were used during the more extensive item development and pilot work, we expect high retest reliability (>80% exact agreement on items; ICC >.70 on dimension and total scores).

Although measures of the nature and number of components have face validity as an estimate of treatment quality, they have never been directly related to standard measures of clinical performance. Therefore, we will test the short-term and longer-term predictive validity of the new measures against patient-level changes during treatment and with standard program-level performance indicators.

Short-term Predictive Validity of the Measurement Battery

Fifty adolescents will be recruited from each of 20 adolescent treatment programs and will complete the program interview along with a battery of psychosocial measures [i.e., the Comprehensive Adolescent Severity Inventory (CASI; Meyers et al., 2006), a modified Adolescent Diagnostic Interview – Youth Version (ADI-Y; Winters & Henley, 1993), and a modified time-line follow-back interview (TLFB; Sobell & Sobell, 1992)] after their first month of treatment. The adolescent patients will also provide a urine specimen and blood alcohol content (BAC) reading. They will be re-interviewed monthly for the next two months using the follow-up CASI and the TLFB if they remain in the treatment program. We hypothesize that if the number of components within a treatment program is a valid short-term indicator of treatment performance, then programs that offer (TCI-D and TCI-A) and patients who receive (TCI-P) the most components during treatment (that are matched to their needs) should show the best performance on the clinical measures (e.g., urine test results, treatment completion, school attendance, arrests). Hence, TCI-D, TCI-A, and TCI-P scores should be positively and significantly (p<.05) related to patient-level rates of drug use during treatment, our primary treatment performance measure, as well as to our secondary measures of treatment completion, school attendance and arrests.

Long-term Predictive Validity of the Measurement Battery

If the number and nature of components within a treatment program is a valid predictor of future treatment quality and effectiveness, then the original measures should continue to be related to program performance at a later time point and in a different sample of patients. Thus, we will return to the same 20 treatment programs six months later to collect standard program performance indicators (i.e., completion rate, length of stay, type of discharge) on a new sample of patients (50 patients per program) admitted after the phase above has been completed. Predictive analyses will assess the strength of the relationships between the three quality measures and the primary program outcome of mean length of time in treatment. Hence, TCI-D, TCI-A, and TCI-P scores will be positively and significantly (p<.05) related to program-level rates of time in treatment as well as to type of discharge, in this new samples of patients who enter the same programs six months following the cohort for the short-term validity study.

DEVELOPING THE PRESENTATION FORMAT OF THE CONSUMER GUIDE

Working with the PTRC’s Parent Advisory Board and communications experts (the Partnership at Drugfree.org) we will develop an accessible web-based presentation of the comparative information collected on these programs. We plan to produce an appealing and useful, web-based presentation of the component information: a one-city (i.e., Philadelphia metropolitan area) version of the Consumer Guide. Once completed, we will assess its utility and value to parents and other purchasers.

What might the Consumer Guide look like?

Our thinking at this point is guided by the Consumer Reports presentation format for a typical comparison among brands providing a specific product. The information would be presented as an Internet site devoted to the programs for a single county or a mid-sized to larger city, in this case, Philadelphia. The website would likely have three kinds of information on every treatment program:

1. Standard Program Descriptors: The home page would contain standard data describing the organizational features of the program collected primarily from the questionnaire portion of the TCI-D: number of treatment slots, the age ranges accepted, types of insurance or payment accepted, accreditation information, admission process, etc.

2. Availability and Likelihood of Receiving Evidence-based Principles and Practices: A second page would likely be comprised of at least two columns, one based upon information from the TCI-D, the second from the TCI-P. The score for an individual component would be a number or shaded circle derived from the TCI-D which would indicate for each of the components the extent to which it was available and of quality. We envision that clicking on any of these components would produce a drop-down listing of the rationale for why the practice was included in the guide and how it was scored. The adjacent column would be derived from the TCI-Ps and present the percent of patients who reported receiving the component.

3. Patient or Parent Comments: We anticipate that experiential information will supplement the research data. This would offer potentially important information that could not be captured in standard surveys. For example, a 16 year-old girl might report that she was the only girl in her therapy groups, or a parent may indicate that they felt excluded from the treatment process.

Procedures for Testing Consumer Value

Once a beta version of the guide is formally placed onto a website, widely advertised in the Philadelphia area and linked to other well-known web sources in the addiction field (e.g., Join Together, SAMHSA Treatment Finder, etc.), we will insert a brief questionnaire into the website seeking input from all viewers. This questionnaire will ask for demographics (parent, referral source, adolescent, etc.), their reasons for using the site and their perceptions of the content, ease of use of the information, and attractiveness. These responses will be collected and tabulated each month to assess the amount of traffic to the site, the number of new and repeated visits, the time spent (including downloads) on the site and of course, the responses to the evaluation questions. (Note: We will take care to display the findings as an experimental effort by the Treatment Research Institute. For study purposes, the website will be available for approximately a 1 year period.)

We will also return to our participating programs and ask parent/guardians and referral sources for admissions to participate in a 10-minute phone interview about whether they used the Guide, how they used it, how they liked it, what else would they have liked. Those that did not use the Guide will be asked how they made their treatment decision and if they knew about the Guide, why had they not used it). Revisions to the Guide will be made as appropriate.

SUMMARY AND NEXT STEPS

The research described herein illustrates a key feature of translational research projects: attention to rigorous research principles while at the same time providing practical value to consumers. This approach also attends to the needs of the research agenda and to the needs of consumers. Researchers will likely sum each of the scores on the items, assess the concordance between the TCI-D and TCI-P, and examine patient and organizational characteristics that predict these total scores. As mentioned, the existing Drug Strategies data has already been used by researchers in such ways. Conversely, the individual items (rather than scores) may be more useful for parents and other consumers. We anticipate that consumers will want useful, accurate, understandable information on individual components so they can compare programs or services on the specific features important to them. In fact, we may summarize the Components into narrative form as a companion piece for parents.

In addition to the guide, we plan to package the TCI-D, TCI-A, and TCI-P with their accompanying manuals and procedures for implementation into a program-level measurement protocol that, when transported to the field for similar work in other geographic locations and implemented as described, will yield a valid estimate of patient performance during treatment. We hope that the use of this protocol and the data that result from it will improve the service marketplace by putting pressure on insurers to cover services shown to be effective in treating adolescent substance abuse and by working with SAMHSA to assure critical components (e.g., family services) are reimbursable within health care reform regulations.

Previous work and our research to date suggests that the measurement tasks we have outlined can be completed in a reliable and face valid manner. Predictive validity is crucial and we look forward to collecting and analyzing the data to inform that issue. Our work up to this point has, however, raised some concerns. First, the array of services in IOP/OP may be severely limited as a result of existing and expected funding cuts and insurance reimbursement mechanisms and hence the variability among programs could be minimal. Second, our experience to date is that programs without a juvenile justice relationship/referral source tend to have very low census and hence minimal staffing (e.g., one counselor). This too may limit their ability to provide an array of services and may also impede our ability to have a reasonable sample of adolescent patients. Additionally, the programs with the juvenile justice relationship may result in outcomes that are artifacts of the role of the judge/juvenile justice system rather than a result of program quality. And finally, the majority of patients within these programs so far have been male. If this gender imbalance remains, we may be able to say little about how these components are related to performance and outcomes with female clients.

Despite these potential limitations, we believe that the information that will result from this study will be useful to parents when they are making treatment decisions for their child. This information should also be useful to purchasers such as the juvenile justice system affording them the opportunity to place a youth within a program that offers services the youth needs. Over time, this information could potentially improve the service marketplace particularly when components of treatment shown to be effective in treating adolescents with substance abuse problems are missing from community treatment programs. Providers are often aware of the need to offer services shown to be effective, but insurers and other funders may not reimburse them. Having information illustrating the disconnect between research and practice as a result of reimbursement issues has the potential to empower parents to demand – and receive - more.