Data Collection Strategies and Measurement Tools for Assessing Academic and Therapeutic Outcomes in Recovery Schools

Abstract

Accurate evaluation and documentation of the efficacy of recovery schools can be vital to the continuation and expansion of these beneficial resources. A very limited data set currently exists that examines the value of specific schools established to support adolescents and young adults in recovery; additional research is necessary. The following article outlines the methodology utilized in a current quasi-experimental study evaluating both academic and therapeutic outcomes of adolescents attending recovery high schools as compared to traditional (non-recovery-based) high schools. The developmental considerations in assessing adolescents in recovery and their parents is delineated in this article, which underscores the need for extensive knowledge of adolescent substance abuse and other mental health issues. In addition, sensitivity around privacy among adolescents, parents, schools, and health providers is highlighted, as well as the validity of assessment. Key assessment strategies, including protocol of recruitment and interviewing techniques, are also presented along with a list of parent and adolescent assessment instruments and their corresponding interpretive variables. Protocol recommendations for future research are also outlined.

Recovery from substance use disorders (SUDs) can be a very fragile stage, during which individuals try to avoid and cope with the triggers that may contribute to a relapse. This fragility may intensify within the adolescent and young adult population. Studies suggest that roughly 66% of youth return to using drugs within 6 months following treatment (Cornelius et al., 2003), and about 85% of adolescent drug abusers report at least some use within the 1st year following treatment (Winters, Stinchfield, Opland, Weller, & Latimer, 2000). Triggers for relapse vary by individual and circumstance but can be as mundane as hearing a certain song on the radio or as significant as hanging out with friends from the old drug crowd (Gonzales, Anglin, Beattie, Ong, & Glik, 2012). For adolescents, two significant triggers for relapse are related to attending school: (a) school stress, including failing classes and facing discipline issues and (b) socialization processes, defined by peer pressure, media influence, social networks, and social norms (Gonzales et al., 2012). The school often serves as the site of social connections from which adolescents gain exposure to drug-using peers, where they obtain or sell their drugs, and also the locationwhere the stressors originate (such as academic struggles or interpersonal struggles with peers or authorities). Thus, a return to the same school environment following a treatment episode may make an adolescent in recovery more likely to relapse.

From this recognition of schools serving as a strong trigger for relapse, the first recovery high school was established in 1979, growing to more than 30 schools in 10 states over the course of two decades (Finch & Wegman, 2012; Moberg & Finch, 2008). Recovery schools exist at both the high school and collegiate levels and aim to provide a place where students can complete their education amidst the company of other peers in recovery, as well as staff that understand the unique challenges for youth in recovery and who provide the necessary support. Although the structure and format of recovery schools can vary widely, they typically include academic classes integrated within a curriculum that focuses explicitly on substance abuse issues and recovery support, thereby targeting success in both education and recovery (Association of Recovery Schools, 2013). For a more thorough description of Recovery High Schools (RHSs), please see the article by Moberg and Finch (2014/this issue) or Finch and Wegman’s (2012) review of the topic.

As directed by local, state, and national academic standards, all high schools have an extensive list of expectations to guide their instructional efforts. For RHSs, those academic standards can be difficult to attain while also providing recovery services, including regular support groups, individual counseling, and remedial instruction to compensate for prior poor performance. In an era of precarious funding and frequent budget cuts to school districts (“School Finance,” 2011), evidence demonstrating the efficacy of these recovery schools is necessary in order to secure funding. Are RHSs providing the same level of academic rigor as the more traditional schools? Do students maintain a more stable recovery pattern as a result of the RHS structure? Do adolescents in recovery schools fare better on mental health and support system outcomes? We know of only one published study that has addressed this issue. Moberg and Finch (2008) conducted a descriptive study of RHSs, in which the researchers conducted in-person site visits to 17 different schools. The researchers conducted oral interviews with key staff as well as a brief written survey completed by both the high school students and staff. Results of their study indicate that students reported a significant reduction in their substance use while attending the RHS. Students also reported experiencing fewer mental health symptoms, suggesting beneficial levels of recovery support provided by RHSs (Moberg & Finch, 2008). Moberg and Finch (2014/this issue) further examines that study.

Similar to RHS research, very few studies have examined the efficacy of collegiate recovery programs. One such published study was conducted in a recovery program at Augsburg College (StepUP program; Botzet, Winters, & Fahnhorst, 2007). This small, exploratory study suggested that students in the program were largely able to maintain sobriety along with a favorable grade point average (GPA). In addition, this study reported that the program provided interpersonal support, a safe and healthy environment, education and counseling, and accountability, all of which greatly facilitate the recovery process (Botzet et al., 2007). The extant and small literature provides encouraging insights that a recovery-based education setting can promote student recovery and academic success. Nonetheless, more research is needed on recovery-supported education and its potential benefits to the academic, mental health, social, and environmental needs of young people in recovery.

Currently, a quasi-experimental study funded by the National Institute on Drug Abuse is under way to start understanding and evaluating the academic and therapeutic outcomes of students attending RHSs. This research study targets high school adolescents who have undergone drug treatment and examines whether students who attend RHSs exhibit better academic and behavioral outcomes than those who attend traditional high schools. Participants are recruited from RHSs, traditional high schools, and drug treatment facilities, then followed for 12 months postintake as they progress throughout their educational choice of attending an RHS (expected n = 225) or a traditional high school (expected n = 225). Both teens and their parent individually complete an approximate 90-min assessment four times over the course of a year (intake, 3 months, 6 months, and 12 months); parent assessments are completed via telephone interviews and adolescent interviews are completed in-person, with the exception of the 3-month follow-up, which is an abbreviated (30-min) telephone interview. All assessments utilize a semistructured interview format, and teen assessments additionally incorporate a questionnaire component and a biological measure (urinalysis) to validate the self-reported drug use. Parent and youth participants are each provided a $30 gift card remuneration following the completion of each assessment. Specific measures and instrument scales are noted in a later section, but all measures target relevant outcomes, including academic achievement, recovery support, mental and physical health, and peer and family support. Additional information on the design of the RHS Project and the data analyses can be found in the article by Tanner-Smith and Lipsey (2014/this issue).

This article summarizes the experiences learned in the RHS Project while outlining research strategies and other important considerations critical to the evaluation of recovery school efficacy. First and foremost in such an evaluation is the recognition of the developmental context of participants. Assessing adolescents and young adults, in general, creates unique research challenges simply because of the age of the population, developmental stage, and parental involvement. Considering the targeted population of youth in recovery, the primary data strategy question asks, “What are the best methods to connect with this population given its specific needs and challenges?” In addition, studying individuals who have a history of addiction calls for a strong understanding of SUDs, related mental health issues, and the intricate sensitivity surrounding the participant’s privacy, as well as the privacy and challenges of the treatment and recovery sites from which they might be recruited. Obtaining a balance of the needs of vulnerable youth while implementing a rigorous research design can be quite complicated and require special considerations by the research team.

Another significant consideration in evaluating a recovery school population involves the assessment strategies utilized. Basic research protocol of recruitment, interviewing strategies, and research measures need to be understood and approached with the developmental context of the population in mind. Data collection for sensitive topics such as drug use and other mental health issues presents some unique challenges, including the validity of self-report (e.g., faking good, faking bad, or response bias) and difficulty with follow-up in a longitudinal design. Collecting data on academic outcomes can pose problems, considering that self-reported school progress may not be accurate, privacy laws in school districts can restrict access to school grades and scores on standardized tests, and differences exist in grading standards across school districts. Drug use and other health outcomes may also be challenging to obtain, given that objective measures are not always available and may not be optimal (e.g., even a urinalysis has limits on its ability to detect drug use). Rapport between the assessor and participant is essential in addressing these obstacles, as is the rapport between the assessor and the recruitment site.

Adolescents in RHS settings are a vulnerable population, and as such, conducting empirical research with this population requires special considerations in terms of data collection and measurement strategies. Because of the intricacies in conducting empirical research with this population, the goals of this article are to address these challenges and to report the lessons learned from the RHS Project. Although this chapter focuses on the examination of academic and recovery outcomes of adolescents who have experienced treatment for chemical dependency, the lessons learned are largely generalizable to other populations and contexts, especially other adolescent behaviors and collegiate recovery programs.

DEVELOPMENTAL CONTEXT

Adolescent Considerations

The developmental period of life recognized as adolescence has been the focus of a multitude of research studies over the past three decades (Morris & Wagner, 2007). Researchers and health practitioners alike are interested in the complex dynamics between the biophysiological systems and the social environments commonly associated with adolescence. A new understanding of brain development in the adolescent years has allowed researchers to regard social behaviors, including motivations, impulses, and reward systems, with a biological perspective (see Dahl & Spear, 2004; Giedd, 2004; Wallis, 2004). In fact, researchers have suggested that adolescence is not necessarily a defined range of ages during the lifespan but a set of developmental transformations that occur within a short span of time (Morris & Wagner, 2007). Some of these transformations, as summarized by Morris and Wagner (2007), include

• biological transformations (such as puberty and hormonal changes, changes in physical appearance, and brain maturation),

• psychological transformations (such as identity formation, ego development, autonomy, and moral reasoning),

• social transformations (including intimacy, peer and media influence, and social problem solving), and

• role transformations (including gender roles, school/educational roles, and independence roles, such as getting a driver’s license or a job).

Because these transformations occur at different times for individuals and most often occur quietly and covertly, it is important for parents and professionals alike to regard and address the teen as an individual at his or her own developmental phase.

The new research and theoretical perspectives on adolescence brings to light the responsibility for professionals who work with young people to have a clear understanding of how these key developmental milestones affect decision making, risk taking, and other self-regulatory process that can lead to SUDs. From a research design and implementation perspective, these developmental milestones and transformations must be taken into consideration when working with an adolescent population. Suggestions of how to incorporate these developmentally conscious considerations into future research designs are presented throughout the article, specific to the various research strategies.

Recovery Population Considerations

Another characteristic of the focus population is the history of substance dependence. Considering the aforementioned perspective on adolescent development, researchers and health professionals must bear in mind those developmental factors when assessing or studying substance use. The psychological transformations occurring during adolescence frequently include various levels of egocentrism, defiance against authority, and risk taking, which can interfere with accurate reporting in the assessment process (Botzet, Fahnhorst, & Winters, 2012) at both the intake and follow-up assessments. It may be difficult to ascertain progress in recovery if the assessment is not sensitive to common adolescent characteristics.

Another consideration when working with a population of individuals who are recovering from substance dependence is the definition of recovery itself. Although the term is widely used to describe the goal of treatment for SUDs, a concrete definition of this term is not widely utilized. According to a panel of experts from The Betty Ford Institute, “Recovery may be the best word to summarize all the positive benefits to physical, mental, and social health that can happen when alcohol- and other drug-dependent individuals get the help they need” (The Betty Ford Consensus Panel, 2007, p. 225). Most often, recovery is regarded as complete abstinence from substances, but considering that adolescents commonly return to drug use (or relapse) within 1 year of treatment, and many do not continue their substance-dependent behaviors into adulthood (Winters, Botzet, Fahnhorst, & Koskey, 2009), recovery at this stage may be regarded as a decrease in symptomology. Thus, researchers need to consider the appropriate definition of recovery according to the research needs and define the measures and scales that best represent that definition.

If diagnostic standards are used to measure progress in recovery, researchers also need to consider aspects of the diagnostic criteria that are unique to adolescents. Current diagnostic criteria (Diagnostic and Statistical Manual of Mental Disorders [4th ed., text rev.; DSM–IV–TR]; American Psychiatric Association, 2000) do not differentiate the experiences of adults and adolescents, though research suggests that adolescents do not experience the criteria of withdrawal and tolerance in the same way as adults (Martin, Chung, & Langenbucher, 2008) nor do they report the diagnostic criterion of using substances in larger amounts or for a long period of time as adults do (Martin et al., 2008). Thus, when examining the concept of recovery, it is worth noting that current diagnostic criteria do not take into account the adolescent experience of SUDs, which may affect the number of endorsed criteria and, ultimately, the SUD diagnosis.

Furthermore, SUDs are commonly accompanied by other mental health disorders; nearly three fourths of adolescents with SUDs have a co-occurring mental illness (Hoffman, Bride, MacMaster, Abrantes, & Estroff, 2004). When we consider treatment and recovery for these disorders, it is important to keep in mind that recovery often targets multiple facets of the individual’s life and can thus be a long-term process. In fact, research has found that treatment of the SUD or the mental illness alone does not remit the other; in other words, treatment only for the SUD will not necessarily diminish the other mental health symptoms and vice versa (Davies, Riggs, & Thurstone, 2012). From a research design perspective, the comorbidity factor in recovering from a SUD needs to be addressed by asking: Do the assessment instruments measure SUDs and other mental health disorders? Are follow-ups spaced at intervals that can measure change over a period of time that is sufficient for progress to occur?

ASSESSMENT STRATEGIES

Recruitment

Early identification of potential recruitment sites is advantageous and largely involves networking within the recovery community through personal contacts and the use of Internet searches. Identification of administrators within adolescent substance abuse treatment programs and high schools is the initial step in developing and maintaining relationships within the recovery community. Moreover, in order to maintain collaborative relationships and ensure research support, it is vital that administrators and directors grasp the potential value of the research.

As previously noted, the subject recruitment sites for the RHS Project are RHS, traditional high schools, and adolescent substance abuse treatment centers. Although each site presents its own unique challenges and benefits in recruitment, some complexities are widely consistent. One common challenge in recruiting from the various sites stems from the need for repeated contact with the administration of the schools and the treatment centers. Regular contact with administrators is necessary to ensure continued recruitment throughout the study. Although most sites are interested in the concept of the research, they often have numerous other issues within their organizations that take precedence. Thus, it can be quite challenging for administrators to uphold a collaborative relationship with researchers after the initial contact is made. Therefore, it is essential for researchers to maintain consistent contact with the schools and treatment centers throughout the life of the study.

Although there are similarities between the three main recruitment sites, each also has its own unique advantages and challenges. For example, RHS students and their parents are often very eager participants. They typically are quite accustomed to sharing their stories of recovery, and these particular students bring invaluable insight about the RHS experience in terms of recovery support and academic strengths and weaknesses. In addition, we have found administrators at the RHSs to be typically very excited about the research and the opportunity to assist with collecting data on the value of these schools. The main obstacle in recruiting from RHSs is their relative scarcity, which can pose practical and logistical limitations in collaborating with a potentially rich recruitment site.

The second primary recruitment venue, traditional high schools, also has distinct advantages and challenges for researchers. These students provide a comparative perspective of recovery in a traditional academic setting to contrast and compare the recovery experience of those in RHSs. However, recruitment of participants from traditional high schools has been the most difficult for several reasons. Due to the high volume of administrative responsibilities, the potential negative connotation of having students “in recovery” and pressure to increase academic performance across all groups of students, it can be difficult to convince school leaders of the importance of the research and obtain permission to recruit students. Also, it is common for some schools to have their own internal review boards for research that oversee and limit the number of research studies in which a school participates. Some school research review committees permit research activities only within the school that can be directly linked to benefiting the participating students; a descriptive study may not be viewed in this light. Another obstacle in recruiting participants from traditional high schools is that eligible adolescents (those who have completed treatment) are spread out among multiple schools; there may be only one or two eligible students at any particular time within a traditional high school. Furthermore, if a school does not have specific substance abuse resources on site, it can be extremely difficult for administrators and staff not only to identify eligible students within their school but also to disseminate the study information in a confidential manner. Similarly, assurance of confidentiality can be a significant obstacle. Although information gathered for research is kept confidential and deidentified, school officials can often be skeptical of the extent of participants’ confidentiality. Working diligently to collaborate with traditional high schools can provide a very valuable source of comparison for the research, but extra effort may be necessary to reduce the impact of multiple barriers to recruitment at traditional high schools.

Because eligibility requirements for the RHS Project entail a high school student who has completed treatment for substance use, adolescent treatment centers (inpatient, outpatient, and day treatment) are primary sites to recruit youth in recovery. Treatment center staff is often willing to introduce the research study to either the parent and/or the youth during the youth’s treatment process, which can increase the family’s interest and investment in the study. Families at treatment centers decide of their own accord what type of school the adolescent will attend following treatment; therefore, no assignment to school condition is directed by the study. This is advantageous because it allows for recruitment of participants from both the RHS and traditional high schools. In fact, due to the obstacles presented in recruiting from traditional high schools, the treatment centers tend to be a source for the majority of our traditional high school participants in the current study.

Interviewing Strategies

Interviewing and collecting research data from youth can be challenging, starting with the initial consent process. Because most teens in high school are under the legal age of consent, parental consent must first be obtained. Empathy and sensitivity are crucial elements of the consent and assessment process, as parents and youth are often rightfully cautious in protecting their privacy, especially considering the possible school and legal consequences adolescents may have experienced prior to treatment. Helping the adolescent understand that none of his or her information will be shared with anyone (mandated reporting excluded) including parents, the current school, treatment center, or legal system is of great importance. If an adolescent fears that the information disclosed will be shared with others, he or she may refrain from providing accurate information or decline to participate in the study altogether. The development of good rapport between the assessor and the parents and adolescents is crucial in securing trust, obtaining accurate information, and reducing attrition. It is also important that the youth understands that he or she can discontinue participation at any point without any penalty and refuse to answer any question at any time, by simply stating “pass.” Thorough training for assessors should be conducted not only in regard to the consent and assessment processes but also to enhance rapport-building skills.

In cases where adolescent participants need to meet the assessor outside of school or the treatment center, it can be difficult to schedule those interviews for several reasons. One obstacle can be transportation. Lack of public transportation, no driver’s license or vehicle, and the need to rely on others for transportation can be problematic. This obstacle can be remedied by meeting the participant at an easy and convenient location, or by conducting phone interviews. The research team will need to determine the most efficient location and assessment practices according to the needs of the study.

Although the target population for the RHS Project is adolescents, parents are often a valuable source of corroborative information to help balance the potential for the youthful misperceptions that can sometimes occur with adolescents. In the RHS Project, parents are also assessed at the same four time points (intake, 3 months, 6 months, and 12 months), though the interviews are conducted over the phone. Although the option of meeting in person is always given, our experience shows that conducting the interviews over the phone is most appealing to parents because it provides flexibility in scheduling, cuts out travel time, and eliminates the need for daycare of younger children in the family. It also gives parents a sense of anonymity, as one may not be as comfortable talking about such sensitive issues in person. Some of the more difficult aspects of parental involvement encountered in this study include the challenges of developing trust and a relationship over the phone, considering the sensitive nature of topics covered in the interviews. In addition, parents may not have had their own experience with substance abuse issues and find it difficult to understand or feel uncomfortable discussing it. Whereas adolescents in recovery are often familiar with and comfortable talking openly about their experiences with drugs and alcohol following treatment, it can be difficult for the parents to openly share that kind of information. Reiteration of confidentiality can help ease hesitancy by parents to share information as well as the reassurance that researchers have the adolescents’ best interest in mind. This is especially important in regard to mandated reporting procedures and adolescent safety. Finally, it is important that the assessor be flexible with parents’ need to cancel or reschedule any interview. It can often be a challenging time in the lives of the families with someone in recovery and frequent rescheduling may occur.

Research Evaluation

Like the population considerations just outlined, significant research consideration needs to address the proposed outcomes, specifically targeting the most effective and thorough ways to obtain information. Heavy reliance on youth self-report of substance use and other mental health issues is a necessity, as parents and other corroborative reporters likely are not fully aware of the adolescent’s experiences and drug use patterns. More objective measures of the adolescent’s substance use, such as peer reports of the participant’s substance use or the use of biological specimens, may be desirable from a research perspective, but these can be quite difficult to obtain. Even in situations where the adolescent is comfortable and responding to the best of his or her ability, assessors need to be aware that an adolescent may either “fake good” (i.e., minimize their problems or drug use to look good) or “fake bad” (i.e., exaggerate their problems or drug use to impress others). Research has suggested that a self-report format is the preferred method of assessment because of the convenience, low cost, ease of administration, and the perception that the individual is the most knowledgeable reporter (Williams & Nowatazki, 2005). However, adolescent clients may distort their responses out of defiance, fear, and possible apprehensiveness to share (Williams & Nowatazki, 2005). In addition, youth may see the assessment as an opportunity to “cry for help,” so the adolescent may exaggerate their responses. Researchers may benefit by incorporating measurement tools that have “lie scales” built into the measure to help detect the presence of infrequent or likely inaccurate responses and by training the assessors to address these types of responses.

Although youth self-report measures are necessary for many outcomes concerning academics and recovery, corroborative reports, especially parental reports, can be very useful when examining adolescent behaviors. Parents may be a better source of reporting for family information, such as socioeconomic status, family health, and the adolescent’s mental health and treatment history. Although parents may not be able to accurately report their adolescents’ drug involvement, they may be aware of negative consequences related to that use, such as legal and school problems. Parent reports also provide corroborating data to compare to the adolescent reports.

Procedural strategies must be considered in order to optimize the collection of reliable and valid data and to promote study participation. For example, in the RHS Project, interview-based data are collected via semistructured interviews. This type of interview provides a modicum of structure in order to promote consistency of data collection and scoring of responses but also allows for some elaboration and clarification by the respondent. As noted earlier, reassurances of confidentiality are emphasized, and parent assessments are completed over the phone to reduce scheduling and transportation barriers. Also, we provide remuneration for the completion of all assessments. In the current study, participants are provided a $30 gift card to a local store at the completion of each assessment period (and anecdotally, some parents have commented that their teenager is participating in the research project largely because of the gift cards).

Another procedural strategy used in the RHS Project is the utilization of a Timeline Follow Back (TLFB) measure. This assessment tool was first developed by M. B. Sobell, Sobell, Klajner, Pavan, and Basian (1986) to obtain a retrospective, detailed account of alcohol use patterns. Participants are directed to recall their daily alcohol use through the use of a calendar as a visual aid, holiday and special event prompts, and other memory cues that assist in triggering memories associated with their alcohol use. The procedure was adapted in this study to include alcohol and other drugs, but the same techniques are used to help trigger the recollection and reporting of substance use over the prior three months. The TLFB has been shown to be a psychometrically sound tool among adolescent and other populations (see L. C. Sobell & Sobell, 1992), and this procedural strategy of triggering memories can enhance the accuracy of the adolescent self-report.

Other issues to consider in obtaining follow-up outcome data on adolescents in recovery pertain to the mobility of the participants. Adolescence (especially for older adolescents who have a history of addictive behavior) can be characterized by high mobility (Dong et al., 2005). Thus, parents may not know the whereabouts of their adolescent, and the use of phone books or other directory assistance programs is futile because living arrangements are likely not legally documented in their names. In addition, though cell phones may provide a stable contact number, parents often turn off cell phones as a consequence for negative behavior, or the adolescent does not have the financial means to pay a monthly cell phone bill. Social media sites, such as Facebook and Twitter, can prove to be fruitful in contacting youth, but Institutional Review Boards may be reluctant to allow such contact in research, due to confidentiality issues.

Academic Outcomes

In examining the efficacy of RHSs, the primary outcomes focus on academic achievement and success in recovery. Academic outcomes can be difficult to obtain again due to self-report issues, privacy laws in school districts, and differences in grading standards across school districts. For example, in the RHS Project, assessors learned that students weren’t always aware of their academic grade point averages, nor did parents frequently know how their teens fared academically. Schools or specific classes often used a pass/fail system rather than the traditional A–F grading scale. In addition, school districts have been quite hesitant to release student grades to researchers due to student privacy policies, as well as the burden on staff to generate student data.

Thus, in the RHS Project, academic progress was measured in multiple ways. Student grades are obtained solely by self-report; schools do not release this information to researchers. As such, several items were added to the instruments to capture the various grading systems. For example, question items asked students to record their grades in core classes, such as English, Math, Social Studies, and Science, utilizing traditional grading response options as well as pass/fail options. Students are also asked to report their overall GPA and number of credits earned, along with a more qualitative measure of grades, such as, “How would you describe your academic performance?” (Responses include above-average student, average student, and below-average student.)

Academic achievement is also measured by youth attitudes toward school and problems with school, including attendance, support, and satisfaction. Parents are also asked to classify their adolescent’s academic ability by reporting their teen’s grades, school problems, and parental satisfaction with the school’s academics. Last, a standardized measure of academic achievement (Wide Range Achievement Test; Wilkinson & Robertson, 2006) is utilized to measure academic achievement. This instrument serves to provide a standardized and direct measure of academic achievement when other school-mandated standardized measures (such as the Iowa Test of Basic Skills, PLAN test, or Stanford Achievement test) are unattainable due to school district confidentiality policies and other restraints.

Another academic-related outcome utilized when evaluating the efficacy of RHSs pertains to the academic rigor of the recovery schools themselves. How do these schools differ in terms of academic curriculum, attendance rates, staffing credentials, and relative emphasis on academics versus recovery? How do these differences in school characteristics impact student outcomes? All of these factors function in tandem, affecting the overall experience of the students. The RHS Project addresses these measures of academic rigor via qualitative review of each participating RHS. School staff is interviewed regarding these items, and research staff tours the school grounds to note the physical characteristics. Furthermore, during participant assessments, both teens and parents are asked what they like and do not like about their school experience. By measuring the academic outcomes utilizing both qualitative and quantitative measures, and by assessing not only the students and parents but also the school staff, a more comprehensive assessment of RHSs more likely follows.

Recovery Outcomes

Obstacles often occur during the course of assessing recovery outcomes as well. As previously mentioned, the definition of the term recovery can be quite ambiguous and interpreted differently among various people. Thus, multiple measures of the youth’s modifications to his or her substance use provide more flexibility in the analysis and reporting phases of the research. In the RHS Project, we considered these multiple viewpoints of recovery and incorporated several outcome measures specific to substance use, including a self-reported frequency of use, responses to DSM–IV diagnostic criteria of SUDs, endorsement of protective factors such as involvement in structured after-school activities, and relapse-risk associated with school and other social environments. Biological specimens are also useful in measuring recent substance use, as they provide an objective measure of drug use for a wide variety of substances. However, biological specimens for drug testing, such as urinalysis and hair testing, have their own limitations (see National Institute on Drug Abuse, 2012) and should be used in conjunction with other sources of reporting when measuring recovery from SUDs.

Mental health and behavioral outcomes may also be considered when examining recovery. The aforementioned corroborative reports by parents may provide a better grasp of the adolescent’s mental health issues and treatment success, so they may be able to better report on some of the behavioral symptomology of mental health disorders. Information on the treatment for mental health concerns is also quite beneficial when measuring recovery outcomes, as the treatment experience can be used as a covariate, helping to see if students who received more treatment, or a different type of treatment, fared better in their SUD recovery outcomes. Similarly, assessing the youth’s legal involvement and/or disruptive behaviors can provide glimpses into the behavioral symptomology associated with some mental health disorders and should be regarded in the wider context of recovery.

Complicating the measurement of drug use and recovery outcomes are the aforementioned self-report issues common in this population. Some youth may have a tendency to “fake good” or “fake bad” in their self-report of recovery, or simply minimize the severity of use if their reference group is skewed (i.e., if their peers are using a similar or greater amount, it may appear to the teen that he or she is using at a minimal level). In this regard, the TLFB tool previously mentioned is very useful as a corroborative measure of alcohol and other drug use; youth are reporting not only their perceptions of their substance use habits but also a day-by-day report of their substance use, which can be very different than their initial responses. Complicating the measurement of drug use and recovery outcomes further are the aforementioned problems with SUD diagnoses in adolescents; SUD diagnosis alone is not a suitable measure of recovery outcome as some of the current DSM–IV criteria are not applicable to the adolescent substance use experience. The utilization of multiple measures of the recovery construct is essential in gaining a comprehensive assessment of the issue at hand.

Measures

In obtaining reliable outcomes regarding academics and recovery among adolescents, the instrument development phase of research is critical. Myriad instruments are available for use in research with adolescents; some are designed as a screening tool to quickly identify youth at risk for a given problem, whereas others provide a much more extensive, diagnostic assessment (see Winters, Fahnhorst, & Botzet, 2007, for a more thorough review of assessment measures of adolescent substance use). Careful evaluation of the available instruments and scales is necessary to adequately prepare the instrument battery.

When considering the measures to be used when studying youth in recovery, the targeted outcome variables must first be determined. Those proposed outcome variables then serve as a guide to how that outcome will be interpreted. For example, as previously mentioned, the outcome of recovery is quite vague and can be interpreted and measured in a number of ways. In the RHS Project, some of the ways recovery is interpreted and measured is through the use of variables such as diagnostic symptomology, patterns of use, urinalysis, substance use expectancies and consequences, and services received. These interpretive variables are targeted in a number of scales and items found in various measures. As an example of the interpretative variables and the instruments in which they are found for the RHS Project, a complete list of these instruments used and the interpretive variables for which they target can be found in Table 1.

TABLE 1.

Instruments and Interpretive Variables Utilized in Assessing Academic and Therapeutic Outcomes in Recovering Adolescents

Interpretive Variables	Adolescent Measures (Format and Source)	Parent Measures (Format and Source)
Demographic information	ADI (interview; Winters & Henly, 1993)	ADI–Parent (ADI-P; interview;  Winters & Stinchfield, 2000)
Substance use history (tobacco,  alcohol, and other drugs  history & diagnostic)	ADI (interview; Winters & Henly, 1993) Time Line Follow Back (interview; adapted from Sobell et al., 1986). Urinalysis (biological specimen) MINI-SCID 5.0.0 (interview; Sheehan et al., 1999) PEI (questionnaire; Winters & Henly, 1989)	ADI-P (interview; Winters & Stinchfield, 2000)
Substance use expectancies and  consequences	PEI (questionnaire; Winters & Henly, 1989)	ADI-P (interview; Winters & Stinchfield, 2000)
Peer attitudes	PEI (questionnaire; Winters & Henly, 1989)	No parent report
Peer substance use	PEI (questionnaire; Winters & Henly, 1989)	No parent report
Perceived availability	Adapted from Monitoring the Future, 2010  (interview; Johnston, O’Malley, Bachman, & Schulenberg, 2011).	No parent report
Services received (includes  alcohol/drug counseling  services, mental health  services, & school services)	Adapted from Timeline Followback (interview; L. C. Sobell & Sobell, 1992). ADI (interview; Winters & Henly, 1993)	ADI-P (interview; Winters & Stinchfield, 2000)
Recovery support	HSQ (interview; Moberg & Finch, 2008)	Adapted from HSQ (interview;  Moberg & Finch, 2008)
Financial burden of substance  use disorders	No youth report	Six items created for this study  to measure cost/benefit  analysis
Parental emotional quality of  life	No youth report	Adapted from Health Utilities  Index (interview; Feeny, Furlong, Boyle, & Torrance, 1995)
Family health history (substance  use and mental health)	ADI (interview; Winters & Henly, 1993)	ADI-P (interview; Winters & Stinchfield, 2000)
Youth mental health	MINI-SCID (interview; Sheehan et al., 1999) GAIN-SS (interview; Dennis, Chan, & Funk, 2006)	ADI-P (interview; Winters & Stinchfield, 2000)
Youth mental health services  (i.e. medication, counseling,  out-of home placement)	ADI (interview; Winters & Henly, 1993)	ADI-P (interview; Winters & Stinchfield, 2000)
Physical health	GAIN-Q3 (interview; Dennis, 2010)	No parent report
Legal involvement	Adapted from ADI (interview; Winters & Henly, 1993)	ADI-P (interview; Winters & Stinchfield, 2000)
Youth stress	GAIN-Q3 (interview; Dennis, 2010)	ADI-P (interview; Winters & Stinchfield, 2000)
Risk behaviors for infectious  disease	GAIN-Q3 (interview; Dennis, 2010)	No parent report
Crime & violence	GAIN-Q3 (interview; Dennis, 2010)	No parent report   (continued on next page)
Life satisfaction	GAIN-Q3 (interview; Dennis, 2010)	No parent report
Social support	GAIN-Q3 (interview; Dennis, 2010)	Adapted from HSQ (interview;  Moberg & Finch, 2008)
Religion/Sprituality	GAIN-Q3 (interview; Dennis, 2010)	No parent report
Free time activities	Adapted from Healthy for Life (interview;  Piper, Moberg, & King, 2000)	No parent report
Problem solving	Problem Solving Inventory (Latimer, Winters, D’Zurilla, & Nichols, 2003)	No parent report
School atmosphere	HSQ (interview; Moberg & Finch, 2008)	Adapted from HSQ (interview;  Moberg & Finch, 2008)
School/Work problems	GAIN-Q3 (interview; Dennis, 2010)	GAIN-Q3 (interview; Dennis, 2010)
School performance	HSQ (interview; Moberg & Finch, 2008)	Adapted from ADI-P (interview; Winters & Stinchfield, 2000) Adapted from HSQ (interview; Moberg & Finch, 2008)
Attitude toward school	Scale on the Behavior Assessment System  for Children (questionnaire; Reynolds & Kamphaus, 1992).	No parent report
School climate	Inventory of School Climate (questionnaire;  Brand, Felner, Shim, Seitsinger, & Dumas, 2003)	No parent report
Parent/Child relationship	Youth Happiness with Parent Scale (questionnaire; DeCato, Donohue, Azrin, & Teichner, 2001) Alabama Parenting Questionnaire–Youth form (questionnaire; Shelton, Frick, & Wootton, 1996)	Parent Happiness with Youth Scale (questionnaire; DeCato, Donohue, Azrin, & Teichner, 2001) Alabama Parenting Questionnaire–Parent form (questionnaire; Shelton, Frick, & Wootton, 1996) Stress Index for Parents of Adolescents (questionnaire; Sheras, Abidin, & Konold, 1998)
Community activities	Adapted from the National Education  Longitudinal Study of 1988  (questionnaire; Ingels, Scott, Taylor, Owings, & Quinn, 1998)	No parent report
Feelings about yourself	HSQ (interview; Moberg & Finch, 2008)	No parent report
Video game addiction	Adapted from the SOGS-RA (questionnaire;  Winters, Stinchfield, & Fulkerson, 1993)	No parent report
Gambling addiction	SOGS-RA (questionnaire; Winters, Stinchfield, & Fulkerson, 1993)	No parent report

Note. ADI = Adolescent Diagnostic Interview; HSQ = High School Questionnaire; SOGS–RA = South Oaks Gambling Screen–Revised Adolescent.

CONCLUSION

A critical question for the recovery school community is whether such schools are as effective as or more effective than the students’ traditional high schools in preventing relapse, facilitating academic achievement, and reducing dropout rates for students recovering from SUDs.

This article addresses the data collection strategies that may be useful in answering this critical question and summarizes an ongoing quasi-experimental study that seeks to answer whether students who have completed treatment for SUDs have significantly better behavioral outcomes (less alcohol and other drug use, fewer mental health symptoms, less delinquent behavior) and academic outcomes (higher GPA, higher standardized test scores, better attendance, lower dropout rates) if they attend recovery high schools compared to similar recovering students who attend traditional high schools. Experiences gained from this RHS Project can serve other researchers in the field by providing insight and tips regarding the best practices when working with this population. Table 2 provides a summary of those tips and lessons learned.

TABLE 2.

Summary of Suggestions When Conducting Research With Adolescents in Recovery From SUDs

Suggestions for Research Design

Suggestions for Working With Targeted Population

Define outcome variables; recovery and academic progress can be measured in many ways Address potential issues in using diagnostic criteria for adolescent SUD and mental health diagnoses Network with SUD treatment and RHS communities so they see value of the research Carefully review potential instruments and scales to increase reliability and validity within the study When examining SUDs as outcome variables, be sure to include measures of co-occurring mental health disorders Use a variety of assessment formats, such as interviews, questionnaires, and biological specimens Use multiple interpretive variables to represent a single construct Utilize corroborative/parental reports whenever possible to increase validity of study Allow sufficient time for progress to occur when creating follow-up protocol

Train research staff on topics of adolescent development Train research staff on substance use disorders and the criteria used for diagnoses Network with SUD treatment and RHS communities to assist with recruitment Be sure that research staff are proficient in the consent process, compliance in confidentiality, and gaining rapport with participants Be aware of participant’s age and legal age of consent; if youth is younger than age 18, consent is first needed from parent Train research staff on how to utilize probing techniques in semistructured interviews; essential for obtaining a valid self-report from participants Be flexible in the assessment scheduling process and interviewing locale Maintain consistency and tenacity when scheduling follow-up interviews Provide remuneration at each assessment period

Note. SUDs = substance use disorders; RHS = Recovery High School.

School is a critically important social environment for adolescents with substance use disorders. Indeed, school sits at the heart of the threat of relapse for these students. Approximately two thirds of high school students say drugs are used, kept, or sold on the grounds of their schools (Johnston, O’Malley, Bachman, & Schulenberg, 2011). Yet schools are a vital vehicle for their recovery. Succeeding academically can help students stay sober, which in turn can help them graduate (Gibson, 1997), whereas involvement in prosocial activities at school can also assist the recovering adolescent (Vaillant, 1988). This is at the heart of the potential benefit for recovery schools. As others and we continue to study these types of schools, their value as a pathway to recovery will hopefully become clearer and the public health benefit they provide will be more evident.

AUTHOR BIOS

Andria M. Botzet, M.A., has worked in the Center for Adolescent Substance Abuse Research at the University of Minnesota for more than 18 years. She helps to coordinate and manage several research projects in addition to conducting assessments and interventions with adolescents and young adults who experience alcohol, drug, and gambling addictions.

Patrick W. McIlvaine is a Research Study Grant Coordinator in the Psychiatry Department at the University of Minnesota. He has worked with the Center for Adolescent Substance Abuse Research since 2011. He has also served as an outside consultant for several agencies in the fields of addiction and mental health, including the Substance Abuse and Mental Health Services Administration in 2012.

Ken C. Winters, Ph.D., is a Professor in the Department of Psychiatry at the University of Minnesota, director of the Center for Adolescent Substance Abuse Research, and a Senior Scientist with the Treatment Research Institute, Philadelphia, PA. He received his B.A. from the University of Minnesota and his Ph.D. in Psychology (Clinical) from the State University of New York at Stony Brook. His primary research interests are the assessment and treatment of addictions, including adolescent drug abuse and problem gambling.

Tamara Fahnhorst, M.P.H., has been employed in the Department of Psychiatry at the University of Minnesota for more than 20 years. She has worked in both administrative and therapeutic capacities on numerous research initiatives for youth who suffer from mental health and drug abuse problems. She has authored several journal articles and book chapters pertaining to adolescent alcohol and drug use prevention, assessment, and intervention.

Christine Dittel received her degree from the University of Wisconsin–River Falls in 1991 and joined the Center for Adolescent Substance Abuse the same year. She has worked at the University of Minnesota for 22 years researching eating disorders, pathological gambling, and substance use disorders.