Advancing the science of evaluating Collegiate Recovery Program processes and outcomes: A recovery capital perspective

Abstract

Collegiate recovery programs (CRPs) offer resources and programming for postsecondary students in addiction recovery to ensure they can initiate or maintain their recovery and complete college. To achieve these goals, CRPs offer a variety of activities that research and theory suggests should produce positive outcomes among their students; yet the lack of systematic evaluation research in this area means it is unknown which programming components may drive outcomes. Recovery capital theory posits a variety of factors at multiple ecological levels that might influence students’ recovery experience and their engagement and success in community programs like CRPs. To address this complexity in research and evaluation work on CRPs, we provide a recovery capital-oriented theory of change and logic model for CRP evaluations, and demonstrate how this model could be used with an exemplar case. This is followed by a recovery capital-oriented data collection toolkit for future research and evaluation. These efforts should help to inform program planners and evaluators interested in understanding the influence of the ecosystem of recovery-oriented systems of care in CRPs for emerging adults.

Introduction

Collegiate recovery programs (CRPs) were instituted to support college students in recovery from an alcohol or other drug use disorder (Harris et al., 2014; White et al., 2012; White & Finch, 2006). The overall goal of CRPs is to offer services, resources, and programming for postsecondary students in recovery to ensure they can initiate or maintain their recovery and complete college (Bugbee et al., 2016; Harris et al., 2007). To achieve these goals, CRPs offer a variety of activities and leverage partnerships with preventionists, clinicians, and healthcare providers to implement services for college students in recovery (Laitman et al., 2014). Research suggests these efforts are beneficial for students who are engaged in CRPs and that the types of activities used by CRPs (i.e., peer recovery coaches and mentors, continuing care treatment programming, and mutual-help groups) should be effective in supporting and maintaining student recovery (Ashford, Brown, & Curtis, 2018; Cleveland et al., 2007; De Martell, 2019; Harris et al., 2007, 2010; Hennessy et al., 2018; Laudet et al., 2016). Yet, many CRPs also implement activities that have not yet been empirically evaluated (e.g., campus-based drop-in centers, substance free/recovery housing; (Vest et al., 2021)). Thus, it remains unknown which programming components may be driving outcomes, and moreover, how these components relate to key recovery constructs (Brownson et al., 2009).

There are challenges to creating a standardized CRP model because these programs operate in diverse institutions of higher education with a wide range of funding and levels of institutional support. Indeed, most CRPs typically operate independently with little centralized leadership from the broader campus and there is a lack of consensus on national standards (Harris et al., 2010; Laudet, Harris, Winters, et al., 2014). Furthermore, differences in research philosophies and human subjects protection policies across campuses may complicate efforts to create a standardized set of data collection processes and procedures that would permit comparisons across multiple CRPs (De Martell, 2019). It is perhaps unsurprising then, that the field currently lacks a comprehensive logic model describing the hypothesized relationships between CRP program resources, activities, and outcomes. As a result, there is no agreed upon set of core constructs (nor measures of those constructs) that reflect the individual, program, and community outcomes CRPs are intended to affect. Without a systematic framework to guide administrators and program evaluators, CRPs are unlikely to produce the strong evidence needed to ensure their sustainability because administrators and staff cannot know which CRP practices lead to positive outcomes, and which practices do not, within the context of their institution.

To address this complexity in research and evaluation work on CRPs, we detail a recovery capital-oriented theory of change and logic model for CRP programming, using an exemplar from the field. This is followed by a proposed data collection toolkit for future CRP research, evaluation, and program planning. The logic model and data collection toolkit is intended to help inform healthcare practitioners, researchers, and other CRP professionals to produce findings that can influence the larger ecosystem of college administration and policy in a recovery-oriented systems of care framework (Bassuk et al., 2016; Laudet, Harris, Kimball, et al., 2014).

Method

This paper was developed as part of an iterative process from a larger study analyzing CRP student outcomes from several different sites (Hennessy et al., 2021; Nichols et al., 2021). The larger study entailed secondary analyses of existing data that were collected at six CRPs operating in the United States. For the larger study, we contacted CRP sites to request de-identified, student-level data that was already being collected regarding participating students’ resources, experiences, and academic/behavioral outcomes. As a result of the data collection process and conversations with site collaborators, we facilitated a roundtable discussion at a national conference to discuss data collection directions for CRPs. We brought our suggestions from the study process to the roundtable, which generated new ideas and discussion. For this manuscript, we have synthesized these ideas and reframed them into a CRP theory of change and logic model using recovery capital as an organizing theoretical framework. We then follow this logic model with a data collection toolkit of best practice suggestions to inform future implementation and evaluation efforts of CRPs.

Exemplar case.

Our exemplar case example is drawn from one of the sites used in the study referenced above (Hennessy et al., 2021); this is a site located at a large, public university that reported a wide range of services for students in recovery, had about 60–80 participating students/year, had multiple sources of internal and external funding, and a large operating team compared to other CRPs (3 paid full-time staff and 4 paid part-time staff, 2 of which were undergraduate students). The services this CRP offered included recovery-specific activities, academic, life skills/enrichment, social activities, and sober living (see Table 1 for details).

Table 1.

Exemplar Case CRP Information and Services Offered

5 Funding Sources

University general fund

Private foundation grant

Individual private donations

State grant

University foundation

12 Paid Staff Support (n)

Full-time paid staff (7)

Part-time paid staff (3)

Undergraduate paid part-time staff (2)

6 Recovery Services Offered

Regular group recovery support meetings

Seminars

12-step meetings

Referrals to other health providers

Individual counseling sessions

Urine/saliva testing

5 Academic Services Offered

Academic advising

Early course registration

Tuition scholarships offered

Student travel to academic conferences

Study abroad

3 Life Skills/Enrichment Services Offered

Meditation/yoga/physical activity

Community service opportunities

Student leadership retreats

5 Social Activities Offered

Weekend social activities

Panel discussion for college classes

Student travel to recovery conferences/events

Summer activities for students

Sober tailgates at sporting events

1 Living Solution Offered

Recovery dorms or separate housing for students in CRP

Recovery Capital and Recovery

Recovery capital is a useful theoretical framework for developing a logic model for a recovery-oriented program as it details the resources an individual could use in their recovery efforts and considers these resources at several ecological levels (Cloud & Granfield, 2001, 2008; Granfield & Cloud, 1999). Recovery capital can be used to consider the individual in context (Hennessy, 2017) and is comprised of human capital (internal resources such as motivation, self-efficacy, cognitive skills), financial capital (tangible resources such as financial stability, access to evidenced based treatment and continuing care supports), social capital (sober and supportive peers, friends, and family), and community capital (available recovery-focused resources in the community). A CRP is one primary form of community recovery capital, which could work to build other forms of recovery capital among its participants.

Common Components of Collegiate Recovery Programs

Although CRPs are generally diverse, there are several common elements of many CRP programs (Ashford et al., 2018; Cleveland et al., 2007; Transforming Youth Recovery, 2018)). One primary CRP component is offering peer support through several avenues including providing a space for meetings and organizing sober social activities (e.g., drop-in center, sober tailgates for sporting events, weekend social events). Many CRPs also offer individual or group counseling sessions related to recovery maintenance and provide links to external mutual help groups (Bugbee et al., 2016; Hennessy et al., 2021). Some CRP programs also offer seminar groups, skill building workshops, academic advising, or early course registration to support students with their college progression (Laudet et al., 2014). Finally, well-resourced programs may offer additional services such as sober living spaces (roommate-pairing or dormitories) or academic scholarships for student members (Reed et al., 2020).

How Collegiate Recovery Programs Bolster Recovery Capital

The recovery capital framework suggests that as an individual builds capital in one domain, they are likely to build capital in another. Figure 1 provides a broad theory of change of how CRP involvement, based on the common CRP components and the process of change suggested by recovery capital, might bolster the four domains of recovery capital to produce meaningful outcomes for college students in recovery. For example, CRPs that implement supports attending to students’ personal needs, such as by providing relapse prevention sessions or life skills training (e.g., stress management), may help to build human recovery capital. Further, by creating a sober environment and fostering sober networks, CRPs may bolster students’ social recovery capital by facilitating the creation of recovery supportive relationships with peers and recovery role models, which can lead to a stronger recovery social identity (Bliuc et al., 2019; Dingle et al., 2019; McIntosh & McKeganey, 2000; Smock et al., 2011). Although CRPs are a form of community capital on their own, they may also provide linkages to recovery capital in the community, such as by connecting students with off-campus mutual help groups or to employment opportunities. These resources can build upon each other in a synergistic process; as students build human capital, they then may be more likely to engage in CRP services and with sober peers, thus strengthening their social recovery capital. Students who develop social recovery capital through their new peer relationships may feel greater self-efficacy for their recovery, thus enhancing their human recovery capital.

Figure 1.

CRP Involvement and Recovery Capital

Recovery Capital and the CRP Programming Logic Model Applied to Exemplar Case

The overall process of change suggested from a recovery capital lens is useful, but still too broad for specifically evaluating the process of change in a CRP. Our proposed logic model, shown in Figure 2, depicts the complexity of CRP programming operations from a recovery capital perspective, which considers each recovery capital domain. We have selected our case exemplar’s programming elements for the logic model figure but acknowledge that CRPs operating in different locations or contexts may need to adapt the components of this model to fit their unique environment.

Figure 2.

Recovery Capital-Oriented Logic Model of CRP Using Exemplar Case

Situation/Problem

From a recovery capital perspective, college students in recovery from an SUD bring their own unique strengths and limitations, which will interact with the resources and barriers in their immediate (e.g., dorm, classes, work, extracurricular activities) and broader (e.g., university resources, location, culture) environmental contexts (Bell et al., 2009; Terrion, 2013). These factors then influence each other, in a bidirectional manner.

Personal recovery capital and barriers.

Students enter institutions of higher education with a variety of strengths and resources that need to be considered (Ashford, Brown, Eisenhart, et al., 2018; Brown et al., 2019; Watts et al., 2019; Wiebe et al., 2018); such as whether they enroll with an existing alcohol or other drug problem and how severe it is, their problem-solving and coping skills and their ability to use them in a new environment, and the social determinants of health that affect their day-to-day actions in the campus setting. From a financial recovery capital perspective, students have varying economic backgrounds and material resources that can influence the types of resources they have access to on campus, including supports for their recovery or other means to navigate challenges. Lastly, some CRP students enter institutions of higher education with previous justice system involvement (Cleveland et al., 2007; Laudet et al., 2015) and may face additional barriers to admissions, enrollment, and housing that non-system impacted students will not encounter.

Social recovery capital and barriers.

There are several relevant aspects of social recovery capital for a college student in recovery. First, the student’s existing friend and family support, outside and prior to entering the campus setting are important influences that may remain in their lives although they may now be some distance away from that home setting. Next are the social influences on campus including faculty, staff, administrators, peers in dorms and in classes, and close peers with whom they spend time (Botzet et al., 2008; Kollath-Cattano et al., 2018; Smith et al., 2018). These new social influences can be a challenge for college students to navigate especially as they seek to create a fresh social network in a new and oftentimes stressful environment. All of these people and groups can be a source of support and social recovery capital, or conversely, a barrier to recovery.

Community recovery capital and barriers.

The CRP and the programming it offers are sources of community recovery capital (Laudet, et al., 2016). Local mutual help or other recovery support groups in the wider community are also relevant sources of community recovery capital. The culture of substance use on campus, including school policies around use and the presence of alcohol or other drugs at social gatherings, are also important sources of recovery capital – or barriers – depending on their perceived role in the environment (Gibson et al., 2017; Holleran Steiker & McElrath, 2017).

Inputs: CRP Activities and Process Objectives

Here we consider six CRP activities that our exemplar case implements and the role these supports may play in a recovery capital informed CRP logic model: (1) group recovery support meetings, (2) academic advising, (3) meditation/yoga, (4) weekend social activities, (5) sober tailgates, and (6) sober living environment.

Process Objectives.

To ensure that CRP activities are implemented as planned and thus can be causally linked to outcomes of interest, a series of process objectives for each CRP activity needs to be identified and evaluated. For example, in the group recovery support meeting, we would expect several process objectives to be examined, including: (1) recruitment of student leaders to run recovery meetings, (2) deciding on meeting structure or mutual help format (e.g., all recovery, Alcoholics Anonymous, Dharma Recovery), (3) how to be inclusive of diverse recovery pathways, and (4) recruitment and advertising strategies. Examining whether and how these activities happened would help to determine whether this broader activity was implemented as planned or if changes to any parts of its implementation are needed to increase engagement or improve chances of participants achieving aims.

Short term outcomes expected from inputs.

Short term outcomes that CRP attendees might show from group recovery support meetings include reporting increased peer support for their recovery, a sense of accountability to others in their recovery, and the ability to practice and improve their coping skills. Engaging in academic advising may support CRP students in creating a tailored course plan that helps them plan each semester to address their overall academic and career trajectory. Furthermore, this dedicated advising may help to identify problems early in the semester or during times of stress so that necessary changes may be quickly implemented. CRP attendees might show several immediate outcomes from engagement in meditation/yoga, such as increasing physical well-being and self-confidence, and reducing the effects of stress. We would expect CRP attendees of weekend social activities to report more experiences of having fun while sober, increased peer support for their recovery, and higher engagement in typical college activities without substances present. Finally, the presence and use of a sober living environment (e.g., sober dorm) might produce positive outcomes for individual CRP attendees as well as the larger campus community: individuals may report increased feelings of support and accountability for their recovery, reduced feelings of stigmatization (Mackert et al., 2014) and fewer triggers in their surroundings on campus.

Long Term Outcomes

CRP programming could result in many long term outcomes related to building student recovery capital. As a result of joining the CRP, we might expect that students who experience short term outcomes (e.g., having fun while sober) are more likely to stay engaged in college, have higher grades, and be more likely to graduate, thus generating human recovery capital and likely improving financial recovery capital. Similarly, we might expect these students to report feeling more engaged in college, choose to participate in additional extracurricular activities (e.g., sports, hobbies, clubs), and demonstrate stronger ties to the college and the local community (e.g., volunteering, interning), all indicators of social and community recovery capital.

CRP Data Collection Initiative: A Toolkit

The logic model we propose highlights the complexity of CRP programming and how difficult it is to capture causal mechanisms of change even with a standardized set of measurement tools. To begin to examine CRP results along a causal pathway, intermediary (i.e., mediators and moderators), short term, and long term outcomes need to be measured. Here we outline what we consider to be a bare minimum data collection strategy for CRP directors, college health providers, CRP student groups, or college administrators based on the dimensions of the recovery capital-oriented theory of change and logic model (Figure 1 and 2). Along with these recommendations, we provide Table 2 with suggestions for how these data could be collected. These data collection recommendations are presented by the type of CRP under examination based on a continuum of services from less student engagement to more student engagement: (1) membership-based, activities during semester, breaks + sober dormitories; (2) membership-based, activities during semester, breaks; (3) non-membership based, drop-in center and activities during semester, breaks; (4) infrequent activities.

Table 2.

Data Collection Suggestions by Type of CRP Structure

CRP Structure	Timing of data collection	Minimum evaluation measures	Minimum baseline measures	Minimum outcome measures
Membership-based, activities during semester, breaks + Sober dormitories	CRP entry/exit; every semester between Dorm living entry/exit	Quality of activities, Satisfaction with activities Quality of living situation, Satisfaction with living situation, Areas for improvement	CRP enrollment date, gender, date of birth or age, grade level, previous treatment services, primary substance of use/abuse, length of time in recovery, other mental health symptoms/diagnoses	GPA, graduation, level of substance use, coping, self-efficacy, motivation, mental health, global measure of well-being*
Membership-based, activities during semester, breaks	CRP entry/exit; every semester in between	Quality of activities, Satisfaction with activities	CRP enrollment date, gender, date of birth or age, grade level, previous treatment services, primary substance of use/abuse, length of time in recovery, other mental health symptoms/diagnoses	GPA, graduation, level of substance use, coping, self-efficacy, motivation, mental health, global measure of well-being*
Non-membership based, drop-in center and activities during semester, breaks	Yearly census; Pre/post activity	Quality of activities, Satisfaction with activities	Gender, date of birth or age, grade level, previous treatment services, primary substance of use/abuse, length of time in recovery,	GPA, graduation, level of substance use, coping, self-efficacy, motivation, mental health, global measure of well-being*
Infrequent activities	Pre/post activity	Quality of activities, Satisfaction with activities	Gender, date of birth or age, grade level, previous treatment services, length of time in recovery	Pre/post measure of outcomes expected from the activity

^*For example: recovery capital, quality of life, psychological well-being

Data Collection Practices

CRPs should strive to measure detailed information about participants at multiple times during their involvement with the CRP. The first data collection point should be a comprehensive assessment of baseline (i.e., at initiation of CRP membership or first encounter with the CRP) characteristics of all students participating in the CRP (i.e., characteristics of participants that would not be expected to change). Though care should be taken to balance timely data collection and minimize student burden, to capture student changes and attribute changes to CRP involvement, data should be collected from students 2–4 times yearly. Some natural times for these follow-up waves are at the beginning and end of every college semester (or quarter or trimester) or before/after specific events.

When possible, CRPs should strive to ensure these data are collected in a longitudinal manner that allows linking of individual student data across waves. This could be achieved, for instance, by asking students to report their student ID, date of birth, school email address, or other known identifier at each wave of data collection. However, because collecting directly identifiable information may introduce concerns around privacy and confidentiality, we strongly recommend that CRPs consult with their home institution’s research compliance and registrar offices to ensure the appropriate steps are taken to protect confidentiality of data from this vulnerable population of students while also complying with relevant institutional, state, and federal policies and regulations.

For CRPs that only offer scheduled activities, data collection may instead occur at discrete time points immediately prior to and after an activity is offered. The content of these assessments would be specific to each activity, with the goal of measuring distinct outcomes expected to change as a result of the programming along with any process questions deemed important to improving the event in the future. For these discrete pre- and post-activity assessments, CRPs will also likely need to collect basic demographic information, which can be used to assess whether activities are reaching the target audience or are particularly useful for different subgroups of students. One way to collect these data would be to ask students to swipe in to these events using their college ID card, assuming CRPs could access those card swipe data from the institution.¹

Data Collection Measures

There are a variety of standardized tools used in addiction recovery research that may be relevant for understanding the population of students served by a CRP and the relationship between CRP programming (inputs) and students’ short, intermediate, and long term outcomes. For the CRP toolkit proposed herein, we have synthesized recommended measures into three primary domains as relevant to assessing the relationship between CRP engagement and student outcomes. First, to understand the “situation,” it is important to collect variables related to (1) student characteristics. It is also necessary to measure the (2) level of student engagement in CRP activities as based on the CRP inputs themselves; thus, engagement measures will vary by activity employed by the CRP. Finally, short, intermediate, and long term (3) student outcomes should be collected based on those specific domains the program component aims to address. It is worth noting here that despite the most developed logic model and rigorous tools, it will be difficult to establish causality between CRP engagement and student outcomes in most real-world data collection scenarios. Students are within a larger college environment that offers a variety of recovery supports and barriers that may not be measured (or measurable) by these CRP efforts. Without an appropriate comparison group (i.e., in this case, students in recovery who do not engage in CRP activities), programs will only be able to establish a correlational relationship between their programming and student outcomes. Identifying and collecting data from a comparison group requires additional resources that many CRPs will not have at their disposal. Nevertheless, observational and correlational evidence about the benefits of CRP engagement can still be incredibly valuable and informative for CRPs and their funders.

In the appendix, we provide tables with suggested tools to capture each type of information proposed in this toolkit. This toolkit primarily discusses quantitative methods for data collection as quantitative data will allow for measurement of specific outcomes related to programming. There also are a range of qualitative and mixed-methods assessments outside the scope of the current manuscript that may be valuable; they may be particularly useful for CRPs interested in implementing a new activity (e.g., new type of activity or support group), working with a new population of students (e.g., diverse population or group with another identified concern), or identifying barriers and facilitators to student participation and progress (for additional resources on this topic, see (Creswell, 2018)).

Student characteristics.

A CRP should plan to measure some basic student characteristics including demographic information such as date of birth or age, race/ethnicity, sex, gender identity, sexual identity, disability status, veteran status, grade level at CRP entry (or at time of CRP engagement in activities), previous treatment service use (any type of mental health and addiction services), primary substance of use/misuse, additional mental health symptoms or diagnoses (Ashford et al., 2019), and date of CRP enrollment. CRPs might also consider collecting data on other factors that are important to understanding a history of use and recovery, such as family mental health and substance use disorder, history of criminal justice system involvement, history of houselessness, quality of life, high school grades and standardized test scores, and physical health and pain.

Level of student CRP engagement.

Engagement in CRP activities (inputs) should be systematically collected to measure how much students are using the services being offered. Level of engagement could be measured in several different ways, which will depend on the nature of the CRP structure. If the CRP has regular (weekly or monthly) meetings, then tracking participant attendance at these meetings over time is valuable and can provide insight into understanding student outcomes. For example, when collecting these data along with student outcome data, this would allow one to empirically answer the question, is more regular involvement positively correlated with better student outcomes? Alternatively, for CRPs with a drop-in center, recording the number of students who visit each week and length of time spent might be a useful metric for understanding level of engagement with the center.

Student outcomes.

The recovery capital framework suggests that there are many short, intermediate, and long term outcomes likely to be of interest to CRP administrators and universities. From a higher education administration perspective, what is likely most valuable is capturing student academic engagement and progress over time related to human, social, community, and financial recovery capital: semester and academic major grade point average (GPA), involvement and leadership in other campus activities, academic persistence, and graduation. Likewise, universities may be interested in capturing more distal outcomes such as gainful employment after graduation or involvement in the local community. These outcomes would be important to demonstrate the value of the use of space and other university resources for CRP activities and could be used in a cost-benefit analysis. On the other hand, many CRPs may be most interested in whether members are maintaining their recovery efforts, such as by reducing/abstaining from substances and whether students experience direct human recovery capital benefits as a result of their programming (e.g., improved coping skills, self-efficacy, motivation for recovery, and reduced mental health symptoms). Further, many CRPs will be interested in understanding whether students experience changes in broader concepts of wellbeing, such as recovery capital, quality of life, and psychological wellbeing.

Lessons Learned and Future Directions

In our work with CRPs, we have discovered there is a large gap in the field and a need for a systematic approach to data collection and evaluation. Oftentimes, these programs are administered by those who have a passion and skillset for the work but may not have the resources or training necessary to conduct a rigorous program evaluation; further, they may not have the time to do so given the time needed to implement key program activities. Thus, we acknowledge that implementing even the most basic data collection toolkit (provided here) will require dedicated time and effort at its initiation and throughout the life of the program. Programs must be willing to set aside personnel time for creating data collection tools and managing data collection (including relevant trainings for staff), obtaining student cooperation in completing data collection measures, preparing and analyzing data, using those results to inform program development and garner administration support, and disseminating results to relevant stakeholders.

To support these efforts, programs may wish to identify potential funding sources to assist with technical assistance or evaluation activities. This might include internal sources of funding, for example from the Provost, President, Student Life, Foundation, or Diversity offices on campus. This might also include external sources of funding, for example from Substance Abuse and Mental Health Services Administration (SAMHSA) block grants, private foundations, or university donors. Finally, many programs may also wish to identify researchers, in or out of their institution, who can serve as evaluation and program planning partners, and who can provide assistance with data collection, data analysis, and dissemination efforts.

This manuscript provides a recovery capital-oriented theory of change and logic model for CRPs as a first step to begin to systematize the collection of CRP process and outcome data. In addition, we provide a set of basic standards and items that CRPs should consider when collecting data, and a list of potential (validated) tools to use to capture these data. We acknowledge that this is a first step in systematizing the study of CRP students and programming in a logic model. Although this proposal is based on extensive work of our team in research and in the field when collaborating with sites that had already attempted to collect some of these data, we anticipate that when the proposed logic model is used in practice others may find elements we missed or nuances important to the particular CRP context under consideration. Further, similar youth recovery-oriented programs, such as recovery high schools or alternative peer groups may find this toolkit to be a useful starting point, but would likely need additional adaptation to fit those unique educational settings. Thus, we suggest that this initial proposal be considered a starting point for the field of CRP program planning and evaluation, and one whose utility can be directly tested in future program evaluation efforts.

Highlights.

• Collegiate recovery programs (CRP) offer programming for students in substance use recovery

• CRPs offer activities intended to produce positive outcomes among their students

• The field lacks a systematic evaluation tool to measure these outcomes

• We provide a recovery capital-oriented theory of change and logic model for CRPs