Protocol for a systematic review of evaluation research for adults who have participated in the ‘SMART recovery’ mutual support programme

Abstract

Introduction

Self-Management and Recovery Training (SMART Recovery) offers an alternative to predominant 12-step approaches to mutual aid (eg, alcoholics anonymous). Although the principles (eg, self-efficacy) and therapeutic approaches (eg, motivational interviewing and cognitive behavioural therapy) of SMART Recovery are evidence based, further clarity regarding the direct evidence of its effectiveness as a mutual aid package is needed. Relative to methodologically rigorous reviews supporting the efficacy of 12-step approaches, to date, reviews of SMART Recovery have been descriptive. We aim to address this gap by providing a comprehensive overview of the evidence for SMART Recovery in adults with problematic alcohol, substance and/or behavioural addiction, including a commentary on outcomes assessed, potential mediators, feasibility (including economic outcomes) and a critical evaluation of the methods used.

Methods and analysis

Methods are informed by the Cochrane Guidelines for Systematic Reviews and the Preferred Reporting Items for Systematic Reviews and Meta-Analysis statement. 6 electronic peer-reviewed and 4 grey literature databases have been identified. Preliminary searches have been conducted for SMART Recovery literature (liberal inclusion criteria, not restricted to randomised controlled trials (RCTs), qualitative-only designs excluded). Eligible ‘evaluation’ articles will be assessed against standardised criteria and checked by an independent assessor. The searches will be re-run just before final analyses and further studies retrieved for inclusion. A narrative synthesis of the findings will be reported, structured around intervention type and content, population characteristics, and outcomes. Where possible, ‘summary of findings’ tables will be generated for each comparison. When data are available, we will calculate a risk ratio and its 95% CI (dichotomous outcomes) and/or effect size according to Cohen's formula (continuous outcomes) for the primary outcome of each trial.

Ethics and dissemination

No ethical issues are foreseen. Findings will be disseminated widely to clinicians and researchers via journal publication and conference presentation(s).

Prospero registration number

CRD42015025574.

Introduction

Addiction is a widespread and serious concern. Addiction can be defined as a behaviour that is habitual, compulsive and continued despite problematic cognitive, behavioural and/or physiological consequences.¹ Addictions formally recognised by current diagnostic systems include substance-related (alcohol, cannabis, hallucinogens, inhalants, opioids, sedatives, hypnotics and anxiolytics, stimulants and tobacco) and gambling.¹ Internet gaming has recently been added as a condition warranting further study.¹ Other common and problematic behavioural addictions yet to receive diagnostic classification include shopping,² internet³ and sex.⁴ Recent data indicates that more than 40% of Australians either smoked daily, engaged in hazardous levels of alcohol use or had used at least one illicit substance in the preceding 12 months.⁵ Although prevalence estimates for many behavioural addictions are complicated by lack of standardised criteria, problem gambling is estimated to affect up to 160 000 Australian Adults per year.⁶

The burden of addiction is considerable. Alcohol and substance use disorders are leading causes of premature mortality and account for over 20% of the 183.9 million disability-adjusted life years lost to mental and substance use disorders worldwide.⁷ In Australia, problem gamblers lose an average of $21 000 per year—approximately one-third of the average salary.⁶ Substance and behavioural addictions also have a profound and detrimental impact on health, relationships, employment and quality of life.^8–10 Together, the harms from alcohol, substances and behavioural addictions such as gambling cost Australians over $28 billion per year.^{6 11 12}

The course of addiction is often chronic and characterised by multiple relapses.¹³ However, sustained recovery is possible. Although the actual definition of recovery will vary according to the individual, the capacity to create and live a meaningful life is key.¹⁴ Recovery-oriented service provision acknowledges the importance of harnessing strengths, maximising self-determination and facilitating self-management such that an individual can recognise and take responsibility for their own well-being and recovery.¹⁴ ‘Mutual aid’ is often central to this process. ‘Mutual aid’ refers to social, emotional and informational support provided by, and to, group members undergoing recovery from addiction.¹⁵

Within the addiction field, 12-step models (eg, alcoholics anonymous, narcotics anonymous) are the largest and most researched source of mutual aid. Within this model, addiction is conceptualised as a medical and spiritual disease, with recovery reliant on relinquishing control to a higher power.¹⁶ Systematic reviews and meta-analyses consistently demonstrate that improvement following 12-step participation is at least equivalent to that of professional interventions for adults with alcohol dependence,^17–19 and in the longer term, active participation increases the likelihood of sustained recovery.^{15 20} Relative to the often time-limited format of formal treatment, mutual aid represents a mechanism for accessing ongoing, long-term support. The importance of mutual aid in promoting and sustaining recovery is also highlighted by the National Institute for Health and Care Excellence (NICE), which recommends that staff routinely provide information about and facilitate access to and engagement in mutual aid groups.^{21 22}

Why it is important to do this review?

Although current findings clearly support the benefits of mutual aid, much of the evidence comes from the study of 12-step models, and focuses on adults with alcohol dependence. However, less is known about the impact of mutual aid on other substance and/or behavioural addictions. Moreover, individuals may fail to engage with 12-step groups, for example, due to a mismatch between personal beliefs and the 12-step philosophy.^{23 24} Indeed, to enhance engagement, clinical guidelines advocate for tailored addiction support that accounts for individual needs and preferences.^{21 22} Choice over mutual aid support options is therefore important—especially given individual variation in presenting concerns and the definition and process of recovery. Alternatives, albeit lower in profile to the dominant 12-step model have been available for a number of years (see ref. 18 for a review). Self-Management and Recovery Training (SMART Recovery) is one model that is cited alongside 12-step as a recommended source of mutual aid by Australian^{25 26} and international^{21 22} clinical guidelines.

SMART Recovery is a not-for-profit organisation that provides group and online mutual aid support. Unlike 12-step groups that are often addiction specific (eg, alcoholics anonymous, narcotics anonymous, gamblers anonymous), SMART Recovery groups offer support for a range of problematic behaviours, including alcohol, substance and/or other addictive behaviours (eg, gambling, eating, technology, pornography).²⁷ SMART Recovery focuses on self-empowerment and adopts key principles (eg, self-efficacy) and therapeutic approaches (eg, motivational interviewing and cognitive behavioural therapy) shown to be effective in promoting recovery from addiction (see²⁸ for a recent review of the efficacy of these approaches and online supplementary file 1 for an overview of SMART Recovery principles/strategies). Although these strategies are clearly evidence based, further clarity regarding direct evidence for the efficacy of SMART Recovery as a mutual aid package is needed.

Relative to the methodologically rigorous reviews summarising the evidence for 12-step models^17–19 to date, reviews of SMART Recovery are descriptive. The focus tends to be on the origins, development and principles of SMART Recovery, with limited analysis of feasibility, efficacy and/or potential mechanisms of action.²⁹ Any changes in healthcare practice and policy should rely on a solid evidence base. This systematic review represents an important step, as it will comprehensively summarise the available evidence on SMART Recovery and identify areas of research need. Results will inform the public health and clinical utility of SMART Recovery as a potentially helpful recovery resource for individuals suffering from addiction disorders.

Objectives

Guided by the review questions listed below, we aim to provide a comprehensive overview of the current state of evidence for SMART Recovery in adults with experience of substance and/or behavioural addiction(s), including a commentary on

1. Population and outcomes assessed, potential mediators and a critical evaluation of the methods used to evaluate SMART Recovery;

2. Feasibility of SMART Recovery, including economic outcomes (eg, cost, resource use, cost-effectiveness), attendance and service user and/or provider satisfaction;

3. Future research directions.

Review question

For adults with experience of substance and/or behavioural addiction(s)

1. Does SMART Recovery result in changes to severity of addiction and its consequences (eg, quantity, frequency and severity of addictive behaviour; quality of life; functioning)?

2. Is the effect of SMART Recovery on the above listed treatment outcomes influenced by:

   1. Treatment engagement (eg, quantity, frequency and/or duration of SMART Recovery attendance);
   2. Process measures/mediators/mechanisms (eg, cognitive (empowerment/self-efficacy/motivation); behavioural (eg, active coping, including managing urges); process (eg, therapeutic alliance))?

3. What is the evidence for the feasibility of SMART Recovery, including commentary on economic outcomes (eg, cost, resource use, cost-effectiveness), attendance and service user and/or provider satisfaction?

Methods and analysis

A systematic review will be conducted in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA³⁰).

Eligibility criteria

Types of studies

In accordance with the objective of providing an overview of the current evidence for SMART Recovery in adults with experience of substance and/or behavioural addiction(s), liberal design criteria will be adopted. The following designs will be included—randomised controlled trials (RCTs; cluster and parallel design); cross-over trial; case series or case controls; one-arm trial; non-randomised trials; cross-sectional or cohort studies and case reports. As broad inclusion criteria may increase risk of bias, this will be assessed using the Collaboration's risk of bias tool, as described in the Cochrane Handbook for Systematic Review of Interventions³¹ (detailed under risk of bias assessment below). Qualitative-only designs will not be included.

Types of participants

Adults (≥18) attending SMART Recovery with current or past problematic experience of at least one addictive behaviour (substance and/or behavioural), identified via patient and/or carer subjective report, self-report assessment and/or clinical interview. ‘Problematic’ will be defined in terms of subjective and/or objective impact on functioning and/or comparison to recommended guidelines. Participants may be residing in the community, rehabilitation, treatment and/or correctional facility.

Types of interventions

The intervention of interest is SMART Recovery, delivered in a group format, of any intensity or frequency, by a trained facilitator. We will include all SMART Recovery approaches, including both conventional mutual aid groups delivered by a non-professional volunteer and SMART Recovery informed groups delivered by a trained professional. SMART Recovery may be a standalone intervention and/or delivered in combination with other treatment components, including pharmacological. Interventions delivered in any setting will be included (eg, online, community, hospital, rehabilitation or residential treatment centre, etc).

Types of comparison conditions

The intervention may be compared with inactive (eg, standard care, waiting list control) and/or active controls (eg, 12-step programmes, psychological interventions) of any intensity, frequency and delivery method (eg, individual, group, technology assisted). Evaluations of SMART Recovery without a comparator group will also be included.

Types of outcome measures

1. Severity of addiction and its consequences (eg, quantity, frequency and severity of addictive behaviour; quality of life; functioning);

2. Treatment engagement (eg, quantity, frequency and/or duration of SMART Recovery attendance);

3. Process measures/mediators/mechanisms (eg, cognitive (empowerment/self-efficacy/motivation); behavioural (eg, active coping, including managing urges); process (eg, therapeutic alliance));

4. Feasibility, including economic outcomes (eg, cost, resource use, cost-effectiveness) and/or attendance/satisfaction/preference. Qualitative outcomes regarding participant and/or provider satisfaction will be reported as described.

Outcomes may be clinician and/or patient rated, assessed by objective and/or subjective indices (eg, blood, urine, actigraph, questionnaire, monitoring form/diary) with or without collateral information (eg, using a family member to validate use) and of any time frame (eg, baseline, short-term and/or medium-term and/or long-term follow-up).

Information sources

Search strategy

Consistent with methods detailed in Cochrane Guidelines for systematic reviews,³¹ the search strategy will be conducted as follows. First, in May 2015, we consulted with a qualified librarian and identified seven relevant scientific electronic databases (MEDLINE, PubMed, EMBASE, CINAHL Complete, Psychinfo and Central) and four electronic non-scientific databases (Google Scholar; Virginia Commonwealth University; Project Cork; Prevention, Information and Evidence Library) to search. Search terms related to SMART Recovery will be combined with addiction-related search terms and then outcome-related search terms (see online supplementary file 2 for the full MEDLINE search strategy).

Abstract, title, key words and subject headings specific to each of the identified database will be searched. All subject headings will be exploded, so that narrower terms are included. No limits will be placed on publication year. Publications must be available in English. Reference lists of identified publications will be hand searched to identify any additional publications. All publications will be organised in reference manager Endnote. The searches will be re-run just before final analyses and further studies retrieved for inclusion. All searches will be performed by AKB.

Classification of studies

The titles and abstracts of identified references will be classified in a three-step process.

Step 1: identification of studies for exclusion

AKB will review the titles and/or abstracts of identified references and exclude articles if they: (1) are duplicates; (2) do not focus on adults with a substance and/or behavioural addiction; (3) do not focus on SMART Recovery; (4) if the outcomes, process and/or predictor variables do not include or specifically relate to SMART Recovery or (5) are not journal articles, reports, book chapters or newsletter articles. If eligibility is unclear from the title and/or abstract, the full-text article will be accessed and assessed.

Step 2: classification of studies

The abstracts and/or full text of the remaining studies will be examined by AKB to identify studies that are (1) evaluation, defined as an evaluation of SMART Recovery as per the PICO (Population, Intervention, Comparator, Outcomes) criteria outlined above; (2) reviews, including summaries, descriptive, critical and/or systematic reviews (3) discussion, defined as general discussion of SMART Recovery, including its development, principles, methods and implementation. References that are not evaluation, review or discussion papers (eg, treatment manuals) will be classified as ‘other’.

Step 3: cross-checking

Publications from step 2 will be cross-checked by having a research assistant blinded to the results of the initial classification, reclassify the publications. In case of disagreement, the final classification will be made by consensus, with the involvement of AB. The articles excluded in step 1 will not be cross-checked because they will not be relevant to the review. The evaluation studies identified in step 2 will be retained for further examination.

Data extraction from evaluation studies

Data extraction will be performed by AKB and checked by EF. Extraction forms will be piloted on several papers and modified as needed before use. When multiple reports of the same study are identified (eg, related journal articles, conference proceedings which are then published), data from each report will be extracted separately and then combined across multiple data collection forms. In accordance with Cochrane Guidelines, methodological critique and assessment of risk of bias will be performed independently by two raters (AKB and EF) and judgements reached by consensus. In the event of disagreement, final ratings will be made via consensus with a third independent rater (following discussion with AB). The presence and resolution of any disagreements will be carefully recorded (ie, original and consensus ratings) to allow for assessment of reliability of coding. In the event that inadequate trial details are reported, study authors will be contacted on no more than two occasions to obtain further information.

To enable methodological critique of observational research and RCTs, criteria for data extraction will be adapted from the Downs and Black Scale³² and the Cochrane Handbook for Systematic Reviews³¹ and include:

1. Participant information, including n-values at each stage of the study (and reasons for non-participation), treatment setting, eligibility criteria, descriptive data including age, gender, ethnicity, socioeconomic status, diagnostic criteria, treatment history;

2. Methods, including study design, country, setting(s), methodological limitations reported, methodological limitations observed (eg, recruitment allocation and data collection methods; blinding; comparability of groups at baseline; appropriateness of analysis methods; bias/selective reporting);

3. Interventions, including number of groups, duration of treatment (number, frequency and duration of SMART Recovery and any additional treatment components), delivery method(s; including professional vs peer facilitation), description of control intervention(s);

4. Primary and secondary outcomes, including data collection sources/methods, percentage of treatment sessions attended, other process measures/mediators/mechanisms, economic outcomes, satisfaction-related outcomes, follow-up period (short-term vs Medium-term vs long-term follow-up; defined as 1–6, 7–12 and >12 months after intervention completion, respectively);

5. Results, including severity of addiction and its consequences, treatment engagement, process measures/mediators/mechanisms, economic outcomes and patient satisfaction collected at all available follow-up time points.

See online supplementary file 3 for proposed data extraction forms (to be managed using Microsoft Excel).

Methodological critique of evaluation research

To provide a thorough overview of the literature, we will implement procedures to evaluate the quality of observational studies and RCTs. A narrative synthesis of the findings from the included studies will be reported, structured around intervention type and content, population characteristics, and outcomes. In order to better inform research and clinical care, we intend to describe the treatment context (eg, SMART Recovery alone vs additional pharmacological and/or non-pharmacological support; professionally managed vs peer-operated community groups) and whether the studies target particular addictive behaviours (eg, alcohol, smoking, illicit substances, other addictive behaviours) and/or clinical presentations (eg, addiction only vs dual diagnosis). This qualitative review will be supplemented with the following quantitative measures.

For observational studies, methodological quality will be assessed against the Downs and Black Scale.³¹ Criteria will be assigned a yes (1 point), no (0 points) or unclear (0 points) rating. All criteria will have the same weight, and a quality score ranging from 0 to 27 points will be calculated for each study.

For RCTs, methodological quality will be assessed against the 11-item Physiotherapy Evidence Database (PEDro) scale.³³ Consistent with published reviews of psychological interventions,^{34 35} two items regarding blinding of participants and therapists will not be scored, as these criteria are not appropriate for the studies under review. The remaining nine criteria will be assigned a yes (1 point) or no (0 points) rating, and a quality score ranging from 0 to 8 points will be calculated for each study (as item 1 is not included in the quality score³³).

Risk of bias (within and across studies) will also be assessed using the Collaboration's risk of bias tool, as described in the Cochrane Handbook for Systematic Review of Interventions.³¹ We will judge each item as being high, low or unclear risk, as per the criteria provided by Higgins and Green³¹ and provide a quote from the study report and a justification for our judgement for each item in the risk of bias table. Given that growing empirical evidence suggests that sequence generation and allocation concealment are particularly important potential sources of bias, studies will be deemed to be at the highest risk of bias if either item is scored as ‘high’ or ‘unclear’.

Measures of treatment effect

A narrative synthesis of the findings from the included studies will be reported, structured around intervention type and content, population characteristics, and outcomes. Where possible, ‘summary of findings’ (SOF) tables will be generated for each comparison (eg, pharmacological/psychological treatment alone vs pharmacological/psychological treatment plus SMART Recovery; SMART Recovery vs other mutual aid support groups; SMART Recovery vs active treatment; SMART Recovery vs inactive control). SOF tables will provide key information regarding evidence quality, the magnitude of effect of the interventions examined (ie, within and between groups effect sizes), and a summary of available data on the outcome variables defined above.

Dichotomous outcome measures

When data are available, a risk ratio (RR) and its 95% CI will be provided for the primary outcome of each trial. RR has been selected in preference to ORs as evidence suggests that RR is more intuitive³⁶ and clinicians tend to misinterpret ORs as RR.³⁷

Continuous outcome measures

When data are available, between-groups effect sizes will be calculated according to Cohen's formula, to allow for comparison across studies. Effect sizes will be interpreted according to published guidelines, where 0.2–0.49 is defined as a small effect size, 0.5–0.79 is moderate and >0.8 is large.

A study will be considered to have a positive outcome if at least 50% of reported outcomes demonstrate a between-group difference in favour of SMART Recovery at the end of the intervention. Positive maintenance outcome(s) will be evidenced when this effect is also evident at short-term and/or medium-term and/or long-term follow-up (defined as 1–6, 7–12 and >12 months after intervention completion, respectively). We anticipate there will be limited scope for meta-analysis due to the range of different outcome measures.

Ethics and dissemination

As no primary data collection will be undertaken, no formal ethical assessment is required. We plan to present the findings of this systematic review for peer review in an appropriate journal. We also intend to present to clinicians and researchers at appropriate conferences, including preliminary findings to the Australasian Professional Society on Alcohol & other Drugs in November 2015.