Reviewing and interpreting the effects of brief alcohol interventions: comment on a Cochrane review about motivational interviewing for young adults

Abstract

Background

Cochrane recently published a systematic review on motivational interviewing (MI) for alcohol misuse in young adults. The review authors concluded that ‘there are no substantive, meaningful benefits of MI interventions for the prevention of alcohol misuse’ (p. 2), as effect sizes were ‘small and unlikely to be of any meaningful benefit in practice’ (p. 27). As most of these interventions were quite brief, we wish to open a dialogue about interpreting effect sizes in this review and of (brief) alcohol interventions more generally.

Analysis

We analyze four methodological aspects of the review that likely influenced the author's conclusions about intervention effects: (1) risk of bias assessments, (2) search strategies, (3) assessing the quality of the body of evidence and (4) definitions of sustainability and clinical significance.

Conclusions

We interpret the effect sizes found in this review to indicate modest yet beneficial and potentially meaningful effects of these interventions, given their brevity and low cost. This interpretation is consistent with other reviews on brief, MI‐based interventions and brief interventions more generally. We therefore encourage the field to re‐open dialogue about the clinical importance of the effects of MI on alcohol misuse by young adults. Rather than dismissing interventions with small effects, we believe a more fruitful way forward for the field would be to catalogue effect sizes for various alcohol interventions. Such a catalogue would help stakeholders themselves to choose which interventions meet their minimum desired impact, and thus may be suitable given their targeted populations, setting and resources.

Introduction

Cochrane has published a systematic review on Motivational Interviewing (MI) for alcohol misuse in young adults 1. Systematic reviews and meta‐analyses of alcohol use interventions provide robust summaries of intervention effects, setting agendas for future research while informing policy and practice 2, 3. We commend the review authors, along with the oversight of the Cochrane Drugs and Alcohol editorial group, for conducting a review on this important topic. MI‐based interventions are common alternatives to more didactic approaches that are more lecture‐based and provide education about alcohol 4, 5. Given the prevalence of MI approaches, understanding their effectiveness is clearly important for the field. The review authors have a notable history of producing systematic reviews instrumental to understanding the effectiveness of various alcohol preventive interventions for young people 6, 7, 8, 9, 10, 11, 12. Similarly, Cochrane is methodologically exemplary in setting international standards for conducting systematic reviews 13.

The review authors concluded that ‘there are no substantive, meaningful benefits of MI interventions for the prevention of alcohol misuse’ (p. 2), stating that effect sizes were ‘small and unlikely to be of any meaningful benefit in practice’ (p. 27) 1. These conclusions were surprising, as they are discrepant with conclusions of other reviews on similar interventions with similar effect sizes 1, 14, 15. This review's conclusions indicate that the field should discuss what should be expected from brief interventions and how best to use them—or to caution their dissemination and focus more research on alternative intervention approaches, as suggested by a recent individual participant data (IPD) meta‐analysis of a subset of trials evaluating brief motivational interventions for college student drinking 16, 17. Careful interpretation of intervention effects is crucial to using systematic reviews for informing future research, policy and practice, as summary conclusions are more likely to attract attention than specific effect sizes from meta‐analyses 18, 19, 20, 21. Critical appraisal of this systematic review is pertinent to understanding its conclusions and why they may be discordant from other reviews: the specific review methods employed can influence review findings and how they are interpreted. Any ‘disciplinary dialogue’ arising from such a critical appraisal and responses to it are also likely to apply to future studies and reviews of brief interventions.

In this paper, we wish to open a dialogue about interpreting effect sizes in this review and of behavioral interventions more generally. Although the review authors’ interpretations of MI effects may eventually be considered the most appropriate conclusions, we believe some of the review's methods prompt constraint in making definitive assertions that MI does not have meaningful benefits for young adult alcohol misuse. We will discuss how the following methodological aspects of the review likely influenced the effect sizes found and their interpretation: (a) risk of bias assessments 22, (b) search strategy, (c) application of the GRADE approach to assess quality of the body of evidence 23 and (d) definitions of sustainability and clinical significance of intervention effects.

Risk of bias

Assessing risk of bias is a key step in conducting systematic reviews. Notably, these assessments play an important role in grading the quality of evidence reviewed and thus directly influence final interpretation of review findings 23. One question to ask when appraising a systematic review is: were appropriate risk of bias assessment procedures used?

This review used the Cochrane risk of bias tool, which prompts review authors to judge risk of bias as high, low or unclear for a series of items that cover established domains of bias in randomized trials (e.g. proper randomization), and to provide their rationale for each judgment 22. We agree with the need for conducting risk of bias assessments in systematic reviews. However, we believe there are some issues concerning how the Cochrane risk of bias tool was used in this review that affected interpretations of review findings.

In re‐analyzing the information reported in the review, it appears that 11% of the risk of bias assessments were inconsistent with either Cochrane guidance (e.g. rating card‐shuffling as ‘unclear risk’ rather than ‘low risk’ for random sequence generation), or with other assessments within the review (e.g. rating reporting of all outcomes as ‘unclear risk’ for selective outcome reporting for some studies and ‘low risk’ for others). In addition, 39% of risk of bias assessments were ‘unclear risk’, meaning the review authors did not have sufficient information to tell whether risks for particular biases were high or low. Information needed to complete various aspects of a systematic review is often missing from published reports of primary studies. To address unclear risks of bias and other missing information, Cochrane guidance requires review authors to contact trial authors for missing data 13. However, the review authors did not report that they contacted the trial authors, and our own contact of the trial authors supported this—of 69 publications, 58 contact authors (84%) replied to our inquiries; of those, 51 (88% of respondents; 74% of all contact authors) either do not recall or have no record of being contacted by the review authors. Despite the responsiveness of trial authors to our e‐mails, it is still possible that the review authors attempted to contact them and simply did not report this information. Regardless, the percentage of ‘unclear’ risk of bias assessments should prompt caution in making definitive assertions about MI effects. It is possible that if more studies were found to be at low risk of bias the quality of evidence would have been graded higher which, in turn, would impact interpretations of effects, as discussed below.

In addition, 94% of trials were judged ‘high risk’ for performance bias because participants and providers were not blind to their assigned intervention condition. Performance bias is standard practice for reviews on pharmaceuticals, and double‐blind procedures are an important indicator of study quality. However, behavioral intervention trials are generally ‘open‐label’ trials 24: it is generally impossible to blind providers and participants because clinicians must know what type of intervention they are delivering, and recipients’ awareness of and engagement with intervention content is often necessary for the intervention to work 25. The review authors even note, in some risk of bias judgments, that it is ‘not possible [to] blind participants to intervention’ (e.g. p. 37). Inability to blind participants and providers is a methodological challenge inherent in most behavioral interventions. Because risk of bias assessments should be relevant to the content area, we propose that lack of blinding providers and participants should be assessed as a limitation only if such blinding is possible. We encourage Cochrane to evaluate whether formal inclusion of performance bias is appropriate for behavioral intervention trials, or make the distinction between ‘lack of blinding’ providers and participants and ‘inability to blind’ them. This precedent has been set by other Cochrane reviews on health behaviors 26, 27, although criteria alternative to blinding are needed to judge performance bias in this area, rather than not judging studies by this risk of bias at all 28. It is worth emphasizing, however, that blinding of outcome assessors is encouraged in behavioral intervention trials, especially given that subjective outcome measures are used prevalently in this area of research 25.

Search strategy

Comprehensive searches for eligible studies from various information sources are a key element of systematic reviews 29. Another question in appraising a systematic review is: were relevant studies missed by the search strategy? Although challenging to identify, ‘grey’ literature—reports that are not controlled by commercial publishers—can provide a more complete view of available evidence by identifying documents linked to journal articles (e.g. dissertations with additional data) and new studies altogether (e.g. studies only reported in dissertations). Searching the grey literature is particularly important in this area, as many MI interventions for young adult alcohol use are evaluated in dissertations 30.

The grey literature was hypothetically eligible in this review. However, the reported search strategy is limited to bibliographic databases of commercially published research and reference lists of included studies and topic‐related systematic reviews. Moreover, the review's references section lists ‘published data only’ as the data source for every included study. This could lead to a significant amount of missing data for studies included in the review, and leave eligible studies unidentified. For example, one author of our paper is conducting a systematic review on Brief Alcohol Screening and Intervention for College Students (BASICS) 31: a manualized, MI‐based intervention for young adult alcohol misuse that meets this Cochrane review's eligibility criteria. Consequently, all randomized trials evaluating BASICS that were published before October 2013 should be included in this review. The Cochrane review identified some manuscripts on BASICS; however, there are 20 studies (11 of which are reported in graduate theses/dissertations) published before October 2013 that are not included in this Cochrane review, as well as 14 papers that provide additional information to studies included in this review that were not cited or referenced in the review (see Supporting information, Supplement 1). The above‐mentioned concerns with risk of bias assessments relying on reported data only are compounded by the lack of specifically searching grey literature sources, where further information about ‘unclear’ risks of bias could potentially be found. Moreover, at least two reports thought by review authors to be from different studies appear to be the same study (Barnett et al., 2010 32 and Monti et al., 2007 33). Limitations in the search strategy's ability to identify and properly classify the primary literature in this area should further temper definitive conclusions.

Quality of evidence

Grading the quality of evidence aims to indicate the likelihood that future research could change the direction or magnitude of estimated intervention effects for the population and outcomes reviewed 34. A relevant appraisal question is: was grading quality of evidence conducted appropriately? Cochrane encourages using the Grades of Recommendation, Assessment, Development and Evaluation (GRADE) working group guidelines 23. This approach assesses quality of evidence for each individual review outcome and details these assessments in a Summary of Findings table 13. Risk of bias assessments are a component—along with directness of evidence, effect size heterogeneity, effect estimate precision and risk of publication bias—that determines quality of evidence. Consequently, concerns about risk of bias assessments and search strategies are not trivial, as these formally influence interpretation of effects using GRADE quality ratings 23. Our concern lies not with the GRADE approach, but with the need for more detailed justifications for downgrading evidence.

The review authors primarily discounted the statistically significant yet small‐to‐moderate effect sizes favoring MI in the Summary of Findings table because of risk of bias assessments. Of the eight outcomes summarized in this table, seven had their quality of evidence downgraded due to ‘risk of bias’ or ‘potential risk of bias’, with no explanation of which biases led to downgrading. Systematic reviewers should delineate the component risks of bias that led to evidence downgrading rather than reporting ‘holistic’ judgments (e.g. downgrading due to ‘risk of bias’). As transparency is a key feature of both the Cochrane risk of bias tool and the GRADE approach, reviewers need to document carefully and justify the procedures used to assess the quality of the evidence (e.g. explicit decision rules for combining risk of bias domains in quality of evidence grades) to allow readers to appraise decisions underlying evidence grades 35.

For this review, greater detail on the use of risk of bias assessments to determine GRADE quality ratings would have been helpful to appraise interpretation of effect sizes critically. Currently, readers cannot discern which risks of bias were used to downgrade evidence, as the review authors reported only ‘risk of bias’ for the rationale where applicable. Moreover, quality of evidence ratings led the review authors to conclude that they ‘cannot rule out the possibility that the effects observed in this review may be exaggerated due to methodological limitations’ (p. 28) 1. Without details on decisions leading to quality‐of‐evidence ratings, we cannot assess clearly whether conclusions are imprecise or even misleading 36, as the aforementioned issues with risk of bias assessments call these evidence grades into question. For example, the review authors may have downgraded evidence for inability to blind providers or participants, which would mean that, at best, behavioral interventions could have moderate quality of evidence if discounted for this reason. We ask experts involved in brief alcohol interventions—and behavioral interventions more generally—to discuss publicly whether lack of blinding participants or providers is appropriate to include in risk of bias or quality of evidence assessments for such interventions.

Sustainability and clinical significance of intervention effects

Summary of Findings tables in systematic reviews intend to present main findings in a transparent, simple format. Sustainability of effects post‐intervention is one important consideration for summarizing interventions with potential public health impact 37, and finding sustainable interventions that have long‐term effects on preventing young adult alcohol misuse is a key goal for the field. However, it is important to reflect upon the nature and purpose of an intervention when defining and assessing its sustainability. MI approaches for young adult populations are often used as brief preventive interventions targeting escalation of drinking during particular high‐risk times. We therefore pose one final critical appraisal question: was the primary focus on effects after 4 months or longer appropriate?

The Summary of Findings table and conclusions for this review—which reported only on effects at 4 months and after—describe MI as not producing effects that reach a minimal clinically important difference (MCID). However, we contend that the review authors have used a particular perspective on MCIDs and expected sustainability of this intervention with which other stakeholders may disagree. First, the MI interventions in the Cochrane review were quite brief: in 49 of 66 included studies (74%) the MI intervention was one session, and seven studies (11%) had two sessions. The ‘longest’ intervention included in the review involved five sessions over a 9‐hour period, while the shortest was 15 minutes, and in 43 trials sessions took 1 hour or less (65%). Given the purpose, brevity and probable cost‐effectiveness of these interventions 38, 39, it may be questionable to rest chief conclusions about effectiveness on ‘sustainable effects’ and define sustainability at 4 months or longer, as many practitioners may not expect or use these brief interventions to have large and long‐lasting effects. For example, most of these interventions were conducted with college students; reducing drinking over a 4‐month period in college (which is typically a semester in the United States) is important, as it may prevent the escalation to heavier drinking patterns and subsequent consequences, such as risky sexual behavior and academic problems 40.

This methodological decision is not trivial, as outcomes differ significantly based on follow‐up period. Intervention effects up to 4 months were larger than effects after 4 months for seven of the eight outcomes in this review—and were at least twice as large for four of these outcomes. As written, it is unclear why review authors defined 4 months as the cut‐off for sustainability. One might argue that it is clinically meaningful for a brief intervention that does not address structural determinants of alcohol misuse (e.g. culture of drinking on college campuses, outlet density and access) to demonstrate observable effects up to 4 months, and smaller yet still observable effects beyond 4 months. Moreover, the effect size from every meta‐analysis in this review favors MI, 19 of 34 (56%) effect sizes demonstrate statistically significant effects, and no studies reported harms for the intervention (see ‘Comparison 1’ in the review). Given the above, we were surprised by the review's strong conclusions unfavorable to MI, especially as the review authors noted that their conclusions differed from another Cochrane review on MI for drug use, even though the other review had similar effect sizes 41. Discrepancies in interpreting similar effect sizes for similar interventions indicate that MCIDs are not defined uniformly in this area, and the review authors’ conclusions represent a particular interpretation of effects (rather than the effects themselves) that differs from other review authors’ interpretations 1, 14, 15.

Suggestions moving forward

We propose several suggestions moving forward. First, the newly developed Methodological Expectations of Cochrane Intervention Reviews (MECIR) provides promising standards for the conduct and reporting of Cochrane reviews 42. We support recommendations to describe the application of the GRADE approach in the Methods section more fully, explain decisions about quality of evidence in reporting results and incorporate information about quality of evidence in discussion sections of Cochrane reviews. For example, authors reviewing behavioral interventions could use a checklist to improve the consistency, reproducibility and transparency of GRADE assessments 43. In addition, although MECIR does not make searching for grey literature mandatory, we recommend that the Cochrane Drug and Alcohol Group consult experts on a given review topic to determine whether grey literature sources (e.g. dissertations) may provide relevant evidence.

There is also a pressing need to adapt existing ‘risk of bias’ and ‘quality of evidence’ assessment tools for behavioral interventions 44, 45, 46. For example, alternative strategies to minimize performance bias (such as preventing participant and provider awareness of primary research hypotheses) could be developed by trial researchers and then used by systematic reviewers in risk of bias assessments when blinding participants and providers is not an option 28. Assessing risk of performance bias becomes more challenging when evaluating automated online interventions that reduce risk of provider performance bias, but not participant performance bias 47. Challenges may also arise when comparing a behavioral intervention to pharmacological interventions—although only when a placebo control is offered, as participants and providers would be aware of the presence of a pill in assigned intervention conditions 48. Cochrane is well positioned to take a leading role in developing and enforcing evidence‐based guidance on assessing risk of bias and quality of evidence tailored to behavioral interventions 44.

Lastly, we believe that further work can be conducted to develop benchmarks for MCIDs. Consensus on MCIDs for various outcomes targeted by substance use interventions is currently lacking. As such, interpreting the meaning of various effect sizes in discussion sections of systematic reviews is left largely to the views of researchers—and to a certain degree peer‐reviewers and editors. Depending on their background and experiences, stakeholders may have different views about the clinical significance of a given effect size. Some providers may find a small effect size meaningful for particular type of intervention used with specific clients, whereas some researchers may find this same effect size as not meaningful if they are focused on long‐term outcomes or widespread dissemination.

Rather than dismissing interventions with small effects as insubstantial, a more fruitful way forward for the field may be to engage stakeholders from various groups in establishing MCIDs for important outcomes in substance use research, given that MI interventions—brief or otherwise—are only one of many interventions available to address alcohol misuse. For example, groups developing core outcome sets could conduct Delphi processes 49 with researchers, practitioners and clients 50 to determine which outcomes each finds most important, the smallest clinical effect that would be meaningful to them and reasons for differences in the selection of core outcomes and definitions of MCIDs by stakeholder group 51. These consensus‐based MCIDs could then be used to inform interpretations of effect sizes in systematic reviews that are sensitive to the views of important stakeholder groups. They could also be used by evidence‐based program registers—such as the US Substance Abuse and Mental Health Services Administration's National Registry of Evidence‐based Programs and Practices 52—to catalogue intervention effect sizes. With hundreds of RCTs on alcohol treatment alone 53, and more than 70 reviews published just by the Cochrane Drugs and Alcohol Group 54, such a catalogue would help stakeholders to choose which interventions meet their own definitions of MCIDs and thus may be suitable given their targeted populations, setting and resources.

Concluding thoughts

As a product of Cochrane, this review has potentially far‐reaching impact. Its conclusions have already been disseminated and discussed in research circles 18, mental health practice and consumer resource circles 19, 20 and lay media 21. Several reviews on MI‐based interventions indicate that these interventions have modest effect sizes. Other reviews in this area have interpreted such effects as beneficial and potentially meaningful, given the brevity and inexpensiveness of these interventions, as well as the focus on changing behavior during high‐risk periods 14, 15. Moreover, several recent reviews provide support for the theory of change underlying MI interventions 55, 56. Effect sizes common to these reviews suggest that these brief interventions are one useful tool in the public health toolkit for preventing young adult alcohol misuse. As such, we encourage the field to re‐open the dialogue about the effects of MI on alcohol misuse in young adult populations. Namely, we ask interested stakeholders to reconsider whether conclusions are commensurate with the information provided in the Cochrane review, effect sizes of these brief interventions are of clinical importance, and future research and policy should prioritize these interventions or others for young adult alcohol misuse. Given the widespread use of MI and brief interventions, such discussion is imperative before making changes to future research, policy and practice.

Declaration of interests

S.G.'s fiancée is a salaried‐employee of Eli Lilly and Company, and owns stock. S.G. has accompanied his fiancée on company‐sponsored travel. E.P., K.O., M.K. and E.J.D. have received and been supported on several grants from NIH to conduct trials on motivational interviewing for substance use. K.O. and E.J.D. are members of the Motivational Interviewing Network of Trainers (MINT). K.O. and E.J.D. train practitioners in motivational interviewing for substance use. K.O. and E.J.D. consult for several organizations regarding motivational interviewing for substance use. All authors have been and/or are currently involved in research evaluating motivational interviewing (MI) and MI‐based interventions.

Supporting information

Online Supplement 1 References of studies identified in BASICS systematic review that meet Foxcroft et al's criteria but are not included

Supporting info item

Grant, S. , Pedersen, E. R. , Osilla, K. C. , Kulesza, M. , and D'Amico, E. J. (2016) Reviewing and interpreting the effects of brief alcohol interventions: comment on a Cochrane review about motivational interviewing for young adults. Addiction, 111: 1521–1527. doi: 10.1111/add.13136.