Study design to examine the potential role of assessment reactivity in the Screening, Motivational Assessment, Referral, and Treatment in Emergency Departments (SMART-ED) protocol

Abstract

Background

Screening, brief intervention, and referral to treatment (SBIRT) approaches to reducing hazardous alcohol and illicit drug use have been assessed in a variety of health care settings, including primary care, trauma centers, and emergency departments. A major methodological concern in these trials, however, is “assessment reactivity,” the hypothesized impact of intensive research assessments to reduce alcohol and drug use and thus mask the purported efficacy of the interventions under scrutiny. Thus, it has been recommended that prospective research designs take assessment reactivity into account. The present article describes the design of the National Institute on Drug Abuse Clinical Trials Network protocol, Screening, Motivational Assessment, Referral, and Treatment in Emergency Departments (SMART-ED), which addresses the potential bias of assessment reactivity.

Methods/design

The protocol employs a 3-arm design. Following an initial brief screening, individuals identified as positive cases are consented, asked to provide demographic and locator information, and randomly assigned to one of the three conditions: minimal screening only, screening + assessment, or screening + assessment + brief intervention. In a two-stage process, the randomization procedure first reveals whether or not the participant will be in the minimal-screening-only condition. Participants in the other two groups receive a more extensive baseline assessment before it is revealed whether they have been randomized to also receive a brief intervention. Comparing the screening only and screening + assessment conditions will allow determination of the incremental effect of assessment reactivity.

Discussion

Assessment reactivity is a potential source of bias that may reduce and/or lead to an underestimation of the purported effectiveness of brief interventions. From a methodological perspective, it needs to be accounted for in research designs. The SMART-ED design offers an approach to minimize assessment reactivity as a potential source of bias. Elucidating the role of assessment reactivity may offer insights into the mechanisms underlying SBIRT as well as suggest clinical options incorporating assessment reactivity as a treatment adjunct.

ClinicalTrials.gov Identifier

NCT01207791.

Background

Since the publication of the Institute of Medicine’s report on broadening the base of treatment for alcohol problems [1], there has been a marked expansion in efforts to identify individuals with hazardous or harmful alcohol use, provide a brief intervention (BI) when indicated, and refer those with more serious levels of alcohol dependence for specialty substance abuse treatment. This process, known as screening, brief intervention, and referral to treatment (SBIRT), has been expanded more recently to also screen for illicit drugs, tobacco, and prescription medications [2,3]. It has also been implemented in a variety of health care settings where the prevalence of alcohol and drug misuse are thought to be relatively high. These have included primary care, trauma center, and emergency department (ED) settings [2,4,5].

The interventions delivered in these settings to individuals who screen positive for hazardous or harmful alcohol or drug use are often quite brief, sometimes as short as 15–20 minutes. These approaches are typically based on the principles of motivational interviewing [6-8], with feedback provided about the individual’s pattern of use and consequences derived from screening questionnaires or interviews. In clinical trials evaluating the efficacy of such interventions, extensive assessments are also conducted to elicit information characterizing the patient sample. These assessments measure substance-related and other variables that potentially moderate or mediate the effects of treatment.

Increasing concerns have been raised in SBIRT clinical trials regarding the potential impact of such assessments. The intensive assessments, relative to the brevity of the SBIRT intervention being delivered, have raised concerns that the treatment outcome may be contaminated by “assessment reactivity.” Assessment reactivity is the process by which increasing an individual’s awareness of potential problem areas by targeted and extensive assessment may, in and of itself, initiate behavior change in the absence of even a minimal intervention. Thus, it is thought that the extensive assessment may have a positive therapeutic effect and contribute to the change process [9]. While this may be beneficial from a clinical perspective, assessment reactivity may reduce the difference between the active intervention effect relative to the “inactive” control condition, thus concealing the therapeutic benefit of the active intervention [10-14]. This may have contributed to the absence of a significant difference between the intervention and “assessment only” comparison conditions in some SBIRT trials and led to an underestimation of the actual beneficial effects of BI [15].

Issues of assessment reactivity were initially noted in regard to the effects of repeated post-treatment follow-up assessments in longitudinal evaluations of alcohol treatment interventions [16-18]. A randomized study that varied both the frequency and intensity/comprehensiveness of post-treatment assessments, for example, found the poorest drinking outcomes among patients in the infrequent/low comprehensive assessment condition [17]. These findings highlighted the clinical influence of the assessment process on outcomes, and the authors also noted the lack of previous methodological designs to control or account for possible assessment reactivity.

Subsequent attention shifted from post-treatment follow-up assessment to the potential impact of assessments that take place prior to the initiation of treatment in clinical trials [19,20]. These studies found that reductions in drinking/drug use occurred between an initial screening, subsequent intake assessment, and the first therapy session. Further, these screening/post-assessment/pretreatment reductions predicted better treatment outcomes. As an example, Epstein and colleagues [19] found significant reductions in percentage of drinking days after each of the following events: a brief 10-minute telephone screening, a 90-minute clinical screening, and a 3–4 hour baseline research interview. However, there were no changes from baseline to the day before treatment or from the day before treatment to the end of the first week of treatment. Thus, the primary change in drinking behavior occurred prior to treatment initiation. While it could be argued that the observed changes reflected regression toward the mean [21], the increased attention to drinking may initiate a change process and may reflect an active therapeutic benefit of assessment [9,22]. Randomized experimental evaluations of the effects of assessment on measures of alcohol consumption among heavy-drinking college students involved in BIs have similarly found a reactivity effect [10,12,14], supporting the view that an active therapeutic process is at play rather than regression to the mean.

In contrast, studies in EDs targeting alcohol use [23,24] and marijuana [25] have not found an assessment reactivity effect. As an example, ED patients identified as having hazardous drinking patterns were randomly assigned to a screen-only, an assessment-only, or an assessment + intervention condition [23]. Drinking outcomes at a 12-month follow-up indicated that patients in both the screen-only and assessment + intervention conditions had greater, but nonsignificant, improvements compared with those in the assessment-only condition. In a study with a similar design, reductions in drinking-related outcomes were found at a 12-month follow-up among individuals identified as hazardous drinkers in the ED [24]. There were no differences, however, between patients who had been randomized to a 10–15 minute BI, a screen + assessment condition, or a screen-only group.

A recent meta-analysis concluded that there have been too few high-quality, appropriately designed studies to allow any firm conclusions about the existence of assessment reactivity bias [11]. Based on these concerns, it has been recommended that researchers consider methodological approaches to identify, quantify, and minimize the potential confounding and biasing effects of assessment reactivity in any analyses targeting treatment outcomes [11,13,15,20]. Most prior studies of SBIRT, including those in EDs, have not done so. As shown in Figure 1, most prior studies have compared a screen + assessment condition with a screen + assessment + intervention condition. Taking into account the above recommendations, more recent approaches have begun to include a screen-only condition that excludes the intensive assessment component, in addition to the other two conditions [23-25]. Such a design allows one to disaggregate the impact of assessment from the combined effect of assessment plus BI. Without the inclusion of a “no assessment” or “minimal assessment” control group, it is not possible to determine how much the assessment contributed to outcome, or, in the case of a null finding, whether it masked the effects of the therapeutic intervention [11,15,26].

Figure 1.

Prior and more recent emergency-department SBIRT designs.

Methods/design

Overview

These concerns were taken into consideration in the design of the NIDA Clinical Trials Network’s protocol, Screening, Motivational Assessment, Referral, and Treatment in Emergency Departments (SMART-ED) [26]. The study, which is being conducted in six emergency departments across the US, is evaluating the efficacy of a motivational interviewing-based BI delivered in the ED setting, targeting individuals who screen positive for using/abusing illicit drugs or prescription medications. All study materials and procedures have been reviewed and approved by the appropriate Institutional Review Boards (IRBs) of participating universities and affiliated EDs. The trial will be conducted in compliance with protocol, International Conference on Harmonisation Good Clinical Practice guidelines, and applicable regulatory requirements. An independent Data and Safety Monitoring Board oversees the study.

In order to assess the independent contributions of assessment and the BI to outcomes in the study population, a three-arm design was adopted. The three treatment groups are (1) a minimal screen-only condition (MSO) in which participants receive as little interaction as possible and do not complete an intensive baseline assessment; (2) screening, assessment, and (if indicated or requested) referral to treatment (SAR); and (3) screening, assessment, and (if indicated or requested) referral, plus BI lasting up to 30 minutes in the ED followed by two telephone booster intervention sessions within 1–4 weeks following the ED visit (BI-B). Figure 2 presents an overview of this design.

Figure 2.

SMART-ED design overview.

Primary hypotheses

There are three primary hypotheses: (1) Participants randomized to BI-B will have fewer days of use of the patient-defined primary problem substance during the 30 days prior to the 3-month assessment relative to participants randomized to MSO; (2) participants randomized to BI-B will have fewer days of use relative to participants randomized to SAR; and (3) participants randomized to SAR will have fewer days of use relative to participants randomized to MSO.

The study also has a number of secondary objectives: (1) To evaluate the effect of BI-B relative to SAR on substance abuse treatment engagement among those participants who are referred to treatment; (2) to evaluate the effect of the experimental interventions on self-reported health-care utilization; (3) to evaluate several putative predictors of outcome (patient demographics, substance use severity at baseline, primary substance of abuse, motivation, therapeutic alliance, perceived substance-relatedness of ED visit, and psychiatric comorbidity) in terms of both main effect and interaction with treatment assignment; and (4) among participants assigned to BI-B, to evaluate dose–response to the number of booster sessions received.

Study population: Inclusion/exclusion criteria

To be included in the trial, individuals need to meet the following criteria: (1) registration as patient in the ED during study screening hours; (2) positive screen (≥3) on the 10-item Drug Abuse Screening Test (DAST-10) for problematic use of a nonalcohol, non-nicotine drug; (3) at least one day of problematic drug use (excluding alcohol and nicotine) in the past 30 days; (4) age 18 years or older; (5) adequate English proficiency and literacy; (6) ability to provide informed consent; and (7) access to phone (for booster sessions). Individuals are excluded from consideration if they meet any of the following criteria: (1) Inability to participate due to emergency treatment; (2) significant impairment of cognition or judgment rendering the person incapable of informed consent (e.g., traumatic brain injury, delirium, intoxication); (3) status as a prisoner or in police custody at the time of treatment; (4) current engagement in addiction treatment; (5) residence more than 50 miles from the location of follow-up visits; (6) inability to provide sufficient contact information (e.g., at least 2 reliable locators); and (7) prior participation in the current study.

Emergency department patients who are subsequently hospitalized may differ from other ED patients in ways that could affect outcome and response to treatment. For example, they would likely have more severe medical conditions, which could affect motivation to change, and may have greater exposure to treatment and referral while in the hospital. Since there is no reliable way to know which patients will be hospitalized at the time of randomization, hospitalized patients will be kept in the study. The ED logs will be reviewed to identify all participants who are ultimately hospitalized as a result of their ED visit. Secondary analyses will explore whether such patients differ from nonhospitalized participants in substance use outcome or treatment response.

Characteristics of participating emergency departments

To be included as potential sites for this trial, EDs should have a large volume of patients who use drugs and that represent the US population in aggregate, have prior research experience, and not currently be routinely conducting SBIRT targeting drug users. Further, they should be able to present a convincing plan for patient flow and use of space, have (or are able to hire) appropriate staff to conduct the study, have a sufficient referral network for patients needing specialty addiction treatment, and have an ED physician who can serve as protocol PI (or otherwise be actively involved in the protocol). From a pool of 17 EDs that responded to an initial site-survey request, and following review of the above criteria and subsequent site visits to a smaller subset of potential sites, six EDs were selected to participate in the trial.

Primary and secondary outcome measures

The primary outcome for the study is the number of days of use of the patient-defined primary problem drug, assessed for the 30-day period preceding the 3-month post-intervention follow-up, as assessed by the Timeline Followback (TLFB) [27,28].

In addition, a number of secondary outcomes are included: (1) number of days of primary problem substance use during the 30 days preceding the 6- and 12-month postintervention follow-ups; (2) days of abstinence from all drug use during the 30 days preceding the 3-, 6-, and 12-month postintervention follow-ups; (3) days of heavy drinking in the past 30 days at 3, 6, and 12 months; (4) quantity of use (dollar value, number of times used) for the primary drug at 3, 6, and 12 months from baseline; (5) change in days of use of the primary drug between baseline and follow-up at 3, 6, and 12 months (only for the SAR and BI-B groups that completed the TLFB at baseline); (5) relative change in hair-sample drug screen results for the primary drug and also for any drug of abuse from baseline to 3, 6, and 12 months postintervention; (6) consequences of drug use (based on the NIDA-Modified Alcohol, Smoking and Substance Involvement Screening Test (NM-ASSIST) [29]; (7) participation in addiction treatment in the past 30 days at 3, 6, and 12 months postintervention among SAR and BI-B participants with probable dependence who are referred for treatment (assessed with the Treatment Services Review [TSR]) [30]; and (8) health-care utilization (number of ED visits, number of hospital days, and number of outpatient visits) in the past 30 days at 3, 6, and 12 months based on TSR results.

Screening of potential participants

During defined recruitment hours, Research Assistants (RAs) will identify possibly eligible patients seen in the ED and will approach them about their willingness to participate in anonymous screening to determine the representativeness of the study sample. Participants will provide verbal (not signed) consent using a brief IRB-approved script for the anonymous collection of screening data, which includes age, gender, and reason for ED visit. The RA will attempt to screen all ED patients until recruitment hours have ended or the maximum recruitment for the week is achieved (no more than 12 participants randomized per week per site). Refusals and inability to participate (e.g., unavailable due to emergency medical treatment, left without being seen) will be recorded.

Study staff will time their involvement with patients to minimize interference with medical treatment. Depending on level of acuity, some patients will be approached prior to the initial evaluation by a physician, and some after. Patients who show signs of intoxication (e.g., somnolence, slurring of speech) will not be screened unless these signs resolve. Screening data will be collected by research personnel and by participant self-report using direct entry into tablet computers to facilitate rapid screening, electronic data capture, and mobility within the busy ED setting.

The logistics of conducting screening in the ED require that the screening process be relatively brief and minimally intrusive. Further, to minimize potential assessment reactivity in the MSO group, substance-related screening questions were to be kept at an absolute minimum. A four-section screening instrument, the Tobacco, Alcohol, and Drug (TAD) questionnaire, will be used to capture the substances assessed. To maximize confidentiality, the TAD will be completed by the participants, who will enter the data directly using the tablet PC unless the participant is not comfortable with this technology, in which case the data may be entered by the RA. The four sections, which can be completed in 5–10 minutes, consist of (1) the 4-item Heavy Smoking Index [31], a brief screen for nicotine dependence that is administered first to minimize participant discomfort with answering more sensitive questions about drug use; (2) the three consumption items of the Alcohol Use Disorders Identification Test (AUDIT-C) [32,33]; (3) the DAST-10 [34-36]; and (4), for individuals scoring ≥3 on the DAST-10, three questions to determine primary substance of abuse, days of use of the primary substance, and substance-relatedness of the ED visit. The screen is considered positive if the DAST-10 score is ≥ 3 and the individual reports past 30-day use of the primary substance.

Participants who meet screening criteria based on the TAD will be further screened for eligibility by the RA, who will ask participants about (1) current engagement in addiction treatment, (2) whether they reside more than 50 miles from the follow-up location, (3) ability to provide at least two locators, and (4) access to a phone. These two prescreening forms (the TAD and eligibility) will not include identifying information. Individuals who are interested and eligible to participate will then undergo the informed consent process. Patients who do not meet inclusion criteria will receive no further intervention, but the anonymous screening data will be kept to allow comparison to those who do meet criteria.

Informed consent process

Patients will be provided with an IRB-approved informed-consent form that will include a description of all significant elements of the study. The consent form will also include assurances of confidentiality and a statement that participation is entirely voluntary, that the decision to participate will in no way influence other aspects of the patient’s treatment, and that the participant is free to withdraw participation at any time. Patients must be able to read the first paragraph of the informed consent and express verbally their understanding of the key elements of the study (e.g., random assignment, possible interventions received, duration of follow-up). They will then indicate their consent to participate in the study by signing and returning the informed consent form. A Health Insurance Portability and Accountability Act (HIPAA) disclosure form will also be required to allow the study to access protected health information in the patient’s medical record in the ED. Because of the time constraints of completing the assessment and intervention in the ED, and because of the relatively low risks associated with the study, the consent form will be as brief as possible within the constraints of adequate human subject protections. Prior to randomization, all consenting participants will complete a demographic questionnaire, provide locator information, and provide a hair sample, which will be used to obtain an objective measure of substance use.

Randomization procedure

After providing informed consent, all eligible participants will be randomly assigned in 1:1:1 ratio to one of the three conditions (MSO, SAR, or BI-B). Randomization will be stratified based on site, presence of an alcohol use disorder (AUDIT-C score ≥ 4), and drug problem severity (DAST-10 score ≥ 8). The randomization procedure will be conducted through a centralized, web-based process set up by the NIDA Clinical Trials Network Data and Statistics Center. The randomization sequence will be unknown to staff, and treatment assignment will be revealed in two stages. Initially, the RA performing the randomization will be informed as to whether the participant is in the MSO group or not. If so, the RA is finished with the research component in the ED and will have no further study involvement until the 3-month follow-up visit. The remaining participants will receive a more extensive baseline assessment. After completion of the baseline assessment, RAs will be informed whether participants are in the SAR or BI-B group. Research assistants conducting the follow-up assessments will be blinded to treatment condition. The randomization scheme was designed to balance patient allocation to each treatment arm. Given the intent-to-treat nature of the study, participant dropouts after randomization will not be replaced.

Baseline assessments (SAR and BI-B groups only)

Given both the concerns about the potential effects of assessment reactivity and the practical difficulties associated with an extensive assessment procedure (e.g., interfering with the rapid pace of clinical treatment in the ED setting), the baseline assessment is shorter than that often found in addiction treatment trials. The semistructured 30-day TLFB interview [27,28] is the measure from which the primary outcome will be derived. The TLFB, which uses a calendar prompt and a number of other memory aids to facilitate accurate recall, provides estimates of the daily quantity, frequency, and pattern of substance use (cannabis, cocaine, methamphetamine and prescription stimulants, heroin and prescription opioids, inhalants, sedatives, hallucinogens, alcohol, tobacco, and other drugs) during a specified time period. For each day of use, the RA also will record both the number of times the substance was used and the dollar value of the amount of substance consumed. The baseline assessment also will include the NM-ASSIST [29,37] to assess frequency of use and associated problems for a number of substances including tobacco, alcohol, cannabis, cocaine, amphetamine-type stimulants, inhalants, sedatives, hallucinogens, opioids, and other drugs. The NM-ASSIST includes a separate assessment of prescription stimulants and methamphetamine and of prescription opioids and heroin. The total time burden for the baseline assessment is approximately 30 minutes.

Hair testing will be used at baseline (following consent and prior to assessments) and at 3-, 6-, and 12-month follow-ups as an objective secondary measure of cannabis, cocaine, amphetamine, methamphetamine, opioid, and benzodiazepine use. Compared with body fluids such as urine or saliva, hair testing enables the detection of substance use over a significantly longer window of time [38,39] and is increasingly being used as an objective approach complementary to self-reported substance use outcomes in clinical trials. A significant benefit of this testing approach is the comparatively nonintrusive nature of collecting a hair sample from the scalp [40].

Study interventions

Minimal screening only (MSO)

Following randomization, participants in the MSO group will receive an informational pamphlet about drug use and its consequences, addiction, and treatment. This pamphlet will consist of information collected from materials produced by NIDA for the general public. They will receive no further intervention.

Screening and referral (SAR)

Participants in the SAR group will receive the same informational pamphlet as the MSO group. Those with probable dependence (NM-ASSIST score ≥27 for any drug or alcohol) will also be provided with minimal scripted feedback to let them know their score is in the high-risk range, and will receive a recommendation to seek treatment. The RA will provide these participants with an information sheet listing treatment and self-help resources in their community. The latter will be standardized in format but will include site-specific information, i.e., names, addresses, and phone numbers of local addiction treatment agencies in the normal clinical referral network of the participating EDs. Based on our initial site survey, the majority of responding EDs are not currently screening or referring systematically, so the SAR condition represents a level of care significantly higher than treatment-as-usual.

Brief Intervention + booster (BI-B)

Individuals randomized to the BI-B condition will receive the same information and referral as those in SAR. However, in addition to these materials, while in the ED the BI-B group will receive a manual-guided BI based on motivational interviewing (MI) principles, including feedback based on screening information, the Feedback, Responsibility, Advice, Menu of options, Empathy, Self-efficacy (FRAMES) heuristic, and development of a change plan, as delivered in previous trauma center trials [6,41-44] and the Washington State SBIRT projects [6,41,42,45-49]. As Sise [50] indicated, SBIRT services such as these can be integrated effectively into all components of a busy urban ED by adding specially trained interventionists to the ED service staff and/or by training existing staff. The BI will be performed by study RAs (rather than regular ED staff members) who have received centralized training, have successfully completed and met criterion skill levels as assessed by objective ratings of tapes of their interventions with a series of subsequent training cases, and who will receive ongoing fidelity monitoring and clinical supervision over the course of the trial. For a given study participant, BI will be performed by a different RA than the one who conducted assessments of that individual. The in-person BI, which will be approximately 30 minutes long, will be delivered while the participant is still in the ED. Participants in BI-B will receive the informational pamphlet and referral for treatment (if indicated) from the BI interventionist.

In addition, as a means of augmenting the intervention in the ED, participants also will receive up to 2 phone “booster” sessions. These sessions, which are also based on MI principles, will check to see whether they have engaged in treatment, review change plans, and seek a commitment from them [51]. This number of booster sessions was chosen to replicate the structure of the standard Motivational Enhancement Therapy (MET) [52], with the goal of maximizing the magnitude of the therapeutic effect while keeping the intervention short enough to be practical. The goal of the first phone session is to re-engage and reinforce the change plan and support continuing change efforts. The second phone session is a check-in and addresses barriers to change or treatment engagement. Each of these booster calls will be approximately 20 minutes long. The content of these boosters is patterned after sessions in MET [52,53]. The phone booster process is similar to those previously used to deal with problem drinkers in EDs, either as part of stepped-care interventions initiated in the ED and continued following discharge [54] or as stand-alone phone-delivered BIs conducted without an initial intervention in the ED [51], and to procedures used in a primary care settings [55]. The target window for the initial first and second phone booster calls is within 3 days and within 7 days of discharge from the ED, respectively. If initial attempts to complete the booster sessions within these windows are unsuccessful, further attempts to engage participants will be made for up to one month post-discharge from the ED. Booster calls will be made from a centralized, study-wide intervention booster call center by interventionists who have received standardized training and will have ongoing fidelity monitoring and clinical supervision.

Follow-up assessments

Follow-up assessments will be conducted at 3, 6, and 12 months following the ED visit. In order to maintain blinded follow-up, different RAs will be used to complete baseline and follow-up assessments. If in-person follow-up is not possible, follow-up visits may be conducted by phone, but this should be avoided if possible, as hair samples would be missing in such cases. In addition to measures of substance use (e.g., NM-ASSIST, TLFB, hair sample), the Treatment Services Review (TSR) [30] also will be administered at these follow-up points to track substance-abuse treatment entry and utilization and medical-treatment service utilization.

Follow-up will be conducted in person at a central location, preferably close to the participating ED. Additionally, if participants are unable to provide their own transportation to or from follow-up visits, sites may provide transportation including bus or taxi fare if they have local IRB approval to do so. As a means to achieve high follow-up rates in the study population, adequate compensation for time and inconvenience is critical. Compensation of $50 for the screening/baseline visit and $75 for the 3-month visit (primary outcome) and subsequent visits is proposed.

Safety assessment

Adverse events, serious adverse events, and substance use events (e.g., worsening of drug use, need for higher level of care, admission to detox or inpatient drug treatment) will be monitored and reported throughout the study. These events will be subject to ongoing monitoring by the study executive committee and will be presented for review by the independent Data and Safety Monitoring Board.

Blinding

As in almost all psychosocial treatment protocols, study participants will not be blinded. However, because of the two-stage method of revealing the randomized group assignments, RAs conducting baseline assessments will be partially blinded until the assessment has been completed. The RAs conducting follow-up assessments will be different from those conducting the baseline assessments and will be blinded to treatment condition. To further minimize potential bias, the RA who conducts the baseline assessment in the ED will not be the same one that conducts the BI-B intervention with that patient.

Sample size determination

There is no straightforward way to estimate the effect size of the proposed contrasts across the three conditions due to differences across critical aspects of prior SBIRT studies (e.g., primary care setting rather than ED, or alcohol versus drug use as the primary outcome). While previously published studies guided our estimates of treatment effect, the power analysis and sample size calculations were based on the minimum clinically significant nonstandardized effect that we wish to be able to detect between SAR and BI-B, and estimates of error variance based on the data collected in the Washington State implementation of an SBIRT model in ED setting [48,56]. In order to detect an absolute difference of three days between the two groups, which was considered to be a clinically meaningful difference, and with an estimated standard deviation of 11 days, an anticipated 15% attrition rate, and a Type I error α = .05/3 (two-tailed), a total of 1285 subjects (429 in each arm) was determined necessary to have 90% power. Estimating conservatively that 5% of screened patients will meet inclusion criteria, and that 75% of these will consent to participate in the study, it will be necessary to screen 34,267 patients (5711 per site) to achieve the proposed sample of 1285 randomized participants. We propose to screen approximately 160 individuals per week per site to achieve the targeted enrollment rate of six participants per week per site. This will allow recruitment at each site to be accomplished within nine months.

Statistical methods for primary analyses

The primary analyses will consist of three prespecified pair-wise comparisons between the MSO, SAR, and BI-B conditions with respect to the primary outcome variable (number of days of use of the primary drug of abuse in the past 30 days at three-month follow-up based on the TLFB), all of which will use a simple closed testing procedure to control family-wide Type I error at no more than 0.05. Linear mixed-model analyses with a random site effect and fixed treatment effect and intercept will be used. These analyses will take into account possible variability in the overall level of abstinence between the sites, possible site-by-treatment interaction, and the level of a baseline covariate (days of use of the primary drug of abuse, as defined at screening by the DAST-10, during the 30 days preceding the baseline assessment). The primary outcome will be analyzed according to the intent-to-treat principle. Patients who refuse treatment will still be followed for outcome in their assigned group, but no attempt will be made to impute outcomes for these patients.

Potential interpretations of prespecified group comparisons

The proposed design provides the opportunity to make a number of meaningful comparisons. First, the contrast between the SAR and BI-B groups allows a determination of the incremental benefit of the assessment + BI + booster sessions over and above the effects of the assessment alone. This contrast has been conducted in most prior SBIRT studies and is the one for which the group differences and effect size may be attenuated due to potential assessment reactivity. The second comparison, between the SAR and MSO, allows the determination of any incremental therapeutic effects of assessment over and above minimal contact. This comparison provides the quantification of the assessment reactivity effect. The final comparison, between the BI-B and MSO groups, provides a measure of the total effect of the intervention, including any therapeutic effects of assessment and referral. The MSO condition actually offers a better approximation of true treatment-as-usual, as opposed to SAR, since the latter condition represents a greater intensity of intervention than is typically found in EDs.

There are many potential equalities and inequalities that can hold between the three arms of the SMART-ED design. Assuming that abstinence is the outcome (and the bigger the better), the most intuitive results of possible comparisons and appropriate conclusions are depicted in Table 1. For example, if MSO < SAR < BI-B, we may conclude that there is both assessment reactivity and a treatment effect. Other rows can be interpreted similarly. Thus, each significant difference, or lack of difference, between groups would provide meaningful evidence of the presence or absence of assessment and/or treatment effects but with different implications for practice.

Table 1.

Interpretation of potential assessment and treatment effects based on outcomes of contrasts among SMART-ED conditions

Equalities and Inequalities					Conclusions
MSO*		SAR**		SAR/BI-B†	Assessment Reactivity	Treatment Effects
	<		<		Yes	Yes
	<		=		Yes	No
	=		<		No	Yes
	=		=		No	No

*MSO = minimal screen only. **SAR = screening, assessment, and referral to treatment if indicated or requested. †SAR/BI-B = screening, assessment, and referral if indicated or requested plus brief intervention and subsequent telephone follow-up booster sessions.

While assessment has typically been an integral component of BI, in a number of general health-care settings it is likely that BI may be conducted without assessment; there is often limited or no time for conducting an assessment beyond a brief screen in such settings, and assessment is often seen as complex by generalist clinicians. The possibility and effectiveness of conducting BIs without assessment is a condition that is not examined in the current design. Conversely, it is possible to have assessment without treatment. If the SAR condition has a better outcome than the minimal screen only (MSO) condition, and the BI-B condition does not outperform SAR, then a case could be made for simply conducting an assessment and subsequent referral to specialty substance abuse treatment. Given the added costs involved, an intervention should be avoided if it does not significantly add to the outcome over and above that of SAR.

Discussion

While the three-arm design follows the recommendation for taking potential assessment reactivity into account, there are a number of limitations that need to be noted. One obvious limitation is that the effects of the minimal screening, which may not be negligible, cannot be assessed [26]. A number of screening instruments that have been developed for use in SBIRT trials consist of as few as one item [57,58]. Although the MSO condition attempts to minimize the length of the screening procedure and measure its impact, it nevertheless consists of 20 items, including the 4-item Heavy Smoking Index [59], the AUDIT-C [32], the DAST-10 [34], and the 3 questions on primary substance of abuse, days of use, and substance-relatedness of the ED visit. A recent meta-analysis and systematic review of BI [13] showed that merely asking questions about alcohol use alters subsequent drinking behavior even in the absence of a more intensive assessment or intervention.

Second, another approach to reducing bias in SBIRT trials may be to blind subjects to study purpose as a way of limiting the potential effects of social desirability [15]. The SMART-ED design does not address this issue specifically; however, participants and staff are blinded to final condition assignment via the two-step process of revealing the randomization results. The RA who conducts the screening and baseline assessment and who is made aware of the randomized treatment assignment “hands off” the patient to a different RA who delivers the assigned intervention, thus minimizing the likelihood that participants will be treated differently (e.g. subtly advising MSO subjects to change). Further, research staff who conduct follow-up assessments are different from those who conduct baseline assessments or interventions in the ED and are blind to participants’ treatment condition.

Blinding subjects to study purpose is difficult given both the need for informed consent and the nature and content of the screening instrument. It is not possible to determine what impact the consent process has on outcomes given that, in the current trial (as in many similar trials), the IRBs require disclosure of the purpose and procedures of the study. The present study was granted IRB approval to use an abbreviated script to approach potential participants in the ED and ask if they would be willing to answer the TAD screening question; only verbal (not signed) consent was required. Even this procedure, however, alerts individuals to the intended focus on substance use. The collection of hair samples following completion of the screening questionnaire by all potential subjects but prior to obtaining formal written consent may alter individuals’ willingness to volunteer for the study. Similarly, the collection of hair samples following the screen but prior to the baseline assessment for the SAR and BI-B groups might alter responses across these two sets of drug-use data.

While research ethics need to be followed, methods to increase the blinding of potential subjects in BI studies have been approved by IRBs. As an example, patients in a Level-I trauma center who screened positive for hazardous or harmful alcohol use were asked to give consent to assess their long-term outcome from trauma [44]. Assessments included 6- and 12-month follow-up interviews and review of their medical records and other databases. Consent was not obtained for randomization into an SBIRT intervention or control group, nor were patients told they were taking part in a study to reduce alcohol consumption. After consent, patients were randomized to an intervention or control group. Such a procedure avoided the bias that otherwise might have occurred because more seriously alcohol-impaired or dependent patients may have refused to participate, and patients who did participate would have been sensitized to the fact that their drinking would be monitored [15]. Finally, it is unclear to what extent the presence or absence of assessment reactivity in SMART-ED can be generalized to other trials with other assessments.

Although there are questions about whether assessment reactivity effects occur in BI studies in general [11], and, more specifically, in those delivered in EDs [23-25], it is important to use designs that take this possibility into account. As noted, a number of methodological factors may contribute to biases that can lead to an underestimation of the impact of BI, and these factors need to be minimized as much as possible. The design of the SMART-ED study allows determination of the incremental contribution of the assessment process to the change in subsequent substance use, although it is not able to address or eliminate other sources of potential bias. The ability to disaggregate the effects attributable to assessment versus the BI is methodologically important and allows a clearer perspective of the mechanism of action in the observed change in alcohol or drug use.

From both a clinical and public health perspective, however, this methodological disaggregation of relative effects may be less important. The SBIRT approach has typically followed a stepped-care approach; some form of screening, often asking as few as one question [58], is used for case-finding purposes. In a number of general health-care settings, a positive response to such a screening item might be sufficient to lead directly to BI and/or advice to change behavior without conducting a more extended assessment. In other settings, an assessment (even brief) is conducted with patients who score positive for problem substance use to provide additional information about the nature and severity of use beyond the initial screen, after which a brief motivationally focused intervention is conducted, usually incorporating feedback based on information derived from the assessment. Assessment is considered to be an integral component of the intervention “bundle” in such an approach. Thus, if integrated into the intervention, and if found to contribute independently to the change process, assessments could be designed to maximize the therapeutic benefit they provide and add to the overall intervention effectiveness [9].

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

All authors participated in the conceptualization and design of the protocol. DMD developed the initial draft of the manuscript; all authors read, provided editorial input, and approved the final manuscript.

Acknowledgments

This work was supported by grants from the National Institute on Drug Abuse as part of the NIDA National Drug Abuse Treatment Clinical Trials Network (U10DA013714, Donovan, PI; U10DA015833, Bogenschutz, PI; 5U10DA020024, Trivedi, PI). The opinions expressed in this paper are solely those of the authors.