A Randomized Clinical Trial of the Health Evaluation and Referral Assistant (HERA): Research Methods

Abstract

The Health Evaluation and Referral Assistant (HERA) is a web-based program designed to facilitate screening, brief intervention, and referral to treatment (SBIRT) for tobacco, alcohol, and drug abuse. After the patient completes a computerized substance abuse assessment, the HERA produces a summary report with evidence-based recommended clinical actions for the healthcare provider (the Healthcare Provider Report) and a report for the patient (the Patient Feedback Report) that provides education regarding the consequences of use, personally tailored motivational messages, and a tailored substance abuse treatment referral list. For those who provide authorization, the HERA faxes the individual’s contact information to a substance abuse treatment provider matched to the individual’s substance use severity and personal characteristics, like insurance and location of residence (dynamic referral). This paper summarizes the methods used for a randomized controlled trial to evaluate the HERA’s efficacy in leading to increased treatment initiation and reduced substance use. The study was performed in four emergency departments. Individual patients were randomized into one of two conditions: the HERA or assessment only. A total of 4,269 patients were screened and 1,006 participants enrolled. The sample was comprised of 427 tobacco users, 212 risky alcohol users, and 367 illicit drug users. Fourty-two percent used more than one substance class. The enrolled sample was similar to the eligible patient population. The study should enhance understanding of whether computer-facilitated SBIRT can impact process of care variables, such as promoting substance abuse treatment initiation, as well as its effect on subsequent substance abuse and related outcomes.

1. Introduction

The burden of tobacco-, alcohol-, and drug-related injuries and diseases is staggering, accounting for more than 500,000 deaths and 510 billion dollars in lost productivity and medical costs every year in the United States.[1, 2] Because screening, brief intervention, and referral to treatment (SBIRT) has proven effective for reducing tobacco use and SBI has proven effective for alcohol abuse, the United States Preventive Health Task Force (USPSTF) and the Substance Abuse and Mental Health Services Administration (SAMHSA) have recommended universal SBIRT for these substances in general medical settings, including emergency departments (EDs) and primary care.[3, 4] Additionally, there is increasing evidence to support SBIRT for illicit drug abuse and SAMHSA state grants to promote SBIRT require assessment and intervention for both alcohol misuse and illicit drug use.[5, 6] The Centers for Medicare and Medicaid Services (CMS) and the American Medical Association have authorized billing codes that reimburse SBIRT for tobacco, alcohol, and illicit drug abuse.[7]

Despite their efficacy, cost-effectiveness, and potential for reimbursement by CMS, the majority of SBIRT programs collapse once grant funding ends. Two themes have emerged that account for the poor sustainability of traditional SBIRT programs:

1. Most physicians and nurses will not perform SBIRT themselves.[8–11] Physicians and nurses almost universally recognize the need for SBIRT but generally feel ill-prepared and too overwhelmed by clinical demands to comply with SBIRT recommendations. A fear of opening “Pandora’s Box” prevents screening, especially in acute care settings like the ED. For these reasons, physicians and nurses strongly prefer the dedicated interventionist model, a team-based approach using on-site counselors or clinicians to provide the interventions. In this model, SBIRT is completed in parallel with medical care and often requires minimal involvement of the treating physicians and nurses. As long as the financial support for a team model is available, providers and patients are satisfied.

2. Dedicated interventionist models, however, have not proven sustainable.[10, 12–14] Despite SBIRT’s ability to reduce healthcare costs[12], the costs and complexities associated with hiring, training, supporting, and scheduling on-site interventionists have been prohibitive in most settings. Consequently, the evidence-based implementation model is scarcely implemented and treatment as usual prevails, represented by idiosyncratic provider screening and interventions.

Technological advances such as computerized assessments, personalized feedback reports, faxed referrals, and electronic health records hold tremendous potential for facilitating the implementation of SBIRT in healthcare settings.[15–23] Given such potential, Polaris Health Directions and the University of Massachusetts Medical School created the Health Evaluation and Referral Assistant (HERA), a web-based program designed to facilitate SBIRT. The prototype demonstrated strong acceptability among providers and patients and evaluation data provided initial support for its feasibility in ED and inpatient medical settings.[24] This paper describes the methods of a randomized controlled trial (RCT) to evaluate the efficacy of the HERA for improving initiation of specialized treatment among ED patients abusing tobacco, alcohol, or illicit drugs and to examine the HERA’s effect on use of and motivation to change these substances. An evaluation of the representativeness of the enrolled sample compared to non-enrolled patients is also presented.

2. Materials and Methods

2.1. HERA Overview

The HERA is comprised of three integrated modules (Figure One): (1) a web-based assessment of tobacco, alcohol, and illicit drug use, as well as other pertinent psychosocial variables, (2) a report generator, and (3) a referral generator. The HERA was funded by a Small Business Technology Transfer (STTR) grant from the National Institute on Drug Abuse (R42DA21455). Phase 1 created and pilot tested the prototype. Phase 2 made further modifications based on Phase 1 results and studied the program’s efficacy, which is the subject of this paper. Because the Phase 1 program is described in depth elsewhere, we will provide a brief overview and describe the changes made in the system during the early stages of Phase 2.[24] Innovations include: integration of patient-facing technology during the medical visit, poly-substance assessment and intervention, highly personalized referral matching capability, and dynamic referral capability, or the ability to automatically send an electronic referral to a “best match” substance treatment provider in the community.

Figure One.

The HERA Modules and Their Hypothesized Mechanisms of Action

2.1.1. Assessment module

The assessment module provides for the self-administered assessment of tobacco, alcohol, and illicit drug use and abuse. This represents a significant innovation, since many computerized systems, as well as most traditional SBIRT studies, only assess one substance category, such as studies of the Tobacco Expert System[25, 26] and SBIRT studies on alcohol.[12, 27, 28] Focusing only on one substance category would limit adoption in clinical practice, where all three substances are prevalent and often co-morbid.

In the original version of the HERA, the foundation of the assessment consisted of an abbreviated version of the Addiction Severity Index (ASI).[29] However, during Phase 2, the assessment was replaced with the Heavy Smoking Index to assess tobacco use, the Alcohol Use Disorders Identification Test (AUDIT) to assess alcohol use and misuse, and the Drug Abuse Severity Test (DAST-10) to assess illicit drug use.[30–32] These scales were judged to be better than the ASI because they are: (1) briefer, (2) have been used extensively in clinical SBIRT research, and (3) are more accepted by credentialing agencies, such as the American College of Surgeons, and funding agencies, such as CMS.

In addition to these scales, the HERA also assesses history of injection drug use, readiness to change, readiness to enter treatment, treatment history, and withdrawal symptoms. For patients reporting any readiness to change, the HERA assesses interest in a faxed referral to a “best match” substance abuse treatment provider (i.e., a dynamic referral) which is further described in Section 2.1.3.

2.1.2. Report generator

The report generator produces two reports using the data collected through the assessment: (1) the Healthcare Provider Report, and (2) Patient Feedback Report. They are described in depth elsewhere.[24] The one-page Healthcare Provider Report, designed to be “charted” on the patient’s ED medical record, summarizes substances used, whether the individual has ever injected drugs, abuse/dependence risk level per the AUDIT and DAST-10, readiness to change, withdrawal symptoms, and referral information. In addition, it includes reference values for the AUDIT and DAST-10 linked to evidence-based recommended clinical action. The ordering and content of the report were optimized based on what ED providers stated was most important for clinical decision making (i.e., the reports are clinically actionable).

The Patient Feedback Report was rearranged for Phase 2 to make it more user-friendly, appealing, and useful based on consultant reviews and feedback in Phase 1. It consists of three to eight pages divided into three sections: (1) the Face Sheet, which includes an overview and tailored referral list for each substance class endorsed, (2) the Patient Assessment Summary, which includes motivationally tailored messages based on the assessment, as well as education related to consequences of use for each substance endorsed, and (3) the Motivation Toolkit, including an overview of factors to consider when deciding to change and exercises for decisional balance, stimulus control, and goal setting. The reports include testimonials of successful changers matched to the participant’s sex, age, and substance used to help promote motivation.

2.1.3. Referral generator

The referral generator draws upon an extensive, nationwide provider library of accredited substance abuse treatment facilities and independent practitioners maintained by Polaris. The core of the library consists of the accredited treatment facilities maintained by SAMHSA, and it is enriched with other providers gleaned from the specific implementation sites. The library is used to generate personally tailored referral lists, as well as to send dynamic referrals. For the purposes of the current study, we established dynamic referral agreements with approximately 50 facilities in the communities wherein the program was being implemented (Worcester MA, Camden NJ). The participating dynamic referral providers agreed to contact the patient within five business days of receiving the fax to complete an initial telephone screening, discuss treatment options, and, if appropriate, schedule an intake assessment. This proactive, faxed referral system is very similar to the faxed referral systems currently implemented in most states with tobacco Quitlines.[33] If patients were not appropriate for the receiving facility after an initial assessment, the facility assisted the patient in identifying other treatment options. The referral was faxed automatically by the HERA and included a statement of release of information, a very brief summary of the assessment, and the patient’s contact information to allow the receiving provider to contact the patient. The referral indicated whether it was acceptable to leave a message with another individual or on an answering machine. The library was routinely updated as information about new facilities or closing of old ones became known.

An individual was offered a dynamic referral if he or she (1) indicated use of tobacco or illicit drugs, or risky use of alcohol; (2) reported not currently being in treatment; and (3) reported at least some interest in changing his or her use. Poly-substance abuse patients who wanted to change more than one substance were given the option for a dynamic referral first for illicit drugs, then for alcohol, then for tobacco. Subjects who were in need of acute detoxification were assessed and managed by the treating clinicians using their usual and customary procedures. The system collected information from the patient on matching parameters, including gender, addiction severity, previous treatment, insurance, and ZIP code. This information was matched to a provider based on a profile generated for each. The referral system underwent extensive testing to ensure that our matching algorithms successfully paired individual patients with a well-matched treatment facility in terms of clinical presentation, substances used, level of care provided, and insurance coverage. Of 240 mock patient assessments varied across the relevant patient matching characteristics, 100% were rated as “appropriate” by the receiving treatment facilities. Although there is a Spanish version of the HERA, and substance treatment providers are coded for whether they can provide services to Spanish speakers, we did not test this during the trial, because we enrolled only English speakers.

2.2. Methods

2.2.1. Procedure

We enrolled patients from four EDs (described below). The HERA assessment and the evaluation interviews were all completed in English and were self-administered during the ED visit using a tablet personal computer (PC) connected wirelessly to the Internet. A trained research assistant telephoned all participants four and 12 weeks after the healthcare visit to assess treatment initiation and to re-assess substance use. For all participants who selected a dynamic referral, research staff contacted the substance abuse treatment provider to which the patient was referred to verify the participant’s self-reported treatment initiation, as well as to identify whether the referral was a good match for the facility. We designed the study to mimic the “real world” as closely as possible. However, because it was an experimental procedure and we required standardization to maximize the internal validity of the study, the RA guided the computer logistics by making sure the patient had the tablet PC and was comfortable using it. In addition, the RA made sure the reports were printed and received by their respective targets, including the treating physician and the patient. The study was approved by the Institutional Review Boards for all performance sites. All participants agreed verbally to the initial screening, and all participants who agreed to participate in the clinical trial component signed a printed consent form. Those who accepted a dynamic referral also signed an authorization to obtain medical information, which allowed us to contact the provider who received the referral.

2.2.2. Setting and population

Participants were recruited from four EDs. Pertinent characteristics of the EDs are summarized in Table One. We chose the ED setting for two primary reasons. First, substance use and its medical consequences are prevalent among ED patients, making it not only an important setting for such a system as the HERA to function but also making enrollment into a large RCT efficient. Second, the ED represents a demanding clinical environment, where patient throughput is paramount. A computerized SBIRT system that can be integrated successfully into the ED setting will likely be feasible in most other medical settings, such as inpatient floors and primary care clinics.

Table One.

Site Characteristics

Type	Annual Volume	Location	Race/Ethnicity
Academic, Urban	90,733	Worcester MA	W 82%, H 11%, B 4%
Community, Urban	47,364	Worcester MA	W 74%, H 14%, B 9%
Community, Suburban	23,217	Marlboro MA	W 80%, H 15%, B 3%, U 2%
Community, Urban	59,482	Camden NJ	W 35%, H 20%, B 45%

Legend: W = White, Non-Hispanic, H = Hispanic, B = Black, U = Unknown, MA = Massachusetts, NJ = New Jersey

2.2.3. Participant selection and randomization

Enrollment occurred 7 days a week, during the hours of 9:00 AM to 10:00 PM. Adult patients were approached at the bedside by the research assistant (RA) after the patient had been evaluated by a physician and was deemed medically stable. Consecutive patients presenting during research shifts were logged and approached. To help avoid selection bias, a standard enrollment script was used which reassured patients that prior computer experience was not needed to participate. Exclusion criteria included severe illness or distress (e.g., intubation, intractable vomiting or pain), cognitive insufficiency (e.g., dementia, psychosis, altered consciousness), in state custody or restraints, begin held involuntarily, and insurmountable language barriers. Verbal consent was obtained to complete the rapid screener, and, if the individual screened positive on at least one of the substances and was eligible and willing to participate in the clinical trial, written consent was obtained.

Because poly-substance use was common, we created a method to classify each participant in order to guide subsample analyses. Tobacco users were defined as those who smoked cigarettes in the past 30 days but did not have risky alcohol use or illicit drug use in the past 12 months (i.e., “clean” smokers with no other substance abuse). Risky alcohol users were defined as having used alcohol above the AUDIT quantity/frequency guidelines, with or without tobacco use but with no illicit drug use in the past 12 months (i.e., they could have been smokers but not illicit drug users). Illicit drug users were defined as using illicit drugs in the past 12 months, with or without alcohol use and with or without tobacco use (i.e., they could have any combination of polysubstance use in addition to illicit drug use).

An individual was randomized to either the intervention or control condition by a random number generator from the Java programming language standard library within the HERA. The standard library is the set of default programming language features that are installed with every copy of the programming language. In our code we set the random number generator to return a random integer, either 0 or 1, and used that to assign a participant to the experimental or control group.

2.2.4. Study conditions

Both the intervention and control conditions were treated the same in all respects related to the study procedures, such as recruitment and completion of the baseline assessment on tablet PCs. Those in the intervention condition were (1) offered a dynamic referral (if the individual met the criteria for such an offer; see Section 2.1.3.), (2) their treating physician was given the Healthcare Provider Report by the RA, and (3) the patient was given the Patient Feedback Report by the RA. We trained the physicians on how to read the Healthcare Provider Reports and encouraged them to discuss results with their patients, including encouraging follow-up with any referrals the patient may have received. However, we did not train the clinicians in any method of counseling, such as motivational interviewing[34], nor did we otherwise require any specific clinical action. The RAs ensured that the participants received the Patient Feedback Report and provided a brief explanation that the report summarized their assessment and provided personalized information, but they did not use the report to otherwise counsel the individual. We chose this minimalist approach because it mirrors the conditions under which the HERA will be implemented. Namely, it will likely be used without extensive training or mandated physician counseling and as a tool adjunctive to usual and customary care.

Those participants assigned to the control condition completed an identical computerized assessment as the intervention group but were not offered the opportunity to choose a dynamic referral or receive the Patient Feedback Report. The Healthcare Provider Report was not printed. However, control group patients did receive a standardized, pre-printed list of local community substance abuse treatment providers.

2.2.5. Blinding

Patients were partially blinded to group assignment. In order to minimize bias which might have influenced patient’s reactions and assessment ratings due to knowing the group to which they were assigned, the consent process emphasized the similarities between the groups. For example, we referred to the two conditions with equipoise by describing them as Group 1 and Group 2, rather than referring to intervention and control conditions. We explained that all participants would complete an assessment on the computer and, at a minimum, would receive the usual and customary care provided at the site, as well as community referral materials. We explained that they had the potential to have the results of their assessment shared with their physicians. We stated that the difference between the two conditions was the method for generating the referral materials -- computer versus pre-printed.

The RAs performing the outcome assessments were partially blinded. Because the HERA was heavily focused on the referral process, and different patients received different kinds of referrals (pre-printed list vs. tailored referral list vs. dynamic referral), the follow-up questions were keyed to the referral type received by the individual to avoid confusion. For example, those individuals who received a dynamic referral were asked whether they received a call from the treatment provider to which they were referred, while those that had received only a printed list were asked if they had called any providers on the list. Consequently, although the RAs were not informed about the group assignment, and did not have access to the baseline assessment data, the nature of the tailored follow-up assessment revealed information about assignment in some cases, particularly if the participant had accepted a dynamic referral. Because of this, the RAs could intuit group membership for those that accepted a dynamic referral based on the way the questions were worded.

2.3. Measures

2.3.1. HERA

The HERA assessment, obtained at baseline during the index ED visit, was described in Section 2.1.1. The HERA’s automatic time stamps were used to calculate assessment completion times in minutes.

2.3.2. Other baseline data

In addition to the HERA substance use assessment, we assessed (1) demographic and socio-economic variables, including insurance status and carrier; (2) general health status using an SF-12 item[35] (In general, I would say my health is: (1) Poor, (2) Fair, (3) Good, (4) Very good, (5) Excellent)); (3) psychiatric diagnoses (Please indicate all the mental health diagnoses you have EVER been given by a doctor or therapist. (1) None, (2) Anxiety or Panic attacks or Post-traumatic stress disorder (PTSD), (3) Depression or Bipolar disorder, (4) Schizophrenia or Schizoaffective disorder, (5) Anorexia or Bulimia, (6) Attention Deficit Disorder (ADD, ADHD), (7) Other); and the Patient Health Questionnaire – 2.[36]

2.3.3. Post-visit interview

Immediately after discharge, the RA who enrolled the participant completed a very short interview with the participant to ascertain whether his or her treating clinicians assessed tobacco, alcohol, or drug use and/or provided counseling, educational materials, or referrals. This allowed us to determine if the provision of the Healthcare Provider Feedback reports prompted changes in the clinician’s behaviors as compared to the control condition, which did not receive the reports. If we could not interview the patient in person immediately upon discharge, the RA called the individual within 48 hours of the ED visit to minimize decay in recollection and confusion between actions taken by clinical staff treating the participant versus the research staff as part of the research protocol.

2.3.4. Follow-up assessment

A single RA unaffiliated with any of the performance sites contacted all participants by phone four and 12 weeks after the index visit. All participants were asked if they had attended an initial assessment session with a substance treatment provider, along with the number of treatment sessions attended, if any. Any participant that reported a pending appointment with a treatment provider was re-assessed two weeks later. Also, participation in self-help groups, like Alcoholics Anonymous, was assessed. The RA assessed self-reported substance use, including abstinence since the ED visit, amount and problem use, efforts to decrease use, and readiness to change. The same measures used in the baseline HERA were re-administered to assess these constructs.

Treatment initiation was defined as being admitted for inpatient drug or alcohol treatment, completing an initial outpatient appointment, or completing an initial intake assessment by phone, for example, with the tobacco Quitline.[37, 38] In addition, we plan to examine the influence on self-help group participation as a secondary outcome. Finally, RAs from each site contacted the referral providers for each dynamic referral and assessed whether: (1) the provider had received the faxed referral, (2) the provider had contacted the participant, (3) the participant had scheduled an initial assessment, and (4) the number of sessions the participant attended. These providers also rated the goodness of fit of the referral.

2.4. Data Analyses

2.4.1. Sample characteristics

We have calculated descriptive statistics for our primary baseline variables, including: substance use indicators, such as HSI, AUDIT, DAST-10, rates of injection use, and rates of polysubstance use; readiness to change, including readiness to change multiple substances; and acceptance rates for dynamic referrals. To confirm sample representativeness, we compared the screened sample to the non-screened sample, and the enrolled sample to those eligible but not enrolled. To confirm randomization success, we will compare intervention and control conditions on baseline characteristics. Should significant differences be observed, these characteristics will be included in the predictive models as a covariate.

2.4.2. Planned RCT analyses overview

The RCT is still being conducted, and future papers will report the results. The data analyses planned for the RCT are described briefly below. All analyses will be conducted with each substance class separately – tobacco, alcohol, and illicit drugs.

Given the longitudinal nature of the study with multiple measurements of outcomes, we will employ extensions of the generalized linear model depending on the distributional characteristics of the outcomes. For categorical outcomes, including detection of substance use, provider counseling, referral to treatment, treatment engagement, and categorical indicators of abstinence, we will use generalized estimating equations (GEE)[39] with Group (Intervention/Control) as the primary predictor. In the examination of the HERA, GEE will be used to account for the inherent correlation between binary measurements nested within individuals over time (baseline, 4-, and 12-weeks). For AUDIT, DAST-10, HSI scores and other continuous variables (e.g., abstinent days), we will conduct linear mixed models with Group as the primary predictor of growth/decline over time. Each of these models utilizes maximum likelihood estimation capable of accounting for non-normality in outcomes. Consistent with intention to treat principles, participants that are lost to follow up will be considered non-treatment engagers, with substance use fixed at baseline levels. We will compare participants retained against those lost to follow-up on demographic and substance use characteristics in order to provide a more complete description of the population to which findings are being generalized.

The primary hypothesis states that the HERA will lead to more substance users actually initiating specialized treatment after the ED visit when compared to the control condition. The secondary hypothesis states that the HERA will lead to greater 7-day point prevalence abstinence when compared to the control condition. We will also explore whether the HERA prompted additional counseling or action from the participant’s ED clinician and the effect of the HERA on other relevant variables, including health status and ED visits.

2.4.2.2. Power analysis

Powering this study is complicated because we enrolled participants from multiple substance use categories. We decided to calculate a sample size for each of the three drug classes, and set our target enrollment for each substance separately. We chose to focus on what we perceived to be the most important of the primary outcomes variables, treatment initiation.

Unfortunately, there are no precise estimates of the base rate of treatment initiation after an index ED visit for illicit drug users under treatment-as-usual conditions. Based on our clinical experience, we estimate that <3% of all patients who report using illicit drugs will engage in treatment within 3-months of their visit. In Phase 1, we found that 3 of 19 illicit drug users, or 16%, initiated treatment within 4 weeks of the ED visit after completing the HERA. As such, we derive an effect size (expected difference in 3-month engagement rates) of 13%. With regard to tobacco users, our best estimate of base rate of treatment engagement within 3-months of an index ED visit under treatment-as-usual conditions comes from Boudreaux, et al., 2011.[40] In a cohort of 591 smokers who were given a passive referral to a smoking cessation clinic, only one smoker actually called the clinic to set up an appointment (<1%). We expect the effect size of the HERA will be somewhat smaller for tobacco and alcohol than illicit drugs, since these are substances that most individuals do not believe they need “treatment” to overcome. Therefore, we set an expected group difference in treatment initiation on 10%. As this is the smallest anticipated effect, we estimated the minimum number of participants needed in each substance category to observe power of 0.80 to find an effect of this size. The statistical power, therefore, for observing the expected effects in the illicit drugs will be higher.

When estimating statistical power for our proposed models, we used an alpha level of 0.05, a relatively high (i.e., conservative) auto-correlation between measurement of 0.50, and a first-order autocorrelation within-subject covariance structure. In order to achieve power of 0.80 under intention-to-treat (ITT) assumptions with three time points, we require enrollment of at least 96 substance users per group (i.e., total n _tobacco = 192, total n _{illicit drugs} = 192, total n _alcohol = 192). In non-ITT analyses, full information maximum likelihood estimation (FIML) will be used to account for missing data over time. FIML approaches make use of all partial data available in the model and provide appropriate estimates when data is missing completely at random or when the relevant causes of missingness (e.g., hospitalization, comorbid psychopathology) are included in the model (e.g., missing at random).

3. Results

Here, we only report on the sample characteristics. Future publications will report on the primary and secondary outcome analyses of the RCT data.

3.1. Sample Characteristics

3.1.1. Screened vs. not screened

Figure Two depicts the enrollment flow. Of the 8,798 patients who entered the ED during enrollment shifts, 4,269 (49%) were successfully screened for substance use; the rest had some clinical factor that prevented approach, were missed (e.g., discharged before the RA could approach), or refused the initial screening interview. These factors made the screened sample less severely ill than the non-screened sample. In addition, the patients that were screened were slightly more likely to be female (2,397/4,269; 56%) than the patients who were not screened (666/1,311; 51%), χ² (1) = 11.59, p < 0.01. Screened patients were also slightly less likely to be White (2,767/4,264; 65%) than not-screened patients (909/1,311; 69%) χ² (1) = 8.82, p < 0.01. Fewer Hispanic individuals were also screened (779/4,264; 18%) than not screened (400/1,311; 31%), χ² (2) = 91.16, p < 0.001. Screened participants were significantly younger (M_age = 48.06, S.D. = 18.41) than patients that were not screened (M_age = 44.12, S.D. = 17.39), t (5,578) = 7.10, p < 0.001.

Figure Two.

Describing the Sample Used to Test the HERA System

3.1.2. Enrolled vs non-enrolled

Of the 1,551 that screened positive for at least one substance, 218 were ineligible because of exclusion criterion, 327 were eligible but declined or were missed, and 1,006 (76% of eligible) were enrolled (Figure Three). The sample was comprised of 427 tobacco users, 212 risky alcohol users, and 367 illicit drug users. Fourty-two percent used more than one substance class (e.g., tobacco and alcohol misuse, alcohol misuse and illicit drug use). A slightly higher proportion of men declined participation (183/686; 27%) than women (137/640; 21%), χ² (1) = 5.02, p < 0.05. A greater proportion of White participants also agreed to participate (710/914; 78%) than non-White participants (288/407; 71%), χ² (1) = 7.30, p < 0.01. Eligible individuals who chose to participate were significantly younger (M _age = 36.82, S.D. = 12.58) than those who declined to participate (M _age = 40.31, S.D. = 13.40), t (1,324) = 4.26, p < 0.001. There were no differences on ethnicity (p > 0.10). Descriptive characteristics of the sample are summarized in Table Two. Comparisons were then made between intervention and control participants on these characteristics to test the effectiveness of the randomization procedure. Results indicated there were no significant differences on these baseline variables across conditions (see Table Three).

Figure Three.

Assignment of HERA Participants

Table Two.

Sample Characteristics

	Frequency (%)	Mean (S.D.)
Sex
Men	503 (50%)
Women	503 (50%)
Race
White	710 (71%)
Black/African American	252 (25%)
Other/Undocumented	44 (4%)
Ethnicity
Hispanic or Latino	155 (15%)
Location
UMass University	503 (50%)
UMass Memorial	19 (2%)
Cooper Hospital	11 (1%)
Marlborough Hospital	473 (47%)
Substance Use
Tobacco Use Only	427 (42%)
Baseline HSI Score		2.45 (1.39)
Alcohol (Overall)	212 (22%)
Alcohol Misuse Only	97 (10%)
Alcohol Misuse and Cigarettes	109 (11%)
Alcohol Misuse and Other Tobacco	6 (1%)
Baseline AUDIT Summary Score		10.96 (6.59)
Drugs (Overall)	367 (36%)
Drug Use Only	65 (7%)
Drug Use and Cigarettes	163 (16%)
Drug Use and Other Tobacco	3 (< 1%)
Drug Use and Alcohol Misuse	23 (2%)
Drug Use, Cigarettes, and Alcohol Misuse	110 (11%)
Drug Use, Other Tobacco, and Alcohol	3 (< 1%)
Misuse		3.56 (2.61)
Baseline DAST-10 Summary Score

Table Three.

Baseline Comparisons

	Frequency (%)		Mean (S.D.)
	Intervention	Control	Intervention	Control
Sex - χ2 (1) = 0.06, p > 0.05
Men	251 (50%)	250 (49%)
Women	249 (50%)	256 (51%)
Race - χ2 (2) = 2.20, p > 0.05
White	355 (71%)	355 (70%)
Black/African American	119 (24%)	133 (26%)
Ethnicity- χ2 (2) = 2.87, p > 0.05
Hispanic or Latino	82 (17%)	73 (15%)
Location - χ2 (3) = 7.37, p > 0.05
UMass University	259 (52%)	247 (49%)
UMass Memorial	3 (1%)	13 (3%)
Cooper Hospital	234 (47%)	239 (47%)
Marlborough Hospital	4 (1%)	7 (1%)
Substance Use a
Tobacco Use Only – t (425) = −0.34, p > 0.05
Baseline HSI Score			2.43 (1.40)	2.47 (1.39)
Alcohol (Overall) – t (210) = −1.18, p > 0.05
Baseline AUDIT Summary Score			10.38 (5.37)	11.45 (7.45)
Drugs (Overall) – t (365) = −1.19, p > 0.05
Baseline DAST-10 Summary Score			3.40 (2.52)	3.73 (2.70)

^aComparisons made within participants enrolled in each substance class (e.g., tobacco intervention participants vs. tobacco control participants).

4. Discussion

The RCT results will provide a robust test of the utility of the HERA in improving process outcomes, like detection of substance abuse, improving provider counseling and referral, and initiating specialized treatment. It will also help to determine whether these process improvements lead to reduced substance use. However, testing such technological interventions using a clinical trial can raise important methodological challenges. We review some of the key challenges facing this study and our attempts to solve or mitigate them.

First, recruiting participants to participate in a trial of a technology intervention can introduce a bias towards recruiting younger, more computer literate individuals. We attempted to address this in two ways. First, the HERA assessment was designed to require no computer literacy. They simply had to be able to read at the 8^th grade level and respond using a numeric keypad or the tablet PC’s stylus, whichever was most comfortable to the individual. Second, our recruitment procedures reassured the potential participant that computer experience was not needed. While we cannot state whether our sample was more or less computer literate than the population from which we sampled because we did not explicitly test computer literacy, it is reasonable to assume at least some bias exists. Consequently, the results will be more generalizable to a more computer literate population. This weakness is partially mitigated by the fact that interventions like the HERA are more likely to be implemented clinically in the same fashion, with less computer literate individuals refusing to complete the assessment, making our obtained sample more likely to reflect true clinical samples to which the program is likely to be applied.

Second, we wanted to study the intervention under conditions that would mimic the “real world” as closely as possible to avoid the problem faced by many clinical trials wherein efforts to maximize internal validity lead to compromised external validity. If the HERA were to be integrated into clinical care, it is likely that healthcare providers would be trained on the nature of the program and how to the read the reports but would not receive extensive training on SBIRT or be required to perform any specific kind of counseling. Rather, they would be encouraged to manage conditions as per their best clinical judgment, based on the clinical data provided in the reports. The site would also have to establish a process to ensure that patients received their Patient Feedback Reports after completing the assessment. It is unlikely that the emergency physician would do this; it would probably be delegated a paraprofessional or clerical staff. In the usual flow of patient care in the ED, the nurse often gives the discharge instructions to the patient and is responsible for patient education. Integrating the reports into this process may augment the likelihood of successful implementation.

We attempted to mimic these conditions but it was impossible to rely upon the clinical staff to manage all parts of the process. Clinical staff are reluctant to adopt new procedures and modify clinical protocols to implement experimental interventions if they have not yet demonstrated efficacy. Consequently, the RA guided the computer logistics and “paper process” by making sure the patient was comfortable using the computer and the provider and patient both received their respective reports. This departed from what would likely happen in the clinical setting since this would have to be done by an employee of the site but we judged that this minimal level of consistency in protocol implementation was deemed indispensible for the success of the RCT. We felt the protocol balanced internal versus external validity.

Third, because we wanted to mimic the real world, we enrolled a heterogeneous sample of ED patients regardless of class of substance used or interest in quitting. The benefit of this approach is that our results are more likely to be applicable to the real world where EDs treat all comers rather than narrowly defining only one substance class with which to intervene. This complicated the analyses, however, and transformed our study into three separate RCTs, since we wanted to be able to examine the impact of the HERA on each substance class while at the same time implementing it in the way it would be implemented clinically (i.e., assessing all substance classes). This led to tripling the sample size. This was compounded by the fact that we predicted a relatively modest effect size; individuals interact with the system one time, and one can only expect modest impact from such an intervention. A modest impact from an easily scalable intervention that can reach millions, however, retains significant public health value.

4.1. Limitations

The partial blinding of the research staff evaluating outcomes could potentially compromise the results. The likely direction of the bias would be to inflate the rate of treatment initiation and reduce the severity of substance use in the intervention condition. To help mitigate the effect on treatment initiation, we confirmed with dynamic referral providers whether the subject actually began treatment. We could have obtained biological assessments of use to help mitigate the influence on substance use, but the complexity and costs associated with multiple biovalidation approaches with a sample that included tobacco, alcohol, and drug users proved prohibitive.

A longer follow-up window would have provided more information on the proportion of participants that entered treatment and on the proportion of those who reported abstinence that eventually relapsed. We reasoned that a brief intervention is more likely to have a stronger impact on short term outcomes. Future studies should follow participants over a longer period to assess the short- and long-term impact on treatment involvement and abstinence.

Finally, our analyses showed that the sample was younger, more likely to be white, and more likely to be female than those who were eligible but not enrolled. While these differences were statistically different, their magnitude was not very large. For example, men were less likely to participate but the absolute difference between the proportion of men and women who declined was only 6% (27% of men and 21% of women declined).

4.2. Conclusions

The HERA is an innovative program designed to assist in the identification, counseling, and connection of risky substance using individuals to appropriate treatment resources. Studying technology interventions for enabling SBIRT in clinical settings poses many challenges to internal and external validity that must be balanced. The HERA has the potential, if implemented as part of routine care, to reach a huge population in a highly efficient manner that reduces provider burden.