Evaluating the impact of Getting To Outcomes-Underage Drinking on prevention capacity and alcohol merchant attitudes and selling behaviors

Abstract

Underage drinking is a significant problem facing US communities. Several environmental alcohol prevention (EAP) strategies (laws, regulations, responsible beverage service training and practices) successfully address underage drinking. Communities, however, face challenges carrying out these EAP strategies effectively. This small-scale, three-year, randomized controlled trial assessed whether providing prevention coalitions with Getting To Outcomes-Underage Drinking (GTO-UD), a tool kit and implementation support intervention, helped improve implementation of two common EAP strategies, responsible beverage service training (RBS) and Compliance Checks. Three coalitions in South Carolina and their RBS and Compliance Check programs received the 16 month GTO-UD intervention, including the GTO-UD manual, training, and onsite technical assistance, while another three in South Carolina maintained routine operations. The measures, collected at baseline and after the intervention, were a structured interview assessing how well coalitions carried out their work and a survey of merchant attitudes and practices in the six counties served by the participating coalitions. Over time, the quality of some RBS and Compliance Check activities improved more in GTO-UD coalitions than in the control sites. No changes in merchant practices or attitudes significantly differed between the GTO-UD and control groups, although merchants in the GTO-UD counties did significantly improve on refusing sales to minors while control merchants did not.

INTRODUCTION

Underage drinking is a significant problem, costing the US about $53 billion annually (Institute of Medicine, 2004). About two thirds of US high school seniors report drinking in the last 30 days, almost half report being drunk within the past 30 days, and youth are drinking at an earlier age than before and consume more than adults when they do (Institute of Medicine, 2004; Johnston, O’Malley, Bachman, & Schulenberg, 2006b). As a result, underage drinking is a leading cause of death from injury. Frequent underage drinking can also lead to poor school and health outcomes, e.g., having unprotected sex with multiple partners (Grunbaum et al., 2004).

These figures suggest much benefit could be gained by community mobilization to curb youth access to alcohol. The Report, Reducing Underage Drinking: A Collective Responsibility, by the National Research Council and Institute of Medicine (Institute of Medicine, 2004) documents 10 environmental alcohol prevention (EAP) strategies considered evidence-based, including conducting alcohol compliance checks and responsible beverage service (RBS) programs. Although federal and state governments spend millions of dollars to promote and fund their use, communities face challenges carrying out EAP strategies effectively. This study tests Getting To Outcomes for Underage Drinking (GTO-UD), a tool kit and implementation support intervention to help communities prevent underage drinking using EAP strategies¹.

The Gap between Research and Practice

Communities often find it difficult to put evidence-based EAP strategies into practice because the strategies can be complicated to choose, plan, implement, evaluate, improve, and sustain; do not always come with the guidance needed (e.g., how much staff time is required to implement them); and require support from local stakeholders, especially law enforcement (Wandersman & Florin, 2003). Also, community practitioners often lack the capacity—defined as the knowledge and skills—needed to successfully plan, implement, self-evaluate, improve, and sustain EAP strategies. Variation in states’ underage drinking laws is one indictor of a gap between research and the community practice needed to enact such laws (Fell, Fisher, Voas, Blackman, & Tippetts, 2008). For example, only 14 states ban sales of alcohol on certain days (mostly Sundays), although a CDC review shows such bans reduce consumption (Guide to Community Preventive Services, 2008). Enforcement of these laws is even more critical (Fell et al., 2008), suggesting that local implementation at the county, city or even smaller community level plays an important role in preventing underage drinking and its negative consequences.

Getting To Outcomes–Underage Drinking

GTO-UD was developed to address the gap between prevention research and practice by building prevention capacity—or the knowledge and skills—needed to conduct critical prevention practices (choosing, planning, implementing, evaluating, and sustaining specific prevention activities) as applied to EAP strategies. Getting To Outcomes (GTO) has also been applied to the prevention of teen pregnancy (Lesesne et al., 2008), drug use (Chinman et al., 2008; Chinman, Tremain, Imm, & Wandersman, 2009), positive youth development (Chinman et al., 2012; Chinman et al., in press), and homelessness (Chinman, Hannah, & McCarthy, 2012). In quasi-experimental and randomized controlled trials, GTO has been shown to improve practitioner capacity (e.g., knowledge and skills of good prevention practice) and standardized ratings of prevention practice (e.g., the carrying out of key tasks associated with high quality prevention: Chinman, Acosta, et al., In press; Chinman et al., 2008; Chinman et al., 2009).

GTO’s model specifies ten steps that have been identified in numerous studies as critical to the success of any prevention program—Step 1: Conduct needs and resources assessments (e.g., Hogan & Murphey, 2002); Step 2: Develop goals and desired outcomes (e.g., Mattessich & Mansey, 1992); Step 3: Choose the best evidence-based strategies to achieve the goals (use of evidence-based prevention is self-evident); Step 4: Ensure the strategies “fit” within the community context (e.g., Stith, 2006); Step 5: Ensure sufficient capacities are in place to carry out the strategies (e.g., Livet & Wandersman, 2005; Kallestad & Olweus, 2003); Step 6: Develop a plan for implementing the strategies (e.g., Roussos & Fawcett, 2000); Step 7: Conduct a process evaluation (e.g., Smith & Ananiadou, 2004); Step 8: Conduct an outcome evaluation (e.g., Asch et al., 2004); Step 9: Engage in continuous quality improvement (e.g., Labonte, 2001; Roussos & Fawcett, 2000); and Step 10: Sustain the strategies over time (e.g., Scheirer, 2005).

Figure 1 shows the logic model for the study. We hypothesized that compared to control coalitions, those receiving the GTO-UD intervention would show greater improvements in their work to enhance local responsible beverage service (RBS) and Compliance Check programs. RBS encompasses various practices merchants can undertake to prevent sales to minors, including training managers and servers. RBS training is a short, classroom-based course (from one to four hours) in which those who directly sell alcohol or those who manage or own establishments that sell alcohol learn how to comply with laws that govern the appropriate sale of alcohol. Compliance Checks involve law enforcement use of “undercover” minors to assess whether an outlet would sell to a minor. GTO-UD achieves improvements in the quality of prevention based on social cognitive theories of behavioral change (Fishbein & Ajzen, 1975), in which exposure to GTO-UD training and TA leads to more knowledge about performing GTO-UD-related activities (e.g., planning, implementation, evaluation), which leads to greater skills in conducting these activities, which in turn lead to the execution of more GTO-related behaviors. These GTO behaviors support the successful implementation of EAP strategies (Durlak & DuPre, 2008). For example in this study, GTO-UD was hypothesized to improve how well coalitions planned, implemented, and self-evaluated RBS training and Compliance Checks to prevent underage drinking, which in turn would lead merchants to improve their attitudes toward, and practices of, underage drinking prevention and enforcement. These are considered “proximal” outcomes in the sense that they are closer to the GTO-UD intervention than more distal outcomes of, for example, alcohol consumption, which was beyond the scope of this study.

Figure 1.

Logic Model

Study Design

This study was a small-scale, three-year (April 2008–March 2011), randomized controlled trial assessing whether providing prevention coalitions with the GTO-UD intervention improved implementation of RBS training and Compliance Checks to prevent underage drinking. Our outcomes were the quality with which the coalitions carried out RBS training and Compliance Checks and whether alcohol merchants changed their attitudes and selling behaviors. We involved six coalitions from across South Carolina, each in a geographically distinct county. The coalitions were chosen because they were the six, and only, recipients of the Enforcing Underage Drinking Laws (EUDL) grant from the state of South Carolina. All six were approached, and all agreed to participate. While there are many other prevention coalitions in the state, there were no other EUDL-funded coalitions in the state. We stratified county by size and then randomly assigned the coalitions: three coalitions and their RBS and Compliance Check programs received the 16 month GTO-UD intervention while the other three maintained routine operations. We chose South Carolina because a) like many states in the US, South Carolina has a strong mandate to put EAP strategies into place; b) locating sites in one state holds constant the different state funding streams, laws and state regulations, and state-level political climates; and c) all the coalitions were funded from the same grant program and therefore had the same funding, goals, and timelines for implementation of EAP strategies. Compliance Checks and RBS training were targeted because these EAP strategies were chosen by all six coalitions, specifically funded by their EUDL grants. All coalitions were using the 2.5 hour RBS training curriculum approved by the Department of Alcohol and Other Drug Abuse Services called PREP (Palmetto Retailers Education Program). Each coalition received $4,000 a year for their participation. Coalition members gave written consent. The study was approved by the RAND Corporation’s Human Subjects Protection Committee. The measures were from an interview assessing the quality of each coalition’s work and a survey of merchant attitudes and practices in the six counties served by the coalitions, both conducted at baseline and after the 16 month GTO-UD intervention.

METHODS

Study Sites

The six counties represented a mixture of socioeconomic profiles, with the more urban counties representing slightly higher socioeconomic status, based on median household income (www.quickfacts.census.gov, retrieved 9/6/09). The population density of the counties varied, from 41.7 persons per square mile to 481 persons per square mile (www.sciway.net, retrieved 4/12/10). The racial composition of the counties also varied. Statewide in 2000, about two thirds were White and 30% were African American. The most heterogeneous county was about half White and 45% African American, while the most homogeneous county was 84% White and 13% African American (www.sccommunityprofiles.org, retrieved 4/12/10).

Participants in this study were the staff of six county-wide coalitions in South Carolina. Coalitions are common public health mechanisms, simultaneously intervening on the multiple levels (individual, organizational, community) and sectors (parents, youth, criminal justice, education) needed to have an impact on community health (Butterfoss & Wandersman, 1993). The six coalitions are similar in that they have a core group of paid staff supporting a large volunteer base made of key stakeholder including the local alcohol and drug abuse agency, law enforcement, and community stakeholders. All six coalitions received the same EUDL funding ($50,000 a year for two years during the study) from the South Carolina Department of Alcohol and Other Drug Abuse Services. All chose RBS training and Compliance Checks.

GTO-UD Intervention

The GTO-UD intervention has three components: a manual, training, and on-site technical assistance (TA). GTO-UD Manual. The manual, published by the RAND Corporation (Imm et al., 2007), provides a detailed conceptualization about the nature of the problem of underage drinking and descriptions, history, and research evidence for ten evidence-based EAP strategies, including RBS and Compliance Checks. It explains the GTO-UD ten step model and specifies how to apply each step to each EAP strategy, often using tools and worksheets. For example, there are many steps that need to be taken to successfully plan (Step 6) a series of Compliance Checks. The manual lists all of the steps unique to Compliance Checks, but using a planning worksheet, prompts practitioners to decide how much of the EAP will be done (e.g., how many checks) and from there: who will carry out these steps, by when, with what resources, overcoming which barriers, and involving which stakeholders. Similar concrete guidance is provided in the other GTO-UD steps and for the other EAP strategies (including RBS training). These tools, mostly Word documents, can be downloaded and tailored.

GTO-UD Training

The intervention began with a one-day (six hours) training on the GTO-UD steps and EAP strategies in February 2009, in which manuals were distributed. The training was attended by representatives from each of the three intervention site coalitions.

GTO-UD Technical Assistance

Over the course of 16 months, members of the three intervention coalitions received ongoing TA from two local TA providers, both PhD level psychologists with experience in prevention. TA providers met with coalition staff overseeing RBS training and Compliance Checks about every other week, providing consultation and feedback on conducting tasks in accordance with the AGTO 10 steps as applied to the RBS training and Compliance Check strategies. The consultation could be considered “facilitation,” in which changes are stimulated through encouragement and clarification of tasks that need to be completed (Stetler et al., 2006). For example, as part of the TA meetings, TA providers prompted coalition staff to more aggressively market their RBS training to alcohol establishments, reward establishments that took the training, augment the training with additional information, monitor the delivery of the training for its fidelity, evaluate the impact of the training on knowledge and intentions, and make changes to the training based on evaluation data. Table 1 shows additional examples of TA activities. In May 2010, TA culminated with a four-hour meeting in which recommendations for ongoing implementation of EAP strategies were provided.

Table 1.

TA Activities and Hours by GTO-UD Step and Coalition

GTO-UD Steps and examples of TA activities conducted	Coalition			Sum	Mean (SD)
	1	2	3
Step 3: Evidence-base: TA providers gave each coalition summaries of best practices in Compliance Checks and RBS training	71	10	8	89	29.55 (35.79)
Step 6: Planning: TA providers helped each coalition better plan future compliance checks	50	14	10	75	24.89 (22.08)
Step 7: Process Evaluation: TA providers helped each coalition create a database, by alcohol outlet, that was used to monitor compliance check activities and outcomes, RBS training activity, and instances of media advocacy	53	4	5	61	20.41 (28.07)
Step 8: Outcome Evaluation: TA providers helped coalitions demonstrate that outlets with RBS training have less compliance check violations	49	1	1	50	16.83 (28.07)
Step 9: CQI: The compliance check database helped coalitions streamline data collection on future checks	33	9	4	46	15.45 (15.27)
Step 10: Sustainability: TA providers helped one coalition influence a policy to spend a portion of taxes from Sunday alcohol sales on underage drinking prevention	55	8	3	66	22.02 (28.42)
Total	311	46	31	387	129.16 (157.43)

Weekly review of TA provision and planning for upcoming TA was a regular item for discussion by the whole GTO-UD team. Each TA provider kept a TA log indicating date, duration of TA, notes on progress and next steps, and the step of GTO that was the primary focus of the TA session. After the intervention, the total number of TA hours and the hours per GTO step were tallied. Over the 16 month TA period, approximately 612 hours of TA were provided to the three intervention coalitions (M=203, SD=237). Six GTO-UD steps were identified as most pertinent and were chosen as 1) the focus of TA as almost two-thirds (63% or 387.47, 129.16 per coalition, SD=157.43) of the hours were spent on these six steps, and 2) the steps in which change in the quality of the coalitions’ Compliance Check and RBS work was measured in the interviews. Table 1 shows the types of activities that comprised TA by GTO step and the number of TA hours delivered by step and coalition. The distribution ranged from 89 hours (evidence base, 23% of the 387 hours) to 46 hours (continuous quality improvement, 12%). Coalition 1 engaged more with the TA providers and thus had significantly more TA hours.

Measures and Data Collection

Interview of Prevention Quality

The Interview was adapted from previous GTO research (Chinman et al., 2008). The interview assessed the quality with which the coalitions conducted tasks in six key areas targeted by the six GTO-UD steps mentioned above, as applied to their Compliance Check and RBS training work: evidence base, planning, process evaluation, outcome evaluation, continuous quality improvement, and sustainability. Interview questions focused on various aspects of the program’s mechanics (e.g., are all merchants checked at least twice a year?) and how well information from one step is applied to another (e.g., using data to make improvements or changes to better meet goals and objectives). Within each area, questions allowed raters to examine several micro-tasks for their presence (=1) or absence (=0). Scores were then summed by area, by program type, and by study group. Each GTO-UD step was standardized to a 0–10 scale. Totals by coalition ranged from 0–60. In a previous project involving ten programs, the Total Score was sensitive to change and had an average inter-rater reliability of .74. Inter-rater reliability for each area ranged from .65-.96 (Chinman et al., 2008).

Compliance Check and RBS training program directors from each of the six coalitions (n=3 GTO-UD, 3 control) were interviewed at baseline (prior to GTO-UD) and follow-up (16 months later). There was no coalition staff turnover. We used two PhD-level raters that had been trained in previous GTO studies and found to be reliable (Chinman, Acosta, et al., In press; Chinman et al., 2008; Chinman et al., 2009). The first rater conducted and audio-recorded the interviews. The second rater double coded all the recorded interviews. We estimated a prevalence-adjusted bias-adjusted kappa (PABAK, Bryt, Bishop, & Carlin, 1993) because of the potential impact on kappa from the relatively high prevalence of positive ratings. Raters had the same exact ratings, resulting in the PABAK at baseline and follow-up both = 1.0.

Merchant Survey

At baseline and follow-up, samples of merchants from the counties served by the six participating coalitions were surveyed to assess the impact of GTO-UD on attitudes and selling practices. A list of alcohol outlets (2,147 at baseline, 2,593 at follow-up) was obtained from the South Carolina Department of Revenue, the state alcohol licensing agency. Stratifying by county and off/on-premises sales status, we drew a random sample of outlets (675 at baseline, 837 at follow-up) based on power calculations. At baseline, 336 (50%) resulted in completed Merchant Survey telephone interviews, 49 (7%) refused, 124 (18%) were ineligible because they were out of business or were not alcohol sales outlets, 4 (1%) were duplicates, and 162 (24%) were incomplete at the end of the field period. At follow-up, 421 (50%) resulted in completed Merchant Survey telephone interviews, 64 (8%) refused, 199 (24%) were ineligible because they were out of business or were not alcohol sales outlets, and 153 (18%) were incomplete at the end of the field period. Deleting ineligible cases from the original sample resulted in a completion rate of 61% at baseline and 66% at follow-up. We were unable to verify the license type for five respondents at baseline and 10 at follow-up, so they were excluded, leaving 331 merchants at baseline and 411 at follow-up in the analytic dataset. The survey was administered by phone, following a standardized Computer Assisted Telephone Interviewing (CATI) protocol. Merchants received in advance a $2 bill incentive.

The merchant survey included two attitudinal and four behavioral outcome measures (some measures are scales or groups of items, others are single items). The first attitudinal measure assessed merchant attitudes toward the seriousness of youth alcohol use. It was the sum of two items chosen through factor analysis: “Alcohol-related accidents among youth are a serious problem in our community” and “Underage drinking is a serious problem in our community” (response scale 1=strongly disagree to 5=strongly agree; alpha=0.67). The second attitudinal measure assessed perceptions of likely consequences of selling to minors (1=no consequence, 2=warning, 3=employee fined, 4=business fined, 5=business’ license suspended).

The first behavioral outcome measure, RBS practices, asked owners and managers about a cluster of six outlet practices (presence of an incident log, staff signed written policies, age verification device, signs stating IDs are required, incentives for employees to refuse minors, other RBS activities). The latter three RBS practice items were adapted from the Communities Mobilizing for Change on Alcohol survey, which have been found to be valid and sensitive to changes in serving and selling practices over time (Wagenaar et al., 1999; Wolfson et al., 1996). The other three RBS practice items were developed for this study based on recommended practices alcohol outlets should follow to minimize the sale of alcohol to minors (Colthurst, 2004). These questions were developed by the study’s authors following the same format as the CMCA survey. The draft questionnaire was reviewed by local practitioners in South Carolina and their comments were incorporated into the final survey. A resulting RBS Practice Index summed the Yes/No (1/0) responses to all six items.

The next behavioral outcome measure assessed whether merchants required RBS training for their servers, and if so, what type (1=state sanctioned formal training for all employees, 2=in-house training for all employees or a mix of training types, 3=no training required of all employees). RBS Training was split off from the RBS Practice Index because it has more evidence (although mixed) compared to the practices in the Index, which have a modest amount of evidence. The third and fourth behavioral outcome measures assessed enforcement of underage drinking laws. One assessed the circumstances under which merchants require age identification for the purchase of alcohol (1= always, 2=when the purchaser appears to be under 35, 3=options less strict). The other was the rate per week of refusing sale to minors (number of reported weekly refusals out of reported weekly attempts by minors to buy alcohol).

Finally, the analyses were adjusted for the number of outlets within 500 meters of each responding outlet (i.e., “density”), which is purported to impact sales practices through increased competition and lax norms that may arise within clusters of alcohol outlets (Truong & Sturm, 2009). We used Microsoft’s online Terra Server database and Google Maps API Geocoding service to geocode merchants (see Chinman et al., 2011).

Merchant Survey Respondents

Merchant characteristics are shown in Table 2. The only difference between GTO-UD and Control groups on these characteristics was at baseline, when the control group had significantly more off-premises merchants than the GTO-UD group (75% vs. 51%). The two groups were very similar at baseline. More than half the merchants in both groups were male and the average age was about 40 years. In both groups, most were either college graduates (27 to 40%), had vocational training or some college (about a quarter), or were high school graduates (about a third). The majority race was White in both groups. Also, about a quarter of both groups were owners, over half were head managers, and about a fifth were assistant managers. Most from both groups had worked for their outlet between one and five years (over a third) or between five and 50 years (over a third). At baseline, 44% of outlets in the GTO-UD group and 33% in the control group were part of a regional or national chain. These baseline figures were comparable to those of the follow-up sample.

Table 2.

Merchant Demographics and Outlet Characteristics

	% at Baseline		% at Follow-up
Demographics	GTO-UD (n=248)	Control (n=83)	GTO-UD (n=243)	Control (n=168)
Male	58	53	64	63
Education
Less than high school	3	5	3	5
High school graduate	31	41	32	29
Vocational/some college	27	27	21	24
College degree or higher	40	27	43	42
Race/ethnicity†
Black	11	19	12	8
White	65	57	61	68
Other	25	24	27	23
Position at outlet
Owner	27	33	28	28
Head manager	54	54	61	55
Assistant manager	19	13	12	17
Years owned/worked for outlet
0–1	19	22	14	17
>1–5	36	40	39	41
>5–50	45	39	48	42
Off-premises outlet	51	75*	56	54
Outlet part of national or regional chain	48	33	36	45
	Mean (SD) at Baseline		Mean (SD) at Follow-up
Age	40.9 (13.2)	42.6 (13.4)	42.2 (13.5)	42.0 (11.9)

^†At baseline, race/ethnicity was strongly associated with county (unit of randomization), so we were not able to test for GTO-UD/control differences.

^*p<0.05 for difference between GTO-UD and control groups.

Data Analyses

Interview of Prevention Quality

Given the small number of programs in the study, we were limited to descriptive analyses. We calculated averages for each of the six GTO areas by time and group, and then summed each area into total Compliance Check and RBS scores. We then calculated the percent change for each area and for the overall scores by group.

Merchant Survey

To test for differences in the baseline to follow-up changes of the control and GTO-UD groups, we fit mixed models to the outlet attitude and behavior measures. We used linear models for the attitude toward seriousness measure and RBS Practice Index, and ordinal logistic models for likely consequences of selling to minors, RBS training, and age identification measures. To model the average weekly refusal rate to minors attempting to purchase alcohol, we fit a binomial regression to reported refusals/attempts, conditioning on reporting at least one weekly attempt. For outlets that reported more refusals than attempts, we set the number of refusals equal to the number of attempts. We also conducted a sensitivity analysis which excluded outlets that reported more refusals than attempts.

We regressed each of the measures on an intervention group indicator (0=control, 1=GTO-UD), a follow-up indicator (0=baseline, 1=follow-up) and their interaction, controlling for outlet density as a covariate. The models included random effects for county and county × follow-up to account for the clustering of outlets within counties and the county-level assignment to the intervention or control condition. Due to clustering, hypothesis tests for the coefficients of interest were on four denominator degrees of freedom (Murray, 1998).² Additionally, the models included a repeated measures adjustment for the 66 outlets that coincidentally responded to the survey at both baseline and follow-up. Analyses were implemented in SAS 9.2, using PROC GLIMMIX for nonlinear models and PROC MIXED for linear models.

RESULTS

Interview of Prevention Quality

Based on interviews with the six program directors (3 GTO-UD, 3 control for each program type), overall, the GTO-UD group improved on the Interview total score by about 12 to 15%, while the control group declined slightly, for both program types (see Table 3). Each Compliance Check program in the GTO-UD group increased between 5 to 22% in their total Interview score compared to control Compliance Check programs, which had between a 5% decrease to a 3% increase. Two of the RBS programs in the GTO-UD group had improvements of 9 and 43 percentage points, while one did not improve. Two control RBS programs declined and one increased a small amount. In the GTO-UD condition, Compliance Check programs improved most in their delivery of best practices and process evaluations and the RBS programs improved most in best practices and sustainability. Coalition 1 improved more overall on both RBS and Compliance Checks than the other two.

Table 3.

Quality of Prevention Tasks applied to Compliance Check and Responsible Beverage Service Training

Areas by GTO-UD Step1		Baseline		Post		Mean % Change
		GTO-UD (n=3)	Control (n=3)	GTO-UD (n=3)	Control (n=3)	GTO-UD	Control
		Mean (SD)	Mean (SD)	Mean (SD)	Mean (SD)
Compliance Checks (CC)
Best Practices		6.11 (1.13)	7.62 (0.48)	7.86 (0.86)	6.75 (0.90)	29%	−11%
Plan		8.24 (0.00)	8.63 (0.34)	8.43 (0.34)	8.43 (0.34)	2%	−2 %
Process Evaluation		7.78 (0.56)	9.07 (0.85)	9.63 (0.32)	8.89 (1.11)	24%	−2%
Outcome Evaluation		8.33 (2.89)	8.33 (2.89)	9.17 (1.44)	8.33 (1.44)	10%	0%
Continuous Qual Imp		6.39 (0.24)	6.39 (0.48)	6.25 (0.42)	6.67 (0.00)	−2%	4%
Sustain		6.25 (0.00)	4.58 (1.91)	7.08 (1.44)	5.42 (0.72)	13%	18%
Total by coalition	1.	38.20 (1.22)	46.78 (2.90)	48.77 (1.61)	43.67 (1.35)	22%	3%
	2.	44.64 (1.57)	43.17 (1.47)	47.49 (1.34)	45.23 (1.64)	6%	−6%
	3.	46.45 (1.43)	45.93 (1.77)	48.99 (1.60)	44.55 (1.66)	5%	1%
TOTAL, CC		43.10 (4.34)	44.63 (1.39)	48.42 (0.81)	44.48 (0.78)	12%	−2%
Responsible Beverage Service (RBS)
Best Practices		6.97 (1.05)	8.56 (0.57)	8.56 (0.35)	7.73 (0.45)	23%	−10%
Plan		8.75 (0.00)	8.33 (0.72)	8.33 (0.72)	8.13 (0.00)	−5%	−2%
Process Evaluation		8.00 (1.00)	9.33 (0.58)	8.33 (0.58)	8.67 (0.58)	4%	7%
Outcome Evaluation		6.67 (1.44)	6.67 (1.44)	8.33 (2.89)	5.83 (2.89)	25%	−13%
Continuous Qual Imp		5.42 (0.42)	6.25 (0.42)	6.11 (0.24)	6.39 (0.48)	13%	2%
Sustain		5.83 (2.60)	5.47 (1.44)	8.33 (1.91)	5.83 (0.72)	43%	7%
Totals by coalition	1.	35.86 (1.78)	43.47 (2.08)	51.22 (1.54)	45.72 (1.03)	43%	5%
	2.	42.03 (1.69)	44.76 (1.72)	45.93 (1.64)	40.27 (2.29)	9%	−10%
	3.	47.02 (1.09)	45.45 (1.38)	46.86 (1.48)	41.73 (1.26)	−0.3%	−8%
TOTAL, RBS		41.64 (5.59)	44.56 (1.01)	48.01 (2.82)	42.57 (2.82)	15%	−4%

¹Each GTO-UD step was standardized to a 0–10 scale. Totals by coalition ranged from 0–60.

Merchant Survey

In Table 4, unadjusted means or frequencies for each time point are shown within the GTO-UD and control groups. Model results are also presented, with differences (or odds ratios) and confidence intervals for baseline to follow-up change within each group as well as the difference of differences (or ratio of intervention OR to control OR). Regression diagnostics supported our choice of linear or logistic regression for each measure. Due to the small number of denominator degrees of freedom (four), statistical power was greatly limited. None of the tests for differences in baseline to follow-up change between the intervention and control groups were significant at the 0.05 level. However, a within-group test for pre-post change was significant. In the GTO-UD group, the odds of refusing minors’ attempts to purchase alcohol were 2.61 times greater after the intervention than before (p = 0.0223), while there was no significant change in the control group. The sensitivity analysis dropping outlets that reported more refusals than attempts yielded similar results.

Table 4.

Comparisons of Baseline to Follow-up Change for GTO-UD and Control Groups

	Descriptive Statistics				Adjusted Differences/Odds Ratios
	Baseline		Follow-up		GTO-UD	Control	Diff of Diffs
Outcomes	GTO-UD n=248	Control n=82†	GTO-UD n=243	Control n=168	Baseline/Follow-up Difference or OR (CI)	Baseline/Follow-up Difference or OR (CI)	Diff of Diffs or Ratio of ORs
Greater perceived seriousness of underage drinking & alcohol-related accidents (2–10)1	M=7.82 SD=2.14	M=7.22 SD=2.71	M=7.62 SD=2.32	M=7.48 SD=2.55	Diff= −0.21 (−0.80, 0.38)	Diff= 0.32 (−0.56, 1.19)	Diff= −0.53 (−1.59, 0.54)
Expects more severe consequences for selling to minors (1–5)2	M=3.76 SD=0.94	M=3.84 SD=0.88	M=3.69 SD=0.98	M=3.68 SD=0.91	OR=0.90 (0.52, 1.53)	OR=0.73 (0.33, 1.64)	OR=1.23 (0.46, 3.25)
Requires sanctioned RBS training for employees3	Sanctioned:13% In-house: 25% None: 62%	Sanctioned: 8% In-house: 25% None: 66%	Sanctioned:14% In-house: 22% None: 64%	Sanctioned: 13% In-house: 26% None: 61%	OR=0.95 (0.52, 1.72)	OR=1.20 (0.52, 2.79)	OR=0.79 (0.28, 2.22)
RBS practices (0–6)4	M=3.19 SD=1.45	M=2.86 SD=1.45	M=3.16 SD=1.38	M=3.18 SD=1.44	Diff=−0.02 (−0.60, 0.57)	Diff=0.30 (−0.41, 1.02)	Diff=−0.32 (−1.25, 0.61)
Requires age identification more often for alcohol purchase3	Always: 27% Buyer<35: 42% Less strict: 31%	Always: 33% Buyer<35: 40% Less strict: 28%	Always: 28% Buyer<35: 44% Less strict: 28%	Always: 33% Buyer<35: 40% Less strict: 27%	OR=1.10 (0.69, 1.75)	OR=0.97 (0.48, 1.94)	OR=1.13 (0.49, 2.63)
Refuses to sell to minors more times per week5	M=3.19 SD=5.01	M=2.31 SD=3.42	M=3.80 SD=9.90	M=2.65 SD=3.86	OR=2.61 (1.25, 5.44)*	OR=1.31 (0.36, 4.74)	OR=1.99 (0.45, 8.74)

^†Outlet density (an adjustor variable) could not be calculated for one respondent, who was excluded from the models.

¹Linear regression; 2=least serious, 10=most serious

²Ordinal logistic regression; 1=no consequence, 2=warning, 3=employee fined, 4=business fined, 5=license suspended

³Ordinal logistic regression

⁴Linear regression; RBS practice index is sum of 6 Yes/No items: age verification device, written policy for employees, incident log, signs stating IDs are required and checked, incentives for employees to refuse minors, and any other RBS activities

⁵Binomial regression modeling refusals/attempts among merchants that reported at least one weekly purchase attempt

^*p<0.05

DISCUSSION

Successful implementation of EAP strategies to prevent the sale of alcohol to minors is challenging for coalitions with limited funding and other capacity constraints. Using a randomized controlled study design in six sites, this study sought to assess an implementation support intervention, Getting To Outcomes-Underage Drinking, designed to improve prevention coalitions’ capacity to implement EAP strategies with quality. Interviews with staff found that the quality of some activities targeted by the GTO-UD steps, applied to RBS training and Compliance Check programs, improved more over time in the GTO-UD sites than in the control sites. Also, Coalition 1, which received substantially more TA hours, improved more overall on the ratings of prevention quality than the other two GTO-UD coalitions. Surveys of alcohol sales merchants did not find that baseline to follow-up change significantly differed between the GTO-UD and control groups for any practice or attitude, although merchants in the GTO-UD counties improved significantly on refusing sales to minors and control merchants did not.

The findings on GTO-UD are similar to prior research showing that GTO can improve the quality of certain tasks key to the success of prevention programming and that greater TA hours may lead to greater improvement (Chinman et al., 2008; Chinman et al., in press; Chinman et al., 2009). Given the proposed logic model in Figure 1, these results are sensible because they most directly reflect the work of the GTO-UD intervention with the coalitions. Further out—i.e., what merchants believe and do—there were markedly less positive findings. It is possible that the reasons Coalition 1 chose to utilize more hours of TA may be a confound for why they also improved the most. For example, one hypothesis could be that Coalition 1 had higher quality programs to start, which could reflect an overall higher ability to simultaneously take advantage of the available TA and to make more improvements (i.e. rather than the greater TA causing the improvements). However, this was not the case as Coalition 1 had visibly lower scores on both CC (38.20 vs 44.63, 46,45) and RBS (35.86 vs. 42.03, 47.02) at baseline. Still, Coalition 1 could have had other characteristics unmeasured in this study which confounded the relationship between TA hours and improvement in prevention quality.

The finding about refusals may have occurred because a significant amount of TA time was spent better organizing and targeting Compliance Checks. While the Compliance Check databases that GTO-UD staff helped the coalitions to create did not lead to an increase in the number of Compliance Checks, it did appear to change how they were carried out. Instead of picking outlets at random (the common practice before), the database was used to target outlets that had not been checked in a long time or had a history of violations. These actions could have led to a perception among merchants that they needed to be more stringent in refusing minors.

It is possible that some other attitude or behavior unmeasured by this study did change and was responsible for the change in refusals. For example, a study by Turrisi, Nicholson, & Jaccard (1999) found that a different set of merchants’ attitudes—namely whether or not adopting RBS practices was a hassle to implement or perceived to have a negative impact on patrons and/or sales—was related to a more negative appraisal and less use of those practices. Although the Turrisi et al. sample included only college bars and the focus was to prevent excessive intoxication and DUI, not underage drinking, it does suggest that there is a range of possible attitudes that could affect merchant behavior.

The modest findings overall could be because of the limited funding given to the coalitions to carry out the EAP strategies. Despite some improvements in the quality of the coalitions’ work, there were many instances in which the lack of funding for the coalitions and law enforcement impeded actions (less media advocacy, compliance checks) that could improve merchant practice. In addition, the choice of the RBS strategy may have also played a role. Although studies have shown that RBS training can produce better server policies and practices, and less consumption and alcohol involved crashes (Guide to Community Preventative Services, 2010) the specific RBS programs in these trials are often more intensive than what is normally done in communities. Advocating for a law that makes strong RBS training mandatory for servers and owners might have been more effective in changing merchant practice than having coalitions try to convince merchants to voluntarily participate in training. Further, advocating for other laws may have been more effective than mandatory RBS training. For example, “dram shop” laws that hold merchants liable for the harms their customers cause and restricting the hours of sales have stronger evidence of impact than RBS training (Guide to Community Preventative Services, 2010). Finally, coalitions themselves have had an uneven history in carrying out effective alcohol and drug prevention. There are many reports in which coalitions have not had an impact (e.g., Hallfors et al., 2002). However, there has been more evidence of impact, especially when coalitions have organized strong EAP strategies (Hingson et al., 1996; Holder et al., 2000; Wagenaar et al., 1999). In those cases, the coalitions had good funding, chose evidence based EAP strategies, and strongly implemented them. In this study, even though the quality of the coalitions’ prevention work did improve some, the coalitions may not have had enough resources to implement EAP strategies with enough intensity to change merchant behavior and could have chosen different EAP strategies that may have had a stronger impact.

This study has limitations that should be noted. First, we sampled alcohol outlets in one state, which may limit generalizability. The relationships between GTO-UD, the quality of the coalitions’ prevention work, and merchant attitudes and practices could vary in states with different alcohol sales regulations. Future research ought to assess whether these relationships are replicated in different geographic areas. Second, although response rates of 61% at baseline and 66% at follow-up are high among commercial establishment surveys, it is possible that non-respondents differed from respondents in their relationships between attitudes, training, practices, and enforcement. Third, the RBS practice items were created specifically for this study. While they are face valid, their psychometric properties are not known. Fourth, a number of statistical challenges may have limited our ability to detect differential change over time between the GTO-UD and control groups. There were only six respondents from six coalitions (3 intervention, 3 control) and this small number suggests that the findings should be interpreted with caution. In addition, because the intervention assignment was done at the county coalition level, random effects for county had to be included in all models comparing the intervention and control groups. This meant that there were only four denominator degrees of freedom for tests of differences of differences. In addition, the analysis required a repeated measures adjustment for the 66 outlets that were coincidentally surveyed at both the baseline and follow-up waves of the survey. Several outcomes were categorical so nonlinear models were used, and differences of differences have larger variances than simple differences, making them more difficult to detect. Fifth, the design was cross-sectional, with a small portion sampled at both times. While not as strong as a pure longitudinal design, it was chosen as a way to maximize sample size (i.e., a longitudinal design would have yielded smaller samples given the potential dropout at the second time point). Finally, both sources of data (Interviews, Merchant Survey) were self-reported, which may have biased the results given the sensitive nature of some of the questions. For example, coalition members may have wanted to present a more favorable picture of their prevention work, although the narrow interview questions about the presence or absence of specific behaviors may have mitigated that potential bias somewhat. Also, merchants’ self-reported stringency in checking IDs could have been impacted by social desirability or a recent citation. Future studies could include compliance checks to help validate the self-reports.

Clearly, more research is needed to understand what leads to the ultimate desired merchant behavior—i.e., the “refusal”—compared to RBS practices whose aim is to support merchants’ refusals. This is because while many minors obtain alcohol through social sources (e.g., siblings), 23% to 30% of youth still access alcohol through commercial sources (Dent, Grube, & Biglan, 2005). However, our results show only 28% of merchants always check IDs and most do not require RBS training. Thus, more research—and on a larger scale to avoid clustering limitations—is also needed to strengthen community based efforts to improve alcohol sales practices among merchants. It may be that stronger compliance check programs and a different community wide approach—for example that includes a strong media advocacy—may lead to more change in merchant attitudes and influence their ID checking and refusal behaviors.