Fifteen-Minute Comprehensive Alcohol Risk Survey: Reliability and Validity Across American Indian and White Adolescents

Abstract

Objective:

American Indians (AIs) suffer from significant alcohol-related health disparities, and increased risk begins early. This study examined the reliability and validity of measures to be used in a preventive intervention trial. Reliability and validity across racial/ethnic subgroups are crucial to evaluate intervention effectiveness and promote culturally appropriate evidence-based practice.

Method:

To assess reliability and validity, we used three baseline surveys of high school students participating in a preventive intervention trial within the jurisdictional service area of the Cherokee Nation in northeastern Oklahoma. The 15-minute alcohol risk survey included 16 multi-item scales and one composite score measuring key proximal, primary, and moderating variables. Forty-four percent of the students indicated that they were AI (of whom 82% were Cherokee), including 23% who reported being AI only (n = 435) and 18% both AI and White (n = 352). Forty-seven percent reported being White only (n = 901).

Results:

Scales were adequately reliable for the full sample and across race/ethnicity defined by AI, AI/White, and White subgroups. Among the full sample, all scales had acceptable internal consistency, with minor variation across race/ethnicity. All scales had extensive to exemplary test–retest reliability and showed minimal variation across race/ethnicity. The eight proximal and two primary outcome scales were each significantly associated with the frequency of alcohol use during the past month in both the cross-sectional and the longitudinal models, providing support for both criterion validity and predictive validity. For most scales, interpretation of the strength of association and statistical significance did not differ between the racial/ethnic subgroups.

Conclusions:

The results support the reliability and validity of scales of a brief questionnaire measuring risk and protective factors for alcohol use among AI adolescents, primarily members of the Cherokee Nation.

American indians (ais) suffer from significant alcohol-related health disparities, with increased risk because of more alcohol-attributable motor vehicle crash fatalities, suicides, and falls than among other racial/ethnic groups (Indian Health Service, 2013; Keyes et al., 2012). Increased risk begins early: AI 8th- and 10th-grade students living on or near reservations reported significantly higher rates of alcohol use, drunkenness, and heavy episodic drinking than national rates (Stanley et al., 2014). Effective preventive strategies are clearly needed for this vulnerable, at-risk population of youth.

Behavioral health surveys are usually relied on for measurement of key outcomes in prevention research and practice. A typical behavioral health survey conducted within schools requires at least one full classroom session for completion (Centers for Disease Control and Prevention, 2004; Glaser et al., 2005; Johnston et al., 2014) and is implemented on an annual or less frequent schedule. However, if feasible, more frequent assessment would provide increased time sensitivity and statistical power to detect intervention effectiveness (Guo et al., 2013).

In addition to the benefits of measurement comprehensiveness and frequency, consistency of measurement reliability and validity across racial/ethnic subgroups is crucial to validly examine preventive intervention effects by important subgroups. The literature includes only two rigorous psychometric studies with adequate sample sizes of AI youth. One study, Glaser et al. (2005), examined the Communities That Care Youth Survey that includes 133 items measuring multiple risk and protective factors for drug use and delinquency. The researchers found that reliability and criterion validity were equivalent across gender and all five racial/ethnic adolescent subgroups studied (White, African American, AI, Hispanic, and Asian or Pacific Islander). However, a limitation of their study was that they did not explore whether the factors were equally predictive of drug use across subgroups. The second study, Rosay et al. (2000), using three data sets, one of which included a national sample with an adequate sample size of AI youth, found that only half of their risk and protective factor measures were equally reliable across ethnic groups, and that the measures were generally more reliable among White youth. Rosay et al. also examined predictive validity and reported that, once reliability differences were controlled for, differences in validity were rare across subgroups. Overall, the results of those two studies support the use of the same measures of adolescent drug use risk and protective factors across racial/ethnic subgroups. However, there remains concern regarding lower reliability of measures among ethnic minority youth, including AI, than among White youth.

We received funding from the National Institute on Alcohol Abuse and Alcoholism for a community trial to design, implement, and evaluate a multilevel preventive intervention for high schools and surrounding communities with high proportions of AI families (Komro et al., 2014). As part of the evaluation of this trial, we implemented quarterly assessments of a cohort of high school students followed over 3 years. We designed a comprehensive 15-minute survey for high school students measuring alcohol risk and protective factors, alcohol and other drug use, and associated consequences. A 15-minute completion requires less classroom time than is typical and facilitates more frequent ability to monitor outcomes in a time-sensitive fashion. This article examines the reliability and validity of measures to be used in the preventive intervention trial.

The study included a unique sample of adolescents from our trial, which included nearly half AI youth who were primarily members of the Cherokee Nation. It was also unique, given that it was a nonreservation sample of AI youth attending public high schools, which include sizable populations of White and racially/ethnically mixed (i.e., AI and White) youth. We conducted this study to assess the consistency of reliability and validity of measures used in a brief, 15-minute alcohol risk survey across three racial/ethnic subgroups, including students who identified as AI only, AI and White, and White only. The study addressed the following three questions: (a) Can a comprehensive alcohol risk survey be designed and administered successfully to measure key proximal and primary outcomes and be completed within 15 minutes, (b) can proximal and primary outcomes be reliably and consistently measured by the same indicators across the three racial/ethnic subgroups, and (c) are associations (criterion validity and predictive validity) between proximal and primary outcomes consistent across the racial/ethnic subgroups?

Method

Design

Our trial was designed to evaluate effects of an alcohol preventive intervention on the primary outcomes of alcohol use and alcohol-related problems and on intermediate (or proximal) outcomes. The trial design combined purposive selection of towns, random assignment to study condition, controlled interrupted time-series design, nested cohorts as well as repeated cross-sectional observations, a factorial design crossing two conceptually distinct interventions, and multiple comparison groups (Komro et al., 2014). In this study, we used three baseline surveys from the trial to assess the reliability and validity of outcome measures by racial/ethnic group.

One of the primary design features was a time-series research design. The large number of repeated measures (a time series) substantially increased internal validity (i.e., strength of causal inference) as well as statistical power over conventional pre-/post-community trial designs (Shadish et al., 2002; Wagenaar & Komro, 2013). The study sample was a cohort of high school students within study towns. Quarterly measurements across all study treatment and control sites produced a time-series design with observations of youth nested within school/town.

Towns were purposively selected from within the 14-county tribal jurisdictional service area of the Cherokee Nation and were randomly assigned to study condition (Komro et al., 2014). The Cherokee Nation is a federally recognized Indian tribe and has sovereign control over 66,000 acres of land as well as 96 miles of the Arkansas riverbed. The Cherokee Nation is not a reservation; it is a 7,000-square-mile jurisdictional service area covering all of eight counties and portions of six additional counties in northeastern Oklahoma. The Cherokee Nation offers services to its citizens throughout its tribal jurisdiction, including housing, education, health and human services, commerce, and career services. The sample for the current study included high school students from the four initial towns participating in the trial (one high school per town). Data from the first three baseline surveys (before implementation of the intervention began) of all 9th- to 12th-grade students within the four high schools were used for the current study. Surveys were implemented in November/December 2011 (Wave 1), January–March 2012 (Wave 2), and April/May 2012 (Wave 3).

Theoretical framework and survey development

Wagenaar and Perry’s (1994) comprehensive theoretical framework of drinking behavior guided development of our integrated community-level intervention, as shown in Figure 1. The expanded Screening, Brief Intervention, and Referral to Treatment (SBIRT) intervention, named CONNECT, focused on promoting screening and motivational interviewing, positive social interactions and role models, and social and multicultural competencies. The CONNECT intervention was designed to affect alcohol cognitions/expectancies, social support and bonding, drinking models and norms, drinking behaviors, and alcohol-related risks. In addition, the CONNECT intervention served the role of SBIRT in identifying and guiding especially high-risk students into counseling or treatment. The environmental intervention, Communities Mobilizing for Change on Alcohol (CMCA), focused on decreasing availability and increasing formal social controls on both access to and consumption of alcohol by youth. Intervention features were hypothesized to affect perceived and observed access to alcohol by youth, perceptions of police enforcement, drinking norms, drinking behaviors, and alcohol-related risks and outcomes.

Figure 1.

Theoretical framework for alcohol preventive intervention and proximal and primary outcome measures. SBIRT = Screening, Brief Intervention, and Referral to Treatment; MI = motivational interviewing; DUI = driving under the influence; SES = socioeconomic status.

The survey was designed to measure each primary, proximal, and moderating variable outlined in the theoretical framework (Figure 1). Questionnaire items were drawn from the national Youth Risk Behavior Survey (YRBS; Centers for Disease Control and Prevention, 2010), the Oklahoma Prevention Needs Assessment survey (Oklahoma Department of Mental Health and Substance Abuse Services, 2010), and surveys used in the Project Northland trials (Komro et al., 2008; Perry et al., 1996). The main outcome of interest was alcohol use, measured primarily with two standard items from the YRBS (Centers for Disease Control and Prevention, 2010), including the frequency of use during the past 30 days and the frequency of heavy use (five or more drinks of alcohol in a row) in the past 30 days. We also included items on smokeless tobacco, cigarettes, marijuana, prescription drugs without a doctor’s prescription, and any other illegal drug (Centers for Disease Control and Prevention, 2010). In addition to alcohol and other drug use behaviors, other primary outcome measures included alcohol-related consequences (5 items; i.e., academic, social, physical) and riding with a drinking driver or driving under the influence of alcohol (DUI Risk, 3 items; Centers for Disease Control and Prevention, 2010; Komro et al., 2008).

Key proximal outcome measures included the following scales: Perceived Alcohol Availability (10 items), Drinking Places (12 items), and Alcohol Acquisition (10 items; Centers for Disease Control and Prevention, 2010; Komro et al., 2008); Perceived Police Enforcement (3 items; Oklahoma Department of Mental Health and Substance Abuse Services, 2010); Alcohol Norms (6 items; Komro et al., 2008; Oklahoma Department of Mental Health and Substance Abuse Services, 2010); Alcohol Expectancies (14 items; Christiansen et al., 1982); Social Support (6 items; Oklahoma Department of Mental Health and Substance Abuse Services, 2010); and Parental Communication (3 items; Komro et al., 2008). In addition to these proximal and primary outcomes, we included measures of hypothesized intervention effect moderators, including demographic characteristics (i.e., gender; race/ethnicity and tribal membership; family composition); Early Alcohol Risk (3 items); Household Adult Problems (2 items; Center on Alcoholism, Substance Abuse, and Addictions Research Division, 1995). Socioeconomic status (5 items) was measured by enrollment in a free or reduced-price lunch program, family composition, parental education, family computer ownership, and vacations; Boyce et al., 2006; Oklahoma Department of Mental Health and Substance Abuse Services, 2010); Racial Discrimination (2-item composite score; Tobler et al., 2013; Wolfe & Kimerling, 1997); Depression (6 items; Kandel & Davies, 1982); and AI (3 items) and White American or Anglo (3 items) Cultural Identity (Oetting & Beauvais, 1991). (A copy of the survey is available on request from the first author.)

Data collection procedures

Brief (10- to 15-minute) self-report questionnaires were administered to 9th- through 12th-grade students in November/December 2011, January–March 2012, and April/May 2012 in four study high schools. The questionnaires were administered with a team of one or two trained research survey staff members per classroom, following standardized procedures. School staff members were not involved in survey administration. Students received a $5 incentive for each survey and an additional $10 during the third administration if they participated in all surveys for which they were eligible during the academic year. Each questionnaire had a unique study ID to link individual student responses over time. Parents were sent a consent letter 4–6 weeks before survey administration and were asked to call a toll-free number or to return a postage-paid postcard if they did not want their child to participate. Before the consent letter mailing, we sent a postcard informing parents that an important letter would be mailed to them; following the consent mailing, we sent a reminder postcard. Students were given an assent form and were given the opportunity to refuse participation at each survey administration. Survey procedures were approved by both the University of Florida and the Cherokee Nation Institutional Review Boards.

Analyses

We evaluated psychometric properties of 16 scales and one composite score (Racial Discrimination) included in the 15-minute alcohol risk survey by estimating the internal consistency, test–retest reliability, criterion validity, and predictive validity of each scale. We further evaluated these measures of reliability and validity separately by race/ethnicity (i.e., AI only, mixed AI and White, White only). However, all students regardless of race/ethnicity were retained for the analyses of the full sample to verify the appropriateness of the scales for later analyses of the intervention.

The consistency of reliability and validity across reported race/ethnicity was evaluated by testing the equivalence of reliability and validity estimates across subgroups using appropriate tests of heterogeneity. The equivalence of the Cronbach’s alphas across racial groups was tested using the method described by Feldt et al. (1987). The Feldt test was implemented using the “cocron” package of R v3.0.2 (Diedenhofen, 2013). The equivalence of test–retest, criterion validity, and predictive validity estimates across racial groups was tested using the method described by Weaver and Wuensch (2013). Specifically, the correlation coefficients for each group were converted to z scores using Fisher’s transformation. These z scores were then used to construct the Cochran’s Q test statistic most commonly seen in meta-analyses. The null hypothesis for this Q statistic is equivalence of the estimated correlation coefficients across all three racial groups; rejection of this hypothesis is indicative of differences across groups.

Reliability.

We assessed internal consistency and test–retest reliability for each scale. An internal consistency estimate (Cronbach’s α) and 95% confidence interval (CI) were calculated for each scale using the Wave 1 survey. Test–retest reliability was estimated by calculating the Pearson’s product-moment correlation and corresponding 95% CI between Wave 2 and 3 values for each scale. Waves 2 and 3 were used to estimate test–retest reliability because they were the survey waves occurring closest together in time during the baseline period (2–3 months between surveys). These analyses were replicated for each demographic subgroup to attain subgroup-specific reliability estimates. All reliability estimates used R v3.0.2.

To interpret the reliability results, we used the conventions for internal consistency and test–retest reliability as set forth in Robinson et al. (1991). For Cronbach’s alpha, Robinson et al. (1991) suggested that internal consistency be categorized as minimal (<.60), moderate (.60–.69), extensive (.70–.79), or exemplary (>.80). Test–retest reliabilities were measured over a period of 2–3 months for each scale and the racial discrimination composite score. Although Robinson et al. (1991) used the same nomenclature, the test–retest scale reliability interpretability heuristics included the strength of association and the temporal proximity of data collection. Thus, r = .20 across less than 1 month is minimal, r = .30 across 1–3 months is moderate, r = .40 across 3–12 months is extensive, and r = .50 across 12 or more months is exemplary.

Validity.

Criterion validity assessed the cross-sectional correlation at Wave 1 of each scale with the frequency of alcohol use during the past month. Predictive validity assessed the longitudinal association of each scale measured at Wave 1 and the frequency of alcohol use during the past month at Wave 3 (approximately 6 months later). Criterion validity for each scale was estimated by calculating the Pearson’s product-moment correlation and corresponding 95% CI between each scale at Wave 1 and the frequency of alcohol use during the past month at Wave 1. Predictive validity for each scale was estimated by calculating the Pearson’s product-moment correlation and corresponding 95% CI between each scale at Wave 1 and the frequency of alcohol use during the past month at Wave 3 (approximately 6 months following Wave 1). All validity estimates used R v3.0.2. We used Cohen’s (1988) convention to interpret the correlation coefficients, with a correlation coefficient of .10 interpreted as a weak or small association, a correlation coefficient of .30 considered a moderate correlation, and a correlation coefficient of .50 or larger interpreted as a strong or large correlation.

Multivariable models using forward stepwise regression were conducted to assess the total variance in the frequency of past-month alcohol use explained by the scales. All tested scales were included as candidate variables for the model, and the forward stepwise procedure was used to build the final model. The R² from the final model is reported as the total variance in the frequency of past-month alcohol use explained by the scales.

As a sensitivity analysis, the validity coefficients were reestimated using heavy alcohol use (five or more drinks in a row) during the past month as the outcome variable. There were no substantive differences found in the magnitude or direction of these estimates; thus, the heavy alcohol use estimates are not presented in the results.

Results

Participants

Response rates for the three survey waves used in the current study were 85% (n = 1,928 completed surveys), 83% (n = 1,828 completed surveys), and 81.7% (n = 1,813 completed surveys), respectively. Noncompleters included parental refusals (2.3%) and absent students (9.2%–12.3%). There were only two student refusals across the three waves of data collection.

Fifty percent of students reported being eligible for free/reduced-price lunch. Forty-nine percent of the students were young women. For self-reported race/ethnicity, students were able to mark all the potential responses that applied. Forty-four percent of the students indicated they were AI (82% of those self-identifying as AI reported being a member of the Cherokee Nation), including 23% who reported being AI only (n = 435), 18% both AI and White (AI/White; n = 352), and 2% AI and another race/ethnicity (n = 32). Forty-seven percent of the students reported being White only (n = 901). For race/ethnicity comparisons, we included the groups that comprised the majority of the sample and with large enough sample sizes for comparison (AI only, both AI and White, White only). The majority of students not classified as AI only, AI/White, or White only were missing responses on the race/ethnicity survey item and were not included in the race/ethnicity subgroup analyses. The mean age of students was 16.0 years. The mean age for AI only, AI/White, and White only students was 16.1, 15.9, and 16.0 years, respectively. Reported lifetime (and past month) alcohol use for the full sample was 53% (23%). Reported lifetime (and past-month) alcohol use for AI only, AI/White, and White only students was 53% (25%), 55% (23%), and 51% (22%), respectively.

Reliability

Presented in Table 1 are the Cronbach’s alphas for the scales for the entire sample and for reported race/ethnicity subgroups. Also presented are the p values for the comparisons of differences in Cronbach’s alphas between the race/ethnicity subgroups.

Table 1.

Internal consistency estimates (Cronbach’s α) at Wave 1 for full sample and by race/ethnicity

		Race/ethnicity
Scale	Full sample Cronbach’s α [95% CI] (N = 1,932)	AI Cronbach’s α [95% CI] (n = 435)	AI and White Cronbach’s α [95% CI] (n = 352)	White Cronbach’s α [95% CI] (n = 901)	p
Alcohol Availabilitya	.83 [.82, .84]	.83 [.80, .85]	.79 [.75, .82]	.84 [.82, .85]	.02
Drinking Places	.89 [.88, .90]	.88 [.86, .89]	.89 [.87, .91]	.90 [.89, .91]	.18
Alcohol Acquisition	.89 [.88, .90]	.88 [.86, .90]	.88 [.86, .90]	.89 [.88, .90]	.60
Police Enforcementb	.79 [.78, .81]	.79 [.76, .83]	.77 [.73, .81]	.78 [.75, .80]	.78
Alcohol Norms	.65 [.63, .68]	.66 [.61, .71]	.61 [.55, .67]	.68 [.64, .71]	.17
Alcohol Expectancies	.73 [.71, .74]	.74 [.70, .77]	.68 [.63, .73]	.74 [.71, .76]	.05
Social Support	.82 [.81, .83]	.81 [.78, .84]	.81 [.77, .84]	.83 [.82, .85]	.49
Parent Communication	.70 [.68, .72]	.73 [.69, .77]	.71 [.65, .76]	.69 [.66, .73]	.57
DUI Risk	.56 [.53, .59]	.46 [.37, .54]	.62 [.54, .68]	.63 [.58, .67]	.01
Alcohol Consequences	.98 [.98, .98]	.98 [.97, .98]	.98 [.98, .99]	.98 [.98, .98]	1.00
Adult Problems	.58 [.54, .61]	.58 [.49, .65]	.57 [.47, .65]	.60 [.54, .65]	.90
Early Alcohol Risk	.87 [.86, .88]	.87 [.85, .89]	.87 [.84, .89]	.88 [.86, .89]	.77
Socioeconomic Status	.59 [.56, .61]	.57 [.50, .63]	.55 [.48, .62]	.62 [.58, .66]	.29
Depression	.84 [.83, .85]	.86 [.84, .88]	.82 [.79, .85]	.85 [.83, .86]	.06
AI Cultural Identity	.89 [.89, .90]	.91 [.89, .92]	.87 [.84, .89]	.84 [.82, .85]	<.01
White Cultural Identity	.93 [.92, .93]	.93 [.92, .94]	.92 [.91, .94]	.92 [.91, .93]	.42

Notes: Full sample includes n = 244 students who were other or missing race/ethnicity. AI = American Indian; CI = confidence interval; DUI: driving under the influence of alcohol.

^aAlcohol availability = perceived alcohol availability;

^bpolice enforcement = perceived police enforcement.

Internal consistency.

Extensive to exemplary internal consistency (Cronbach’s α = .70–.98) was observed for 10 of the 16 scales for the full sample and for each reported race/ethnicity subgroup: Perceived Alcohol Availability, Drinking Places, Alcohol Acquisition, Perceived Police Enforcement, Social Support, Alcohol Consequences, Early Alcohol Risk, Depression, and both the AI and White Cultural Identity sub-scales (Table 1). For all but 2 of these 10 scales, there were no statistically significant differences by race/ethnicity. The Perceived Alcohol Availability scale was significantly lower among AI/White youth, but still extensive (Cronbach’s α = .79). The AI Cultural Identity subscale was significantly less reliable among White youth, but still extensive (Cronbach’s α = .84).

For two additional scales, internal consistency was extensive except for one subgroup. Internal consistency for Alcohol Expectancies was extensive for the full sample but was significantly lower among AI/White youth (Cronbach’s α = .68) compared with AI (Cronbach’s α = .74) and White (Cronbach’s α = .74) students. Internal consistency for the Parent Communication scale was extensive for the full sample. It was below the cut point with Cronbach’s α = .69 among White youth, but this was not significantly different from that for AI (Cronbach’s α = .73) or AI/White (Cronbach’s α = .71) students.

Moderate internal consistency was observed for Alcohol Norms for the full sample and all three subgroups. Moderate internal consistency was observed for AI/White and White youth for the DUI Risk scale; however, it was significantly lower among AI youth. Moderate internal consistency was observed among White youth for the Household Adult Problems and Socioeconomic Status scales but not among AI/White and AI students, although the differences between subgroups were not statistically significant.

Test–retest reliability.

Table 2 presents results of the test–retest reliabilities for the full sample and by race/ethnicity subgroup. Test–retest reliabilities for all of the scales for the entire sample and across the three reported race/ethnicity subgroups were in the extensive to exemplary range (r = .44–.90). There were significant differences by race/ethnicity for four of the scales (DUI Risk, Household Adult Problems, Depression, AI Cultural Identity) and the Racial Discrimination composite, even though they were all in the extensive to exemplary range. Test–retest reliability for the DUI Risk scale was higher among White youth than among AI/White and AI youth. Household Adult Problems and Depression test–retest reliabilities were lowest among AI youth, followed by AI/White youth, and then White youth. Discrimination and the AI Cultural Identity subscale test–retest reliabilities were highest among AI youth, followed by AI/White youth, and then White youth.

Table 2.

Test–retest reliabilities from Wave 2 to Wave 3 (2–3 month time lag) for full sample and by race/ethnicity

		Race/ethnicity
Scale	Full sample r [95%CI] (n = 1,592)	AI r [95%CI] (n = 306)	AI and White r [95%CI] (n = 263)	White r [95%CI] (n = 705)	Group diff. p
Alcohol Availabilitya	.70 [.67, .73]	.66 [.59, .72]	.71 [.64, .77]	.72 [.68, .76]	.12
Drinking Places	.52 [.49, .56]	.51 [.42, .59]	.58 [.49, .66]	.52 [.46, .57]	.28
Alcohol Acquisition	.53 [.49, .57]	.54 [.45, .61]	.51 [.41, .60]	.58 [.53, .63]	.25
Police Enforcementb	.52 [.48, .56]	.51 [.42, .59]	.49 [.39, .58]	.54 [.48, .59]	.50
Alcohol Norms	.72 [.69, .74]	.69 [.63, .75]	.68 [.61, .74]	.73 [.70, .77]	.16
Alcohol Expectancies	.73 [.70, .75]	.73 [.67, .78]	.74 [.68, .79]	.74 [.70, .77]	.88
Social Support	.63 [.60, .66]	.62 [.55, .69]	.60 [.52, .68]	.62 [.57, .67]	.89
Parent Communication	.69 [.66, .71]	.66 [.59, .72]	.70 [.63, .76]	.70 [.66, .73]	.43
DUI Risk	.49 [.45, .53]	.47 [.38, .56]	.44 [.34, .53]	.61 [.56, .66]	<.01
Alcohol Consequences	.52 [.48, .56]	.58 [.50, .65]	.46 [.36, .55]	.52 [.47, .58]	.08
Adult Problems	.55 [.51, .58]	.46 [.36, .54]	.53 [.43, .61]	.59 [.54, .64]	<.01
Early Alcohol Risk	.84 [.83, .86]	.83 [.80, .87]	.85 [.81, .88]	.85 [.83, .87]	.63
Socioeconomic Status	.90 [.88, .91]	.89 [.86, .92]	.88 [.84, .91]	.90 [.89, .92]	.13
Discriminationc	.73 [.70, .76]	.71 [.63, .77]	.62 [.53, .69]	.58 [.53, .64]	<.01
Depression	.63 [.60, .66]	.55 [.48, .62]	.62 [.55, .69]	.68 [.64, .72]	<.01
AI Cultural Identity	.77 [.75, .79]	.82 [.78, .86]	.73 [.66, .78]	.65 [.60, .69]	<.01
White Cultural Identity	.72 [.69, .75]	.75 [.69, .81]	.69 [.61, .75]	.70 [.65, .74]	.08

Notes: Full sample includes n = 318 students who were missing or other race/ethnicity. AI = American Indian; CI = confidence interval; diff. = differences; DUI: driving under the influence of alcohol.

^aAlcohol availability = perceived alcohol availability;

^bpolice enforcement = perceived police enforcement;

^cdiscrimination = racial discrimination.

Validity

Tables 3 and 4 present results for criterion validity and predictive validity, respectively. For each scale, the Pearson product-moment correlation and CI are presented for the full sample and then by reported race/ethnicity subgroup. Results of the statistical test for equivalence between subgroup estimates are also presented.

Table 3.

Criterion validity estimates at Wave 1 for full sample and by race/ethnicity

		Race/ethnicity
Scale	Full sample r [95% CI] (N = 1,932)	AI r [95% CI] (n = 435)	AI and White r [95% CI] (n = 352)	White r [95% CI] (n = 901)	Group diff. p
Alcohol Availabilitya	.30 [.26, .34]	.39 [.30, .47]	.38 [.28, .47]	.28 [.21, .34]	.06
Drinking Places	.71 [.69, .73]	.66 [.60, .71]	.71 [.66, .76]	.77 [.74, .80]	<.01
Alcohol Acquisition	.64 [.62, .67]	.65 [.59, .70]	.69 [.62, .74]	.69 [.66, .73]	.39
Police Enforcementb	-.20 [-.24, -.15]	-.24 [-.33, -.15]	-.25 [-.35, -.15]	-.19 [-.26, -.13]	.56
Alcohol Norms	.47 [.43, .51]	.53 [.46, .59]	.44 [.35, .53]	.49 [.43, .53]	.30
Alcohol Expectancies	.36 [.32, .40]	.41 [.32, .48]	.39 [.30, .48]	.36 [.30, .41]	.56
Social Support	-.10 [-.14, -.05]	-.02 [-.12, .07]	-.03 [-.14, .07]	-.18 [-.24, -.12]	.01
Parent Communication	-.07 [-.12, -.03]	-.07 [-.16, .03]	-.05 [-.15, .06]	-.11 [-.17, -.04]	.63
DUI Risk	.39 [.35, .43]	.32 [.23, .40]	.37 [.28, .46]	.47 [.41, .52]	.01
Alcohol Consequences	.67 [.65, .70]	.64 [.59, .70]	.70 [.64, .75]	.71 [.67, .74]	.12
Adult Problems	.05 [.00, .10]	-.02 [-.12, .08]	.09 [-.02, .20]	.07 [.00, .14]	.21
Early Alcohol Risk	.55 [.52, .58]	.55 [.48, .61]	.55 [.47, .62]	.56 [.51, .60]	.99
Socioeconomic Status	-.05 [-.10, .00]	-.07 [-.18, .03]	-.02 [-.14, .10]	-.04 [-.12, .03]	.77
Discriminationc	.07 [.02, .12]	.03 [-.07, .13]	.08 [-.03, .19]	.15 [.08, .21]	.12
Depression	.03 [-.01, .08]	.06 [-.04, .15]	.00 [-.11, .11]	.06 [-.01, .12]	.63
AI Cultural Identity	-.03 [-.08, .01]	-.05 [-.15, .05]	.10 [-.01, .21]	-.06 [-.13, .01]	.03
White Cultural Identity	.00 [-.04, .05]	-.01 [-.11, .09]	.08 [-.03, .19]	-.02 [-.09, .05]	.28

Notes: Full sample includes n = 244 students who were other or missing race/ethnicity. AI = American Indian; CI = confidence interval; diff. = differences; DUI: driving under the influence of alcohol.

^aAlcohol availability = perceived alcohol availability;

^bpolice enforcement = perceived police enforcement;

^cdiscrimination = racial discrimination.

Table 4.

Predictive validity estimates of scales at Wave 1 with past-month alcohol use at Wave 3 [6-month time lag] for full sample and by race/ethnicity

		Race/ethnicity
Scale	Full sample r [95% CI] (n = 1,602)	AI r [95% CI] (n = 343)	AI and White r [95% CI] (n = 301)	White r [95% CI] (n = 762)	Group diff. p
Alcohol Availabilitya	.21 [.16, .26]	.22 [.12, .33]	.30 [.19, .40]	.20 [.13, .27]	.30
Drinking Places	.46 [.42, .50]	.41 [.31, .49]	.61 [.53, .68]	.44 [.38, .49]	<.01
Alcohol Acquisition	.38 [.34, .42]	.36 [.26, .45]	.53 [.45, .61]	.36 [.29, .42]	<.01
Police Enforcementb	-.14 [-.19, -.10]	-.14 [-.24, -.03]	-.12 [-.23, -.01]	-.17 [-.24, -.10]	.70
Alcohol Norms	.35 [.30, .39]	.44 [.35, .52]	.29 [.18, .39]	.32 [.26, .39]	.03
Alcohol Expectancies	.26 [.21, .31]	.23 [.13, .33]	.29 [.18, .39]	.29 [.22, .35]	.56
Social Support	-.09 [-.13, -.04]	-.03 [-.13, .08]	.11 [-.22, .01]	-.15 [-.22, -.08]	.11
Parent Communication	-.08 [-.13, -.03]	-.04 [-.15, .07]	-.1 [-.12, .10]	-.11 [-.18, -.04]	.20
DUI Risk	.27 [.22, .31]	.25 [.14, .34]	.41 [.31, .50]	.26 [.19, .32]	.02
Alcohol Consequences	.38 [.34, .42]	.30 [.20, .39]	.42 [.33, .51]	.42 [.36, .48]	.03
Adult Problems	.03 [-.02, .08]	.04 [-.08, .14]	-.03 [-.15, .09]	.07 [-.01, .14]	.30
Early Alcohol Risk	.44 [.40, .48]	.47 [.39, .55]	.40 [.30, .50]	.45 [.38, .50]	.47
Socioeconomic Status	-.01, [-.06, .05]	.02 [-.10, .14]	.14 [.01, .26]	-.08 [-.15, .00]	<.01
Discriminationc	.07 [.01, .12]	.12 [.00, .23]	.03 [-.09, .15]	.08 [.00, .15]	.50
Depression	.01 [-.04, .06]	.02 [-.09, .13]	-.04 [-.15, .08]	.06 [-.02, .13]	.34
AI Cultural Identity	-.04 [-.10, .01]	-.04 [-.15, .08]	.07 [-.06, .19]	-.10 [-.18, -.03]	.03
White Cultural Identity	.04 [-.01, .09]	.08 [-.04, .19]	.09 [-.04, .21]	.01 [-.07, .08]	.32

Notes: Full sample includes n = 196 students who were missing or other race/ethnicity. AI = American Indian; CI = confidence interval; diff. = differences; DUI: driving under the influence of alcohol.

^aAlcohol availability = perceived alcohol availability;

^bpolice enforcement = perceived police enforcement;

^cdiscrimination = racial discrimination.

Criterion validity

Table 3 presents results from the criterion validity analyses. Four of the scales had a strong positive correlation (r > .50), with an additional one close to a strong correlation, and were significantly associated with frequency of alcohol use during the past month (Drinking Places: r = .71; Alcohol Acquisition: r = .64; Alcohol Norms: r = .47; Alcohol Consequences: r = .67; Early Alcohol Risk: r = .55; Table 3). An additional three scales were moderately correlated (r > .30) and significantly associated with frequency of past-month alcohol use (Perceived Alcohol Availability: r = .30; Alcohol Expectancies: r = .36; DUI Risk: r = .39). Three scales were statistically significantly associated with past-month alcohol use but with a weak correlation (r < .10; Social Support: r = -.10; Parent Communication: r = -.07; Racial Discrimination: r = .07). Five of the seven hypothesized moderators of the intervention were not significantly associated with past-month alcohol use (Household Adult Problems, Socioeconomic Status, Depression, AI Cultural Identity, White Cultural Identity).

For the majority of scales, interpretation of the strength of the association and statistical significance did not differ between the reported racial/ethnic subgroups. However, in the subgroup analyses, Drinking Places, Social Support, and DUI Risk were more associated with past-month alcohol use among White students compared with AI and AI/White students. In cross-sectional multivariable models, inclusion of the statistically significant scales accounted for much of the variance in the frequency of alcohol use in the past month: 67% in the model with the full sample, 65% in the model with AI students, 74% in the model with AI/White students, and 73% in the model with White students.

Predictive validity

Similar, albeit less strong, patterns were seen for predictive validity (Table 4). There were moderate correlations between Drinking Places (r = .46), Alcohol Acquisition (r = .38), Alcohol Norms (r = .35), Alcohol Consequences (r = .38), and Early Alcohol Risk (r = .44) measured during Wave 1 and the frequency of alcohol use during the past month measured during Wave 3. These findings indicated that these scales independently accounted for approximately 10%–20% of the variance in alcohol use 6 months later. There were weaker, but still statistically significant, correlations between Alcohol Expectancies (r = .26), DUI Risk (r = .27), Alcohol Availability (r = .21), Perceived Police Enforcement (r = -.14), Social Support (r = -.09), Parent Communication (r = -.08), and Racial Discrimination (r = .07) at Wave 1 and past-month alcohol use at Wave 3. Longitudinal associations between past-month alcohol use and Household Adult Problems, Socioeconomic Status, Depression, AI Cultural Identity, and White Cultural Identity were not statistically significant.

AI/White students had higher correlations between past-month alcohol use and Drinking Places, Alcohol Acquisition, and DUI risk, although the interpretation and statistical significance did not vary by reported race/ethnicity group. Socioeconomic status was significantly and positively correlated with past-month alcohol use among AI/White students but not among AI or White students. Among AI youth compared with the other two groups, the correlation between Alcohol Norms and alcohol use was higher and the correlation between Alcohol Consequences and alcohol use was lower, but the direction and statistical significance did not differ from those of the other two groups.

In longitudinal multivariable models, the statistically significant scales accounted for a sizable percentage of the variance in the frequency of alcohol use in the past month: 28% in the model with the full sample, 35% in the model with AI students, 42% in the model with AI/White students, and 27% in the model with White students.

Discussion

The 16 multi-item scales and the one composite score in the 15-minute alcohol risk survey were adequately reliable for the full sample and across race/ethnicity. Among the full sample, all scales had acceptable internal consistency, with minor variation across reported race/ethnicity subgroups. The three scales with the lowest internal consistency (DUI Risk, Household Adult Problems, and Socioeconomic Status) were just below the .60 cut point of minimal internal consistency (Robinson et al., 1991). Internal consistency of the three DUI Risk scale items was lowest among AI youth. Internal consistency of the two items in the Household Adult Problems and five items of the Socioeconomic Status scale did not differ by any of the subgroups. All scales had extensive to exemplary test–retest reliability, again with minimal variation across reported race/ethnicity.

The eight proximal and two primary outcome scales were each significantly associated with the frequency of alcohol use during the past month in both the cross-sectional and longitudinal models, providing support for both criterion validity and predictive validity. For the majority of scales, interpretation of the strength of the association and statistical significance did not differ between the reported racial/ethnic subgroups and provided support for their use in evaluations of preventive interventions.

The strongest correlations observed between the scales and past-month alcohol use were among the scales measuring where and how adolescents get alcohol. These results support the importance of community interventions to reduce youth access to alcohol, which is the target of our community environmental change intervention, CMCA.

Five of the seven hypothesized moderators of the intervention were not significantly associated with past-month alcohol use in either cross-sectional or longitudinal models. Only Early Alcohol Risk and Racial Discrimination were significantly associated with the frequency of alcohol use during the past month. Therefore, caution will be used in the decision as to whether to include these scales in the trial’s outcome analyses and will depend on whether these constructs influence the intervention effectiveness.

The results from this study are consistent with those of two large previous studies that investigated the reliability and validity of substance use and delinquency measures across racial/ethnic groups, including adequate samples of AI youth (Glaser et al., 2005; Rosay et al., 2000). These studies, along with the current study, provide support for the reliability and validity of scales measuring risk and protective factors for alcohol and other drug use among AI adolescents.

The current study included a unique sample of nonreservation AI and both AI and White youth. Most (82%) of the AI youth in the sample reported being members of the Cherokee Nation, and the results may not be generalizable to other AI tribes, especially those living in remote, isolated areas. The alcohol risk survey will be used to evaluate the effectiveness of two distinct interventions for this population of youth, the CONNECT intervention targeting alcohol cognitions, social support, and alcohol norms, and the CMCA intervention targeting alcohol access, enforcement, and norms (Komro et al., 2014). Planned mediation analyses will investigate the effects of each intervention on the hypothesized proximal and primary outcomes.

The comprehensive survey was successfully completed within 10–15 minutes, with approximately 20 minutes of class time used for survey administration. The brevity, yet comprehensiveness, of the survey provides a strong tool for the routine monitoring of adolescent alcohol use and the utilization of time-series designs. Collecting one observation before and one observation after an intervention, traditionally a common practice, results in weak inference because any differences observed might simply reflect natural variation in the outcome over time, particularly in rapidly developing adolescents (Wagenaar & Komro, 2011). Collecting many repeated measures before and after the introduction of a new intervention makes it easier to see whether changes in the outcome of interest occur consistently as an intervention is introduced and implementation ramps up, are larger than typical background variation over time, and substantially enhance confidence in causal inference (Wagenaar & Komro, 2013). Many repeated observations also provide more immediate feedback on proximal and primary outcome trends as the intervention is being rolled out, so that intervention adjustments can be made. Incorporating time-series design elements into the research trial facilitates a sustainable longer-term continuous quality-improvement ethos within the prevention practitioner community.

The targeted proximal and primary outcomes of our alcohol preventive interventions were deemed reliably and validly measured with the 16 scales and the one composite score that were included in the brief 15-minute alcohol risk survey. There were minimal differences between AI race/ethnicity subgroups, providing confidence in the interpretation of the trial outcomes across demographic subgroups. Differences that were observed in criterion validity and predictive validity reinforce the importance of mediation analyses (Komro et al., 2001; MacKinnon, 2008) by demographic subgroups of intervention effectiveness, and of attention to differences in etiology and intervention effectiveness across demographic subgroups.