The Protective Behavioral Strategies Scale–20: Improved Content Validity of the Serious Harm Reduction Subscale

Abstract

Excessive drinking in college leads to serious harms, but students who use protective behavioral strategies (PBS) avoid negative consequences by drinking in a safer manner. This study aimed to increase the content validity of the Serious Harm Reduction subscale of the most widely used measure of PBS, the Protective Behavioral Strategies Scale (PBSS: Martens et al., 2005). An initial item pool was developed from literature on college student drinking, existing lists of drinking control strategies, and an online pilot survey of college students (N = 1832). Items were also evaluated by focus groups of experts and members of the target population. Next, 1376 students (57.9% women; M_age = 18.5 years; 86.3% White) completed questionnaires online. A subset (n = 170; 12.4%) completed a second survey 4–6 weeks later to examine test-retest reliability and criterion-related validity. The remaining students (n = 1206) were divided into equal development and validation samples. Analyses included exploratory and confirmatory factor analysis of the development sample, factor structure replication and gender invariance testing in the validity sample. Results of this process produced the PBSS-20, with an expanded SHR scale of 8 items. The revised SHR scale demonstrated improved internal consistency and was associated with a broader range of alcohol-related negative consequences at follow-up. Given its focus on reducing serious harms, the SHR scale is arguably the most clinically relevant PBSS factor, and those who use this measure will benefit from the expanded breadth of SHR content and improved psychometric properties of the PBSS-20.

Excessive drinking in college leads to serious harms, including unintentional injury, hospitalization, fatal crashes, risky sexual behavior, and sexual assault. There is considerable evidence that levels of heavy drinking in college are on the rise (Hingson, Zha, & Weitzman, 2009; White, Hingson, Pan, Yi, 2011), which suggests that serious consequences will increase also. A harm-reduction approach to student drinking shifts the focus from drinking itself and emphasizes the importance of preventing serious, life-threatening harms (Marlatt, Larimer, & Witkiewitz, 2011). Accordingly, measures have been developed to assess utilization of protective behavioral strategies (PBS) students can use to control their drinking or drink in a safer manner. Adequate measurement of PBS is clinically relevant as information on PBS is often included in brief motivational interventions (Cronce and Larimer, 2011), and PBS may mediate the efficacy of some brief motivational interventions (Barnett et al., 2007; Larimer et al., 2007).

The Protective Behavioral Strategies Scale (PBSS: Martens et al., 2005) is the most widely-used and well-validated measure of PBS (Prince, Carey, & Maisto, 2013). The PBSS consists of 15 Likert-type items with response categories ranging from 1 “never” to 6 “always.” Factor analytic studies have supported a three-factor structure consisting of the following subscales: Stopping/Limiting Drinking (SLD), Manner of Drinking (MOD), and Serious Harm Reduction (SHR) (Martens et al., 2005; Martens, Pedersen, LaBrie, Ferrier, & Cimini, 2007). A number of studies have shown that scores on the PBSS are associated with less alcohol use and fewer alcohol-related problems (D’Lima, Pearson, & Kelley, 2012; LaBrie, Lac, Kenney, & Mirza, 2011; Martens, et al., 2005; Martens, et al., 2007).

The SHR subscale is particularly clinically relevant, given its specific focus on reducing significant negative consequences from drinking. One longitudinal study found that increases in SHR strategies are more predictive of reduced alcohol-related problems than changes in MOD or SLD (Martens et al., 2011). Another recent longitudinal study examining each subscale individually found that MOD is associated with reduced drinking/consequences whereas the SHR is specifically associated with reduced consequences (Napper, Kenney, Lac, Lewis, & Labrie, 2014). Despite its importance, the SHR subscale of the PBSS includes only three items: Use a designated driver, Make sure that you go home with a friend, and Know where your drink has been at all times. It is unlikely that the component items fully encompass the targeted construct of SHR strategies used by students. Put simply, the SHR subscale lacks content validity (Haynes, Richard, & Kubany, 1995).

The present study aimed to increase the content validity of the SHR subscale of the PBSS. Given that the PBSS is essentially a behavioral measure (i.e., list of behaviors used by college students to limit their drinking and avoid harms) used both for assessment and intervention, it is particularly crucial that the items be representative of the full range of the targeted behaviors (Foster & Cone, 1995). Other lists of PBS strategies have included eating before or during drinking (Delva et al., 2004), “hanging out” with trusted friends (Benton et al., 2004), and choosing not to “pre-game” (Sugarman & Carey, 2007). These and other potential protective strategies are not included in the PBSS, and thus, important SHR strategies are not provided as options in intervention trials using the PBSS measure.

Method

An initial pool of new items for the SHR subscale was created based on a review of the literature on college student drinking, existing lists of drinking control strategies, and a large pilot survey of college students recruited from introductory psychology courses at a large Midwestern university (N = 1832). An overly inclusive item pool was retained to promote broad content coverage. In the pilot survey, open-ended questions asked students to list three strategies they have used to avoid harm when using alcohol or “partying.” Thirteen non-redundant items were identified and evaluated by two focus groups for clarity and relevance to college-student drinking. Group consisted of five members of the target population (i.e., undergraduate students) and five graduate-student experts with training in brief interventions for college-student drinking. Group members were also asked to make suggestions for improvement of items that lacked clarity and to indicate any items that may have been missed. Focus group ratings and discussions with the authors led to the removal of one item, alteration of the wording of three items, and the addition of two items. The final set of 14 items is available from the first author.

A new sample of college students (N = 1713), also recruited from introductory psychology courses, completed a voluntary online survey for course credit. Students who had never consumed alcohol (more than a sip or taste) in their life (n = 278; 16.2%) and those who had not drank in the past year (n = 59; 3.4%) were excluded. The reduced sample of 1376 students included 797 women (57.9%); 1074 were freshmen (78.1%), with a mean age of 18.5 years (SD = 1.3); 1188 (86.3%) identified as White, 130 (9.4%) Black, 62 (4.5%) Asian, 19 (1.4%) Native American, and 23 (1.7%) indicated another racial identity. Percentages do not sum to 100% as participants were allowed to identify themselves in any or all of these categories. Additionally, 51 (3.7%) indicated Hispanic ethnicity. Of the pool of 1376 eligible students, 1206 (87.6%) completed a baseline survey only and 170 (12.4%) completed both a baseline survey and a follow-up survey 4–6 weeks later for the purposes of examining test-retest reliability and criterion-related validity. Students who self-selected into the follow-up survey received additional course credit. The remaining students (n = 1206) were divided into equal development and validation samples.

National Council on Alcohol Abuse and Alcoholism Task Force Recommended Alcohol Questions (NIAAA, 2003) were modified to assess drinking in the past month. These items are asked in a forced-choice response format, with higher numbers indicating greater drinking. The Young Adult Alcohol Consequences Questionnaire (YAACQ: Read, Kahler, Strong, & Colder, 2006) assessed eight different domains of negative alcohol-related consequences at follow up and was used to compare criterion-related validity of the current and revised SHR (see Figure 1 for descriptors). PBSS and drinking variables were assessed at baseline and follow up; consequences were only assessed at follow up.

Figure 1.

Standardized coefficients from a simultaneous, prospective test of the association of PBSS factors at baseline with drinking and related problems at follow up, controlling for drinking variables at baseline. For simplicity, item measurement model loadings and nonsignificant structural paths are not shown.

Results

Descriptive information about the alcohol use and consequences of the samples used in this development is given in Table 1.

Table 1.

Means (and Standard Deviations) of Alcohol Use and Consequence Variables

	Sample 1 n = 603	Sample 2 n = 603	Sample 3 Baseline n = 170	Sample 3 Follow-up n = 170
PBSS-20
Serious Harm Reduction	39.64 (7.84)	38.72 (8.04)	39.65 (8.29)	41.52 (6.51)
Manner of Drinking	17.57 (6.14)	17.30 (6.43)	18.12 (5.87)	18.25 (5.86)
Limiting/Stopping Drinking	24.33 (8.50)	23.63 (8.57)	24.84 (8.73)	24.71 (8.44)
Total	81.53 (19.14)	79.64 (19.79)	82.61 (19.43)	84.48 (17.64)
Drinking
Frequency	2.82 (1.81)	3.04 (1.88)	2.49 (1.67)	2.66 (1.56)
Quantity	3.23 (2.04)	3.37 (2.12)	2.77 (1.78)	3.30 (1.85)
Heavy (4+ men, 5+ women)	2.34 (1.85)	2.46 (1.88)	1.94 (1.74)	2.15 (1.86)
Consequences
Social-Interpersonal				2.04 (1.65)
Impaired Control				1.51 (1.51)
Self-Perception				0.86 (1.23)
Self-Care				1.26 (1.88)
Risk Behaviors				1.80 (1.90)
Academic-Occupational				0.61 (1.20)
Physical Dependence				0.36 (0.78)
Blackout Drinking				3.19 (2.33)

Note. PBSS = Protective Behavioral Strategies Scale

Sample 1 (n = 603): Development

Item Analysis

Item distributions, interitem correlations, change in alpha if item deleted, and item-total correlations were examined for all new SHR items. The item, Keep your cell phone with you, was highly positively skewed (skewness = 2.55) indicating high endorsement. Make sure you spend the night where you decide to party, and, Bring some form of contraception (e.g., condom) with you, had both low interitem correlations (ranging from .11 to .28) and relatively low item-total correlations, rs = .34 and .27, respectively (all other item-total correlations > .50). These three items were deleted.

Exploratory Factor Analysis (EFA)

An initial EFA for categorical outcomes was conducted in Mplus 6.12 (Muthén & Muthén, 1998–2010). Promax rotation and weighted-least-square mean and variance (WLSMV) estimation were used. Twenty-six items were included (15 original; 11 new). All original PBSS items loaded highest on their respective factors, with loadings from .58 to .83. The lowest-loading item was Drink shots of liquor, the only reverse-coded item of the scale. Eight new items loaded with the three previous SHR items, with loadings from .62 to .82. The item Keep track of how many drinks you have loaded almost equally on all three factors (SHR = .66, MOD = .66, SLD = .60). Avoid “pre-gaming” (i.e., drinking before going out) loaded highest on the MOD subscale (MOD = .74; SHR = .45). Pace the number of drinks you have per hour loaded almost equally on the MOD and SLD factors (.72 and .70, respectively) and low on SHR (.50). Factor correlations ranged from 0.40 to 0.51.

Confirmatory Factor Analysis (CFA)

Next, a series of CFAs were conducted to further inform decisions on item retention. As a preliminary step, the fit of the factor structure of the existing 15-item PBSS was examined; all item loadings were freely estimated and factor variances set to one for identification purposes (χ² = 831.24, df = 87; CFI = .92, TLI = .90, RMSEA = .12). Next, a model was fit including the eight new items that loaded on the SHR factor (χ² = 2450.11, df = 296; CFI = .86, TLI = .84, RMSEA = .12). The loading for Drink shots of liquor was low (.37), as in the EFA. Modification indices indicated that the item Avoid “pre-gaming” loaded on the MOD factor, as suggested by the EFA. In a subsequent model, Drink shots of liquor, Keep track of how many drinks you have, and Pace the number of drinks you have per hour were removed, and Avoid “pregaming” was modeled as an indicator of MOD rather than SHR (χ² = 1435.80, df = 227; CFI = .90, TLI = .89, RMSEA = .097). Modification indices indicated that Avoid combining alcohol with medication or other drugs was strongly correlated with Avoid combining alcohol with marijuana. In addition, Avoid drinking when sad, angry, or otherwise upset loaded highly on the MOD and SLD factors. Next, Avoid drinking when sad, angry, or otherwise upset, and Avoid combining alcohol with medication or other drugs were deleted (χ² = 1125.64, df = 186; CFI = .92, TLI = .91, RMSEA = .094). Modification indices indicated that Limit the amount of money you spend on alcohol loaded on the MOD and SLD factors. This item was removed, resulting in a final, best-fitting model (χ² = 952.73, df = 167; CFI = .93, TLI = .92, RMSEA = .091). Table 2 includes the factor loadings for this model.

Table 2.

Standardized Factor Loadings (Standard Errors), Internal Consistency Estimates and Fit Statistics of the PBSS-20 across Development, Validation, and Retest Samples

Item	Sample 1 n = 603	Sample 2 n = 603	Sample 3 n = 170
Serious Harm Reduction	α = .85	α = .86	α = .82
1. Use a designated driver	.67 (.03)	.58 (.04)	.53 (.07)
7. Make sure that you go home with a friend	.81 (.02)	.82 (.02)	.72 (.05)
8. Know where your drink has been at all times	.83 (.02)	.81 (.02)	.75 (.050
15. Refuse to ride in a car with someone who has been drinking	.70 (.03)	.74 (.03)	.80 (.04)
16. Only go out with people you know and trust	.82 (.02)	.81 (.02)	.79 (.05)
17. Avoid combining alcohol with marijuana	.66 (.03)	.72 (.03)	.59 (.07)
19. Make sure you drink with people who can take care of you if you drink too much	.78 (.02)	.87 (.02)	.82 (.04)
20. Eat before or during drinking	.64 (.03)	.65 (.03)	.68 (.06)
Limiting/Stopping Drinking	α = .86	α = .87	α = .88
2. Determine not to exceed a set number of drinks	.83 (.02)	.79 (.02)	.83 (.03)
3. Alternate alcoholic and nonalcoholic drinks	.77 (.02)	.74 (.02)	.79 (.03)
4. Have a friend let you know when you’ve had enough to drink	.76 (.02)	.74 (.02)	.76 (.04)
6. Leave the bar/party at a predetermined time	.80 (.02)	.83 (.02)	.84 (.02)
9. Stop drinking at a predetermined time	.78 (.02)	.84 (.01)	.87 (.02)
10. Drink water while drinking alcohol	.64 (.03)	.71 (.02)	.67 (.05)
11. Put extra ice in your drink	.55 (.03)	.61 (.03)	.58 (.05)
Manner of Drinking	α = .83	α = .85	α = .81
5. Avoid drinking games	.73 (.02)	.78 (.02)	.77 (.04)
12. Avoid mixing different types of alcohol	.76 (.02)	.81 (.02)	.70 (.04)
13. Drink slowly, rather than gulp or chug	.81 (.02)	.85 (.01)	.78 (.03)
14. Avoid trying to keep up or out-drink others	.77 (.02)	.76 (.02)	.79 (.03)
18. Avoid “pre-gaming” (i.e., drinking before going out)	.75 (.02)	.72 (.02)	.65 (.05)
Fit Statistics
χ2 (df = 167)	952.7	1087.7	365.84
RMSEA	.09	.10	.08
CFI	.93	.93	.94
TLI	.92	.92	.94

Note. RMSEA = root mean square error of approximation; CFI = comparative fit index; TLI = Tucker-Lewis index. Bold typeface denotes new items. Items are numbered in the order they should appear in the new 20-item measure.

Sample 2 (n = 603): Validation

Replication

The newly modified PBSS included five additional SHR items and substituted one new item for one old MOD item; none of the items of the SLD subscale were altered. An important step in providing evidence for validity of the modified PBSS was to replicate the best-fitting model on a new sample (Sample 2: χ² = 1087.67, df = 167; CFI = .93, TLI = .92, RMSEA = .099). Table 2 includes the factor loadings for this model. Internal consistency of the SHR and MOD subscales improved from the original measure (SHR_old: α = .71, 3 items; SHR_new: α = .86, 8 items; MOD_old: α = .79, 5 items; MOD_new: α = .83, 5 items).

Gender Invariance Analysis

Measurement invariance across men and women was tested in three steps: configural, scalar, and partial scalar invariance. All analyses were conducted in Mplus 6.12 with categorical indicators and a threshold structure, WLSMV estimation, and the delta parameterization. Invariance of factor loadings and thresholds were tested together, as the item probability curve is dependent on both parameters for multi-group CFAs in Mplus (Muthén & Muthén, 2010, p. 434). Per Asparouhov & Muthén’s (2006) guidelines for testing measurement invariance in Mplus, an initial multi-group CFA fixed scale factors to one and factor means to zero for both gender groups, while factor loadings, thresholds, variances, and covariances were allowed to vary across groups. This model was compared to one where factor loadings and thresholds were constrained across groups while scale factors were fixed to one and factor means to zero for men only. Models were compared with the DIFFTEST option as traditional chi-square difference tests are not accurate when WLSMV estimation is used in Mplus (Muthén & Muthén, 2010). A significant DIFFTEST, χ²_diff (97) = 134.38, p = .007, suggested that constraining parameters to equivalence across groups worsened model fit (i.e., lack of scalar invariance across gender). Next, the equality constraints of loadings and thresholds were relaxed item by item, and comparisons to the configural model were made using DIFFTEST. One old and one new SHR item exhibited lack of partial measurement invariance at p < .01 (see Table 3).

Table 3.

Partial Measurement Invariance of the 20-item PBSS Across Men and Women

	Men	Women	Chi-square difference test
Items	λ (SE)	λ (SE)	χ2diff	p
SLD
2.	.79 (.03)	.77 (.03)	3.56	.615
3.	.78 (.02)	.70 (.03)	3.68	.596
4.	.76 (.03)	.66 (.04)	14.36	.014
6.	.80 (.02)	.83 (.02)	2.98	.702
9.	.83 (.02)	.83 (.02)	2.07	.839
10.	.72 (.03)	.71 (.03)	8.05	.154
11.	.66 (.03)	.58 (.04)	7.82	.166
MOD
5.	.77 (.03)	.77 (.03)	3.78	.582
12.	.80 (.02)	.80 (.03)	3.34	.647
13.	.86 (.02)	.83 (.02)	1.68	.892
14.	.77 (.03)	.70 (.03)	11.76	.038
18.	.69 (.04)	.76 (.03)	14.07	.015
SHR
1.	.57 (.05)	.59 (.05)	7.36	.195
7.	.80 (.03)	.76 (.04)	14.67	.012
8.	.78 (.03)	.74 (.04)	16.75	.005
15.	.73 (.03)	.73 (.04)	6.49	.261
16.	.76 (.03)	.78 (.03)	9.08	.106
17.	.67 (.04)	.73 (.04)	11.27	.046
19.	.83 (.02)	.87 (.03)	3.90	.564
20.	.64 (.04)	.67 (.04)	15.74	.008

Note. λ = standardized factor loadings; SE = standard error; SLD = Stopping/Limiting Drinking; MOD = Manner of Drinking; SHR = Serious Harm Reduction; χ2diff = Chi-square difference test using DIFFTEST.

Sample 3 (n = 170): Test-Retest Reliability and Criterion-Related Validity

Test-Retest Reliability

Pearson correlations examined consistency of scores of the new, 20-item PBSS from scale administrations approximately one month apart. Correlations between scores at time 1 and time 2 were: SHR r = .67; MOD r = .64; SLD r = .59; PBSS Total Score r = .67. All correlations were significant, ps < .001, supporting stability of scores over time.

Criterion-Related Validity

Criterion-related validity was tested in two ways: (1) we tested whether the 8-item SHR outperformed the 3-item SHR in the concurrent prediction of alcohol use and related negative consequences at follow up, and (2) we tested how the 8-item SHR factor compared to the other PBSS factors when included in a model that prospectively predicted drinking behaviors and consequences.

Concurrent

Table 4 presents results from path models relating the original, 3-item SHR subscale and revised, 8-item SHR subscale to drinking frequency, quantity, and heavy episodic (binge) drinking as well as eight alcohol consequences. Applying a Bonferroni correction for multiple tests, the revised SHR was significantly associated with 7 of 8 alcohol-related consequence domains, where the original SHR was significantly associated with only self-care and academic-occupational domains. The revised SHR was also significantly related to binge drinking, where the original SHR was not. With regard to the MOD subscale, the original and revised versions were both significantly related to all drinking outcomes and consequences, except self-perception.

Table 4.

Criterion-related Validity of Revised 8-item Serious Harm Reduction Subscale in Relation to Heavy Drinking and Related Harms

	3-item		8-item
Outcome	β	95% CI	β	95% CI
Drinking
Frequency	−.08	−.40, .18	−.15	−.08, .01
Quantity	−.07	−.51, .25	−.14	−.10, .01
Heavy (4+ men, 5+ women)	−.10	−.57, .14	−.18*	−.12, −.01
Consequences
Social-Interpersonal	−.21*	−.68, −.10	−.31**	−.13, −.05
Impaired Control	−.16	−.54, .01	−.28**	−.11, −.03
Self-Perception	−.08	−.32, .10	−.14	−.07, .01
Self-Care	−.28**	−.88, −.25	−.34**	−.16, −.06
Risk Behaviors	−.26*	−.94, −.19	−.40**	−.19, −.08
Academic-Occupational	−.23**	−.49, −.13	−.29**	−.09, −.03
Physical Dependence	−.29*	−.41, −.13	−.33**	−.07, −.02
Blackout Drinking	−.14	−.72, .00	−.28**	−.17, −.05

Note. β = standardized path estimate; CI = bootstrapped confidence intervals.

^*p < .05.

^**p < .002 (Bonferroni-corrected significance level).

Prospective

Figure 1 shows an SEM model with drinking outcomes and alcohol-related problems assessed at follow up regressed on PBSS factors assessed at baseline, controlling for baseline drinking. Outcomes were regressed on latent subscale factors formed from final items listed in Table 2 (individual items not pictured in figure). Over and above other subscales and controlling for the effect of baseline drinking, the revised SHR was the only subscale factor to be significantly negatively associated with alcohol-related negative consequences. The SHR was not significantly associated with drinking outcomes over and above the influence of other PBSS factors and baseline drinking.

Discussion

This study improved the content validity of the Serious Harm Reduction subscale of the most frequently used measure of protective behavioral strategies. We undertook a stepwise process of measure refinement (Smith & McCarthy, 1995) to improve the breadth of content coverage of this scale. Results of this process produced the PBSS-20, with an expanded SHR scale of 8 items. In addition to improved content coverage, the internal consistency improved over the original, and we examined test-retest reliability. The revised scale also demonstrated improved criterion-related validity over the previous version. It was associated with a broader range of alcohol-related negative consequences, as evidenced by significant, concurrent associations with 7 of 8 consequence domains at follow-up and prospective associations with 5 of 8 consequences, accounting for baseline drinking and other PBSS factors. To our knowledge, this is also the first study to disaggregate alcohol-related consequence domains when examining the relative criterion-related validity of the PBSS subscales.

This study has limitations to be addressed in future psychometric work. First, we used traditional latent variable models (e.g., EFA, CFA) to be consistent with the existing conceptualization of PBS use as a trait-like construct, akin to neuroticism or empathy. However, this list of behaviors could also be conceptualized as causal indicators of PBS, just as income and education level may be causal indicators of SES (see Bollen & Bauldry, 2011). Similarly, consistent with prior studies of the PBSS, we specified a three-factor structure for the PBSS-20, and results suggested some lack of fit (RMSEA values > .08). Future studies are needed to examine the utility of revising the measurement model (causal vs. effect indicators) and/or the factor structure of the PBSS. Next, we found lack of partial measurement invariance for the SHR items Know where your drink has been at all times and Eat before or during drinking. These findings suggest that differences in endorsement of these items may not be due to true differences in PBS use across gender groups, but that these items have a different conceptual meaning or relevance for college men and women (see Treloar, Martens, & McCarthy, 2014). Last, our removal of the only reverse-coded item for fit issues precludes an easy check for a yea-saying response style.

These improvements to the measurement of PBS can have important clinical implications, as several efficacious interventions incorporate PBS assessment. PBS can be used to provide personalized feedback to students about those strategies that they currently do/do not regularly use and encouragement to consider implementing strategies not often used (Martens, Smith, & Murphy, 2013). Whether used for informative purposes, as personalized assessment feedback, or as a focal outcome measure of change, those who use the PBSS will benefit from the expanded breadth of SHR content and improved psychometric properties of the PBSS-20.