The Rutgers Alcohol Problem Index: Measurement Equivalence among College Students in the U.S. and Mexico

Abstract

The Rutgers Alcohol Problem Index (RAPI) is widely used to assess alcohol-related problems among college students within the U.S. and internationally. Despite its wide usage, whether the RAPI similarly assesses alcohol-related problems among students in different countries has not been established. We begin to address this issue by evaluating responses to the RAPI for measurement equivalence across college students in the U.S. (European Americans and Mexican Americans, treated as separate groups) and Mexico (Mexicans). Towards this end, we evaluated the RAPI for Differential Item Functioning (DIF) within an item response theory framework. Our results showed DIF for six item severities, all but one of which differed as a function of country (U.S. vs. Mexico). Additional analyses showed that using a latent RAPI variable with no DIF assumed had no substantive consequences in terms of group mean differences and zero-order correlations with self-reported drinking behaviors. Similarly, when using observed RAPI scale scores, there were no substantive differences in terms of correlations. The observed scale scores, however, led to inaccurate mean comparisons. Based on our results, we recommend that scholars model the RAPI as a latent variable when conducting analyses.

Alcohol use is common among college students and represents a significant public health concern (National Institute on Alcohol Abuse and Alcoholism, 2014). As such, the accurate assessment of alcohol-related problems among college students is of particular importance – for example, to estimate the prevalence of alcohol-related problems or evaluate the effectiveness of alcohol prevention and intervention efforts. The Rutgers Alcohol Problem Index (RAPI; White & Labouvie, 1989) is commonly used for these purposes and has become one of the most widely used measures in studies that focus on alcohol use and alcohol-related problems among college students (Devos-Comby & Lange, 2008).

The RAPI is a self-report measure that assesses 23 negative consequences that can result from one’s alcohol use. The RAPI items are provided in Table 1. Participants respond to the items on a 5-point scale, anchored by 0 (never) to 4 (more than 10 times; White & Labouvie, 1989). Due to low endorsement of the higher response options a dichotomous scoring scheme is often used (i.e., never vs. 1 or more times), especially for studies that evaluate the measurement properties of the RAPI (Cohn, Hagman, Graff, & Noel, 2011; Neal, Corbin, & Fromme, 2006). The time-frame specified in the RAPI instructions varies across studies – for example, when used to evaluate program effectiveness, the time-frame is often based on elapsed time since the end of the intervention.

Table 1.

Items from the Rutgers Alcohol Problem Index

1. Not able to do your homework or study for a test*

2. Got into fights with other people (friends, relatives, strangers)

3. Missed out on other things because you spent too much money on alcohol

4. Went to work or school high or drunk

5. Caused shame or embarrassment to someone

6. Neglected your responsibilities*

7. Relatives avoided you

8. Felt that you needed more alcohol than you used to in order to get the same effect

9. Tried to control your drinking (tried to drink only at certain times of the day or in certain places, that is, tried to change your pattern of drinking)

10. Had withdrawal symptoms, that is, felt sick because you stopped or cut down on drinking

11. Noticed a change in your personality

12. Felt that you had a problem with alcohol

13. Missed a day (or part of a day) of school or work*

14. Wanted to stop drinking but couldn’t*

15. Suddenly found yourself in a place that you could not remember getting to

16. Passed out or fainted suddenly*

17. Had a fight, argument or bad feeling with a friend

18. Had a fight, argument or bad feeling with a family member

19. Kept drinking when you promised yourself not to

20. Felt you were going crazy

21. Had a bad time*

22. Felt physically or psychologically dependent on alcohol

23. Was told by a friend, neighbor or relative to stop or cut down drinking

Note: Items that demonstrated DIF are identified with an asterisk (*). The specific patterns of differences are reported in Appendix B and shown in Figure 2.

Of particular relevance to the present study, the RAPI has been used with college students in the U.S., where the measure was developed, and internationally (e.g., Baer, Kivlahan, Blume, McKnight, & Marlatt, 2001; López-Núñez, Fernández-Artamendi, Fernández-Hermida, Álvarez, & Secades-Villa, 2012; Orona, Blume, Morera, & Perez, 2007; Skewes, Dermen, & Blume, 2011). Although some efforts have been made to verify that the items of the RAPI function similarly in assessing alcohol-related problems among women and men (Cohn et al., 2011; Earleywine, LaBrie, & Pedersen, 2008; Neal et al., 2006), no published studies have verified that the items of the RAPI function similarly across countries. To the extent that the items of the RAPI function differently across countries, the results of data analyses focusing on country-based associations (e.g., group mean differences) can be biased and potentially misleading (Chen, 2008; Liu, Magnus, & Thissen, 2015; Tay, Meade, & Cao, 2015; Vandenberg & Lance, 2000). In order to address this issue, we evaluated the measurement properties of the RAPI among college students in the U.S. and Mexico. Our U.S. sample included both European Americans and Mexican Americans, which were considered as separate groups in our analyses. Thus, we evaluated the items of the RAPI for measurement equivalence among European American and Mexican American college students in the U.S. and Mexican college students in Mexico. We conducted our analyses using item response theory (IRT), which we briefly describe.

Item Response Theory

IRT is a class of analytic approaches used to model and evaluate measures with ordinal or nominal response options (de Ayala, 2009; Embretson & Reise, 2000). In the present study, responses to the RAPI items were dichotomized (i.e., never vs. 1 or more times) and we used a 2-parameter logistic (2-PL) IRT model, which is appropriate for dichotomously scored items. For the 2-PL model, responses to items are modeled as a logistic function of the underlying construct that the items ostensibly tap (e.g., alcohol-related problems). This model provides details on two key measurement parameters: item severities and item discriminations. These parameters can be illustrated via item characteristic curves (ICCs), which visually depict how changes in the levels of an underlying construct (e.g., alcohol-related problems) relate to the probability of giving specific responses to the items of a measure (e.g., no vs. yes). The ICCs for a hypothetical 3-item measure with no vs. yes response options are shown in Figure 1. In the figure, the continuum of the underlying construct is labeled theta (θ), which, for illustrative purposes, has a mean of 0 and standard deviation of 1. Theoretically, individuals may be placed somewhere along theta – that is, they have a specific level of the underlying construct (e.g., alcohol-related problems). The points at which the curves meet the dotted line represent the item severities and the steepness of the curves at their inflection points represent the item discriminations.

Figure 1.

Item characteristic curves for hypothetical 3-item measure with no/yes response options

For dichotomous response options, the severity parameter indicates the point along theta where an individual with that specific level of theta (e.g., the specific level of alcohol-related problems) has a .50 probability of giving a yes (vs. no) response. Correspondingly, individuals with lower levels of theta (e.g., less severe alcohol-related problems) are more likely to provide a response of no while individuals with higher levels of theta (e.g., more severe alcohol-related problems) are more likely to provide a response of yes. To illustrate, in Figure 1, item 2 has a higher severity than item 1. This indicates that, relative to item 1, a yes response to item 2 would be indicative of a higher level of the underlying construct (e.g., more severe alcohol-related problems). The discrimination parameter indicates how accurately a given item differentiates between individuals who are below and above the severity estimate. In Figure 1, the severities for items 2 and 3 are the same, but the discrimination is stronger for item 2. This indicates that, relative to item 3, item 2 would more accurately differentiate between individuals who are below and above the same level of theta.

Measurement Equivalence within IRT

The primary goal of any self-report measure is to provide estimates of participants’ scores on some underlying construct. In testing group-based associations (e.g., group mean differences, moderation effects), problems can arise if the psychometric properties of a measure are not equivalent across the groups being compared – that is, if the items of a measure function differently across the groups. For example, if a yes response to a given item indicates a high level of alcohol-related problems for women but a moderate level for men, failing to take this difference into account can lead to inaccurate estimates of participants’ scores on the underlying construct, and ultimately affect tests of group-based associations. Within the IRT literature, an item that functions differently across groups is said to demonstrate group-based differential item functioning (DIF).

Present Study

No published studies have tested for DIF in the RAPI among college students across countries. Given the widespread use of the RAPI among college students, both within the U.S. and internationally, analyzing potential measurement bias in the RAPI across countries is necessary to insure accurate results. We begin to address this issue by testing the items of the RAPI for DIF as a function of country and, for our U.S. participants, ethnicity. We focused specifically on three groups: European Americans and Mexican Americans in the U.S. and Mexicans in Mexico.

For our analyses, we used a 2-PL IRT model and tested for DIF in the item severities and discriminations for the three groups. Using the latent RAPI variable with any identified DIF taken into account, we then tested for group mean differences in alcohol-related problems (i.e., latent RAPI scores). We conducted additional analyses to examine whether any substantive bias was introduced if DIF is not taken into account (i.e., using a latent IRT variable with no DIF assumed) and, to be comprehensive, when using observed scale scores (i.e., summed responses to the RAPI items). For the former, we tested for group mean differences using a latent variable with all of the measurement parameters constrained to be equivalent across the groups. Further, we estimated the zero-order associations between the three RAPI variables and three drinking behaviors (i.e., average drinks/day, average drinking days/week, and average binge drinking days/week), tested whether the associations between the RAPI and drinking behaviors were moderated by country/ethnicity, and evaluated whether the magnitudes of the associations differed as a function of the RAPI variable used.

Method

Participants and Procedure

Undergraduate students at four universities in Texas, U.S., and two universities in Chihuahua, Mexico, were recruited for the study via flyers, class announcements, or subject pools; all participants freely chose to participate in this study. Interested students were given a personal password and a link to a brief (~ 2 min) online screening survey to determine their eligibility to be in the study. Eligible participants were 18–25 years in age; spoke English, Spanish, or both as their primary language; and self-identified as either European American/White (U.S.), Mexican/Mexican American (U.S.), or Mexican (Mexico).

Eligible participants then provided their informed consent to be in the study and completed an online survey, which took approximately 50 minutes; participants were free to choose where they completed the study materials. The survey was programmed using Qualtrics (http://www.qualtrics.com) and was hosted on a secure server at the University of Missouri. In exchange for their time, the participants received a small cash payment, a ticket to a cash prize lottery drawing, research participation credit, or extra credit in one of their courses; the incentive differed by university as per institutional norms for research participation. Other than university affiliation, no personally identifying information was collected for this study.

All of the U.S. participants spoke and completed the study materials in English. All of the Mexico participants spoke and completed the study materials in Spanish. All measures were translated from English to Spanish and back-translated from Spanish to English by two Mexico-born, native Spanish speakers who completed their doctoral training in the U.S. Discrepancies in translations were resolved via discussions among the research team. This study was conducted in compliance with the ethical standards outlined by the American Psychological Association (2010) and was approved by the ethics committee at the University of Missouri, which served as the coordinating institution.

Our sample included 1,434 undergraduate college students between the ages of 18 and 25 who reported that they had an alcoholic beverage at least once “since the beginning of the school year.”¹ The sample consisted of 525 European Americans (M age = 19.57, SD = 1.67; 59.8% women; 98.3% US-born) and 563 Mexican Americans (M age = 20.16, SD = 1.95; 68.4% women; 86.5% US-born) in the U.S., and 346 Mexicans (M age = 20.26, SD = 1.77; 61.7% women; 99.7% Mexico-born) in Mexico.² To conserve space, we hence forth use the following abbreviations: EA = European American, MA = Mexican American, and MX = Mexican.

Measures

Of specific relevance to the present paper, participants completed the 23-item RAPI (White & Labouvie, 1989), which included the following instructions: Thinking about your drinking experiences since the beginning of the school year, how many times did the following things happen to you as a result of drinking? There was no to low endorsement of the higher response options for one or more of the three groups on several of the items. We therefore dichotomized the responses, with 0 indicating never and 1 indicating one or more times. Observed scale scores were computed by summing across the responses to the 23 dichotomously scored items.

The participants also reported their typical daily drinking behavior, “since the beginning of the school year,” for each day of the week separately (Collins, Parks, & Marlatt, 1985). Based on these responses we computed three variables. First, we computed the average drinks consumed on a typical day by averaging across the responses for the 7 days of the week (average drinks/day). Because very few participants reported having more than 12 drinks on any given day, we truncated the individual daily drinking variables at 12 before computing the average drinks/day scores. Second, we computed a variable to reflect the average number of days per week that participants had at least one alcoholic beverage (average drinking days/week). Finally, we computed a variable to reflect the average number of days per week that participants engaged in binge drinking behavior (average binge drinking days/week – that is, 4 or more alcoholic beverages on a single day for women and 5 or more for men; U.S. Department of Health and Human Services and U.S. Department of Agriculture, 2015).

Descriptive statistics for the study variables, separated by country/ethnicity, are provided in Table 2. As can be seen, each of the variables had at least moderate levels of skewness and/or kurtosis for one or more of the groups. We thus log-transformed the variables, after adding a constant of 1, for use in our analyses.

Table 2.

Descriptive Statistics for Study Variables

	Mean	SD	Minimum	Maximum	Skewness	Kurtosis	KR-20
Rutgers Alcohol Problem Index (Sum)
European Americans	4.82	5.56	0	23	1.58	2.21	.93
Mexican Americans	4.76	6.13	0	23	1.69	2.20	.95
Mexicans	4.45	4.91	0	23	1.44	2.01	.90
Average Drinks/Day
European Americans	1.20	1.32	0	9.14	1.81	4.46	--
Mexican Americans	.86	.95	0	5	1.64	2.91	--
Mexicans	1.30	1.54	0	12	2.26	7.97	--
Average Drinking Days/Week
European Americans	1.99	1.55	0	7	.89	1.29	--
Mexican Americans	1.66	1.38	0	7	1.10	2.13	--
Mexicans	1.75	1.38	0	7	1.14	2.02	--
Average Binge Drinking Days/Week
European Americans	.93	1.28	0	7	1.49	2.84	--
Mexican Americans	.60	.93	0	4	1.37	.85	--
Mexicans	.89	1.14	0	7	1.43	4.47	--

Notes: SD = standard deviation; minimum = smallest observed value; maximum = largest observed value; KR-20 = Kuder-Richardson formula 20 coefficient.

Analytic Strategy

All analyses were conducted within Mplus Version 6.1 (Muthén & Muthén, 1998) using maximum likelihood estimation with robust standard errors and the expectation maximization algorithm to account for the <.001% of item-level missing data (Enders, 2010). We used the Huber-White sandwich estimator (Huber, 1967; White, 1980) to account for the non-independence of the data due to the nesting of participants within universities. Our use of the Huber-White sandwich estimator required us to conduct our tests of DIF by estimating a series of 2-PL IRT latent class mixture models with three known classes, defined by country/ethnicity (i.e., EAs, MAs, and MXs). For descriptive purposes, the percent of participants in each group who reported experiencing the 23 problems included in the RAPI are provided in Table 3.

Table 3.

Percentage of Cases that Reported Experiencing the 23 Problems Included in the RAPI

	European American (n = 525)	Mexican American (n = 563)	Mexicans (n = 346)
Item 1	41.3	35.5	29.5
Item 2	27.0	24.9	24.3
Item 3	19.2	20.3	34.3
Item 4	22.0	19.4	19.1
Item 5	27.3	26.6	34.7
Item 6	44.2	38.9	28.4
Item 7	5.1	9.3	7.8
Item 8	25.9	23.5	16.8
Item 9	29.1	24.7	20.8
Item 10	9.3	12.1	9.0
Item 11	26.1	24.3	33.8
Item 12	13.0	12.9	11.3
Item 13	32.0	29.5	19.4
Item 14	21.3	25.9	35.8
Item 15	22.9	23.3	22.0
Item 16	14.1	14.9	4.6
Item 17	24.4	25.3	27.5
Item 18	8.4	13.1	12.4
Item 19	18.7	21.9	18.2
Item 20	13.1	14.6	7.5
Item 21	35.8	24.5	29.6
Item 22	7.2	10.7	8.1
Item 23	10.3	14.9	13.9

Note: n = sample size.

Differential item functioning.

For our tests of DIF, we followed the procedures suggested by Tay et al. (2015). Tay et al.’s procedures were developed to test for DIF when an anchor item is not known. For the first step, we estimated a model in which all of the item discriminations and severities were constrained to be equivalent across the three groups; the latent variable mean and variance for EAs (our arbitrarily selected reference group) were fixed to 0 and 1, respectively (for model identification); and the latent variable means and variances for MAs and MXs were allowed to vary. For the second step, we estimated a model in which the item discriminations and severities were allowed to vary for all three groups; the latent variable mean and variance for EAs were fixed to 0 and 1, respectively (for model identification); and the latent variable means and variances for MAs and MXs were fixed using the respective parameter estimates obtained in the first model (unconstrained model). For the third step, we estimated 46 models, each with a single parameter (i.e., discrimination or severity) constrained to be equivalent across the three groups (iteration 1 models). A parameter was identified as demonstrating DIF if the constraint resulted in a poorer fitting model, relative to the unconstrained model. For the fourth step, we estimated a model in which the non-DIF parameters were constrained to be equivalent across the groups; the DIF parameters were allowed to vary across the groups; the latent variable mean and variance for EAs were fixed to 0 and 1, respectively (for model identification); and the latent variable means and variances for MAs and MXs were allowed to vary (final iteration 1 model). For the fifth step, we constrained the previously identified DIF parameters, one at a time, to be equivalent across the groups (iteration 2 models). We continued this process until all non-DIF parameters were identified. Readers are referred to Tay et al. (2015) for a detailed description of this approach.

We evaluated the relative fit of each successive model using changes (Δ) in Consistent Akaike Information Criterion (CAIC; Bozdogan, 1987), Bayesian Information Criterion (BIC; Schwarz, 1978), and Sample-size Adjusted Bayesian Information Criterion (ABIC; Sclove, 1987) values. Information criteria values do not provide details regarding overall model fit but are appropriate for model fit comparisons, with lower values indicating better model fit. Raftery (1995) suggested that BIC changes of 0–2 provides weak evidence of model fit differences, changes of 2–6 provides positive evidence, changes of 6–10 provides strong evidence, and changes over 10 provides very strong evidence. We followed these suggestions, and determined that a model resulted in a drop in model fit if constraining a parameter led to a CAIC, BIC, or ABIC increase of 2 or more.

Substantive analyses.

After conducting our DIF analyses, we tested for group mean differences using a latent IRT variable with any DIF taken into account (RAPI-DIF variable), a latent IRT variable with all measurement parameters fixed to be equivalent across the groups (RAPI-EQ variable), and observed scale scores (RAPI-OS variable). We then estimated the zero-order correlations between the three RAPI variables and the three drinking variables, tested whether the associations were moderated by country/ethnicity, and evaluated whether the magnitudes of the associations differed as a function of the RAPI variable used.

Based on a data simulation study in which different ways to proceed with data analyses when DIF is detected were evaluated, Cho, Suh, and Lee (2016) concluded that a single-factor multi-group approach (used for our DIF analyses) provided the most accurate estimates of measurement parameters in nearly all of the conditions they considered (e.g., uniform DIF with few DIF parameters). A purification approach, however, was less biased when considering substantive questions; specifically, group differences in latent variable means and variances. Correspondingly, for our mean group comparisons, we used a multi-dimensional (i.e., multi-factor) purification approach recommended by Cho and her colleagues, in which all of the measurement parameters (i.e., item discriminations and severities) are constrained to be equivalent across groups, while additional latent variables are model in order to account for any observed DIF. The specifications for this model are described in the Results section.

Results

Differential Item Functioning

In order to reduce clutter, the information criteria and changes in information criteria values for all model comparisons in this paper are provided in Appendix A. For the iteration 1 models, eight of the item severities, but none of the item discriminations, evidenced DIF. Our final iteration 1 model, in which all of the discriminations and the 15 non-DIF severities were constrained to be equivalent across the groups, did not result in a poorer fitting model, relative to the unconstrained model. For our iteration 2 models, six of the eight item severities continued to demonstrate DIF. Our final iteration 2 model, in which the 2 additional severities were constrained to be equivalent across the groups, did not result in a poorer fitting model, relative to the final iteration 1 model. In our iteration 3 models, evidence of DIF remained for the six item severities when they were constrained to be equivalent across the three groups. We subsequently estimated models with the item severities constrained to be equivalent for each two-group pairing (i.e., EA = MA, EA = MX, and MA = MX).

As shown in Appendix A (iteration 3 models), the severities for items 1, 6, 13, 14, and 16 demonstrated equivalence for EAs and MAs while the severity for item 21 demonstrated equivalence for MAs and MXs. Placing this pattern of constraints on the severities did not result in a poorer fitting model (final iteration 3 model), relative to the final iteration 2 model. Placing any further constraints on the severities resulted in a poor fitting model (see iteration 4 models). The final iteration 3 model was thus selected as our RAPI-DIF model, and was used in our subsequent analyses.

The unstandardized (i.e., logit), IRT, and standardized parameter estimates from our RAPI-DIF model are included in Appendices B-D, respectively. Within Mplus, when model constraints are added, they are a placed on logit coefficients, while the IRT and standardized parameters estimates are calculated post-hoc. As such, the equality constraints are only reflected in the unstandardized logit coefficients (Appendix B). The standardized estimates are reported as they may be more intuitive for readers who are more familiar with measurement analyses within a confirmatory factor analytic framework (e.g., standardized item discriminations are conceptually similar to standardized factor loadings). For illustrative purposes, the ICCs for the six DIF items are depicted in Figure 2 using the standardized coefficients.

Figure 2.

Item characteristic curves for items that demonstrated differential item functioning

Note: EA = European Americans; MA = Mexican Americans; MX = Mexicans; theta = continuum of latent construct.

As can be seen in Figure 2, the severities for items 1 (i.e., not able to do your homework or study for a test), 6 (i.e., neglected your responsibilities), 13 (i.e., missed a day [or part of a day] of school or work), and 16 (i.e., passed out or fainted suddenly) were higher for MXs relative to EAs and MAs; the severity for item 14 (i.e., wanted to stop drinking but couldn’t) was higher for EAs and MAs relative to MXs; and the severity for item 21 (i.e., had a bad time) was higher for MAs and MXs relative to EAs.

Group Mean Differences

We tested for group mean differences in alcohol-related problems using the RAPI-DIF, RAPI-EQ (IRT variable with all measurement parameters constrained to be equivalent across the groups), and RAPI-OS variables. As noted in the Method section, the RAPI-OS variable was logged-transformed (after adding a constant of 1), which we further transformed into Z-scores to place the values on an easily interpretable metric. Moreover, as noted in the Analytic Strategy, for the RAPI-DIF variable, we used the multi-factor purification approach suggested by Cho and her colleagues (2016). Towards this end, a three-factor model was estimated. The first factor included all 23 RAPI items as indicators, with all of the item severities and discriminations constrained to be equivalent across the three groups (RAPI variable). Because all of the severity parameters for MAs in our final DIF model were constrained to be equivalent to EAs (items 1, 6, 13, 14, and 16), MXs (item 16), or EAs and MXs (all other item severities), we specified MAs as the reference group for the latent RAPI variable. Correspondingly, for model identification the latent variable mean and variance were fixed to 0 and 1, respectively, for MAs, and allowed to vary for EAs and MXs.

For EAs, the second factor (DIF Factor 1) included item 16 as an indicator, with the latent variable mean and variance fixed to 0 and 1, respectively; the remaining 22 indicators were fixed to 0. For MXs, the third factor (DIF Factor 2) included items 1, 6, 13, 14, and 16 as indicators, with the latent variable mean and variance fixed to 0 and 1, respectively; the remaining 18 indicators were fixed to 0. For DIF Factor 1, the 23 indicators, latent variable mean, and latent variable variance all were fixed to 0 for MAs and MXs. For DIF Factor 2, the 23 indicators, latent variable mean, and latent variable variance all were fixed to 0 for EAs and MAs. Readers are referred to Cho et al. (2016) for further details on this approach.

The group mean differences using the (purified) RAPI-DIF, RAPI-EQ, and RAPI-OS variables are provided in Table 4, with MAs as the reference group. We used the Wald’s test to compare the mean difference between EAs and MXs, and note that the results were substantively equivalent when EAs or MXs were specified as the reference group. As shown in the table, the results for the RAPI-DIF and RAPI-EQ variables were virtually identical. For both of these models, the only statistically significant mean difference was between MAs and MXs, with the latter having higher scores than the former. There were no statistically significant differences when using the RAPI-OS variable.^3/4

Table 4.

Mean Comparisons (Reference Group = Mexican Americans)

Model	b	SE	p	β	Wald χ2	df	p
RAPI-DIF
European Americans	.09	.09	.29	.12
Mexicans	.10	.04	.01	.13	<.01	1	.96
RAPI-EQ
European Americans	.10	.09	.26	.12
Mexicans	.10	.04	.02	.14	<.01	1	.98
RAPI-OS
European Americans	.09	.11	.41	.04
Mexicans	.04	.05	.41	.02	.36	1	.55

Note: RAPI = Rutgers Alcohol Problem Index; RAPI-DIF = latent RAPI variable with differential item functioning taken into account; RAPI-EQ = latent RAPI variable with all measurement parameters constrained to be equivalent across groups; RAPI-OS = observed RAPI scale scores; b = unstandardized coefficient; SE = standard error; p = two-tailed probability value; β = standardized coefficient; χ² = chi-square; df = degrees of freedom; Wald χ² test evaluates whether the regression/path coefficients differ for European Americans and Mexicans (i.e., a mean difference test).

Zero-order Correlations

We estimated the zero-order correlations between the three drinking variables and the three RAPI variables, with a separate model for each RAPI variable. In order to evaluate whether the correlations differed as a function of country/ethnicity (i.e., a moderation effect), we first estimated an unconstrained model in which the correlations between the target RAPI variable and the three drinking variables were allowed to vary across the groups. One at a time, we next constrained the correlation between the RAPI variable and the drinking variables to be equivalent across the groups, resulting in three constrained models. As shown in Appendix A (tests of moderated correlations), none of the model constraints resulted in a poorer fitting model, indicating that country/ethnicity did not moderate any of the correlations. This was the case for all three RAPI variables.

As shown in Table 5, each RAPI variable was positively and strongly correlated with average drinks/day, average drinking days/week, and average binge drinking days/week. The correlations were identical for the RAPI-DIF and RAPI-EQ variables, which in turn were slightly stronger than they were for the RAPI-OS variable. In order to evaluate whether the magnitudes of the associations for the RAPI-OS variable were significantly lower than the associations for the RAPI-DIF variable, we estimated our RAPI-DIF model again, with the correlations constrained to be equivalent across the groups (baseline model). We compared the fit of this model to models in which the magnitude of the correlations were, one at a time, fixed to be equivalent to the magnitude of the correlations for the RAPI-OS models (fixed models). As shown in Appendix A (RAPI-DIF with fixed coefficients), fixing the correlations did not result in a poorer fitting model, indicating that the RAPI-OS variable did not significantly underestimate the correlations. We should note, however, that using a non-log-transformed RAPI-OS variable led to significantly weaker correlations. The results of these analyses are available upon request.

Table 5.

Zero-order Correlations for Final Models

	RAPI Variable
Drinking Variable	RAPI-DIF	RAPI-EQ	RAPI-OS
Average Drinks/Day	.55	.55	.52
Average Drinking Days/Week	.51	.51	.48
Average Binge Drinking Days/Week	.46	.46	.44

Note: RAPI = Rutgers Alcohol Problem Index; RAPI-DIF = latent RAPI variable with differential item functioning taken into account; RAPI-EQ = latent RAPI variable with all measurement parameters constrained to be equivalent across groups; RAPI-OS = observed RAPI scale scores; all correlations are statistically significant at p < .001.

Discussion

For the present study we tested a dichotomously scored version of the 23-item RAPI for differential item functioning (DIF) as a function of country/ethnicity among college students in Texas, U.S. (European Americans [EAs] and Mexican Americans [MAs]), and Chihuahua, Mexico (Mexicans [MXs]). Our results showed evidence of partial DIF in six item severities, but no DIF in the item discriminations. The specific pattern of the DIF can be seen in Table 5 and Figure 2 (items provided in Table 1). Unlike prior studies (Earleywine et al., 2008; Neal et al., 2006), we conducted additional analyses to consider whether there were substantive consequences (i.e., differences in tests of mean group differences and moderated associations) when measurement equivalence was assumed (i.e., constrained measurement parameters without testing and accounting for DIF; RAPI-EQ variable) and, to be comprehensive, when summed observed scale scores were used (RAPI-OS variable). For these analyses, we assumed that the latent IRT variable with DIF taken into account (RAPI-DIF variable) provided the most accurate estimates of alcohol-related problems; thus, the results for the RAPI-EQ and RAPI-OS variables were compared to the results for the RAPI-DIF variable.

For the RAPI-DIF variable, our results showed that MXs had significantly higher alcohol-related problems than their MA counterparts, and that alcohol-related problems were positively and strongly correlated with self-reported average drinks/day, average drinking days/week, and average binge drinking days/week. The results were virtually (and in most cases numerically) identical for the RAPI-EQ variable. In contrast, for the RAPI-OS variable, no statistically significant mean differences in alcohol-related problems were identified. The correlations between the RAPI-OS and the three drinking variables, however, were statistically equivalent to correlations using the RAPI-DIF variable. We note that some scholars may not consider the comparison between the RAPI-DIF and RAPI-OS variables to be appropriate, as the latter, but not the former, includes measurement error. Nonetheless, we find these results to be important as the use of composite scale scores is a common practice.

Limitations

Our study is not without limitations, some of which should be specifically noted. First, our data do not come from a representative sample and were only collected from EA and MA college students in the U.S. and MX college students in Mexico. As such, our results may not generalize to college students in different countries, college students with different ethnic backgrounds, or the general population from which our data were collected. Second, there were too few foreign-born participants to consider potential DIF as a function of nativity. Given that close to one-third of the U.S. Hispanic population is foreign-born (U.S. Census Bureau, 2015), further studies with adequate sample sizes are warranted.

Finally, and most importantly, all of our U.S. participants spoke and completed the RAPI in English, and all of our Mexico participants spoke and completed the RAPI in Spanish. Because of this, we are unable to speak to whether the identified DIF reflects true DIF (i.e., the same experiences are indicative of different levels of alcohol-related problems) or are the results of translational inaccuracies. This is not a trivial concern, and will require further consideration. This might be done, for example, by randomly assigning English-Spanish bilingual students (in the U.S. or Mexico) to complete the RAPI in either English or Spanish, and evaluate the RAPI for DIF as a function of language.

Summary and Conclusion

Despite the limitations to our study, we provide a critical first step towards understanding the measurement equivalence of the RAPI among college students in the U.S. and Mexico. As our results indicate, there was no DIF in the item discriminations and relatively little DIF in the item severities for the RAPI items. Perhaps more importantly, our results showed that there were no substantive consequences when the DIF was not taken into account – that is, when measurement equivalence was assumed by constraining all of the measurement parameters to be equivalent across the groups. This was the case in terms of both group mean differences and correlations with self-reported drinking behaviors. These results should not be too surprising, as scholars have noted that minor deviations in equivalence at the level of individual measurement parameters are not likely to have any substantive consequences (e.g., Little, 1997).

Interestingly, in terms of correlations with self-reported drinking behaviors, there also were no substantive consequences when using observed scale scores. Indeed, although the correlations were slightly weaker, they were not significantly different in magnitude from the correlations using the latent RAPI variable with DIF taken into account. The significant group difference between MAs and MXs identified with the RAPI-DIF and RAPI-EQ variables, however, was not identified when using the observed scale scores. For this reason, we urge scholars to model the RAPI as a latent variable when evaluating group mean differences and, for the general purpose of obtaining the most accurate results, when testing substantive hypotheses.

Public Significance Statement:

The results of this study indicate that the Rutgers Alcohol Problem Index holds similar measurement properties (i.e., has minimal measurement bias) for college students in the United States and Mexico. This suggests that the Rutgers Alcohol Problem Index may be used by scholars and practitioners who are interested in assessing and evaluating differences in alcohol-related problems among these groups.

APPENDIX A

Model Fit Comparisons

	CAIC	BIC	ABIC	ΔCAIC	ΔBIC	ΔABIC
TESTS OF DIFFERENTIAL ITEM FUNCTIONING
Unconstrained Model	26307.12	26742.75	26298.02	--	--	--
Iteration 1 (Constrained) Models
Item discriminations
Item 1	26303.89	26733.30	26294.92	−3.23	−9.45	−3.10
Item 2	26303.55	26732.96	26294.58	−3.57	−9.79	−3.44
Item 3	26302.12	26731.54	26293.16	−5.00	−11.22	−4.86
Item 4	26299.73	26729.14	26290.76	−7.39	−13.61	−7.26
Item 5	26301.12	26730.54	26292.16	−6.00	−12.22	−5.86
Item 6	26302.65	26732.06	26293.69	−4.47	−10.69	−4.34
Item 7	26306.85	26736.26	26297.88	−0.27	−6.50	−0.14
Item 8	26300.35	26729.76	26291.38	−6.77	−13.00	−6.64
Item 9	26301.76	26731.17	26292.79	−5.36	−11.58	−5.23
Item 10	26301.50	26730.91	26292.53	−5.62	−11.84	−5.49
Item 11	26298.81	26728.22	26289.84	−8.31	−14.53	−8.18
Item 12	26299.62	26729.03	26290.65	−7.50	−13.72	−7.37
Item 13	26300.59	26730.00	26291.62	−6.53	−12.75	−6.40
Item 14	26300.03	26729.44	26291.07	−7.09	−13.31	−6.96
Item 15	26299.05	26728.46	26290.08	−8.07	−14.29	−7.94
Item 16	26301.88	26731.29	26292.91	−5.24	−11.46	−5.11
Item 17	26304.82	26734.23	26295.85	−2.30	−8.52	−2.17
Item 18	26301.40	26730.81	26292.43	−5.72	−11.94	−5.59
Item 19	26301.56	26730.97	26292.59	−5.56	−11.78	−5.43
Item 20	26302.80	26732.21	26293.83	−4.32	−10.55	−4.19
Item 21	26299.04	26728.45	26290.07	−8.08	−14.31	−7.95
Item 22	26306.09	26735.51	26297.13	−1.03	−7.25	−0.89
Item 23	26299.55	26728.96	26290.58	−7.57	−13.79	−7.44
Item severities
Item 1	26309.35	26738.76	26300.38	2.23	−3.99	2.36
Item 2	26299.64	26729.05	26290.67	−7.48	−13.71	−7.35
Item 3	26308.07	26737.48	26299.10	0.95	−5.27	1.08
Item 4	26300.29	26729.70	26291.32	−6.83	−13.05	−6.70
Item 5	26309.02	26738.44	26300.06	1.90	−4.32	2.04
Item 6	26319.99	26749.40	26311.02	12.87	6.64	13.00
Item 7	26313.92	26743.33	26304.96	6.80	0.58	6.93
Item 8	26307.34	26736.75	26298.37	0.22	−6.00	0.35
Item 9	26306.51	26735.92	26297.54	−0.61	−6.83	−0.48
Item 10	26301.17	26730.58	26292.20	−5.95	−12.17	−5.82
Item 11	26307.14	26736.55	26298.17	0.02	−6.20	0.15
Item 12	26299.93	26729.34	26290.96	−7.19	−13.41	−7.06
Item 13	26314.04	26743.45	26305.07	6.92	0.70	7.05
Item 14	26320.35	26749.76	26311.38	13.23	7.01	13.36
Item 15	26298.86	26728.27	26289.89	−8.26	−14.48	−8.13
Item 16	26309.84	26739.25	26300.87	2.72	−3.50	2.85
Item 17	26305.96	26735.37	26296.99	−1.16	−7.39	−1.03
Item 18	26307.90	26737.32	26298.94	0.78	−5.44	0.92
Item 19	26301.27	26730.68	26292.30	−5.85	−12.07	−5.72
Item 20	26304.08	26733.50	26295.12	−3.04	−9.26	−2.90
Item 21	26310.16	26739.58	26301.20	3.04	−3.18	3.18
Item 22	26307.54	26736.95	26298.57	0.42	−5.80	0.55
Item 23	26303.30	26732.71	26294.33	−3.82	−10.04	−3.69
Final Iteration 1 (Partially Constrained) Model	26204.98	26416.57	26200.56	−102.14	−326.18	−97.46
Iteration 2 (Constrained) Models	26222.50	26427.87	26218.21	17.52	11.30	17.65
Item severities	26202.80	26408.17	26198.51	−2.17	−8.40	−2.04
Item 1	26242.71	26448.08	26238.42	37.73	31.51	37.86
Item 5	26206.27	26411.64	26201.98	1.30	−4.92	1.43
Item 6	26230.07	26435.45	26225.79	25.10	18.88	25.23
Item 7	26222.06	26427.43	26217.77	17.08	10.86	17.22
Item 13	26226.62	26431.99	26222.33	21.65	15.42	21.78
Item 14	26219.77	26425.14	26215.48	14.79	8.57	14.92
Item 16	26222.50	26427.87	26218.21	17.52	11.30	17.65
Item 21	26202.80	26408.17	26198.51	−2.17	−8.40	−2.04
Final Iteration 2 (Partially Constrained) Model	26203.69	26402.84	26199.53	−1.28	−13.73	−1.02
Iteration 3 (Constrained) Models
Item severities	26224.46	26417.38	26220.43	20.77	14.54	20.90
Item 1	26203.20	26399.24	26199.11	−0.49	−3.60	−0.43
EA = MA	26228.32	26424.36	26224.23	24.63	21.52	24.69
EA = MX	26212.41	26408.45	26208.32	8.72	5.61	8.78
MA = MX	26246.23	26439.16	26242.20	42.54	36.31	42.67
Item 6	26201.96	26397.99	26197.86	−1.73	−4.85	−1.67
EA = MA	26247.46	26443.50	26243.37	43.77	40.66	43.83
EA = MX	26229.52	26425.56	26225.43	25.83	22.72	25.90
MA = MX	26232.57	26425.49	26228.54	28.88	22.65	29.01
Item 13	26200.23	26396.27	26196.14	−3.46	−6.57	−3.39
EA = MA	26233.07	26429.10	26228.97	29.37	26.26	29.44
EA = MX	26224.38	26420.41	26220.28	20.69	17.57	20.75
MA = MX	26218.14	26411.06	26214.11	14.44	8.22	14.58
Item 14	26203.92	26399.96	26199.83	0.23	−2.88	0.29
EA = MA	26222.26	26418.30	26218.17	18.57	15.46	18.63
EA = MX	26207.30	26403.34	26203.20	3.61	0.49	3.67
MA = MX	26227.91	26420.83	26223.88	24.21	17.99	24.35
Item 16	26199.64	26395.68	26195.55	−4.05	−7.16	−3.98
EA = MA	26227.72	26423.75	26223.62	24.03	20.91	24.09
EA = MX	26224.46	26417.38	26220.43	20.77	14.54	20.90
MA = MX	26224.44	26420.48	26220.35	20.75	17.64	20.82
Item 21	26219.47	26412.40	26215.44	15.78	9.56	15.91
EA = MA	26222.45	26418.49	26218.36	18.76	15.65	18.82
EA = MX	26209.15	26405.18	26205.05	5.46	2.34	5.52
MA = MX	26201.34	26397.37	26197.24	−2.35	−5.47	−2.29
Final Iteration 3 (Partially Constrained) Model	26191.88	26372.35	26188.11	−11.82	−30.49	−11.43
Iteration 4 (Constrained) Models
Item severities
Item 1	26213.97	26391.33	26210.26	22.09	18.98	22.16
Item 6	26237.42	26414.79	26233.72	45.55	42.44	45.61
Item 13	26225.12	26402.48	26221.41	33.24	30.13	33.30
Item 14	26205.55	26382.92	26201.85	13.68	10.57	13.74
Item 16	26220.63	26397.99	26216.92	28.75	25.64	28.81
Item 21	26208.85	26386.22	26205.15	16.98	13.86	17.04
TESTS OF MODERATED CORRELATIONS
RAPI-DIF Models
Unconstrained model	28422.09	28720.81	28415.85	--	--	--
Constrained models
Average drinks/day	28422.61	28715.10	28416.50	0.51	−5.71	0.64
Average drinking days/week	28416.90	28709.39	28410.79	−5.19	−11.42	−5.06
Average binge drinking days/week	28419.79	28712.29	28413.68	−2.30	−8.52	−2.17
RAPI-EQ Models
Unconstrained model	28571.04	28851.09	28565.19	--	--	--
Constrained models
Average drinks/day	28571.57	28845.40	28565.85	0.53	−5.69	0.66
Average drinking days/week	28565.83	28839.66	28560.11	−5.21	−11.43	−5.07
Average binge drinking days/week	28568.75	28842.57	28563.03	−2.29	−8.51	−2.16
RAPI-OS Models
Unconstrained model	9513.87	9650.79	9511.01	--	--	--
Constrained models
Average drinks/day	9513.45	9644.14	9510.72	−0.42	−6.64	−0.29
Average drinking days/week	9507.34	9638.03	9504.61	−6.53	−12.75	−6.40
Average binge drinking days/week	9512.24	9642.93	9509.51	−1.63	−7.85	−1.50
TESTS OF CORRELATION DIFFERENCES
RAPI-DIF Models with Fixed Coefficients
Base model	28410.84	28690.89	28404.99	--	--	--
Fixed models
Average drinks/day (fixed to 0.52)	28408.24	28685.18	28402.46	−2.59	−5.70	−2.53
Average drinking days/week (fixed to 0.48)	28407.87	28684.81	28402.09	−2.96	−6.07	−2.90
Average binge drinking days/week (fixed to 0.44)	28407.47	28684.41	28401.68	−3.37	−6.48	−3.30

NOTE: CAIC = Consistent Akaike’s Information Criterion; BIC = Bayesian Information Criterion; ABIC = Sample-size Adjusted Bayesian Information Criterion; Δ = change; RAPI = Rutgers Alcohol Problem Index; RAPI-DIF = latent RAPI variable with differential item functioning taken into account; RAPI-EQ = latent RAPI variable with all measurement parameters constrained to be equivalent across groups; RAPI-OS = observed RAPI scale scores.

APPENDIX B

Logit Parameter Estimates for RAPI-DIF Model

	Item Discriminations				Item Severities
	EAs	MAs	MXs		EAs	MAs	MXs
Item 1	2.14	2.14	2.14		0.73	0.73	1.61
Item 2	2.54	2.54	2.54		1.98	1.98	1.98
Item 3	2.47	2.47	2.47		2.16	2.16	2.16
Item 4	1.85	1.85	1.85		2.09	2.09	2.09
Item 5	2.37	2.37	2.37		1.61	1.61	1.61
Item 6	2.38	2.38	2.38		0.55	0.55	1.80
Item 7	3.19	3.19	3.19		5.32	5.32	5.32
Item 8	2.23	2.23	2.23		2.07	2.07	2.07
Item 9	1.66	1.66	1.66		1.56	1.56	1.56
Item 10	2.58	2.58	2.58		3.94	3.94	3.94
Item 11	1.64	1.64	1.64		1.42	1.42	1.42
Item 12	4.01	4.01	4.01		4.81	4.81	4.81
Item 13	2.28	2.28	2.28		1.35	1.35	2.49
Item 14	2.21	2.21	2.21		1.94	1.94	1.20
Item 15	3.19	3.19	3.19		2.62	2.62	2.62
Item 16	2.21	2.21	2.21		2.97	2.97	4.58
Item 17	2.62	2.62	2.62		2.02	2.02	2.02
Item 18	2.78	2.78	2.78		3.96	3.96	3.96
Item 19	2.01	2.01	2.01		2.22	2.22	2.22
Item 20	2.78	2.78	2.78		3.75	3.75	3.75
Item 21	1.89	1.89	1.89		0.91	1.60	1.60
Item 22	3.51	3.51	3.51		5.26	5.26	5.26
Item 23	2.76	2.76	2.76		3.63	3.63	3.63

Note: EAs = European Americans; MAs = Mexican Americans; MXs = Mexicans; bold values indicate parameters with differential item functioning.

APPENDIX C

IRT Parameter Estimates for RAPI-DIF Model

	Item Discriminations				Item Severities
	EAs	MAs	MXs		EAs	MAs	MXs
Item 1	2.14	2.67	1.83		0.34	0.36	0.74
Item 2	2.54	3.16	2.17		0.78	0.71	0.77
Item 3	2.47	3.08	2.12		0.87	0.79	0.88
Item 4	1.85	2.30	1.58		1.13	0.99	1.18
Item 5	2.37	2.95	2.03		0.68	0.63	0.66
Item 6	2.38	2.96	2.04		0.23	0.27	0.74
Item 7	3.19	3.97	2.73		1.67	1.43	1.81
Item 8	2.23	2.78	1.91		0.93	0.83	0.95
Item 9	1.66	2.07	1.42		0.94	0.84	0.96
Item 10	2.58	3.21	2.20		1.53	1.31	1.65
Item 11	1.64	2.04	1.40		0.87	0.78	0.88
Item 12	4.01	5.00	3.43		1.20	1.05	1.26
Item 13	2.28	2.84	1.95		0.59	0.56	1.14
Item 14	2.21	2.76	1.89		0.88	0.79	0.50
Item 15	3.19	3.97	2.73		0.82	0.75	0.82
Item 16	2.21	2.76	1.89		1.34	1.16	2.28
Item 17	2.62	3.26	2.24		0.77	0.70	0.76
Item 18	2.78	3.47	2.38		1.42	1.23	1.52
Item 19	2.01	2.51	1.72		1.10	0.97	1.15
Item 20	2.78	3.46	2.38		1.35	1.17	1.44
Item 21	1.89	2.35	1.62		0.48	0.77	0.85
Item 22	3.51	4.37	3.00		1.50	1.29	1.61
Item 23	2.76	3.44	2.36		1.31	1.14	1.40

Note: EAs = European Americans; MAs = Mexican Americans; MXs = Mexicans; bold values indicate parameters with differential item functioning.

APPENDIX D

Standardized Parameter Estimates for RAPI-DIF Model

	Item Discriminations				Item Severities
	EAs	MAs	MXs		EAs	MAs	MXs
Item 1	0.76	0.83	0.71		0.26	0.23	0.62
Item 2	0.81	0.87	0.77		0.63	0.54	0.70
Item 3	0.81	0.86	0.76		0.70	0.60	0.77
Item 4	0.71	0.79	0.66		0.81	0.71	0.87
Item 5	0.79	0.85	0.75		0.54	0.46	0.59
Item 6	0.80	0.85	0.75		0.18	0.16	0.66
Item 7	0.87	0.91	0.83		1.45	1.22	1.62
Item 8	0.78	0.84	0.73		0.72	0.62	0.79
Item 9	0.68	0.75	0.62		0.64	0.57	0.68
Item 10	0.82	0.87	0.77		1.25	1.07	1.38
Item 11	0.67	0.75	0.61		0.58	0.52	0.62
Item 12	0.91	0.94	0.88		1.09	0.90	1.24
Item 13	0.78	0.84	0.73		0.46	0.40	0.94
Item 14	0.77	0.84	0.72		0.68	0.59	0.46
Item 15	0.87	0.91	0.83		0.71	0.60	0.80
Item 16	0.77	0.84	0.72		1.04	0.90	1.75
Item 17	0.82	0.87	0.78		0.63	0.54	0.70
Item 18	0.84	0.89	0.80		1.19	1.01	1.32
Item 19	0.74	0.81	0.69		0.82	0.72	0.89
Item 20	0.84	0.89	0.80		1.13	0.96	1.25
Item 21	0.72	0.79	0.67		0.35	0.54	0.66
Item 22	0.89	0.92	0.86		1.33	1.11	1.50
Item 23	0.84	0.89	0.79		1.10	0.93	1.22

Note: EAs = European Americans; MAs = Mexican Americans; MXs = Mexicans; bold values indicate parameters with differential item functioning.