Effects of Language of Assessment on the Measurement of Acculturation: Measurement Equivalence and Cultural Frame Switching

Abstract

The present study used a randomized design, with fully bilingual Hispanic participants from the Miami area, to investigate two sets of research questions. First, we sought to ascertain the extent to which measures of acculturation (heritage and U.S. practices, values, and identifications) satisfied criteria for linguistic measurement equivalence. Second, we sought to examine whether cultural frame switching would emerge – that is, whether latent acculturation mean scores for U.S. acculturation would be higher among participants randomized to complete measures in English, and whether latent acculturation mean scores for Hispanic acculturation would be higher among participants randomized to complete measures in Spanish. A sample of 722 Hispanic students from a Hispanic-serving university participated in the study. Participants were first asked to complete translation tasks to verify that they were fully bilingual. Based on ratings from two independent coders, 574 participants (79.5% of the sample) qualified as fully bilingual and were randomized to complete the acculturation measures in either English or Spanish. Theoretically relevant criterion measures – self-esteem, depressive symptoms, and personal identity – were also administered in the randomized language. Measurement equivalence analyses indicated that all of the acculturation measures – Hispanic and U.S. practices, values, and identifications – met criteria for configural, weak/metric, strong/scalar, and convergent validity equivalence. These findings indicate that data generated using acculturation measures can, at least under some conditions, be combined or compared across languages of administration. Few latent mean differences emerged. These results are discussed in terms of the measurement of acculturation in linguistically diverse populations.

Acculturation – referring to cultural adaptation at the individual level – has attracted considerable scholarly interest in the past 35 years. Due in part to unprecedented rates of international migration (Steiner, 2009), the total number of English-language journal articles on acculturation increased nearly sevenfold (from 107 to 727) between the 1980s and the 2000s (Schwartz, Unger, Zamboanga, & Szapocznik, 2010). Additionally, three edited books on acculturation were published between 2000 and 2010 (Berry, Phinney, Sam, & Vedder, 2006; Chun, Organista, & Marin, 2003; Sam & Berry, 2006).

This increased attention to acculturation has been accompanied by core definitional questions about what acculturation is and how it functions. Although it has been generally defined as cultural change following intercultural contact (Berry, 1980), exactly what changes as a result of acculturation has been more difficult to pinpoint. The most commonly referenced domains have included language use and other cultural behaviors (Kang, 2006; Szapocznik, Kurtines, & Fernandez, 1980), values and attitudes (Berry, 1997; Berry et al., 2006), and ethnic/heritage identity (Phinney & Ong, 2007). Schwartz and colleagues (e.g., Schwartz, Park, et al., 2012; Schwartz, Weisskirch, et al., 2011), among others, have added national identity (identification with one’s country of settlement) to this list of domains.

Acculturation has also been defined as a bidimensional construct, where receiving-culture acquisition and heritage-culture retention are cast as separate dimensions¹ (Gonzales, Knight, Morgan-Lopez, Saenz, & Sirolli, 2002; Ryder, Alden, & Paulhus, 2000). A person can, for example, acquire receiving-cultural orientations while still retaining her or his culture of origin. Combining bidimensionality with the use of multiple domains yields a model where heritage and receiving cultural streams operate within each psychological domain (see Figure 1). Schwartz et al. (2010) have proposed such a model, where heritage and receiving cultural streams are assumed to operate within the domains of practices, values, and identifications. The domain of practices includes behaviors such as language use, culinary preferences, choice of friends, and use of media. The domain of values refers to beliefs about the relative importance of the individual person and of the social group (e.g., family, community, national ingroup). The domain of identifications refers to a sense of solidarity with a cultural group and/or with the country in which one resides. As applied to Hispanic immigrants in the United States, for example, cultural practices refer to speaking English and Spanish, eating traditional Hispanic foods and American foods, and associating with heritage-cultural and Americanized friends². Given the large disparity between U.S. culture and most Hispanic cultures in terms of endorsement of individualism (prioritizing the needs of the individual person) and collectivism (prioritizing the needs of the family or other social group; Hofstede, 2001), individualism and collectivism represent prominent components of acculturation. Cultural identifications can include (a) identifying with one’s country of own/familial origin or with pan-ethnic groups such as “Hispanic” or “Latino” (e.g., ethnic identity) and/or (b) identification with the United States and characterization of oneself as American (e.g., U.S. identity).

Figure 1. Conceptual Model.

Nets: Each cell in this table represents an acculturation component.

Numerous standard scales have been developed for assessing the various domains and dimensions of acculturation. Several behavioral acculturation (i.e., cultural practices) measures have been introduced and validated – including the Bicultural Involvement Questionnaire (Szapocznik et al., 1980) and the Acculturation Rating Scales for Mexican Americans (Cuéllar, Arnold, & Maldonado, 1995). In-depth psychometric analyses of these scales suggest that language use is empirically separable from other cultural behaviors (Guo, Suarez-Morales, Schwartz, & Szapocznik, 2009; Kang, 2006). Language is an extremely powerful transmitter and activator of culture (Luna, Ringberg, & Peracchio, 2008), so much so that bilingual people may express somewhat different personalities in each language (Ramírez-Esparza et al., 2006) and that language of administration can affect the ways in which bicultural people respond to ambiguous cultural cues (Ji, Zhang, & Nisbett, 2004).

Ethnic identification has typically been assessed using the Multi-Group Ethnic Identity Measure (Phinney, 1992), and more recently the Ethnic Identity Scale (Umaña-Taylor, Yazedjian, & Bámaca-Gómez, 2004). Cultural values scales have been less common – especially scales that assess general cultural values for which an “American” equivalent can also be measured. For example, collectivism can be paired with individualism as bidimensional-model indicators of cultural values.

Measurement Equivalence in Indices of Acculturation

In many studies of immigrant populations, some participants complete their assessments in English whereas others complete their assessments in their native languages (e.g., Marsiglia, Kulis, Hecht, & Sills, 2004; Martinez, McClure, Eddy, & Wilson, 2011). In most cases, data are pooled and analyzed across language of assessment, with the assumption that the measures operate in the same way across languages. In most cases, this assumption is not explicitly tested. As a result, there may be some unknown amount of error variance created when data are pooled across language of assessment – especially when observed scores are evaluated. It is therefore necessary to empirically examine the extent to which it may be valid to combine participants across languages of assessment.

Threats to the Validity of Pooling Participants across Languages of Administration

A number of threats are of concern when pooling participants across language of administration in studies of acculturation and other culturally based constructs where participants are asked to self-select into languages of administration. These threats include cultural frame switching, stereotype threat, translation quality, and differential amounts of linguistic competency among respondents choosing a particular assessment language. We describe each of these threats immediately below.

Cultural Frame Switching

Cultural frame switching (Hong, Benet-Martínez, Chiu, & Morris, 2000; Verkuyten & Pouliasi, 2006) refers to the phenomenon where specific cultural cues – including being exposed to language, symbols, and behaviors characteristic of one’s heritage culture or of the society in which one or one’s family has settled – can affect bicultural individuals’ responses to culturally related questions by activating schemata related to the cultural stream in question. For example, Benet-Martínez, Leu, Lee, and Morris (2002) found that priming bicultural Chinese American participants with either Chinese icons (e.g., Great Wall) or U.S. icons (e.g., Statue of Liberty) led these participants to provide characteristically more American (i.e., internal and based on personal characteristics) or characteristically more Chinese (i.e., external and based on situational factors) attributions of an ambiguous situation. Cultural frame switching may also affect the ways in which cultural constructs are organized within the person’s cultural repertoire, which in turn could potentially affect the factor structures (or measurement intercepts) generated by measures of cultural constructs (Pouliasi & Verkuyten, 2007). Large differences in factor structures or item intercepts across assessment languages suggest that the scores produced by the measure in question do not operate equivalently in the two languages, and would contraindicate pooling participants across languages.

Stereotype Threat

The concept of stereotype threat (Steele, 1997) refers to the experience of anxiety or concern in a situation where a person has the potential to confirm negative stereotypes about her/his social group – and to the resulting tendency to behave in a way that supports the stereotype. For example, the societal belief that a particular ethnic group is intellectually inferior may lead to poorer academic performance among member of this group when that belief is primed by cues in the situation (Nguyen & Ryan, 2008). Although stereotype threat has been used primarily to explain disparities in academic performance and related outcomes, it stands to reason that this perspective might be expanded to apply to acculturation-related behaviors such as language use as well. Barker et al. (2001), for example, have found that many Americans view Spanish as a threat to U.S. national identity; and commentators such as Huntington (2004) have explicitly labeled the spread of Spanish as a danger to the solidarity of the United States. Hispanics in the United States often operate within a bicultural reality – they are part of a minority ethnic group with strong ties to Latin America, and they are also an increasingly important segment of the U.S. population. Within bicultural groups, there is often perceived pressure to avoid becoming assimilated and to avoid remaining within one’s ethnic group and separated from the cultural mainstream (Rudmin, 2003). There may therefore be unconscious pressures to avoid stereotypes both of Hispanics and of Americans. Stereotype threat theory may therefore be used to advance a prediction opposite of that advanced by the cultural frame switching perspective. Nevertheless, according to both the cultural frame switching and stereotype threat hypotheses, language of administration unconsciously activates specific cultural orientations (Steele, 1997; Verkuyten & Pouliasi, 2006).

Translation Quality

A set of procedures has been proposed for translating measures from one language into another (Sireci, Yang, Harter, & Ehrlich, 2006). One translator translates the measure from English into the target language, a second translator translates the first translator’s version back into English, and then the two translators work together to (a) resolve discrepancies in the original and back-translated English versions and to (b) translate the final English version into the target language. Optimally, a committee of bilingual individuals reviews the final translated version, ensures that it is appropriate for the target population, and refers back to the original English version to ensure that any corrections do not change the intended meaning of the items. However, following these steps does not guarantee that scores produced by a measure will operate in the same way across languages – this must be established empirically (e.g., Hong, Malik, & Lee, 2003; Knight, Roosa, & Umaña-Taylor, 2009).

Language Competency

Individuals may self-select into assessment languages for any number of reasons. Although the most obvious such reason is likely that the person is most competent or comfortable in that language, there may be other reasons as well. For example, research has indicated that both Hispanic immigrant adolescents (Romero & Roberts, 2003) and Hispanic immigrant adults (Rodriguez, Myers, Mira, Flores, & Garcia-Hernandez, 2002) may feel pressured to improve their English (or to hide the fact that they are not proficient in English), and that that later-generation Hispanic adolescents and adults may feel pressured to improve their Spanish. Immigrant adolescents, for example, might choose to complete their assessments in English even though their competency in the language is not sufficient for them to understand the items and response choices, perhaps because they perceive a stigma associated with requesting or selecting a Spanish-language survey. Although there is no published literature on effects of language competency on psychometrics among “acculturating” populations, it stands to reason that, when the individuals completing assessments in a given language vary greatly in terms of their competency in that language, measurement error may be increased, the validity of the resulting data may be questionable, and the equivalence of factor structures and measurement intercepts across languages may be compromised.

Empirically Assessing Measurement Equivalence across Languages

Methodologists have devised ways to determine whether a measure operates in the same way across languages. A concept that has received a great deal of recent attention in the cultural and cross-cultural literatures is measurement equivalence (Knight et al., 2009). Briefly stated, measurement equivalence refers to the extent to which self-report items convey the same meaning, and to which responses to those items cluster onto the same set of factors, across languages of administration. That is, the item responses should pattern onto the same constructs, and cluster together in the same way, regardless of the language in which the measure is administered. For example, using the Ethnic Identity Scale, White, Umaña-Taylor, Knight, and Zeiders (2009) examined several types of cross-language measurement invariance: (a) configural invariance, where a multigroup model, with cases grouped by language of administration, fit the data adequately; (b) metric/weak invariance, where the factor loadings for each item on its respective subscale were equivalent across languages; (c) scalar/strong invariance, where each item intercept (in addition to each factor loading) was equivalent across languages; and (d) strict invariance, where each item error variance (in addition to each item intercept and factor loading) was equivalent across languages. Some authors (e.g., Vandenberg & Lance, 2000) have described strict invariance as an overly restrictive assumption – and it is often not included in the measurement invariance testing process. In addition, some authors (e.g., Knight et al., 2009; Lonner, 1986) have suggested that construct validity equivalence (Lonner uses the term “conceptual equivalence”), where correlations of latent factor scores for each subscale with other theoretically relevant measures were equivalent across languages, is also an important component of evaluating measurement equivalence.

Linguistic measurement equivalence studies have generally compared individuals who chose to complete a given measure in English versus those who chose to complete the measure in another language (e.g., Spanish). However, individuals choosing to complete measures in English versus in their native languages may differ on a number of important variables, including fluency and literacy in English and extent to which they use English and use their native language in their daily activities. Indeed, Schwartz, Des Rosiers, et al. (in press) found considerable variability in language use among Hispanic adolescents choosing to complete acculturation measures in English, but extremely little variability among those choosing to complete acculturation measures in Spanish.

As a result, it may be reasonable to evaluate measurement equivalence across “naturally occurring” language groups (i.e., participants self-selecting into languages of administration) so long as the construct on which equivalence is being evaluated does not map directly onto the differences in language use or other cultural practices between these language groups – and so long as cultural frame switching and stereotype threat do not affect the factor structures of the instruments being examined. For example, as Schwartz, Des Rosiers, et al. (2013) found, it may not be possible to use naturally occurring language groups to evaluate measurement equivalence of acculturation measures – some of which may include items assessing language use. When individuals self-select into languages of administration, language-related effects (e.g., cultural frame switching, stereotype threat) cannot be uncoupled from other potential confounders such as language competency and translation quality.

Acculturation as a Dynamic Construct: Effects of Cultural Frame Switching

The literature on cultural frame switching has generally used experimental primes, such as assessment languages and cultural icons, to demonstrate the dynamic properties of culture. That is, in bicultural individuals, cultural cues may activate the corresponding cultural schemata. Luna, Ringberg, and Peracchio (2008) found that individuals’ self-ascribed personality characteristics (e.g., self-sufficient) varied depending on the language in which they were assessed. Ramírez-Esparza, Gosling, Benet-Martínez, Potter, and Pennebaker (2006) reported a similar finding for three of the Big Five personality traits – extraversion, agreeableness, and conscientiousness. Processes related to stereotype threat have not yet been examined within the context of experimentally priming languages of assessment.

Only a small number of published studies have examined indices of acculturation across languages of assessment within a single cultural group. Lechuga (2008) randomly assigned Hispanic participants to complete measures of Hispanic and U.S. cultural practices, values, and identifications in either English or Spanish. She found that individuals assessed in Spanish reported greater engagement in Hispanic cultural practices, greater endorsement of collectivist values, and stronger ethnic identity compared to those assessed in English. Lechuga interpreted her findings as suggesting that acculturation is a dynamic construct – influenced by contextual cues such as language. However, she only reported observed means (using analysis of variance) and did not conduct measurement equivalence analyses to ensure that the factor structures, intercepts, and construct validity are similar across the English and Spanish versions. Without empirical evidence of measurement equivalence, observed mean differences could have resulted from differences in factor structures or item intercepts across languages.

When one’s goal is to examine measurement equivalence of acculturation instruments, it may be advisable to recruit a sample of bilingual individuals and randomly assign them to complete the study measures in either English or Spanish. Especially when only fully bilingual participants are included in the sample, random assignment has the potential to equate language groups on linguistic competency, cultural orientations, and other potential confounders. Once the sample has been equated on these potential confounders, measurement equivalence analyses may be better able to examine the possibility that the two language groups might be able to be pooled or compared. Provided that this possibility remains tenable, a further question would be under what conditions such equivalent psychometric properties might emerge in self-selected or non-bilingual language groups. Put differently, examining whether a measure generates scores with equivalent psychometric properties across language of assessment under ideal conditions (i.e., fully bilingual participants randomized to languages of assessment) is a prerequisite for ascertaining cross-language equivalence under less ideal, real-world conditions (e.g., less than fully bilingual participants self-selecting their assessment language). The present study focuses on this first, prerequisite step of this sequence.

The Present Study

In the present study, bilingual first and second generation immigrant Hispanic college students were randomly assigned to complete acculturation measures in either English or Spanish. The study was guided by two primary goals. Both of these goals combine the strengths of random assignment and of measurement equivalence techniques. Our first goal was to evaluate the extent to which measures of Hispanic and U.S. cultural practices, values, and identifications are indeed structurally similar across languages of assessment. Our second goal was to examine the effects of language of administration on latent means for Hispanic and U.S. practices, values, and identifications. Assuming that at least partial metric/weak and scalar/strong invariance are found, one might then ask whether scale means for indices of Hispanic and U.S. acculturation differ significantly for bilingual Hispanic individuals assessed in Spanish versus those assessed in English. To the extent to which the answer to this question is yes, cultural frame switching or stereotype threat effects would be assumed (Luna et al., 2008).

We tested four hypotheses. Because the same acculturation constructs are assessed in both languages, we hypothesized that, first, the factor structure of each acculturation measure would be statistically equivalent across languages (i.e., metric/weak invariance); and second, the item intercepts for each acculturation measure would be statistically equivalent across language of administration. Third, we hypothesized that, if comparing latent scale means across languages was possible (i.e., if a sufficient number of factor loadings and item intercepts were equal across languages), a cultural frame switching effect would emerge – that is, U.S. acculturation indices would be higher in participants assigned to complete the survey in English, and Hispanic acculturation indices would be higher in participants assigned to complete the survey in Spanish. Alternatively, from a stereotype threat perspective, the opposite pattern might be expected: being assessed in English could elicit a defensive response where the person distances her/himself from stereotypically American behaviors, values, and identities (Sanchez, Chavez, Good, & Wilton, 2012); and being assessed in Spanish could elicit a response characterized by distancing oneself from stereotypically Hispanic cultural orientations.

Fourth, we hypothesized that construct validity equivalence would emerge – that is, that latent variables representing the acculturation indices would be equivalently correlated with criterion variables (self-esteem, depressive symptoms, and personal identity) across languages. These specific criterion variables were selected based on their theoretical associations with acculturation, with the expectation that these associations would be equivalent across languages of assessment. Social identity theory (Tajfel & Turner, 1986) holds that being attached to a cultural stream (the U.S., one’s country of origin, or both) serves as a source of self-esteem and protects against internalizing symptoms (cf. Nguyen & Benet-Martínez, 2013; Smith & Silva, 2011; Yoon, Langrehr, & Ong, 2011). Personal identity (identity exploration, commitment, synthesis, and confusion) has been linked with acculturation in a number of studies (e.g., Branch, Tayal, & Triplett, 2000; Schwartz, Kim, et al., 2013), and some authors (e.g., Benet-Martínez & Haritatos, 2005) have framed acculturation as an identity process. In particular, cultural concerns may represent a domain of personal identity (Schwartz, Zamboanga, Luyckx, Meca, & Ritchie, 2013), although it should be noted that identity also operates within many other domains – such as gender, sexuality, and religion.

Method

The Community Context for the Present Study

The university where the present data were collected has been designated as a Hispanic-Serving Institution by the U.S. Department of Education (2010). The university, like the cultural context of Miami in general, is highly bicultural and equally supportive of U.S. and Hispanic practices (e.g., Stepick & Stepick, 2002). Although schooling occurs in English, many other transactions (both formal and informal) occur in both English and Spanish. Beginning in the late 1960s, Cuban immigrants transformed Miami from a port and vacation getaway into a major city. Beginning in the 1980s, other Hispanic groups began to arrive from various countries in Central and South America – especially Nicaragua, Colombia, Peru, Venezuela, Honduras, and Argentina. Today, Miami is home to the most diverse Hispanic population of any U.S. city, and the university where the present data were collected mirrors this diversity.

Participants

The present sample consisted of 722 Hispanic college students attending a university in the Miami area with a heavily Hispanic student population. Participants were recruited from introductory psychology courses, and the recruitment announcement specified that participants should be fluent in both English and Spanish. Consistent with national trends in college attendance, and especially in the social sciences (Marklein, 2010), the majority (77%) of participants were women. Sixty percent of participants, 12% of their mothers, and 12% of their fathers were born in the United States. The most common countries of origin for immigrant participants and parents were Cuba (35%), Venezuela (6%), Colombia (6%), Nicaragua (6%), Peru (3%), the Dominican Republic (3%), Ecuador (2%), and El Salvador (2%). The remaining participants and/or parents were from other countries or did not specify their countries of familial origin. Four percent of participants indicated familial origins in Puerto Rico (a U.S. commonwealth). The vast majority of participants (76%) resided at home with family members, with the remainder living in off-campus houses or apartments (16%) or on campus (6%). In terms of annual household income, 34% of participants reported less than $30,000, 27% between $30,000 and $50,000, 25% between $50,000 and $100,000, and 15% above $100,000. Participants who were living with, or supported by, family members were asked to provide their family members’ income levels, whereas those who were supporting themselves were asked to provide their own income levels.

Procedures

Participants were recruited for the study using an online research participation system, which directed interested participants to the study website. Participants were asked to read an online consent form and to check a box if they were willing to participate in the study. The Institutional Review Boards at the senior author’s home institution and at the institution where the study was conducted approved the study. Following the consent process, to ensure fluency in both English and Spanish, participants were asked to translate four Spanish sentences into English, and to translate four English sentences into Spanish. A sample sentence was “I wanted to go to my mother’s house, but she was not home, so I went to the baseball game instead.”

Two bilingual raters – one from Cuba and one from Venezuela – evaluated each of the eight translations on a scale of 1 (completely inaccurate) to 5 (completely accurate). Raters’ proficiency in both English and Spanish was evaluated through conversations with trained research assistants in both languages. Ratings for each translation direction were then averaged, within each of the two raters, across the four sentences in that direction. Only participants who averaged at least 4.0 in each translation direction (Spanish to English and English to Spanish) for both raters were retained in the sample for analysis. Although the two raters worked during different semesters, never met, and were not told about one another, agreement levels between them were 92% for English-to-Spanish translations and 97% for Spanish-to-English translations. Eighty percent (574 of 722) of participants met the threshold for fluency in both languages and were retained for the second part of the study. Chi-square analyses indicated that retention in the sample was not significantly related to gender, immigrant generation, annual family income, or living arrangements.

The 574 participants who qualified as bilingual were then randomly assigned, using a random number generator, to complete the study measures in either English or Spanish. Slightly more participants (295 versus 279) were assigned to complete the measures in English than in Spanish. After completing all of the study measures, participants were thanked and debriefed about the purpose of the study.

Spanish translations of each of the measures were created as part of a funded research project (Schwartz, Unger, et al., 2012). Because the measures would be administered to respondents from various Hispanic nationalities, the English versions were simultaneously translated into Spanish by bilingual individuals from four different Hispanic countries (see Caetano, Vaeth, & Rodriguez, 2012, for an example of this translation approach). The four translators then discussed and resolved discrepancies in the different Spanish versions, with “broadcast Spanish” (i.e., without idioms or local dialects, as would appear in a national newspaper) used to the extent possible. A committee of experts in Hispanic cultural adaptation met with the translators to review the final versions. These measures were then pilot-tested on a sample of 20 Hispanic parent-adolescent dyads (10 in Miami and 10 in Los Angeles). Based on their feedback, small wording changes were made to the versions administered in the present study.

Measures

Acculturation

Given our multidimensional perspective on acculturation, we assessed Hispanic and U.S. cultural practices, individualist and collectivist values, and Hispanic and U.S. identity. Cultural practices were assessed using the Bicultural Involvement Questionnaire (Guo, Suarez-Morales, Schwartz, & Szapocznik, 2009; Szapocznik, Kurtines, & Fernandez, 1980), which was designed specifically for Hispanics. Individualism and collectivism were assessed using corresponding scales developed by Triandis and Gelfand (1998). Triandis and Gelfand separate both individualism and collectivism into horizontal (in relation to peers, classmates, and coworkers) and vertical (in relation to parents, employers, and other authority figures). Cultural identifications were assessed using the Multi-Group Ethnic Identity Measure (Roberts et al., 1999) for Hispanic identifications and the American Identity Measure (Schwartz et al., 2012) for U.S. identifications. These two measures use parallel item structures and identical response scales; the only difference between them is whether the items refer to “my ethnic group” or “the United States.” Detailed information on these measures can be found in Table 1. It is worth noting that 32 of the 36 alpha coefficients (89%) were above .70.

Table 1.

Summary of Measures and Psychometric Properties

Construct	Measure Name	Number of Items on Subscale	Sample Item	Cronbach’s Alpha (English)	Cronbach’s Alpha (Spanish)
Acculturation
U.S. Practices (language use and other behaviors)	Bicultural Involvement Questionnaire (Szapocznik et al., 1980)	12	“I like American food”	.86	.86
Hispanic Practices (language use and other behaviors)	Bicultural Involvement Questionnaire	12	“I speak Spanish at home”	.91	.91
Horizontal Individualism	Individualism-Collectivism Scales (Triandis & Gelfand, 1998)	4	“I’d rather depend on myself than others”	.77	.77
Vertical Individualism	Individualism-Collectivism Scales	4	“Competition is the law of nature”	.77	.65
Horizontal Collectivism	Individualism-Collectivism Scales	4	“I feel good when I collaborate with others”	.74	.81
Horizontal Collectivism	Individualism-Collectivism Scales	4	“Family members should stick together, no matter what sacrifices are required”	.78	.78
U.S. Identity	American Identity Measure (Schwartz et al., 2012)	12	“I have a lot of pride in the United States.”	.91	.91
Hispanic Identity	Multi-Group Ethnic Identity Measure (Roberts et al., 1999)	12	“I am happy that I am a member of my ethnic group”	.86	.86
Construct Validity
Self-Esteem	Rosenberg Self-Esteem Scale (Rosenberg, 1968)	10	“On the whole, I am satisfied with myself”	.86	.81
Depressive Symptoms	Center for Epidemiologic Studies Depression Scale (Radloff, 1977)	20	“This week, I felt like crying.”	.81	.81
Personal Identity Synthesis	Erikson Psychosocial Stage Inventory (Rosenthal et al., 1981)	6	“I’ve got it together”	.83	.83
Personal Identity Confusion	Erikson Psychosocial Stage Inventory	6	“I don’t really know who I am”	.74	.68
Commitment Making	Dimensions of Identity Development Scale (Luyckx et al., 2008)	5	“I know what I want to do with my future”	.94	.94
Identification with Commitment	Dimensions of Identity Development Scale	5	“My future plans give me self-confidence”	.89	.89
Exploration in Breadth	Dimensions of Identity Development Scale	5	“I spend time thinking about what to do with my life”	.85	.85
Exploration in Depth	Dimensions of Identity Development Scale	5	“I am working out for myself whether the goals I have put forward in life really suit me”	.69	.69
Ruminative Exploration	Dimensions of Identity Development Scale	5	“I keep wondering which direction my life has to take”	.87	.87
Passive Commitment	Dimensions of Identity Development Scale	5	“My future plans are set”	.84	.84

Construct Validity Measures

We used four constructs to ascertain the extent to which the construct validity of the acculturation measures was equivalent across languages (cf. Knight, Roosa, & Umaña-Taylor, 2009). These constructs were self-esteem, depressive symptoms, and personal identity.

Self-esteem was assessed using the Rosenberg (1968) Self-Esteem Scale, which is one of the most commonly used self-esteem instruments (Quilty, Oakman, & Risko, 2006). Depressive symptoms were assessed using the Centers for Epidemiologic Studies Depression Scale (CES-D; Radloff, 1977). Personal identity was measured using two different measures to capture the breadth of this construct (Schwartz, 2001). Personal identity synthesis and confusion – the two poles of Erikson’s (1950) conceptualization of personal identity – were assessed using the 12-item Erikson Psychosocial Stage Inventory (Rosenthal, Gurney, & Moore, 1981; Schwartz, Zamboanga, Wang, & Olthuis, 2009). Identity exploration and commitment, as defined by Marcia (1966, 1980) and refined by Luyckx, Goossens, Soenens, and Beyers (2006), were measured using the Dimensions of Identity Development Scale (Luyckx et al., 2008). This measure assesses three variants of identity exploration and three variants of identity commitment: (a) exploration in breadth (sorting through multiple sets of goals, values, and beliefs to identify one or more to which one may adhere; (b) commitment making (deciding to adhere to one or more sets of goals, values, and beliefs); (c) exploration in depth (thinking and talking to others about commitments one has enacted); (d) identification with commitment (internalizing one’s commitments into one’s sense of self); and (e) ruminative exploration (obsessing over and worrying about making the “perfect” choices, such that one becomes “stuck” in the exploration process). Luyckx (2011) has added an additional scale – passive commitment, to referring to commitments enacted without one’s volitional control (e.g., by internalizing commitments from other people).

Results

As part of our analyses, we tested each acculturation subscale for configural, metric/weak, scalar/strong, and convergent validity invariance, and if possible, we conducted latent mean comparisons across language condition. Because invariance testing corresponds to our first research goal and mean comparisons correspond to our second research goal, we report these sets of analyses separately.

Descriptive Statistics

Descriptive statistics for the observed subscale scores are presented in Table 2.

Table 2.

Observed Means and Standard Deviations by Language of Assessment

Component	Language of Assessment
	English (M, SD)	Spanish (M, SD)
U.S. Culture Acquisition
English Language Use	23.03 (3.29)	22.81 (3.48)
U.S. Cultural Practices	31.10 (4.42)	29.31 (4.68)
Horizontal Individualist Values	15.29 (2.51)	14.99 (2.60)
Vertical Individualist Values	14.33 (2.39)	13.74 (2.71)
U.S. Identity	44.56 (7.75)	45.37 (7.43)
Hispanic Culture Retention
Spanish Language Use	20.19 (4.93)	19.74 (4.88)
Hispanic Cultural Practices	26.21 (6.54)	24.45 (6.24)
Horizontal Collectivist Values	11.17 (2.75)	11.42 (2.74)
Vertical Collectivist Values	14.34 (2.67)	14.37 (2.55)
Hispanic Identity	45.13 (8.51)	44.84 (8.53)

Invariance Tests

As outlined in the introduction, we tested for four types of invariance – configural invariance, weak/metric invariance, strong/scalar invariance, and construct validity equivalence (Vandenberg & Lance, 2000; Widaman & Reise, 1997). All of these steps involve estimating measurement models with the subscales specified latent variables and the individual items specified as indicators. Given that language use has been found to be empirically distinguishable from other types of cultural behaviors (Kang, 2006), for both U.S. and Hispanic practices, we specified language use (English or Spanish) as one latent variable and other cultural behaviors (e.g., culinary preferences, media use, choice of friends) as a second latent variable. For both individualistic and collectivistic values, we specified horizontal individualism or collectivism as one latent variable and vertical individualism or collectivism as a second latent variable. U.S. and Hispanic identity were each specified as single latent variables. A single set of invariance tests was conducted for each acculturation domain; in cases where two latent variables were used to represent an acculturation domain (e.g., English-language use and other cultural practices used to represent U.S. cultural practices), these latent variables were allowed to correlate and were included within a single set of invariance tests.

The invariance testing process begins with estimating a configural invariance model, where the measurement algorithm for the acculturation dimension in question is fit to the data separately for the English and Spanish language conditions. The remaining invariance testing steps entail sequentially imposing a set of constraints on model parameters and evaluating the difference in model fit with versus without these constraints (see Dimitrov, 2010; Widaman & Reise, 1997, for reviews). For each step of invariance testing, we first consider full invariance, and if that is not achieved, we attempt to free model parameters (loadings or intercepts) until partial invariance is achieved (Vandenberg & Lance, 2000). We used Lagrange multiplier tests to identify the specific parameter(s) that needed to be freed to achieve partial invariance. Note that Knight et al. (2009) have suggested that there are some highly unique situations in which partial measurement invariance may be a better indicator of measurement equivalence than full measurement invariance. For example, when comparing monolingual English speakers to monolingual Spanish speakers on measures of acculturation, items assessing language use would not be expected to meet criteria for invariance across languages (Schwartz, Des Rosiers, et al., 2013). We report those model parameters that varied across languages, so that the reader is aware of the items (and therefore aspects of each acculturation component) that appear to operate differently across English and Spanish versions of the survey instruments.

We apply each of these steps in the recommended sequence – configural, weak/metric, strong/scalar, and construct validity – separately for each acculturation dimension (e.g., U.S. practices, Hispanic identifications). Because convergent validity equivalence is not part of the formal invariance testing sequence, we test for this form of equivalence as the last step. For the sake of clarity, we present the results separately by type of invariance.

Configural Invariance]

Testing for configural invariance involved estimating a two-group model and ascertaining its fit to the data. Model fit to the data was examined using the comparative fit index (CFI), the non-normed fit index (NNFI), the root mean square of approximation (RMSEA), and the standardized root mean square residual (SRMR). The CFI and NNFI are incremental fit indices, meaning that they reflect the improvement in fit for the specified model versus a null model with no paths or latent variables (Kline, 2010). The RMSEA and SRMR are absolute fit indices, reflecting the extent of divergence between the model and the data (Keith, 2006). Methodologists (e.g., Little, 2013) suggest that least one index of each type should be used to evaluate model fit. The chi-square index is reported but not used in interpretation, because it tests the null hypothesis of perfect fit and is overpowered in reasonably sized samples and in models with more than minimal degrees of freedom (Kline, 2010).

Although the use of absolute fit index cutoffs has been discouraged (Lance, Butts, & Michels, 2006), general rules might suggest that CFI ≥ .95, NNFI ≥ .90, RMSEA ≤ .05, and SRMR ≤ .06 represent a well-fitting model, and that CFI ≥ .90, NNFI ≥ .85, RMSEA ≤ .08, and SRMR ≤ .10 represent an adequately fitting model (West, Taylor, & Wu, 2012). The assumption of configural invariance is satisfied if the multiple-group model fits the data at least adequately. We found evidence of configural invariance for each of the six acculturation components (see Table 3).

Table 3.

Configural Invariance Findings

Component	Fit Index
	χ2 (df)	CFI	NNFI	RMSEA (90% CI)	SRMR
U.S. Practices	271.02 (102)	.96	.95	.077 (.066–.088)	.056
Individualist Values	64.57 (36)	.97	.96	.053 (.031–.074)	.051
U.S. Identity	278.91 (102)	.94	.92	.078 (.067–.089)	.053
Hispanic Practices	261.37 (96)	.96	.95	.078 (.067–.090)	.048
Collectivist Values	62.43 (36)	.98	.97	.051 (.029–.072)	.034
Hispanic Identity	280.80 (100)	.95	.94	.080 (.069–.092)	.047

Weak/Metric Invariance

The assumption of weak/metric invariance was evaluated by constraining each factor loading to be equal across languages and comparing the fit of the resulting weak/metric invariance model to the fit of the configural invariance model (Vandenberg & Lance, 2000). Differences in a number of fit indices are used to evaluate the tenability of weak/metric invariance. Several fit indices have been examined with regard to their appropriateness for use in invariance testing (Chen, 2007). However, Knight, Roosa, and Umaña-Taylor (2009), in their guidelines for measurement equivalence testing, have recommended that the CFI and RMSEA be used, where differences of .01 or less between the constrained and unconstrained models indicate evidence of equivalence (see also Widaman & Reise, 1997). The chi-square difference is also reported, but it is not used in interpretation because it tests the null hypothesis of identical fit between the constrained and unconstrained models – the absence of which does not necessarily indicate lack of invariance (Dimitrov, 2010). Results indicated evidence for full weak/metric invariance for each of the six acculturation components (see Table 4).

Table 4.

Weak/Metric Invariance Findings

Component	Fit Index
	Δχ2 (df)	ΔCFI	ΔRMSEA
U.S. Practices	17.23 (10)	.002	.002
Individualist Values	9.35 (6)	.003	.001
U.S. Identity	35.53 (11)	.008	.001
Hispanic Practices	17.19 (10)	.002	.002
Collectivist Values	12.90 (6)	.005	.002
Hispanic Identity	17.54 (11)	.002	.002

Strong/Scalar Invariance

The assumption of strong/scalar invariance was tested by beginning with the weak/metric invariance model and constraining each item intercept to be equal across languages. The fit of the strong/scalar invariance model is then compared against the fit of the weak/metric invariance model, using the same fit indices used to test for weak/metric invariance. In testing for strong/scalar invariance, in some cases the assumption of full invariance did not hold. However, partial invariance may still be tenable if freeing a relatively small number of intercepts provides a model for which Δ CFI ≤ .01 and Δ RMSEA ≤ .01 (Cheung & Rensvold, 2002). In our tests for strong/scalar invariance, we retained the assumption of partial invariance as long as fewer than half of the intercepts in the model had to be allowed to vary across languages.

Full strong/scalar invariance emerged for three of the six dimensions of acculturation – U.S. practices, individualist values, and Hispanic practices. Partial scalar invariance emerged for U.S. identity, collectivist values, and Hispanic identity (see Table 5). For U.S. identity, intercepts referring to spending time learning about U.S. customs and history, being active in social groups that include mostly Americans, and talking to others about the United States had to be freed for the assumption of partial scalar/strong invariance to hold. For collectivist values, the assumption of partial scalar/strong invariance required freeing the intercept for the item referring to family members needing to stick together. For Hispanic identity, the assumption of partial scalar/strong invariance required freeing the intercept for thinking about how one’s life will be affected by one’s ethnic group membership.

Table 5.

Strong/Scalar Invariance Findings

Component	Fit Index
	Δχ2 (df)	ΔCFI	ΔRMSEA
U.S. Practices	23.05 (10)	.003	.001
Individualist Values	16.59 (6)	.001	.004
U.S. Identitya	30.95 (8)	.009	.002
Hispanic Practices	17.55 (10)	.002	.002
Collectivist Valuesa	16.61 (5)	.008	.005
Hispanic Identitya	25.50 (10)	.004	.001

^aPartial scalar invariance was obtained for these dimensions. Values reported in the table reflect results of invariance tests after non-invariant parameters were freed.

Construct Validity Equivalence

Construct validity equivalence is suggested by Knight et al. (2009) as an additional step in establishing measurement equivalence across languages. Specifically, if a latent subscale factor relates to a criterion variable in the same way across languages, then the construct validity of the subscale can be considered to generalize across languages of assessment. To test for construct validity equivalence, we examined four models, each with two or three criterion variables (models with more than three criterion variables would not converge): one with self-esteem and depressive symptoms, one with personal identity synthesis and confusion, one with exploration (in breadth, in depth, and ruminative), and one with commitment making, identification with commitment, and passive commitment. Within each of the models tested, the assumption of convergent validity equivalence was evaluated by starting with the final strong/scalar invariance model (either full or partial strong/scalar invariance, depending on what was obtained for each dimension of acculturation) and constraining correlations between the acculturation dimension and each convergent validity variable to be equal across languages. We then used standard invariance testing criteria (Δ CFI ≤ .01 and Δ RMSEA ≤ .01) to test for convergent validity equivalence. We found evidence for full construct validity equivalence for all of the models tested (see Table 6). It is worth noting that, of the 228 parameters tested across all types of invariance (64 factor loadings, 64 item intercepts, and 100 construct validity correlations), only five were significantly unequal across languages.

Table 6.

Construct Validity Equivalence Findings

Component/Correlate	Fit Index
	Δχ2 (df)a	ΔCFI	ΔRMSEA
U.S. Practices
Adjustment	5.82 (4)	<.001	.001
Synthesis/Confusion	6.39 (4)	< .001	< .001
Exploration	1.22 (6)	.001	.002
Commitment	8.73 (6)	< .001	< .001
Individualist Values
Adjustment	10.22* (4)	.005	.001
Synthesis/Confusion	7.44 (4)	.003	.001
Exploration	1.39 (6)	.004	.004
Commitment	21.76** (6)	.009	.003
U.S. Identitya
Adjustment	3.15 (2)	< .001	.001
Synthesis/Confusion	0.26 (2)	< .001	< .001
Exploration	2.55 (3)	< .001	.001
Commitment	1.55 (3)	.001	< .001
Hispanic Practices
Adjustment	7.50 (4)	< .001	< .001
Synthesis/Confusion	2.76 (4)	< .001	.001
Exploration	10.44 (6)	.001	< .001
Commitment	5.11 (6)	< .001	.001
Collectivist Values
Adjustment	12.09* (4)	.005	.002
Synthesis/Confusion	6.56 (4)	.001	.001
Exploration	5.59 (6)	< .001	.002
Commitment	14.18* (6)	.003	.002
Hispanic Identity
Adjustment	1.83 (2)	< .001	<.001
Synthesis/Confusion	2.97 (2)	.001	< .001
Exploration	4.90 (3)	< .001	.001
Commitment	3.78 (3)	< .001	.001

^aDegrees of freedom for each construct validity model are obtained by multiplying the number of subscales tested within the acculturation component by the number of correlates included within that given construct validity model.

^*p < .05

^**p < .01

^***p < .001

Latent Mean Differences

Because at least partial metric/weak, scalar/strong, and construct validity equivalence was found for all six components of acculturation, we proceeded to conduct latent mean comparisons across language of administration. Latent mean comparisons (the latent-variable equivalent of analyses of variance; Hancock, Lawrence, & Nevitt, 2000) are conducted by constraining the mean for one of the groups to be zero, and comparing the other group mean to the reference group mean using a standard z-test (Hancock, 2004). For latent mean comparisons, the mean for a reference group (in this case the English condition) was set to zero, and the mean for the other group was freely estimated (Hancock, 2004). Effect sizes for these mean differences are reported as Cohen’s d, which represents the number of pooled standard deviations between the English and Spanish means.

Because effects of measurement error and of potential differences in factor structures and item intercepts have been removed from the latent variables representing the acculturation dimensions (Little, 1997; Raykov, Marcoulides, & Millsap, in press), latent mean comparisons can be considered error-free (Hancock, 2004). As a result, we did not control for inflated Type I error risk resulting from conducting multiple mean comparisons.

U.S. Practices

Latent mean comparisons indicated that the latent mean for U.S. cultural behaviors (-0.20, z = 4.14, p < .001, d = 0.38) were significantly higher in the English language conditions. English-language use (−0.02, z = 0.41, p = .69) did not differ significantly across the two language conditions.

Individualist Values

Latent mean comparisons indicated that the latent mean for horizontal individualism (−0.14, z = 2.20, p < .03, d = 0.21) was significantly higher in English language condition, but that the latent mean for vertical individualism (0.12, z = 2.12, p < .04, d = 0.21) was significantly higher in the Spanish language condition.

U.S. Identity

Latent mean comparisons indicated that the English and Spanish conditions did not differ significantly on U.S. identity, mean = −0.01, z = 0.13, p = .90.

Hispanic Practices

Latent mean comparisons indicated that the latent mean for Hispanic cultural behaviors (mean = −0.22, z = 2.86, p < .005, d = 0.26) was significantly higher in the English language condition. Spanish-language use (mean = −0.06, z = 0.95, p = .34) did not differ significantly across the two language conditions.

Collectivist Values

The latent mean for vertical collectivism (−0.11, z = 1.75, p = .08, d = 0.17) was marginally significantly higher in the English language condition. Horizontal collectivism (−0.03, z = 0.69, p = .49) did not differ significantly across the two language conditions.

Hispanic Identity

Latent mean comparisons indicated that the English and Spanish conditions did not differ significantly on Hispanic identity, mean = −0.08, z = 1.65, p < .10.

Discussion

The present study was conducted to ascertain the extent of linguistic measurement equivalence in acculturation measures among bilingual Hispanic college students. The present study was also designed to ascertain the extent to which either cultural frame switching – higher endorsement of U.S. acculturation indices when participants are assessed in English, and higher endorsement of Hispanic acculturation indices when participants are assessed in Spanish – or stereotype threat (the opposite pattern of effects) would emerge once measurement equivalence had been established. Bilingual Hispanic participants were randomly assigned to complete the acculturation and criterion (convergent validity) measures in either English or Spanish. In the context of evaluating measurement equivalence and cultural frame switching, random assignment (along with empirical checks for fluency in both languages) serves to control for potential confounders such as language competency. In contrast, when participants are allowed to self-select into languages of administration, there is no way of knowing whether participants completing their measures in English are equally competent in reading and understanding English (and likewise for those completing their measures in Spanish). Random assignment to language permits us to examine the equivalence of the factor structures, item intercepts, and construct validity under ideal conditions (i.e., all participants are fully bilingual, and factors that might lead participants to choose one language over the other are not permitted to operate).

Measurement Equivalence Findings

Following the measurement invariance/equivalence literature (Dimitrov, 2010; Knight et al., 2009; Raykov et al., in press), for each measure we tested for four types of invariance/equivalence: (a) configural, where the subscale fits the data well in both language groups; (b) metric/weak, where factor loadings are set equal across languages; (c) scalar/strong, where factor loadings and item intercepts are set equal across languages; and (d) construct validity, where factor loadings, item intercepts, and correlations with variables that would be theoretically expected to relate to acculturation are set equal across languages. For U.S. practices, individualist values, and Hispanic practices, we found full measurement equivalence at all four levels. For U.S. identity, collectivist values, and Hispanic identity, we found full configural, metric/weak, and construct validity equivalence, but some intercepts had to be freed for the assumption of scalar/strong invariance to hold. Table 7 summarizes the measurement equivalence and cultural frame switching findings.

Table 7.

Summary of Measurement Equivalence and Latent Mean Difference Analyses across Language^a

Component/Subscale	Measurement Invariance/Equivalence				Mean Differences
	Configural	Metric/Weak	Scalar/Strong	Convergent
U.S. Acculturation
English Language	Yes	Yes	Yes	Yes	None
U.S. Practices	Yes	Yes	Yes	Yes	English > Spanish
Horizontal Individualism	Yes	Yes	Yes	Yes	English > Spanish
Vertical Individualism	Yes	Yes	Yes	Yes	Spanish > English
U.S. Identity	Yes	Yes	Partial	Yes	None
Hispanic Acculturation
Spanish Language	Yes	Yes	Yes	Yes	None
Hispanic Practices	Yes	Yes	Yes	Yes	English > Spanish
Horizontal Collectivism	Yes	Yes	Yes	Yes	None
Vertical Collectivism	Yes	Yes	Partial	Yes	English > Spanish
Hispanic Identity	Yes	Yes	Partial	Yes	None

^aA “Yes” means that criteria for full invariance/equivalence were satisfied.

Note: Configural invariance refers to the subscale fitting the data well in both language groups. Metric/weak invariance refers to factor loadings being equal across languages. Scalar/strong invariance refers to both factor loadings and item intercepts being equal across languages. Convergent validity equivalence refers to factor loadings, item intercepts, and correlations with variables that would be theoretically expected to relate to acculturation being equal across languages.

At first glance, these findings may appear to suggest that acculturation data can be pooled across language of assessment in Hispanic samples. Our findings strongly suggest that, in bicultural and bilingual samples, aggregating across language of assessment may be appropriate. However, it might be more appropriate to conceptualize the present study as the first step in a new line of research on the conditions under which data should or should not be aggregated across assessment languages. The present sample was restricted to fully bilingual participants living in a highly bicultural context and attending a post-secondary institution where the majority of students are Hispanic. Our results therefore provide evidence that linguistic measurement equivalence among acculturation instruments is possible under these conditions. Additional research, following up on the present findings, is needed to ascertain whether measurement equivalence would also emerge in participants who were less fluent in one language than in the other, who reside in areas that are unwelcoming (either overtly or covertly) to Hispanic immigrants, or who work or attend universities, companies or institutions where Hispanics are not the numerical and political majority. Even if the bilingual participants in our study had been permitted to select their language of administration, the measurement equivalence findings may have been different from what we have reported here. Additional studies are needed, each of which should vary specific factors such as research location, fluency in both languages, Hispanic groups included in the sample, and representation of Hispanics within the local context. Given the increasing prevalence of online studies (e.g., Buhrmester, Kwang, & Gosling, 2011; Reips, 2002), differences by mode of administration (in person versus online) should also be examined. Such research will help us determine the precise conditions under which data can and cannot be pooled across assessment languages. We now discuss research location and bilingualism as potential influences on the extent to which measurement equivalence is likely to be obtained in acculturation scores across languages.

Research Location

Because Miami is a unique cultural context – not only because of the predominance of Hispanics but also because of the diversity of Hispanic groups that are represented in Miami – it is important to replicate the present study in other contexts, such as the Southwest and Northeast, where specific Hispanic groups (e.g., Mexican Americans, Puerto Ricans, or Dominicans) comprise the majority of Hispanic residents. As Schwartz and Unger (2010) have suggested, biculturalism may be most adaptive in a bicultural context, and less adaptive in contexts where one or both cultural streams are devalued or stigmatized. The concentration (or lack thereof) of Hispanics in the local context may also be important to consider – suggesting that nontraditional immigration destinations may provide very different results compared to traditional gateway areas. Differences in the affective valence of Spanish-language use might affect the factor structures or item intercepts generated by acculturation measures. Replicating the present study in other settings is essential.

Bilingualism

The most striking difference between the present study and most other acculturation research is the extent to which the majority of participants are fluent in both their heritage languages and the language of the country or region where they have settled. Now that we have shown that measurement equivalence can emerge among fully bilingual samples, a next step would be to implement a hybrid design including both bilingual participants and those who are monolingual (or more proficient in one language or the other). Such future work might include monolingual/primary English speakers completing measures in English, and monolingual/primary Spanish speakers completing measures in Spanish, in addition to bilingual participants who are randomly assigned to languages of assessment, and perhaps additional bilingual participants who are allowed to select their assessment language (cf. Pouliasi & Verkuyten, 2007; Ramírez-Esparza et al., 2006). These additional groups would permit examination of whether, across both naturally occurring and artificially created language groups, acculturation measures’ factor structures, item intercepts, and links with criterion measures are equivalent across languages of assessment. Such research would provide much-needed information regarding differences between random assignment and self-selection into assessment languages, as well as regarding differences between bilingual participants and those who are more proficient in one language than the other. The sum total of this additional knowledge would provide researchers and clinicians with guidance regarding how to proceed in studies or situations where individuals have self-selected into languages of administration, and where one must decide whether, or how, to combine or compare participants who have completed their assessments in English and in Spanish. Another potential scenario that our findings – and those of studies building on our work – may be able to inform is longitudinal studies where recent immigrants are assessed in Spanish at earlier timepoints but switch to English at later timepoints (see Schwartz, Unger, et al., 2012, for an example of such a study). Conducting longitudinal analyses on data where participants switch assessment languages requires linguistic measurement equivalence (Little, 2013).

Cultural Frame Switching/Stereotype Threat Findings

Although our measurement equivalence findings were quite strong, our mean difference findings were mixed between cultural frame switching and stereotype threat. Of the five significant latent mean differences that emerged, two were consistent with cultural frame switching – scores for both U.S. practices and horizontal individualism were significantly higher in the English-language condition than in the Spanish-language condition. The other three significant findings were consistent with stereotype threat: vertical individualism was significantly higher in the Spanish-language condition, and both Hispanic practices and vertical collectivism were significantly higher in the English-language condition.

At first glance, these findings do not appear to tell a coherent story. However, considering the cultural context of Miami may provide some perspective. By definition, cultural frame switching occurs when one cultural stream is most appropriate in one setting and the other cultural stream is most appropriate in another setting – such as when Hispanic behaviors are highly encouraged at home or with family but U.S.-oriented behaviors are encouraged at work or school (cf. Cheng et al., 2006; Verkuyten & Pouliasi, 2006). However, in the bicultural context of Miami, both Hispanic and U.S.-orientations – or a blend of the two, such as speaking “Spanglish” – are encouraged across a range of settings. Because both English and Spanish are used widely in Miami, neither language is likely to be associated with a specific setting or set of behaviors – and therefore switching languages may not prime a specific set of cultural behaviors, values, and identifications associated with a given language. A somewhat different pattern may have emerged in other contexts where each language is more closely associated with a specific setting (or group of settings).

An alternative explanation – based in an extension of stereotype threat theory – might suggest that, for bicultural individuals who face competing pressures from the heritage and U.S. cultural communities, completing an assessment in one language unconsciously primes values associated with the other. U.S. culture is stereotyped as extremely individualistic (Hirschman, 2003), such that priming the English language may activate this stereotype and lead Hispanics to endorse greater degrees of collectivism. On the other hand, Hispanic cultures are stereotyped as highly collectivistic and familistic (Galanti, 2003), and being assessed in Spanish may prime these stereotypes and lead Hispanics to endorse greater degrees of individualism. It is noteworthy that these effects emerged only for vertical individualism and collectivism, which refer to how participants think about hierarchical relationships (e.g., the importance accorded to individual achievements versus obligations to family). Indeed, cultural stereotypes of the United States and of many Hispanic countries appear to be based primarily on vertical values (e.g., how parents and other family members should be regarded, and what one’s obligations are to them) – suggesting further that these stereotypical values may be primed by varying the language of assessment. The present findings may call into question the extent to which individuals can be accurately expected to report their cultural values regarding hierarchical relationships (e.g., with parents and other family members) in either English or Spanish. As a result, it may be advisable for clinicians to establish working and trusting relationships with Hispanic individuals before attempting to assess their cultural values in either language. Researchers should also be mindful of potential cultural frame switching or stereotype threat effects on reports of vertical values, although latent mean differences do not contraindicate pooling data across languages (as long as the other types of invariance are present).

Because many cultural frame switching studies are conducted in laboratory-based settings (e.g., Benet-Martínez et al., 2002; Luna et al., 2008; Ramírez-Esparza et al., 2006), the effect of the local cultural context on cultural frame switching (and stereotype threat) needs to be more widely studied. Because different Hispanic groups have different histories in the United States (Suárez-Orozco & Páez, 2002), cultural frame switching and stereotype threat may operate differently across Hispanic nationalities. Mexicans and Puerto Ricans, the two largest Hispanic subgroups in the United States (Ennis, Ríos-Vargas, & Albert, 2011), but who are not well represented in Miami, tend to come from relatively low socioeconomic backgrounds and to be targets for discrimination (Acosta-Belen & Santiago, 2006; Henderson, 2011). Steele (1997) suggests that stereotype threat is most relevant for marginalized groups and those low in socioeconomic status. Cultural frame switching may also be more salient for Puerto Ricans in the Northeast or for Mexican Americans in the Southwest, but less so for Cubans and South Americans in the Miami area – because use of Spanish in public places is less strongly encouraged in areas where Mexican Americans and Puerto Ricans have settled. Cubans and South Americans tend to enjoy more political and economic advantages in the Miami area – Cubans because most political and economic leadership positions in Miami are held by individuals of Cuban descent (Stepick & Stepick, 2002), and South Americans because many of them are educated professionals who immigrated to escape political problems (Sabogal, 2005). Again, the potential moderating effects of local setting and Hispanic nationality on cultural frame switching and stereotype threat warrant investigation. If we are to understand cultural frame switching and stereotype threat more fully, we must understand the statistical dimensions that these phenomena affect, the conditions in which they occur, and the specific prompts that activate them within a given setting.

Limitations and Directions in Future Research

The present findings should be interpreted in light of at least four important limitations. First, although all of our participants demonstrated themselves to be bilingual, the use of a college student sample is not representative of the Hispanic population at large, either in Miami or in the U.S. as a whole. Our findings might have been somewhat different had we included non-college participants as well. It is worth noting that 60% of our sample reported annual family incomes of $50,000 or less – suggesting that our sample was not especially socioeconomically advantaged. However, the gender imbalance that often characterizes college samples may have caused language use patterns specific to women to be overrepresented in our results. Replication with community samples is important to increase confidence in the generalizability of our findings.

Second, as we noted above, although the use of random assignment to languages of administration required a bilingual sample, we do not know the proportion of truly-bilingual participants in most studies where data are to be combined across languages of assessment. The extent to which our findings of measurement equivalence apply to participants who are not bilingual cannot be ascertained from the present data – additional groups of non-bilingual English and Spanish speakers would be required to test for measurement equivalence across the full range of linguistic competence.

Another approach that future research might include involves a within-subjects experimental design, where each participant completes the full battery of measures in English and in Spanish (with the order of languages counterbalanced). In such a design, each participant serves as her or his own control. This strategy considerably increases the precision with which the effects observed can be attributed to language of assessment, rather than to individual differences between participants assigned to two separate conditions (Maxwell & Delaney, 2004).

Third, we did not include potential moderating variables that might have identified subgroups for whom cultural frame switching and stereotype threat effects did and did not occur. For example, Benet-Martínez et al. (2002) found that, among Chinese American bilingual individuals, those high in bicultural identity integration (ability to reconcile and combine their two cultural streams into an individualized cultural mosaic) responded to Chinese or U.S. primes in the expected direction. However, Benet-Martínez et al. found that Chinese American bilinguals who were low in bicultural identity integration responded to Chinese or U.S. primes in a direction opposite of that which would be expected. The relatively small number of cultural frame switching and stereotype threat findings in our study may have resulted from not having included bicultural identity integration or other possible moderators.

In conclusion, and despite these limitations, the present study has indicated that data generated using acculturation measures can, at least under some conditions, be combined or compared across languages of administration. In our results at least, factor structures, item intercepts, and associations with theoretically related variables were all strongly consistent between English and Spanish versions of the acculturation measures that we used. This finding of measurement equivalence appeared to apply to measures of Hispanic and U.S. practices, values, and identifications – suggesting that it may be possible to combine or compare data on multiple dimensions and domains of acculturation across languages, at least among bilingual individuals. Our results may be especially important given the need to provide research materials in multiple languages for participants from acculturating or bicultural populations. We hope that the current study will inspire a research literature on linguistic measurement equivalence, cultural frame switching, and stereotype threat in acculturation-related instruments.