INSTRUMENTS MEASURING PERCEIVED RACISM/RACIAL DISCRIMINATION: REVIEW AND CRITIQUE OF FACTOR ANALYTIC TECHNIQUES

Abstract

Several compendiums of instruments that measure perceived racism and/or discrimination are present in the literature. Other works have reviewed the psychometric properties of these instruments in terms of validity and reliability and have indicated if the instrument was factor analyzed. However, little attention has been given to the quality of the factor analysis performed. The aim of this study was to evaluate the exploratory factor analyses done on instruments measuring perceived racism/racial discrimination using guidelines from experts in psychometric theory. The techniques used for factor analysis were reviewed and critiqued and the adequacy of reporting was evaluated. Internet search engines and four electronic abstract databases were used to identify 16 relevant instruments that met the inclusion/exclusion criteria. Principal component analysis was the most frequent method of extraction (81%). Sample sizes were adequate for factor analysis in 81 percent of studies. The majority of studies reported appropriate criteria for the acceptance of un-rotated factors (81%) and justified the rotation method (75%). Exactly 94 percent of studies reported partially acceptable criteria for the acceptance of rotated factors. The majority of articles (69%) reported adequate coefficient alphas for the resultant subscales. In 81 percent of the studies, the conceptualized dimensions were supported by factor analysis.

Perceived racism is a variable relevant to the health and well-being of minority populations. As indicated by Drevdahl, Taylor, and Phillips (1), there is no question that living in a racialized society can lead to biologic differences in health. Thus, the study of perceived racism across disciplines is important in minority and vulnerable populations who perceive racism or discrimination in their lives.

In recent years, researchers have made great strides in developing instruments to measure perceived racism to facilitate developing a knowledge base on the topic. Compendiums of these instruments have been published, starting with the works by Utsey (2) and Krieger (3). To update these works, Kressin, Raymond, and Manze (4) presented a review of the psychometric properties of 34 measures of perceived racism and/or discrimination, most of which focused specifically on the experiences of African-American patients.

More recently, Bastos, Celeste, Faerstein, and Barros (5) did a systematic review of instruments measuring racial discrimination and health, with a focus on their psychometric properties. Their review included 24 instruments published in the last 12 years, of which 23 originated in the United States. The instruments reviewed had been administered to multiple racial or ethnic groups, such as Blacks, Asians, West Indians, and Whites.

In several of the above reviews, the authors critiqued psychometric properties of the instrument, such as reliability and validity, and indicated when the instrument was factor analyzed (4, 5). However, little attention was given to the quality of the factor analyses procedures performed. The purpose of this present work is to evaluate the exploratory factor analyses (EFA) done on psychometrically sound instruments measuring perceived racism, using guidelines from experts in psychometric theory. This critique helps to determine whether the underlying premises of the instrument have been supported by factor analysis results and whether the hypothesized subscales of the instrument actually emerged. Confirmatory factor analysis will not be included in this article.

CRITERIA FOR FACTOR ANALYSIS

EFA is a modern mathematical and statistical tool used by researchers to determine the number and kinds of factors underlying their instrument (6) and for data reduction purposes, whereby a large number of variables (hereafter referred to as items) are reduced into a smaller number of factors (7). The anticipation with multidimensional instruments is that the results will be compatible with the hypothesized underlying dimensions.

The first consideration for factor analysis procedures is sample size relative to the number of items on the instrument. Nunnally and Bernstein (6) recommend having at least 10 times as many subjects as items on an instrument to minimize sampling error and to ensure stability of factor analysis results across studies. Gorsuch (8) takes a less stringent approach to sample size by recommending an absolute minimum ratio of at least 5 subjects per item, “but no less than 100 individuals for any analysis” (p. 332). In practice, researchers seldom justify sample size with factor analysis procedures.

The second consideration with factor analysis is determining which factor analysis procedure to use to extract factors from the correlation matrix prior to rotation. Nunnally and Bernstein (6) prefer principal components analysis (PCA) as an optimum approach, because PCA factors explain more variance than loadings obtained from any other method of factoring. Gorsuch (8) adds principal axis factoring (PAF) as an option in the principle factor solutions. PCA is more widely used in the literature than principal axes or other factor extraction methods such as maximum likelihood estimation (MLE). Additional criteria are used in factor analysis extraction procedures to determine the number of substantive factors underlying the instrument. The root one criterion states that factors with eigenvalues greater than one should be rotated (9) and the scree test criterion states that factoring should cease when the plotted graph of the eigenvalues levels off, forming a straight line with an almost horizontal slope (10). These latter criteria are often applied to factor extraction procedures by researchers who also report the percent of variance explained by factors.

The third consideration in factor analysis concerns factor rotation. Nunnally and Bernstein (6) suggest that the major reason to rotate factors is to obtain a more interpretable solution, referred to as simple structure. Researchers need to determine the type of rotation to use, which is either an orthogonal or oblique rotation, with a variety of techniques for each. Orthogonal rotation assumes that factors are uncorrelated and thus are rotated for independence (6, 8). Oblique rotation assumes factors are correlated at some level and is appropriate when the theoretical expectation is a general factor (8). According to Gorsuch (8), higher order factors are implicit in all oblique rotations and should be extracted, but Nunnally and Bernstein (6) express ambivalence regarding the necessity of extracting higher order factors. Tabachnick and Fidell (11) argue that the choice of orthogonal or oblique rotation is determined by the theory underlying the instrument, but researchers seldom report this rationale. Tabachnick and Fidell also suggest that oblique rotation be accepted only when factor correlations are 0.32 or greater. Researchers also have to decide whether to allow the rotated factors to emerge on their own or to force (order) the number of factors rotated based on the number of substantive factors identified in factor extraction. This decision is seldom reported.

Another a priori decision that should be made is what item-factor loading will be accepted in the factor rotation. Nunnally and Bernstein (6) indicate that a 0.30 loading of an item on a factor is the minimum acceptable level, but suggest that a factor is better defined when the criterion is set higher for accepting loadings on a factor. Gorsuch (8) suggests that trivial factors are those that do not have at least two or three item loadings above a certain specified level, such as 0.30. Some researchers also indicate the difference needed between a factor loading on the primary and any other factor for acceptance of item-factor loadings (cross-loadings). For example, occasionally researchers might say an item-factor loading of at least 0.40 on the primary factor and a difference of at least 0.20 between a loading of the item on the primary factor and any other factor will be used to determine item retention in rotated factors. Once factors are accepted, they should be tentatively named. Coefficient alpha reliabilities should be reported for items loading on each accepted factor and should be 0.70 or greater (6).

METHODS

Literature Search

Racism and racial discrimination measures were found using several search engines and databases from years 1980 to 2010, including CINAHL, PsychInfo, Medline, PubMed, and general Internet searches. Key words such as “perceived racism,” “racism,” “racism instruments,” and “perceived discrimination” identified six tools measuring perceived racism and/or discrimination. Eight additional tools were found via comprehensive literature and Internet searches. Their titles were used to locate articles using the tools.

Ancestry searching and journal hand searching also took place and 32 additional racism/discrimination measures were found. Of the total of 46 measures, 16 met the inclusion/exclusion criteria set for this review, which are presented in this analysis. The 16 instruments were published in 15 articles.

Inclusion/Exclusion Criteria

Since perceiving racism involves sensing racist attitudes (beliefs) and racist actions (discrimination), instruments measuring both aspects of perceiving racism are presented. Criteria for inclusion were multiple-item and self-report instruments that: (a) measure at least one aspect of perceiving racism, (b) measure perceived discrimination due to one’s race/ethnicity, and (c) have scales or subscales that have been factor analyzed in one or more studies. Excluded were single-item measures of perceived racism/discrimination.

INSTRUMENTS

Perceived Racism Scale

The Perceived Racism Scale (PRS) is a 51-item instrument measuring the multidimensional experience of White racism, as perceived by African-American adults (12). This scale measures perceived exposure to racism defined as “a belief or attitude that some races are superior to others and discrimination based on such a belief …attitudinal and behavioral manifestations of the belief in the inherent inferiority of individuals of certain ethnicities” (12, p. 155). The first section of the PRS has 43 items with a 7-point summated rating scale, measuring the frequency of exposure to racist events. The second assessed coping with racism.

Three emotional responses and six behavioral responses are assessed over five domains: on the job, in academic settings, in public places overtly, in public places covertly, and racist statements from Whites over the past year and one’s life. The emotional responses scale asks respondents the extent to which they feel a particular emotion during a racist encounter (i.e., feeling anger/frustration, depressed, and strengthened). The behavioral coping scale measures responses to racism (i.e., working harder/trying to change things, avoiding/ignoring, praying, forgetting it, getting violent, and speaking up). Scores for the PRS frequency of exposure scale can range from 0 to 301; higher scores indicate more frequent perceptions of exposure to racism.

McNeilly and colleagues (12) assume that racism experiences occur in a variety of unrelated domains. Sections 1 and 2 of this scale were factor analyzed separately. PCA of 43 items and 8 items, respectively, on a sample of 273 African-American college students (57 male and 108 female) and community members (10 male and 15 female) resulted in the acceptance of five factors for the first section (frequency of exposure) and nine factors for the second section (coping responses) based on a scree plot analysis of eigenvalues greater than one. After both PCAs, orthogonal and oblique rotation were employed, and all items with factor loadings above 0.50 were retained. Adequate coefficient alphas were reported for all factors, except one (alpha = 0.64; getting violent).

Although it was theorized that racism frequency would occur in four domains (employment, academic, in public, by racist statements), five factors were accepted since the analysis resulted in the separation of the public domain into two separate factors (overt public and subtle public). In addition, although it was theorized that there would be 17 emotional and behavioral coping responses to racism, only nine factors emerged as several emotions and behaviors collapsed into one factor (sad/powerless/hopeless/ashamed became depressed affect, anger/frustration, avoiding/ignoring, working harder/trying to change things).

Items from two behavioral responses (accepting it and keeping it to myself), failed to load on any one factor so these factors were eliminated. However, the items were retained due to their potential to provide useful clinical information. Though the factors did not emerge exactly as theorized, the multiple dimensions of racism were supported.

The Telephone Administered Perceived Racism Scale (TAPRS)

The TAPRS is a 61-item multidimensional instrument developed from the PRS (12); it is designed to measure both perceptions of and responses to racist experiences among employed Black women by telephone in approximately 15 minutes (13). The scale contains five main subscales that include: (a) experience of racism as a group or by the individual, (b) emotional responses, (c) behavioral responses, (d) concern for child(ren), and (e) past experiences of racism. Racism was defined as “a belief or attitude that some races are superior to others and discrimination based on such a belief” (13, p. 252) and perceiving racist or discriminatory incidents is a chronic stressor. The emotional and behavioral coping response items were factor analyzed in this study.

Vines and colleagues (13) used the original 51-item PRS and added 10 items to construct the TAPRS. Principal Axis Factoring (PAF) of 61 items was used on a sample of 476 Black women aged 36 to 53 to determine the underlying dimensions of the emotional and behavioral responses subscales. Correlated dimensions were expected so oblique rotation was used. A five-factor structure representing separate coping responses was accepted based on scree plot analysis of eigenvalues greater than one. These factors were named active emotions, passive emotions, passive behaviors, internal active behaviors, and external active behaviors. The percent of variance accounted for by each factor was reported and items with factor loadings of 0.35 or higher were retained. One behavioral response item (i.e., acting out angrily) failed to load on any one factor and was therefore not retained. The coefficient alphas for most factors were reported as adequate (≥0.70), except for the one factor that had an alpha of 0.68 (passive behaviors).

The Perceptions of Racism Scale (TPRS)

The TPRS is a 20-item measure of African-American adults’ lifetime experience of racism by asking them to indicate the degree to which they agree with statements about racism (14). Items are rated on four-point summated rating scale. Scores can range from 20 to 80; higher scores indicate a higher perception of racism. Perceived racism was defined as the individual judgment that racism exists when an individual perceives differential treatment, experiences, or attitudes and acts on the basis of those perceptions, which can be affective, behavioral, and cognitive.

Green (14) conceptualized that racism is multidimensional and occurred in two unrelated domains: (a) the health care system and (b) general societal racism. Therefore, PCA with orthogonal rotation was used on 20 items in a sample of 109 African-American women, with another PCA on 136 African-American women. One factor was accepted and accounted for 32 percent of the variance, which did not support the multidimensional conceptualization (14). Even though three items did not have factor loadings above 0.43, all items were retained in the scale. The coefficient alpha for the total scale score was adequate for both samples.

The Index of Race-Related Stress (IRRS)

The IRRS is a 46-item instrument designed to measure race-related stress, experienced by African-American adults as a result of specific daily events of racism and discrimination (15). Race-related stress was defined as the “occurrence and perceived magnitude of specific events of racism and discrimination that African Americans potentially experience in their daily lives” (15, p. 491). The instrument takes into account both frequency of exposure and the appraisal of racist events. Subjects are asked to indicate which racist event has been experienced by them or family members in their lifetimes. The scale items have a five-point summated rating scale ranging from 0 to 4. Scores can range from 0 to 184; higher scores indicate more stress related to racism. The scale is composed of four subscales of racism (cultural, institutional, individual, collective) and a global racism measure, each scored separately.

Utsey and Ponterotto (15) theorized that racism has multiple dimensions and is perceived in four distinct domains named: (a) institutional, (b) cultural, (c) individual, and (d) collective racism. PCA was performed on 59 items in a sample of 377 African-Americans (203 women and 163 men). A second analysis was performed on a sample of 302 African-Americans. Of this sample, 167 were females, 115 were males, 90 percent reported being American-born Blacks and 8 percent reported being Caribbean-born Blacks. Four factors with eigenvalues greater than one were highly interpretable after scree plot analysis of both samples. One, two, three, and four component extractions were forced, using both oblique and orthogonal rotation methods. The four-factor model was most interpretable, accounting for 38 percent of the variance. Item loadings of 0.40 or higher were retained if they had at least a 0.15 difference between loadings. Overall, the results conformed to the theoretical definition and factors expected. Coefficient alphas for the factors were adequate.

Index of Race-Related Stress (IRRS)-Brief Version (B)

The IRRS-Brief version (16) is a 22-item shorter version of the original IRRS (15), designed to measure race-related stress experienced by African-Americans as a result of specific events of racism and discrimination. This scale is a four-point summated rating scale and asks about experiences of specific racist events occurring over their lifetime and indicate the degree of stressfulness of the incidences. Scores range from 0 to 22, with higher scores indicating more race-related stress. The scale has three subscales: (a) Cultural Racism, (b) Institutional Racism, and (c) Individual Racism. A global racism measure can also be computed based on a combined score from the three subscales.

According to Utsey (16), perceiving racism is a multidimensional experience occurring in four unrelated domains (individual, institutional, collective, and cultural racism). Therefore, PCA with orthogonal rotation was performed on 59 items in a sample of 310 African-American university and community adults (207 were female and 92 were male). After orthogonal rotation, items were retained if they had loadings greater than 0.57 and exceeded the next highest loading by a margin of 0.15. Based on these criteria, one proposed factor (collective racism) had only two items loading on it and was therefore eliminated. Three factors were accepted, representing Institutional, Individual, and Collective Racism. Coefficients alphas were adequate, except for the Institutional Racism subscale (alpha = 0.69).

The Index of Race-Related Stress for Adolescents

The IRRS was modified for use in adolescents (17). This is a 32-item instrument designed to measure race-related stress, as originally defined theoretically and operationally by Utsey and Ponterotto (15). Subjects indicate which racist event they or family members have experienced in their lifetimes and their appraisal of these events. Subjects respond to a five-point summated rating scale ranging from 0 to 4, with higher scores indicating more stress (17). Subscales are scored separately.

Based on the theory used for the original IRRS (15), four uncorrelated factors were anticipated by Seaton (17). PCA of 36 items on a sample of 324 urban African-American adolescents (171 females and 153 males) resulted in the acceptance of three factors (individual racism, cultural racism, collective/institutional racism) based on scree plot analysis of eigenvalues greater than one. Orthogonal rotation resulted in the acceptance of three interpretable factors accounting for 36 percent of the variance. Item loadings of 0.35 or higher on a factor and at least 0.15 times greater than loadings on other factors were accepted. Four items were deleted from the scale because they did not meet the aforementioned criteria. Coefficient alphas were reported as adequate for each factor.

Perceived Ethnic Discrimination Questionnaire (PEDQ)

The Perceived Ethnic Discrimination Questionnaire is a 22-item instrument designed to measure ethnicity-related stress by measuring the frequency of various acts of ethnic discrimination in all ethnicities. Ethnic discrimination was defined as unfair treatment attributed to one’s ethnicity (18). This seven-point summated rating scale asks respondents to indicate how often in the previous three months certain racist events were directed at them. This scale has four subscales: (a) Disvaluation Action, (b) Avoidance, (c) Verbal Rejection, and (d) Threat Aggression. Scores can range from 22 to 154, with higher scores indicating more experiences of ethnic discrimination/racism.

Items were developed to reflect seven content areas (disvaluing actions, denial of equal treatment, threat, aggression, exclusion, avoidance, and verbal aggression) of ethnic discrimination, representing separate underlying dimensions (18). PAF was performed on 22 items in a sample of 333 White (n = 208), African-American/Black (n = 34), Hispanic/Latino (n = 31), and Asian/Pacific Islander (n = 60) individuals (91 males and 242 females). Five factors were produced with eigenvalues greater than one, accounting for 69 percent of the variance. However, four factors with eigenvalues greater than one were accepted, accounting for 60 percent of the variance. These items were named threat aggression, verbal rejection, avoidance, and disvaluation. After orthogonal rotation, items loading less than 0.40 were excluded. Factors with two items loading greater than 0.40 were not accepted. The denial of equal treatment and exclusion items, along with their proposed factors, were deleted because they did not meet the aforementioned loading criteria. Coefficient alphas for each accepted factor were adequate.

Perceived Ethnic Discrimination Questionnaire Community Version (PEDQ-CV)

The Perceived Ethnic Discrimination Questionnaire Community Version (PEDQ-CV) assesses perceived racial or ethnic discrimination in community-dwelling adults and any other ethnic group (19). This originally 70-item questionnaire measures multiple dimensions of everyday racism and lifetime experiences of ethnic discrimination. Perceived racism and ethnic discrimination were defined as unfair treatment received because of one’s ethnicity, race, or culture of origin (19). The scale has five subscales: (a) Lifetime Exposure Discrimination Scale (34 items), (b) Discrimination in the Media (3 items), (c) Discrimination Against Family Members (6 items), (d) Discrimination in Different Settings (9 items), and (e) Past Week Discrimination (10 items). After factor analysis, some items were deleted to form the four sub-scales resulting in 62 items; however, 70 items were retained for the entire tool. For the first four scales, participants indicate how often they experience ethnic discrimination during their lifetime on a five-point summated rating scale. The Past Week discrimination scale inquires about everyday discrimination experiences rated on a four-point summated rating scale. Two additional items assess inter-group versus intra-group ethnic discrimination. The lifetime exposure discrimination scale is the focus of this analysis.

Items were taken from the original version of the PEDQ (18) and revised to fit the experiences of community-dwelling adults (19). The theoretical definition of ethnic discrimination suggests different discrimination exists, and therefore distinct and uncorrelated dimensions were expected (19). PCA was performed on the 34-item Lifetime Exposure Discrimination scale. The sample consisted of 301 community-dwelling multiethnic adults who self-identified as either Black (n = 174), Latino (n = 18), White (n = 7), Asian (n = 4), Native American (n = 9), or mixed (n = 9). There were 232 females and 66 males.

This analysis initially produced seven factors with eigenvalues greater than one. After orthogonal rotation, four factors were accepted after all of their items met the criteria for the acceptance of rotated factors. The factors included: (a) exclusion/rejection, (b) stigmatization discrimination, (c) discrimination at work/school, and (d) threat/aggression. These factors had three or more items, were interpretable, and explained 52 percent of the variance. Item-factor loadings were 0.55 or higher. The coefficient alpha for the total scale was adequate.

Four separate subscales were created based on factor analysis of the Lifetime Exposure Discrimination items. Items were accepted if they had factor loadings of 0.50 or greater. The factors were named: (a) exclusion/rejection (nine items), (b) stigmatization discrimination (six items), (c) threat/aggression (four items), and (d) discrimination at work (four items). These scales are scored separately; their coefficient alphas were adequate.

Perceived Ethnic Discrimination Questionnaire Community Version-Brief (PEDQ-CV-B)

A brief version (17 items) of the PEDQ-CV has been created from the 34-item Lifetime Exposure Discrimination scale (19). The theoretical definition of ethnic discrimination proposes that multiple uncorrelated forms of ethnic discrimination are perceived. Therefore, four items with the highest factor loadings (at least 0.50) were taken from the original four subscales of the PEDQ-CV to make the 17-item brief version of the PEDQ-CV, after a final item was added (19). PCA of items in a sample of 301 multiethnic individuals was appropriately employed. Four factors were accepted and these were named exclusion/rejection, stigmatization/disvaluation, discrimination at work/school, and threat/aggression.

The coefficient alpha for the entire Lifetime Exposure Brief PEDQ-CV was adequate. Coefficient alphas for all but one factor were adequate, except for the discrimination at work subscale (r = 0.69).

The Schedule of Racist Events (SRE)

The SRE is an 18-item instrument that measures the frequency of perceived racist discrimination (20). Of these 18 items, 17 additionally measure the extent to which these events were appraised as stressful for African-Americans (20). Racist events were defined as culturally specific, negative life events or stressors that happen to African-Americans because they are African-Americans. This scale has three subscales, each with 18 or 17 items: (a) Frequency of Current (in the past year) racist events, (b) Frequency of Lifetime racist events, and (c) Appraisal of the degree of stressfulness of the events. The items are scored on a six-point summated rating scale. Scores can range from 18 to 108 for the frequency subscales and 17 to 102 for the stressful appraisal subscale (20); higher scores indicate higher frequency of stress produced by these racist events.

The theoretical definition suggests that racism is a multidimensional experience (20). Each 17- to18-item subscale is a different scale to measure a different aspect of racism. To explore separate factors, PCA was used on each scale in a sample of 520 adult African-Americans (277 women and 243 men) (21). However, scree plot analysis of eigenvalues and orthogonal rotation indicated a one-factor structure for each subscale. The percent of variance explained by each single factor was reported. Loading and cross-loading criteria were also adequate. Each subscale is scored separately and adequate coefficient alphas were reported. The results of factor analysis did not conform to the multidimensional conceptualization of racist events proposed.

Workplace Racial Bias Measure (WRB)

The WRB is a 13-item measure of racial bias in the workplace in two forms: (a) institutional perceived discrimination (the perception of the extent to which system-level transactions are unfavorably biased against African-American female workers) (five items) and (b) interpersonal prejudice (the extent to which the individual perceives encounters of workplace racial bias in daily interpersonal interactions (eight items) (22). There was a prior expectation that items on this scale assessed two distinct, unrelated latent dimensions of workplace bias (22). PAF on 13 items in a sample of 79 African-American women confirmed these two dimensions. Item-factor loading above 0.6 on one factor and below 0.45 on another factor were retained. Adequate coefficient alphas for each subscale were reported (22).

Workplace Prejudice/Discrimination Inventory (WPDI)

The WPDI is a 15-item self-report measure of perceptions of racial prejudice/discrimination in the workplace defined as workers’ perceptions that colleagues or managers dislike them or treat them badly due to their race/ethnicity (23). Subjects indicate their level of agreement with items measuring prejudice discrimination on a seven-point Likert-type scale. Scores can range from 15 to 105; higher scores indicate higher perception of workplace prejudice/discrimination.

James and colleagues (23) theorized that this scale would be unidimensional, forming a single factor. PCA was performed of 15 items in a sample of 89 multiethnic minorities who self-identified as Mexican-American (64%), Afro-American (18.1%), Asian-American (10%), Native American (3.4%), and mixed heritage (4.5%) (60 female and 28 male). This analysis resulted in one main factor accounting for 64 percent of the variance. Items loading at 0.50 or higher were retained, resulting in a 15-item measure. The coefficient alpha for this factor was adequate.

The Asian-American Racism-Related Stress Inventory (AARRSI)

The AARRSI is a 29-item measure of racism-related stress for Asian-Americans defined as, “A psychological response specifically resulting from direct or indirect exposure to racism” (24, p. 104). Racism can be active or passive and may occur at various levels, including institutional, societal, and individual. Respondents indicate their level of agreement with statements that describe racist incidents that they themselves or others that they have known have experienced on a five-point summated rating scale. This scale is comprised of three subscales: (a) Socio-Historical Racism, (b) General Racism, and (c) Perpetual Foreigner Racism. Scores can range from 29 to 203; higher scores indicate higher levels of race-related stress for Asian-Americans.

The theoretical conceptualization of racism suggested that it has many unrelated dimensions (24). PCA of 62 items was used in a sample of 161 Asians who self-identified as Koreans (23%), Chinese (16.5%), Asian-Indians (15.1%, Japanese (10.1%), Laotians (6.5%), Filipinos (7.2%), Vietnamese (5.8%), multiethnic Asians (4.3%), Hmong (2.9%), Taiwanese (2.9%), Thai (1.2%), Indonesians, (0.6%), and other Asian ethnicities (3.1%) (88 women and 73 men). Initially, 15 factors with eigenvalues greater than one resulted. After orthogonal rotation, the three aforementioned factors were accepted with item loadings of 0.40 or greater and the percent of variance was explained for each factor. Twelve factors not meeting the loading criteria were deleted. Coefficient alphas for the entire scale and the subscales were adequate.

Race-Related Stressor Scale for Asian-American Vietnam Veterans (RRSS)

The RRSS is a 33-item self-report instrument that measures race-related stress in the military and war zone defined as direct personal experiences in which one perceives that one has been discriminated against or excluded by virtue of race, or subject to denigration, harassment, dehumanization, or stigmatization on the basis of race (25). Respondents indicate how often they have experienced specific examples of race-related incidences on a five-point summated rating scale. This scale has three subscales: (a) racial prejudice and stigmatization, (b) bicultural identification, and (c) exposure to racist environment. Total subscale scores are summed; higher scores indicate more experiences of race-related stress.

Item development was guided by a conceptual model of race-related stressors that has three proposed overlapping dimensions: (a) racial stigmatization prejudice, (b) bicultural identification and conflict, and (c) exposure to a racist environment (25). MLE was used to allow the dimensions to emerge (25). Analysis of 39 items in a sample of 300 multiethnic Asian-American Vietnam veterans resulted in the acceptance of three interpretable factors, explaining 65 percent of the variance. Item-factor loadings above 0.50 were accepted after oblique rotation. Six items not meeting the loading criteria were deleted, resulting in the final 33-item scale. Adequate coefficient alphas for the total and subscales scores were reported.

Measure of Indigenous Racism Experiences (MIRE)

The MIRE is a 31-item measure of indigenous racism experiences developed to assess racism for Indigenous Australians (26). It measures self-reported racism defined as the racism that is experienced or perceived and then reported by respondents in survey or interview settings. The MIRE assesses racism across a range of dimensions together with reactions and responses to racism. This instrument consists of six subscales: (a) interpersonal racism, (b) racism responses and reactions, (c) internalized racism and recognition of systemic racism, (d) race consciousness, (e) salience of indigeneity within social groups, and (f) salience of indigeneity among strangers. Individuals responds to subscales a, b, and c with a three-point response format, subscale d with a five-point response format, and subscales e and f with a four-point response format. The scales are to be scored separately; higher scores indicate more experiences of the respective underlying construct.

Two of the six subscales (a and b) underwent PCA to isolate the underlying dimensions (26). Each subscale had two components that underwent analysis separately, resulting in four different factor analyses. Even though one factor was expected, two factors resulted for the responses to interpersonal racism subscale (six items) in a sample of 212 participants. The factors explained 30 percent and 26 percent of the variance and the coefficient alpha was only 0.48. Since one latent construct was expected, an oblique rotation was utilized.

Two items loaded strongly on both factors but were not discarded. The reactions to the interpersonal racism subscale (four items) also underwent PCA, producing a one-factor structure explaining 45 percent of the variance in a sample of 212 participants. This factor had a very low coefficient alpha of 0.55. However, all but one factor loading was above 0.6. The item that loaded low (0.31) was not discarded. The internalized racism subscale (four items) also had a two-factor structure explaining 35 percent and 25 percent of the variance, respectively, even though one factor was expected in a sample of 287 participants. All factor loadings were above 0.5 after orthogonal varimax rotation. This factor had a very low alpha coefficient (0.23) for this entire scale. Finally, factor analysis of the systemic racism subscale (three items) produced a one-factor structure explaining 50 percent of the variance in a sample of 288 participants. The coefficient alpha of this factor was 0.47. Item-factor loadings were 0.57 or greater on this factor. Since only two of the six originally proposed subscales underwent factor analysis, the factor structure of the other four subscales of the MIRE is still unknown.

Adolescent Discrimination Distress Index (ADDI)

The ADDI is a 15-item measure designed to study adolescent discrimination distress defined as strains children experience in multiethnic ecologies in response to perceived instances of racial discrimination (27). Adolescents respond to statements indicating whether they had experienced certain types of discrimination because of their race or ethnicity on a five-point summated rating scale. This scale is made up of three subscales: (a) institutional discrimination (six items), (b) educational discrimination (four items), and (c) peer discrimination (five items). Responses are summed to yield full-scale and subscale scores, with higher scores indicating more discrimination distress.

The authors conceptualized three distinct dimensions of discrimination distress and used PCA of 15 items in a sample of 177 multiethnic adolescents aged 13 to 19 years who identified as African-American (21%), Hispanic (23%), East Asian (25%), South Asian (8%), and White (23%) (27) (78 males and 98 females). Three distinct factors emerged with eigenvalues greater than one accounting for 10 percent or more of the variance each. Item-factor loadings were all higher than 0.47. Two out of three coefficient alphas were not adequate (a = 0.60 and 0.60).

SAMPLING ADEQUACY

According to psychometric experts (6, 8), sample size is important to yielding stable factor analysis results. Yet, none of the articles provided justification for sample size. Only one instrument (20) met the 20:1 subject-to-item ratio found best to reduce errors of inference regarding the factor structure (28). Of the remaining 15 instruments, only three (18, 26, 27) met the 10:1 subject-to-item ratio recommended by Nunnally and Bernstein (6). Of the remaining 12 analyses (in 11 articles), 11 met the 5:1 subject-to-item ratio (12–17, 19, 22, 23, 25) recommended by Gorsuch (1983). One study did not meet this criterion (24), but did have the minimum of 100 subjects needed for factor analysis per Gorsuch (8), whereas two articles did not meet this latter criterion (22, 23). Based on the recommendations of Gorsuch (8) or Nunnally and Bernstein (6), 94 percent (15/16) of the instruments (see Table 1) were factor analyzed using adequate sample sizes. Because sample size recommendations vary across experts, authors should provide justification for the size of the sample used for factor analysis.

Table 1.

Summary of components of factor analysis procedures

Instrument	Sample size adequate	Factor extraction method	Criteria for accepting factors offered	Method of rotation reported and justifiable	Criteria for accepting rotated factors	Reliabilities reported for total scale and subscales	Conceptual dimensional structure supported by factor analysis
Perceived Racism Scale (McNeilly et al., 1996)	++	PCA	+	+	√	+(−)	+
Telephone Administered Perceived Racism Scale (Vines et al., 2001)	++	PAF	+	+	√√	+(−)	+
Perceptions of Racism Scale (Green, 1995)	++	PCA	–	+	0	+	–
Index of Race-Related Stress (Utsey & Ponterotto, 1996)	++	PCA	+	+	√√	+	+
Index of Race-Related Stress-Brief (Utsey, 1999)	++	PCA	+	+	√√	+(−)	+
Index of Race-Related Stress for Adolescents (Seaton, 2003)	++	PCA	+	+	√√	+	+
Perceived Ethnic Discrimination Questionnaire (Contrada et al., 2001)	*	PAF	+	+	√	+	+
Perceived Ethnic Discrimination Questionnaire Community (Brondolo et al., 2005)	++	PCA	+	+	√	+	+
Perceived Ethnic Discrimination Questionnaire Community Version Brief (Brondolo et al., 2005)	++	PCA	+	+	√	+(−)	+
Schedule of Racist Events (Klonoff & Landrine, 1999)	*	PC	+	+	√	+	–
Workplace Racial Bias Measure (Hughes et al., 1997)	++	PAF	–	0	√√	+	+
Workplace Prejudice/Discrimination Inventory (James et al., 1994)	++	PCA	+	0	√	+	+
Asian-American Racism-Related Stress Inventory (Liang et al., 2004)	–	PCA	+	–	√	+	+
Racism-Related Stress Scale for Asian- American Vietnam Veterans (Loo et al., 2001)	++	MLE	–	+	√√	+	+
Measure of Indigenous Racism Experiences (Paradies & Cunningham, 2008)	*	PCA	+	+	√	+(−)	–
Adolescent Discrimination Distress Index (Fisher et al., 2000)	*	PCA	+	0	√	+(−)	+

Sample Size: ++ = Met Gorsuch’s (1983) 5:1 recommendation; * = Met Nunnally and Bernstein’s (1994) 10:1 recommendation; minus = sample size not adequate per Gorsuch or Nunnally & Bernstein; bold = not at least 100 subjects (Gorsuch)

Factor Extraction Method: PCA = Principal Component Analysis; PAF = Principal Axis Factoring; MLE = Maximum Likelihood Estimation

Un-rotated Factors: + = Eigenvalues and/or scree test reported; minus = no criteria reported

Rotation Method: 0 = rotation not mentioned; minus = rotation not justified; + = rotation method justified by theory or data

Rotated Factors: √ = acceptable loading criteria; √√ = acceptable loading and cross-loading criteria; 0 = Unacceptable criteria for loading and cross-loading

Reliability: + = reported at acceptable level; + Bold +(−) = reported but unacceptable level

Proposed Dimensions Supported: + = Supported; minus = not supported

METHOD OF EXTRACTION

The primary method of extraction across studies was principal component factor analysis (PCA), while MLE and PAF were used less often (see Table 1). PCA is the most widely used method of extraction that yields the most parsimonious results (6). About 75 percent (12/16) of the analyses used PCA to extract the factors (see Table 1). Three analyses used PAF (13, 18, 22) and another used MLE (25) to allow the dimensions to emerge unconstrained.

Criteria for Retention of Unrotated Factors

Scree test analysis of eigenvalues is the preferred criterion for determining the number of unrotated factors to retain (10, 28). More than 81 percent of studies (13/16) reported the acceptance of factors with eigenvalues greater than one (see Table 1). Three of 16 scales (19%) did not mention the use of eigenvalue criteria (14, 22, 25). Half of the articles (8/16) mentioned the use of scree test analysis as a criterion for the acceptance of factors (12, 13, 15–17, 21, 23, 26). Two articles not using scree or eigenvalue analysis reported the percent of variance accounted for by each factor (14, 25). Seven analyses (in six articles) reported the use of conceptual interpretability (14–17, 19, 24). The majority (13/16) of the studies provided justification, such as scree plot analysis and eigenvalue criteria for the acceptance of unrotated factors, while three studies did not (14, 22, 25).

Rotation Method

Exactly 50 percent (8/16) of articles reported use of only orthogonal rotation (see Table 1). Based on careful analysis of the reports, it can be assumed that authors of six of these studies based their choice of a particular rotation method on theory (14, 17–19, 21); authors of one article reporting orthogonal rotation for some subscales and oblique rotation for others based their decision on inter-factor correlations (26). In another study, the rationale for rotation was unable to be determined (24).

Three studies reported the use of oblique rotation based on theory (13, 25, 26). Two of the three studies provided the inter-factor correlations (≥0.32) (11) necessary for interpreting oblique rotations (25, 26). In addition, none of the authors performing oblique rotation reported the extraction of higher order factors as recommended by Gorsuch (8), but the necessity of this additional extraction is debatable (6).

Since both oblique and orthogonal rotation should produce the same factor structure with the same data, if there is a clear factor structure underlying the instrument (6), authors of three scales reported use of both rotation methods (12, 15, 16). The factor structure was the same for each rotation method. Authors of two of these studies reported low to moderate inter-factor correlations to support the method of rotation (15, 16). These authors also report that the extractions were forced based on interpretability and theory (15, 16). Three scales reported no method of rotation (22, 23, 27). Taken together, it can be assumed that 75 percent (12/16) of the studies used the correct type of rotation based on the theory underlying the instrument. One article gave no theoretical or data-based rationale for the selection of a rotation method (24).

Varimax orthogonal rotation is used when authors assume a priori that the resultant factors are independent (6). Theoretically, these authors should not advocate the use of total scale scores in subsequent analyses. According to Tabachnick and Fidell (11), only independent subscale scores should be used to evaluate the relationship between the resultant factors and other variables. Despite these recommendations, two authors continued to use total scale scores in subsequent analyses despite the acceptance of multiple factors after varimax orthogonal rotation (18, 19). However, this is a common practice by researchers in the literature.

Criteria for Retention of Items on Rotated Factors

Researchers a priori should set minimally acceptable criteria for item-factor loadings (> 0.30) (6) and make the magnitude for cross-loadings (differences between items loading acceptably on more than one factor) as low as possible (30). All but one author reported or displayed adequate loadings (14; see Table 1).

However, only six articles report or display a minimally acceptable cross-loading criteria (13, 15–17, 22, 25). Three articles displayed item loadings across all of the accepted factors; however, some of the differences for the cross-loading items were less than 0.15 (24, 26, 27). These items were not deleted despite the fact that the factors already had more than three other items loading strongly (>0.5) and clearly (i.e., did not cross-load). Four articles failed to display item loadings across all factors, but only displayed the loadings for the main factor accepted (12, 18, 19, 21). Even though the loadings were strong (> 0.50), it is not known if cross-loading items were retained or deleted. Based on the information reported, all but one article reported or displayed acceptable loadings (14). However, in eight analyses it cannot be determined if the criteria for the acceptance of cross-loading items were set or followed (see Table 1).

Reliability

Adequate coefficient alphas show that items representing that factor are reliable, a prerequisite for validity (6). All of the studies reported adequate coefficient alphas for the total scales (see Table 1). However, more than 37 percent (6/16) of the studies reported less than adequate alpha coefficients for the resultant factors (12, 13, 16, 19, 26, 27), making these factors essentially useless for research. This often occurred when the subscales had few items on them and/or the samples were demographically homogeneous (e.g., all Blacks). Few items and subject homogeneity reduce coefficient alpha reliability (29).

Factor Analysis Results Congruent with Number of Factors Expected?

Evidence of construct validity is shown when the number of factors resulting from the analysis is consistent with the number of theoretical dimensions proposed (29). Overall, in 81 percent (13/16) of the studies presented, the factor structures reported generally conform to the factors expected and the theoretical conceptualizations of the respective perceived racism/discrimination constructs (see Table 1). In more than 60 percent (10/16) of the scales presenting theory proposing multiple dimensions of the construct being measured, factor analysis confirmed multiple factors represented as subscales (see Table 1). In 37 percent (6/16) of these cases, the factor structure resulted in the exact number of subscales expected (15, 16, 19, 22, 25, 27). In 25 percent of other studies (4/16), multiple unexpected factors resulted (12, 17, 19, 26). In one study, a single dimension was expected and a single factor structure was found (23).

However, the factor structures resulting did not always conform to what was expected. In two studies, multiple dimensions of a construct was expected but only a single factor resulted (14, 21). In another study, a single dimension was expected, but two factors resulted from factor analysis of some of the subscales (26). Reconceptualization of the constructs being measured is warranted for those that did not. In three studies, no structure was hypothesized, but multiple factors resulted (13, 18, 24). Overall, the majority of studies had factor analysis results congruent with the theoretical conceptualizations and the multiple dimensions proposed were supported by factor analysis (see Table 1).

CONCLUSION/RECOMMENDATIONS

As indicated in Table 1, only the authors of the analyses for the IRRS (15) and the IRRS for adolescents (17) reported adherence to all of the factor analysis procedural guidelines from experts in psychometric theory. As reported in these articles, the EFAs were performed accurately, indicating that the underlying premise of the instruments was supported by factor analysis. This thorough reporting provides confirmation of the accuracy of the factor analysis performed and thereby provides additional sound evidence of the validity of these instruments for use in African-American adults and adolescents, respectively. Many other analyses did not report adherence to several factor analysis guidelines, thereby calling into question the accepted factor structures for the samples used (12, 14, 19, 21–24, 26, 27) (see Table 1). For these analyses, more information is needed in order to ensure that the accepted factor structures will remain stable over different samples.

This review has presented 16 instruments currently available to measure racism/racial discrimination, and analyzes and critiques the techniques used for factor analysis. PCA was the main method of extraction used. Since debates weighing the appropriateness of factor extraction methods are ongoing, researchers developing new racism measures should consider seriously their rationale for the selection of a particular method of extraction and discuss this rationale in research reports (8, 28, 30). The basis for the selection of a particular rotation method was able to be determined in more than 75 percent of the articles (see Table 1). Adequately reporting data-based approaches combined with theory will help readers more clearly understand the justification for the use of a particular rotation method. Two methods of rotation could also be used to validate the accepted factor structure. In addition, racism/discrimination instrument developers should continue to report appropriate criteria for the acceptance of rotated factors. This should include both minimum loadings and cross-loading criteria for items to ensure that readers can clearly determine if items clearly represent one factor and not others.

Though the sample sizes used in the analyses presented are generally accepted as adequate (6, 8, 30, 31), no justification for the sample size was offered in any study. Given the high degree of error with EFA (28), those developing new racism measures should strive to have even larger sample sizes (20:1 or at least 10:1 subject-to-item ratio) to further ensure the stability of resultant factor structures (28).

Some of the coefficient alphas for the factors were inadequate (see Table 1). Researchers developing racism measures designed for specific minority groups (e.g., only Blacks or only Latinos) may be limited in their ability to influence sample ethnic variability, thereby creating the conditions for lower alpha coefficients (29). However, these researchers can attempt to enhance sample heterogeneity by: (a) diversifying the subcultures (e.g., a mixed Black sample of African-Americans, Caribbeans, Haitians, Jamaicans, or a mixed Latino sample); and/or (b) diversifying other demographic characteristics (i.e., socioeconomic status). Increasing the number of items on the subscales will also enhance factor reliability (29). Researchers developing measures to assess racism for minorities in general should use multiethnic minority samples to enhance heterogeneity. Researchers can also apply the recommendations outlined previously and listed in Table 2. These measures will contribute to better interpretations of factor structure, help readers discern that instruments measure the racism constructs they are intended to measure (29), and facilitate the study of racism in health-related research.

Table 2.

Recommendations for future factor analytic studies of racism/discrimination measures

Recommendations

Provide justification for sample size. Use very large sample sizes (i.e., at least 10:1 to 20:1 subject-to-item ratio). Use heterogeneous samples (i.e., diverse demographics within cultures, diverse sub-cultures within a specific culture). Select an extraction method consistent with theoretical underpinnings of the instrument (i.e., PCA, PAF, MLE). Report the use of scree test analysis of eigenvalues as the criteria for the acceptance of unrotated factors. Report the rationale for the selection of a rotation method. Report the rotation method and, whenever possible, use oblique and orthogonal rotation for validation. Report the minimum item-loading criteria for the acceptance of items on rotated factors. Report minimum item cross-loadings if the acceptable loading criteria is less than 0.6. Report the coefficients alpha for all subscales. Increase the number of subscale items to enhance reliability. Validate accepted factor structures using CFA.

LIMITATIONS

The factor analytic techniques were critiqued based solely on the specific criteria for factor analysis presented at the outset of this analysis. The use of different criteria deemed acceptable by other researchers may therefore produce conclusions/recommendations different from those of this analysis. In addition, self-reported experiences of racism are inherently limited in their ability to objectively determine if actions taken or judgments made in particular situations are the result of racial/ethnic distinctions. Therefore, these instruments cannot be used to determine the prevalence of actual incidences of racism in a person’s life. However, since the stress of subjectively perceived racism has been associated with negative health consequences (1, 32–34), these instruments are useful measures for analyses aimed at predicting health-related outcomes potentially influenced by the perception of racism.