Personality Assessment in a Diverse Urban Sample

Abstract

The present research examined the data quality and replicability of the Revised NEO Personality Inventory (NEO-PI-R) factor structure in a sample that varied in ethnicity, socioeconomic status, and literacy. Participants (N=546), drawn from the Healthy Aging in Neighborhoods of Diversity across the Life Span study, were African American (58%) and White (42%) urban dwellers living above (49%) and below (51%) 125% of the federal poverty line. The NEO-PI-R, administered via telephone, was evaluated for data quality (percent valid, acquiescence, internal consistency), congruence with the normative factor structure, and readability. All indices of data quality and factor congruence were excellent in the full sample. Literacy was the most consistent predictor of data quality. A slightly worse structure was found for the Openness to Experience and Extraversion factors among lower SES African American and White participants. The overall index of factor congruence, however, supports replication of the normative structure well beyond chance levels even among those with lower literacy. Despite the challenges of low literacy, the present findings indicate that personality traits can be assessed reliably in socioeconomically diverse populations that include those living in poverty.

Personality traits – our characteristic ways of thinking, feeling, and behaving – have been implicated in disease progression and health outcomes. In particular, the traits that define the Five-Factor Model (FFM) of personality, Neuroticism, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness, are associated with inflammation (Chapman et al., 2009; Sutin et al., 2010), obesity (Sutin, Ferrucci, Zonderman, & Terracciano, 2011), cardiovascular disease (Strike & Steptoe, 2004), and, ultimately, longevity (Kern & Friedman, 2008; Terracciano, Löckenhoff, Zonderman, Ferrucci, & Costa, 2008). The pervasiveness of these traits and their relevance to health outcomes make them a promising avenue for examining the role of psychological factors in health in vulnerable populations. Some traits, for example, may ameliorate health disparities, whereas others may exacerbate them.

To address questions about the role of personality in health outcomes across different demographic groups, it is imperative to have a measurement tool that assesses these traits reliably in the populations of interest. Personality measures have typically been constructed and validated using college students or other educated adults (Costa & McCrae, 1992; John & Srivastava, 1999). Less research has addressed the psychometric properties of established personality measures in ethnically and socioeconomically diverse populations, especially among those with low levels of literacy.

The Revised NEO Personality Inventory (NEO-PI-R) is a well-replicated measure of FFM personality traits (Costa & McCrae, 1992). Cross-cultural evidence suggests that the NEO-PI-R reliably assesses personality traits across different populations and that nations as diverse as Burkina Faso and Iceland share a similar factor structure to the normative American structure (McCrae et al., 2005). These studies, however, despite their diversity of culture, typically rely on college students at respondents. Thus, despite providing knowledge about the universal structure of personality, they do not address personality assessment in populations that vary greatly in literacy.

Data quality and the factor structure of the NEO-PI-R tend to be similar across sex and age (Costa & McCrae, 1992) but may vary across other demographic groups. African American samples, for example, tend to have a higher percentage of NEO-PI-R assessments judged invalid and higher acquiescence (i.e., the tendency to agree with items regardless of their content; Savla, Davey, Costa, & Whitfield, 2007). Invalidity and acquiescence, however, may be more strongly related to education than to ethnicity (Löckenhoff et al., 2008). Internal consistency of the five personality factors tend to be slightly lower among African Americans (Savla et al., 2007) and in populations with lower levels of education, regardless of ethnicity (Löckenhoff et al., 2008). Finally, congruence with the normative factor structure tends to be fairly high, but African American samples have slightly lower congruence coefficients for Extraversion and Openness to Experience, suggesting that perhaps the items in these domains do not measure these traits as well for African Americans as for White participants (Salva et al., 2007).

Less research has addressed the issue of literacy in personality assessment. The factor structure of the NEO-PI-R has been clearly replicated in samples with lower levels of education (Allik & McCrae, 2004; Trobst et al., 2000). Although related, literacy and education are not equivalent, and in some domains literacy may be more important than education in predicting outcomes of interest (e.g., cognitive performance; Byrd, Jacobs, Hilton, Stern, & Manly, 2005). Personality assessment relies heavily on the respondent’s ability to comprehend and respond to the meaning of the items. Data quality and factor structure may thus be compromised for individuals with lower levels of literacy who may have difficulty understanding questions about their personality.

The present research examines the data quality and factor structure of personality traits measured with the NEO-PI-R in a socioeconomically diverse urban sample. In addition to the total sample, we test data quality in a number of demographic groups – African American and White participants above and below 125% of the federal poverty line – and in those with higher and lower literacy. We evaluated the assessment on several dimensions, including percentage of invalid NEO-PI-Rs, acquiescence, internal consistency, and factor congruence with the normative sample. We also performed a readability analysis on the items of the NEO by factor to determine reading level of each factor. We expected these indices to be somewhat lower in those with lower SES and lower literacy.

Method

Sample

Participants were drawn from the Healthy Aging in Neighborhoods of Diversity across the Life Span (HANDLS) study (Evans et al., 2010). HANDLS is a prospective population-based longitudinal study designed to disentangle the effects of race and socio-economic status on morbidity and mortality. Participants were recruited as a fixed cohort from an area probability sample of 12 census segments in Baltimore, MD. To be included, participants had to be between 30 and 64 years old, be able to give informed consent, be able to perform at least five of the measures (medical history, physical performance, cognitive testing, dietary recall, audio questionnaire, body composition, carotid Doppler, or pulse wave velocity), and have a valid picture identification; exclusion criteria included pregnancy at time of entry and being within 6 months of cancer treatment. A total of 3,721 participants were recruited into the study at baseline; this sample was 55% female, 59% African American, and 41% were below 125% of the federal poverty level.

A trained staff member administered the personality questionnaire (see below) by telephone. A total of 546 participants from the HANDLS cohort completed the personality measure. This subsample was selected such that participants were contacted in the same order that they were recruited for the initial wave of testing. Although not every participant was available for a telephone interview, we attempted to contact all participants for which we had resources. The NEO-PI-R took about 40 minutes, on average, to administer over the telephone, and participants were paid $20 for their participation. This sample was 65% female, 58% African-American, and 51% were living below 125% of the poverty line. The sample was, on average, almost 50 years old (M_age = 49.37, SD = 8.62, range 30–64).

Measures

Personality assessment

Participants completed the Revised NEO Personality Inventory (NEO-PI-R; Costa & McCrae, 1992), a 240-item questionnaire measure of the five major domains of personality: Neuroticism, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness. Each of these domains contains six facet scales that provide a comprehensive and detailed assessment of normal adult personality in terms of emotional (N), interpersonal (E), experiential (O), attitudinal (A), and motivational (C) styles. Participants responded on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Items on the NEO-PI-R are roughly balanced to control for the effects of acquiescence.

Literacy

Literacy was assessed with the reading subtest of the Wide Range Achievement Test-3rd Edition (WRAT; Wilkinson 1993), which measures the ability to recognize and name letters and words. WRAT scores were available for 490 participants with NEO-PI-R data. A WRAT score below 40 is considered functionally illiterate; 30% of the present sample fell below this threshold (M = 43.10, SD = 7.82, range 14 to 57).

Analysis

We scored the NEO-PI-R according to the protocol described in the Manual (Costa & McCrae, 1992). The questionnaire can be judged invalid if the participant had more than 40 items missing or had repetitive responses (e.g., 10 consecutive neutral responses). We also assessed acquiescence and nay-saying, dichotomized according to the cutoffs listed in the Manual (≥150 agree or strongly agree and ≤ 50 agree or strongly agree, respectively). We evaluated internal consistency with Cronbach’s alpha and split-half correlations.

To test the NEO-PI-R factor structure across the different demographic and literacy groups we used targeted orthogonal rotation (McCrae, Zonderman, Costa, Bond, & Paunonen, 1996). In this procedure, factors are rotated to minimize the sums of squares of deviations from the reference matrix while maintaining orthogonality. This method is more appropriate than standard confirmatory factor analysis (CFA) approaches because the FFM does not have a simple structure. That is, several facets of the NEO-PI-R have meaningful secondary loadings on factors other than their principle one. In CFA, such a secondary loading would be set to zero, artificially reducing fit. Targeted orthogonal rotation overcomes this limitation of CFA and is also more appropriate when evaluating whether a replication sample matches the hypothesized structure. In the present research, we rotated the factor structure to the normative factor structure sample reported in the Manual (Costa & McCrae, 1992).

Following McCrae and colleagues (1996), we evaluated structural fit in several ways. First, we examined the congruence coefficient for each factor, calculated as the sum of the cross-products of two sets of column-normalized factor loadings. In addition to estimating the congruence coefficients for the five factors, we calculated a variable congruence coefficient for each of the 30 facets. We computed an overall index of congruence between the two factor matrices. A threshold of .85 is typically considered indicative of factor replicability (Lorenzo-Seva & ten Berge, 2006).

Finally, to further address the effect of literacy, we conducted a readability analysis (Kincaid, Fishburne, Rogers, & Chissom, 1975) on the items that comprise each of the five factors. For each factor, we calculated the Flesch Reading Ease score, which is a weighted combination of the average sentence length and the average number of syllables per word. Scores can range from 0–100, with higher scores indicating easier text. In addition, we calculated the Flesch-Kincaid Grade Level, which is also a weighted combination of the average sentence length and the average number of syllables per word. It is, however, scaled to reflect U.S. educational grade levels (i.e., a score of 5 indicates the text is at a 5^th grade reading level).

Results

Data quality

We first assessed data quality and screened for invalid questionnaires. In the current data, there were no missing responses, likely due to oral administration via the telephone. A total of 23 completed questionnaires (4%), however, were judged to be invalid based on repetitive responding. Participants with invalid responses were more likely to be functionally illiterate (OR = 7.57, 95% CI = 2.08–27.51); poverty status, ethnicity, sex, and age were not significant predictors of having an invalid NEO. Participants with invalid data were excluded from subsequent analyses.

Of the remaining 523 participants, 54 (10%) met the threshold for acquiescence, but none met the threshold for nay-saying. Acquiescence was more likely among men (OR = 3.28, CI = 1.72–6.25), African Americans (OR = 5.13, CI = 2.01–13.05), and those with lower literacy (OR = 3.92, CI = 1.98–7.78), but it was unrelated to poverty status and age. Since acquiescence is not one of the criteria for invalidity, we retained these 54 participants in subsequent analyses.

Across both the whole sample and within the four demographic groups (African American and White, above and below the poverty line), alpha reliabilities were satisfactory (see Table 1). The median alpha coefficient across the four demographic groups was .88 and ranged from .78 to .93. The alpha reliabilities by literacy were likewise adequate, with a median alpha coefficient of .86 (range .77 to .93). In general, the test scores of low SES African Americans and those with lower literacy had slightly, but not substantially, lower reliabilities than the test scores in the other groups examined. The split-half correlations were slightly lower, but generally mirrored the alpha reliabilities (see Table 1).

Table 1.

Alpha reliability coefficients and split-half correlations for the five domains for the full sample and by ethnicity, poverty, and literacy subgroup

Personality Trait	Full sample	African American		White		Literacy
		Below	Above	Below	Above	Low	High
	Alpha Coefficients
Neuroticism	.92	.89	.87	.93	.93	.86	.93
Extraversion	.87	.84	.86	.89	.88	.83	.87
Openness to Experience	.87	.79	.84	.90	.91	.77	.88
Agreeableness	.83	.78	.80	.86	.87	.79	.84
Conscientiousness	.91	.92	.90	.90	.89	.89	.91
Mean	.88	.86	.86	.90	.90	.83	.89
	Split-half Correlations
Neuroticism	.87	.82	.82	.89	.90	.79	.89
Extraversion	.82	.80	.82	.86	.83	.80	.83
Openness to Experience	.80	.70	.71	.86	.86	.69	.82
Agreeableness	.72	.70	.71	.66	.82	.75	.73
Conscientiousness	.86	.84	.84	.90	.84	.84	.87
Mean	.84	.77	.78	.83	.85	.77	.83

Note. N = 523, n = 200 for African-Americans below the 125% poverty line, n = 97 for African-Americans above the 125% poverty line, n = 62 for Whites below the 125% poverty line, n = 164 for Whites above the 125% poverty line, n = 135 for low literacy (WRAT < 40) and n = 339 for high literacy (WRAT ≥ 40). Below = income below 125% of the federal poverty line. Above = income above 125% of the federal poverty line.

Factor structure

The total sample replicated the normative structure well at the factor level: All five factors were equal to or greater than .95 (see Table 2). Dividing the sample by ethnicity and socioeconomic status revealed that the congruence coefficients for Extraversion and Openness for African Americans living in poverty fell below the .85 threshold. Congruence coefficients for Extraversion and Openness were also below the .85 threshold for those who were functionally illiterate. These coefficients, however, were still well above what would be expected by chance (i.e., .55; McCrae et al., 1996). Total congruence coefficients were above .85 for all of the groups examined. The slight degradation in structure among those lower in SES seems to be due to literacy; when considering participants living in poverty who were literate (n=131), congruence coefficients for the five factors were all greater than .90.

Table 2.

Factor and total congruence in the full sample and by ethnicity, poverty, and literacy subgroup

Personality Trait	Full sample	African American		White		Literacy
		Below	Above	Below	Above	Low	High
Neuroticism	.95	.95	.95	.86	.97	.89	.96
Extraversion	.95	.74	.93	.86	.94	.84	.95
Openness to Experience	.96	.67	.85	.86	.95	.79	.96
Agreeableness	.97	.92	.95	.95	.98	.93	.97
Conscientiousness	.96	.93	.93	.94	.99	.91	.97
Total congruence	.96	.86	.92	.89	.96	.88	.96

Note. N = 523, n = 200 for African-Americans below the 125% poverty line, n = 97 for African-Americans above the 125% poverty line, n = 62 for Whites below the 125% poverty line, n = 164 for Whites above the 125% poverty line, n = 135 for low literacy (WRAT < 40) and n = 339 for high literacy (WRAT ≥ 40). Below = income below 125% of the federal poverty line. Above = income above 125% of the federal poverty line. Coefficients below .85 are in bold.

Similar to the factors, in the total sample, all of the variable congruence coefficients were greater than .85, indicating a good fit with the normative factor structure (see Table 3). Differences did emerge, however, across the demographic and literacy groups. Across the four demographic groups, 13% (16 out of 120) of the variable congruence coefficients fell below .85. In general, those living in poverty, both African American and White, had more facets that did not exceed the .85 threshold than those with higher SES. O6: Values had the lowest congruence coefficient, especially among those with lower literacy, and E2: Gregariousness also had notably low congruence, especially among low SES African Americans and those with lower literacy. Similar to the broad domains, participants living in poverty who were literate only had three variable congruence coefficients that fell below .85 (E6: Positive Emotions = .83, O6: Values = .71, and A5: Modesty = .77).

Table 3.

Facet congruence in the full sample and by ethnicity, poverty, and literacy subgroup

Facet	Full sample	African American		White		Literacy
		Below	Above	Below	Above	Low	High
N1: Anxiety	.98	.95	.97	.96	.97	.90	.97
N2: Angry Hostility	.98	.97	.95	.96	.97	.89	.98
N3: Depression	.98	.96	.97	.98	.97	.97	.98
N4: Self-Consciousness	.93	.93	.90	.84	.97	.93	.94
N5: Impulsivity	.99	.98	.92	.93	1.00	.98	.99
N6: Vulnerability	.97	.91	.94	.93	.99	.93	.99
E1: Warmth	.99	.85	.97	.97	.99	.94	.99
E2: Gregariousness	.95	.34	.94	.89	.95	.54	.94
E3: Assertiveness	.99	.98	.96	.92	.98	.98	.99
E4: Activity	.96	.82	.97	.90	.98	.86	.98
E5: Excitement-Seeking	.97	.78	.93	.70	.97	.84	.97
E6: Positive Emotions	.87	.85	.93	.85	.91	.85	.88
O1: Fantasy	.96	.63	.94	.96	.97	.86	.98
O2: Aesthetics	.94	.89	.91	.62	.96	.78	.95
O3: Feelings	.98	.84	.89	.91	.98	.95	.99
O4: Actions	.95	.89	.96	.92	.95	.94	.93
O5: Ideas	.96	.77	.88	.83	.98	.92	.96
O6: Values	.86	.11	.91	.63	.94	−.17	.94
A1: Trust	.95	.77	.89	.97	.95	.87	.97
A2: Straightforwardness	.96	.92	.96	.98	.87	.88	.93
A3: Altruism	.95	.92	.87	.91	.94	.93	.96
A4: Compliance	.95	.89	.87	.85	.96	.85	.97
A5: Modesty	.95	.85	.77	.87	1.00	.89	.91
A6: Tender-Minded	.87	.86	.66	.97	.91	.91	.83
C1: Competence	.99	.98	.99	.96	.96	.98	.99
C2: Order	.96	.83	.98	.88	.98	.72	.98
C3: Dutifulness	.95	.93	.90	.86	.99	.92	.97
C4: Achievement Strive	.99	.98	.97	.94	.98	.98	.99
C5: Self-Discipline	.97	.98	.98	.96	.98	.97	.97
C6: Deliberation	.96	.95	.97	.85	.97	.89	.97

Note. N = 523, n = 200 for African-Americans below the 125% poverty line, n = 97 for African-Americans above the 125% poverty line, n = 62 for Whites below the 125% poverty line, n = 164 for Whites above the 125% poverty line, n = 135 for low literacy (WRAT < 40) and n = 339 for high literacy (WRAT ≥ 40). Below = income below 125% of the federal poverty line. Above = income above 125% of the federal poverty line. Coefficients below .85 are in bold.

Finally, the readability analysis indicated that the items for Openness to Experience were more complex and required a higher level of reading ability than the other four factors: The Flesch Reading Ease score for Openness was 61.2, compared to 73.2, 74.6, 72.2, and 74.4 for Neuroticism, Extraversion, Agreeableness, and Conscientiousness, respectively. Likewise, the Flesch-Kincaid grade level for the Openness items was 7.5, compared to about 5 for the other four factors (range 4.9–5.4). The lower congruence coefficient of Openness compared to the other four factors likely derives from the reading difficulty of the items.

Discussion

The present study examined the psychometric characteristics, factor structure, and readability of the NEO-PI-R in an urban sample that varied in ethnicity, socioeconomic status, and literacy. Overall, less than 5% of the personality assessments were judged invalid and measures of data quality were generally good, both at the sample level and across the different demographic and literacy groups. Factor congruence with the normative sample was excellent in the full sample. Although factor congruence for Extraversion and Openness to Experience was slightly lower among lower SES African Americans and those with lower literacy, the congruence coefficients were nonetheless acceptable and well above what would be expected by chance. Further, the readability analysis suggested that the items measuring Openness required a higher reading level than the other four factors.

Data quality was higher in the HANDLS cohort than in other socioeconomically diverse samples that have asked participants to read and complete the personality measure individually (e.g., Salva et al., 2007). Oral administration in the present study likely increased data quality, in part because participants may have been less likely to not answer questions and because the administrator was able to define any unfamiliar words/colloquial expressions that could be misinterpreted or skipped if participants completed the test using the traditional paper-and-pencil method. Oral administration may help to significantly improve data quality in populations not accustomed to responding to personality measures or other paper-and-pencil tests.

Despite good overall congruence with the normative factor structure, there was slight degradation of personality structure in some of the subgroups examined. Similar to cross-cultural studies in African countries (McCrae et al., 2005), congruence with the normative structure was lowest for lower SES African Americans. McCrae and colleagues (2005) tested the hypothesis that African nations, because of their history and culture, may share a personality structure that differs from populations of European descent. In the five African cultures they examined, however, congruence was greater with the normative American sample than with each other, leading the authors to conclude that lower index of replicability of the factor structure was more due to lower data quality in these samples than to actual differences in personality structure.

Consistent with this interpretation, in the present study lower data quality, particularly among those with lower literacy, was responsible for the lower congruence coefficients. That is, factor congruence for higher SES African Americans was very similar to the higher SES Whites, whereas the lower SES Whites had degradation in structure that was similar to the lower SES African Americans. In addition, lower literacy was the strongest predictor of invalid questionnaires and the low literacy group had the lowest internal consistencies. Indeed, the factor structure of those living in poverty and who were literate converged well with the normative factor structure, again suggesting that literacy is a strong predictor of data quality. Thus, the differences in personality structure across the different groups appear to be more a reflection of lower SES and, in particular, lower literacy, than of ethnicity.

In cross-cultural research (McCrae et al., 2005) and across different demographic groups in the United States (Löckenhoff et al., 2008), Openness to Experience consistently has slightly lower congruence with the normative sample than the other four traits. We likewise found Openness to have the lowest congruence, particularly for lower SES African Americans and those with lower levels of literacy. As discussed above, it is unlikely that the lower congruence is because Openness is not represented in cultures of non-European descent. It is more likely that the items that measure Openness are linguistically complex and less intelligible to people with lower literacy. Consistent with this argument, the readability analyses indicated that the reading-level of the Openness items was about 2.5 grade levels above the reading level of the items of the other four factors. The Openness items may also be less relevant to demographic groups other than the normative one. Openness is the most complex of the five traits to measure, and it cannot be summarized by single words as well as the other traits. Thus, the expression of Openness may be more context-dependent, and the items that measure Openness on the NEO-PI-R may be more appropriate for higher SES individuals.

Despite somewhat lower data quality, we obtained a reasonably good personality assessment for those living in poverty and those with lower levels of literacy. Recent improvements in the NEO instrument will likely help to improve data quality for individuals with lower literacy. In particular, the NEO-PI-3 (McCrae & Costa, 2010) was developed to be a more readable version of the NEO-PI-R. Even so, items from some of the most problematic facets in the present research (e.g., E2: Gregariousness) were not replaced, which suggests that literacy could still be a concern. Confidence in the integrity of the personality assessment is essential for health research on vulnerable populations. Of note, the traits most consistently associated with morbidity and mortality – Neuroticism, Agreeableness, and Conscientiousness –had the strongest congruence with the normative structure and these test scores had the highest alpha reliabilities across all of the subgroups examined. Thus, the potentially most relevant traits for health-related outcomes are measured very well by the NEO-PI-R.

This study had several strengths, including a comprehensive measure of personality traits on a relatively large sample that included populations typically underrepresented in personality research. The present research also had some limitations that could be addressed in future research. For example, although our overall sample was large, the sample sizes in the demographic subgroups were less than ideal, especially for higher SES African Americans and lower SES Whites. It is remarkable, however, that even with the small sample sizes for these groups, the data quality and congruence were very good. In addition to a larger sample, longitudinal data would be informative to test the structural integrity of the factor structure. Longitudinal data is particularly important, given that retest reliability is the strongest predictor of validity (McCrae, Kurtz, Yamagata, & Terracciano, 2011). Finally, although administration via telephone may be a limitation, our data quality measures were higher than others have reported using an individual administration (e.g., Löckenhoff et al., 2008; Savla et al., 2007), which suggests that this administration method likely does not compromise the assessment. In fact, the telephone administration of the NEO may have somewhat attenuated the effect of literacy on data quality because the administrator was able to clarify words/phrases if asked. Despite these limitations, the present research indicates that personality traits can be measured successfully in socioeconomically and ethnically diverse populations.