Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2011 Feb 1.
Published in final edited form as: J Res Pers. 2010 Feb 1;44(4):53–61. doi: 10.1016/j.jrp.2009.10.005

The measurement equivalence of Big Five factor markers for persons with different levels of education

Beatrice Rammstedt 1, Lewis R Goldberg 2, Ingwer Borg 1
PMCID: PMC2854544  NIHMSID: NIHMS161341  PMID: 20401177

Abstract

Previous findings suggest that the Big-Five factor structure is not guaranteed in samples with lower educational levels. The present study investigates the Big-Five factor structure in two large samples representative of the German adult population. In both samples, the Big-Five factor structure emerged only in a blurry way at lower educational levels, whereas for highly educated persons it emerged with textbook-like clarity. Because well-educated persons are most comparable to the usual subjects of psychological research, it might be asked if the Big Five are limited to such persons. Our data contradict this conclusion. There are strong individual differences in acquiescence response tendencies among less highly educated persons. After controlling for this bias the Big-Five model holds at all educational levels.

Keywords: Big Five, measurement equivalence, acquiescence response bias, factor structure

In personality psychology, most research is based on samples of college students, especially undergraduates taking a course in psychology. This narrow subject population leads to the question of the extent to which the findings from such studies can be generalized to a broader population. Expressed in a more formal way: Are multi-item personality inventories transportable in the sense of measurement equivalence over subsamples of the population? If measurement equivalence (Vandenberg & Lance, 2000; Drasgow & Probst, 2004) does not hold for specific subsamples, it becomes dubious to compare the scale scores of persons from these different subsamples, because their scores are indicators of constructs that differ in terms of their structure and, thus, ultimately also in their meaning.

Over the last two decades, the Big Five factors have become the most prominent model for describing the structure of personality traits. The Big Five have been found to replicate across different methods of data collection such as self-ratings, peer ratings, and behavioral descriptions, as well as across at least some different languages and cultures. However, personality research is still largely based on the typical “psych 100” student samples, which suggests that test norms may be inappropriate for other samples of individuals. Moreover, even the assumed five-factor structure may not hold in general. Indeed, several researchers investigating the Big Five in samples other than undergraduate populations using standardized and well-established inventories found that the five factor structure did not replicate in their data sets (e.g., Körner, Geyer, & Brähler, 2002; Lang, Lüdtke, & Asendorpf, 2001; Mõttus, Allik, & Realo, 2007a; Rolland, Parker, & Stumpf, 1998; Tokar, Fischer, Snell, & Harik-Williams, 1999; Toomela, 2003a). Table 1 provides an overview of the samples investigated and the instruments assessed in these studies, as well as the findings. As can be seen from the table, a variety of different well-established Big Five questionnaires such as the NEO Personality Inventory (NEO PI-R; Costa & McCrae, 1992), the NEO Five Factor Inventory (NEO-FFI; Costa & McCrae, 1989), or the Big Five Inventory (BFI; John, Donahue, & Kentle, 1991; John & Srivastava, 1999) were used as measures of the Big Five in these studies. Moreover, because results not supporting the assumed model are less likely to be published, the studies presented in Table 1 could be just the tip of an iceberg of similar findings. In the present study, we investigate the structure of a Big Five factor marker in subsamples that vary in their educational level. If the five-factor structure does not replicate in population-representative samples, it may nevertheless still hold in subsamples of persons with a higher educational background.

Table 1.

Overview of studies failing to replicate the Big Five structure in non-student samples

Authors Year Big Five Instrument Sample Results
Körner, Geyer, & Brähler 2002 NEO-FFI N = 1908; population representative Poor fit of varimax solution
Lang, Lüdtke, & Asendorpf 2001 BFI N = 480; quoted with regard to age Poor fit of varimax solution
Mõttus, Allik, & Realo 2007 NEO-FFI N = 1342; population representative Differences in explained variance and in congruence with simple structure among five educational groups
Rolland, Parker, & Stumpf 1998 NEO-PI-R, NEO-FFI N = 447 college students and N = 268 military recruits Weaker fit of the varimax solution in the military sample for both NEO-PI-R and NEO-FFI
Tokar, Fischer, Snell, & Harik-Williams 1999 NEO-FFI N = 485 non-student adults Poor fit of the varimax solutation
Toomela 2003 NEO-PI-R N = 912 male military members Weaker fit of varimax solution than in the normative sample
Open in a new tab

Specifically, in the present study we investigate whether the five-factor structure replicates in other than student populations, namely in samples that are representative of the general population. Therefore, we analyze the factor structure of a Big Five measure in two large data sets, both of which are representative of the general German adult population, allowing us to cross-validate our results simultaneously.

Individual Differences in Response Bias

Individuals can differ in their mean response across all items, in the dispersion of their responses around their personal means, and/or in the shape of their response distributions as reflected in measures of skewness and kurtosis. The results of several recent studies suggest that the psychometric quality of questionnaires decreases in general with lower levels of intelligence, education, and age (e.g., Allik, Laidra, Realo, & Pullmann, 2004; Mõttus, Allik, & Pullmann, 2007b; Soto, John, Gosling, & Potter, 2008). This effect of lower psychometric quality seems to be associated with individual differences in subjects’ use of the response options (cf. Gudjonsson, 1986; Mõttus et al., 2007b; Sigelman, Budd, Spanhel, & Schoenrock, 1981; Soto et al., 2008; Tokar et al., 1999), in particular by individual differences in acquiescent responding. Acquiescence refers to the tendency of an individual to consistently agree to questionnaire items, regardless of the content of the items (Jackson & Messick, 1958; Javeline, 1999). The tendency for acquiescent responding can be investigated by comparing the individual’s responses to true- and false-keyed items assessing the same construct (e.g., Prefer to be with others, and Like to be all by oneself). When there are the same number of true- and false-keyed items per dimension, persons with a high tendency for acquiescence will have comparatively higher means scores than those with a lower such tendency.

Results of previous studies have shown, that such “yea-saying” appears to be more frequent among persons with lower social status, lower educational levels, and/or lower intelligence (e.g., Ayidiya & McClendon, 1990; Gove & Geerken, 1977; Krosnick, Narayan, & Smith, 1996; Lenski & Leggett, 1960; Narayan & Krosnick, 1996). It has been suggested that persons with relatively low education have less clear self-concepts, smaller vocabularies, and less verbal comprehension skills than more highly educated persons. This may make them relatively uncertain when it comes to responding to questionnaire items and thus leaves more room for the influence of systematic response biases (i.e., Goldberg, 1963). In the extreme, a respondent’s answers could be completely independent of the content of the particular item. Even more moderate tendencies to agree with all items provides an additional source of variance, specific to each individual, that damages the psychometric properties of the scales and blurs the correlational patterns among them.

Controlling for Response Bias

One method to control for individual differences in acquiescence response tendencies is to compute each individual’s mean across all items, subtract that mean from each response, and then analyze these “deviation” scores.1 An alternative and more common procedure is to “ipsatize” the responses of each individual by using standard (z) scores (dividing an individual’s deviation scores by the standard deviation of those scores), and thus simultaneously controlling for individual differences in both means and dispersions. With ipsatized data, every participant has the same mean and standard deviation across the total item pool, although individual differences in profile shape remain. In the present study, we compared both methods of controlling for individual differences in response bias.

We hypothesize that the usual Big Five factor structure may not hold well in samples that represent the population of an entire country and, in particular, it is expected not to hold for samples of persons with lower educational levels. Moreover, we hypothesize that lower educated persons will have a stronger tendency for acquiescence response bias. Finally, when statistically controlling for acquiescence, we hypothesize that the Big Five structure can be replicated in all educational subsamples.

Method

Samples and Procedure

Analyses are based on the German surveys of the International Social Survey Program (ISSP) conducted in 2003/2004 and in 2005/2006, respectively. The ISSP household surveys were conducted together with the 2004 and 2006 versions of the German General Social Survey (ALLBUS). The ALLBUS was administered as a 45-minute face-to-face interview, and the ISSP was a drop-off questionnaire to be completed by the respondents on their own after the personal interviews. The interviewer remained in attendance and collected the completed forms.

The ISSP survey was based on a representative sample of German adults (age ≥ 18) living in private households in Germany. Foreigners residing in Germany and able to complete the questionnaire in German were included. The sample was drawn in a two-stage design from official registers of inhabitants kept by municipalities throughout Germany. First, the communities and sample points were selected randomly, and then individuals were randomly selected from each sampling point. Full details of the sampling are presented in the methods report on ALLBUS 2004 and ALLBUS 2006, respectively (Haarmann, Scholz, Wasmer, Blohm, & Harkness, 2006; Wasmer, Scholz, & Blohm, 2007).

Sampling and data collection were conducted by a commercial vendor (TNS-Infratest). Participation in the study was voluntary and not financially rewarded. The response rate was 41% both in 2004 and in 2006. Compared to the German Microcensus (Statistisches Bundesamt, 2005, 2007) the resulting samples were slightly biased with regard to the same socio-demographic characteristics as most register samples (cf. Groves, 1989; Koch 1998), somewhat under-representing older respondents, respondents with lower levels of education, and persons living alone. The total sample of ISSP 2003/2004 comprised 2,567 respondents (50% females); age varied in this sample between 18 and 91 years, with a mean of 48 (SD = 17) years. The sample of ISSP 2005/2006 comprised 3,421 respondents (52% females); age varied between 18 and 94 years, with a mean of 49 (SD = 17) years.

Questionnaire

The BFI-10 (Rammstedt & John, 2007) is an abbreviated version of the well-established Big Five Inventory (BFI; John et al., 1991; see also Benet-Martínez & John, 1998; John & Srivastava, 1999; for the German version see Lang et al., 2001; Rammstedt, 1997), consisting of ten of the 44 standard BFI items. It assesses the Big Five with two items per factor, one keyed in the positive and one in the negative direction. In the ISSP 2003/2004, an earlier version of the BFI-10 was used, in which one item assessing Agreeableness differed from the final BFI-10 questionnaire (cf. Rammstedt & John, 2007). This exchange was made because the Agreeableness scale proved to possess somewhat higher validity with the item “…tends to find fault with others” included instead of “…can be cold and aloof”.2

In accordance with the response format used throughout the ISSP questionnaire, all items used five-point Likert-type response options ranging from fully agree to fully disagree. Although this response scale differs in its orientation from the standard BFI-10 format (fully disagree to fully agree), a previous study showed that reversing the direction of the response scale does not change the quality of the resulting responses (Rammstedt & Krebs, 2007). All ten items were re-coded for the present analyses so that fully disagree was scored as 1 and fully agree as 5.

Results

Factor Structure of the Original Responses

The BFI-10 variables from the ISSP2003/2004 and the ISSP 2005/2006 were each subjected to exploratory factor analysis. For the ISSP2003/2004 data, scree tests clearly supported a five-factor solution, as expected for this inventory. For the ISSP 2005/2006, the scree test was less clear, suggesting either a five- or a six-factor solution. To keep our analyses aligned with the Big Five model, we chose to extract five factors in both cases. As it is standard in Big Five research, all solutions were subsequently rotated to simple structure using the varimax criterion. The explained variance was 68% in both samples. The first ten columns of Table 2 show the rotated factor loading matrices of the ten BFI-10 items in each of the two data sets; in this and all subsequent tables, pairs of items measuring the same factor are re-ordered and listed in adjacent rows.

Table 2.

Varimax rotated factor structures of the BFI-10 items (raw data): Total samples

Varimax Rotated Factor Loadings
 Procrustes Rotated Factor Loadings
ISSP2003/2004 (N = 2567) ISSP2005/2006 (N = 3421) ISSP2003/2004 (N = 2567) ISSP2005/2006 (N = 3421)
I see myself as someone who … 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
… is reserved −.70 .17 −.05 .24 .27 −.49 .63 .18 −.23 .34 .78 −.01 −.10 −.09 −.17 .84 −.28 .21 .03 .11
… is outgoing, sociable .73 .10 .18 .04 .08 .84 −.09 −.21 .05 .07 −.46 −.02 −.07 .25 .55 −.65 −.21 −.27 .22 .42
… is generally trusting .08 .00 .02 −.20 .87 −.01 .16 .16 .90 .03 .26 −.74 .09 .05 .42 −.22 −.65 .08 −.03 −.63
… tends to find fault with others .01 −.03 −.03 .84 −.21 −.13 −.89 .12 −.09 .07 .27 .75 .14 −.04 .31 −.29 .71 .17 .35 −.30
… does a thorough job .14 .78 .15 .21 .17 .57 .13 −.47 −.11 .30 .19 .06 −.69 .21 .40 −.20 −.22 −.51 .31 .46
… tends to be lazy −.03 −.77 .04 .34 .19 −.19 .16 .74 −.09 .21 .16 .10 .84 −.00 .12 .27 −.13 .75 .12 .02
… is relaxed, handles stress well .05 .05 .86 .05 .13 .07 −.05 .01 .07 .96 .09 −.04 .02 .87 .07 .19 −.24 −.02 .92 −.00
… gets nervous easily −.22 −.17 −.62 .37 .31 −.23 .69 .21 .31 −.05 .43 .04 .24 −.64 .22 .37 −.62 .17 −.32 −.13
… has an active imagination .71 .18 .07 .23 .19 .68 −.13 .29 −.52 −.12 −.30 .05 −.10 .14 .71 −.44 .18 .28 .04 .73
… has few artistic interests −.49 −.14 .34 .18 −.04 −.02 −.02 .74 .18 −.11 .42 .16 .18 .29 −.31 −.13 −.13 .73 −.09 −.15
Vector congruence .74 .95 .92 .67 .16 .69 .56 .68 .43 .65 .72 .98 .94 .91 .60 .79 .76 .70 .78 .52
Matrix congruence .70 .61 .83 .71
Open in a new tab

Note. Highest loadings of each item set in bold; significant congruence coefficients underlined.

In both samples, the rotated five-factor structure did not correspond to the typical Big Five pattern. The first factor of the ISSP2003/2004, for example, shows high loadings of the two Extraversion items, but also of the two Openness items. Moreover, the loadings patterns differ substantially between the two data sets. The overall congruence of the two factor solutions with an idealized Big Five pattern (consisting of +1, −1, and 0 loadings only) was only .70 and .61 in each of the two data sets.3

Factor Structure of Individuals with Different Levels of Education

The results for the total sample reveal that the Big Five factor structure does not replicate in population-representative samples. To see if this structure can at least be replicated for some educational strata of the population, we split each of the ISSP samples into three distinct groups: (1) no or lower secondary education (N = 1,063 and N = 1,330, respectively); (2) intermediate secondary education or entrance qualification for universities of applied sciences (N = 1,026 and N = 1,358, respectively); and (3) higher secondary education, i.e. those with an entrance qualification for general universities or with a university degree (N = 438 and N = 689, respectively). We then investigated the factor structure of the BFI-10 separately within each of these subsamples.

Tables 2, 3, and 4 provide the factor loadings for each of three subsamples differing in level of education: little or no formal education (Table 3), an intermediate degree of formal education (Table 4), and higher levels of education (Table 5). In the two subsamples of respondents with the least education, scree tests suggest factor structures with six rather than five dimensions. Extracting only five factors explained 68% of the variance in both samples. The factor loadings (the first ten columns of Table 3) of the 10 items do not conform well to a Big Five pattern in either sample. For both subsamples with intermediate educations, the scree tests also suggest six rather than five factors. Five factor solutions again explained 68% of the variance in both samples. As can be seen from the rotated factor loading matrices shown in Table 4 (the first 10 columns), a clear five factor structure does not emerge in either sample.

Table 3.

Varimax rotated factor structures of the BFI-10 items (raw data) for persons with no or little formal education

Varimax Rotated Factor Loadings
 Procrustes Rotated Factor Loadings
ISSP2003/2004 (N = 1063) ISSP2005/2006 (N = 1330) ISSP2003/2004 (N = 1063) ISSP2005/2006 (N = 1330)
I see myself as someone who … 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
… is reserved −.70 .03 −.11 .28 .26 −.72 .05 .00 −.11 .38 .79 −.00 −.07 −.04 −.12 .67 −.46 .08 −.06 .05
… is outgoing, sociable .74 .22 −.07 .10 .17 .75 −.18 .17 −.08 .25 −.44 −.06 −.06 .20 .63 −.76 −.15 −.14 .11 .25
… is generally trusting .06 .00 −.03 −.18 .88 .02 −.11 −.02 .06 .81 .23 −.77 −.01 −.08 .40 −.14 −.79 −.02 −.17 .09
… tends to find fault with others .01 −.05 .05 .86 −.20 .52 .43 −.25 −.13 .02 .28 .73 .09 −.13 .38 −.46 .19 .41 −.28 .22
… does a thorough job .20 .24 −.67 .19 .29 .15 −.61 .12 −.09 .33 .12 −.08 −.64 .20 .45 −.20 −.37 −.56 .08 .16
… tends to be lazy .00 .06 .85 .19 .13 −.02 .87 .05 .08 .04 .10 .02 .87 .00 .14 .02 .05 .87 .00 −.05
… is relaxed, handles stress well .05 .88 .01 .04 .09 .00 .11 .86 −.05 .18 .12 −.04 .04 .85 .20 −.05 −.30 .19 .80 .12
… gets nervous easily −.16 −.60 .22 .40 .34 −.13 .28 −.72 .05 .32 .36 .02 .24 −.69 .23 .12 −.18 .27 −.78 −.04
… has an active imagination .69 .18 −.12 .30 .16 .50 −.10 .20 −.52 .21 −.32 .09 −.10 .14 .71 −.42 −.09 −.08 .15 .63
… has few artistic interests −.43 .20 .19 .18 .02 .03 .08 .00 .92 .11 .44 .10 .21 .16 −.14 −.22 −.22 .13 −.02 −.87
Vector congruence .77 .91 .95 .08 .71 .81 .86 .94 .94 .52 .73 .99 .94 .94 .48 .82 .63 .86 .94 .92
Matrix congruence .69 .82 .81 .84
Open in a new tab

Note. Highest loadings of each item set in bold; significant congruence coefficients underlined.

Table 4.

Varimax rotated factor structures of the BFI-10 items (raw data) for persons with an intermediate degree of formal education

Varimax Rotated Factor Loadings
 Procrustes Rotated Factor Loadings
ISSP2003/2004 (N = 1026) ISSP2005/2006 (N = 1358) ISSP2003/2004 (N = 1026) ISSP2005/2006 (N = 1358)
I see myself as someone who … 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
… is reserved .79 .15 −.06 .25 .10 .80 .01 −.06 −.07 .19 .84 −.01 −.07 −.11 −.10 .70 −.25 −.05 −.05 −.35
… is outgoing, sociable −.76 −.03 .08 .12 .09 −.51 .26 .23 .42 .14 −.60 −.05 −.00 .20 .44 −.67 .03 −.22 .21 −.16
… is generally trusting −.07 .11 .03 .10 .85 .12 .02 −.01 −.09 .80 .15 −.73 .17 .12 .40 −.02 −.79 −.01 −.05 −.20
… tends to find fault with others .02 .55 .04 .32 −.57 −.02 −.12 −.12 .82 −.19 .20 .77 .18 .05 .26 −.30 .47 .12 −.11 −.63
… does a thorough job .00 −.20 .10 .82 −.01 .08 .81 .14 .20 .10 .23 .12 −.60 .24 .45 −.08 −.05 −.81 .15 −.22
… tends to be lazy .04 .82 .03 −.30 .08 .19 −.79 .00 .32 .05 .19 .13 .84 −.03 .14 .08 .07 .78 −.01 −.38
… is relaxed, handles stress well −.06 .05 .84 .17 .11 .07 .00 .90 .04 .10 .08 −.02 .04 .86 .02 −.00 −.13 .02 .89 −.09
… gets nervous easily .29 .47 −.59 .02 .09 .34 −.22 −.68 .12 .15 .39 .05 .31 −.60 .22 .26 −.08 .19 −.68 −.30
… has an active imagination −.59 .16 −.06 .46 .19 −.31 .27 .23 .44 .40 −.27 −.02 −.07 .11 .70 −.57 −.20 −.20 .20 −.30
… has few artistic interests .37 .13 .43 −.26 −.14 .03 .08 .00 .92 .11 .26 .12 .24 .31 −.43 −.37 .21 −.07 .00 −.82
Vector congruence .82 .63 .90 .45 .94 .85 .93 .94 .24 .72 .83 .98 .90 .89 .69 .78 .88 .94 .94 .26
Matrix congruence .74 .71 .85 .75
Open in a new tab

Note. Highest loadings of each item set in bold; significant congruence coefficients underlined.

Table 5.

Varimax rotated factor structures of the BFI-10 items (raw data) for persons with high degrees of formal education

Varimax Rotated Factor Loadings
 Procrustes Rotated Factor Loadings
ISSP2003/2004 (N = 438) ISSP2005/2006 (N = 689) ISSP2003/2004 (N = 438) ISSP2005/2006 (N = 689)
I see myself as someone who … 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
… is reserved .89 .01 −.02 −.03 .01 .88 .08 .02 .08 .08 .89 . 09 .01 −.03 −.05 .88 −.09 .08 .07 .04
… is outgoing, sociable −.84 −.10 .10 .18 −.02 −.73 .21 .29 .05 .02 −.83 −.02 −.10 .11 .20 −.73 −.00 .04 .21 .28
… is generally trusting −.01 .22 .05 .14 −.73 −.05 .00 .18 .14 .84 −.01 −.73 .22 .04 .14 −.06 −.83 .17 −.01 .20
… tends to find fault with others .02 .27 −.04 .05 .78 −.15 −.11 .19 .31 −.69 .02 .78 .27 −.05 .05 −.15 .71 .27 −.11 .18
… does a thorough job −.04 −.73 .16 .09 .09 .16 .10 .31 −.67 −.05 −.03 .09 −.72 .18 .10 .15 .03 −.68 .11 .29
… tends to be lazy .08 .81 .04 −.13 .13 .19 −.09 .04 .84 −.10 .08 .13 .81 .02 −.14 .19 .13 .83 −.11 .08
… is relaxed, handles stress well .03 .09 .88 .11 −.08 .12 .89 .03 .00 .05 .05 −.08 .12 .87 .14 .13 −.05 .01 .89 .05
… gets nervous easily .20 .31 −.74 .07 .02 .28 −.69 −.04 .24 −.05 .19 .02 .29 −.75 .04 .27 .06 .23 −.70 −.04
… has an active imagination −.30 −.16 .24 .71 .03 −.18 .21 .76 .00 −.01 −.28 .03 −.15 .23 .73 −.19 .03 −.03 .20 .76
… has few artistic interests .00 .09 .10 −.86 .11 .03 .11 −.75 .15 −.05 −.02 .11 .09 .13 −.86 .04 .04 .18 .12 −.74
Vector congruence .96 .90 .96 .96 .98 .92 .94 .90 .92 .98 .96 .98 .91 .96 .95 .92 .98 .92 .94 .91
Matrix congruence .95 .93 .95 .93
Open in a new tab

Note. Highest loadings of each item set in bold; significant congruence coefficients underlined.

For the two highest educated subsamples, scree tests suggest extracting five factors. The proportion of explained variance based on the five-factor solution was 71% and 69%, respectively. The rotated factor loadings for the two samples are listed in the first ten columns of Table 5. For both samples, the factor loadings can unambiguously be interpreted in terms of the Big Five. All items load highest on their corresponding factors, forming clear simple structures in both samples. The congruence coefficients with the idealized structure were .95 and .93, respectively, for the two ISSP subsamples, and all congruence coefficients of single factors were significant and at least .90 or higher. Thus, the factor structures for samples with higher education are in excellent agreement with the target five-factor structure.

To test whether the data from the less well educated subsamples might not be able to be wrenched into shape by the use of “Procrustes” rotational procedures so as to optimally fit the ideal loading matrix (as has been advocated by McCrae, Zonderman, Costa, Bond, and Paunonen, 1996, and Allik and McCrae, 2004), we carried out such Procrustean rotations of the varimax solutions to fit the idealized Big Five pattern (consisting of +1, −1, and 0 loadings). Our findings from these rotations are presented in the last ten columns of Tables 2 to 5. Although the Procrustean transformations lead to higher congruence coefficients, they do not eliminate the problem. Using a standard for congruence coefficients of at least .85 to infer “fair” replication (Lorenzo-Seva & ten Berge, 2006), only in the highest educated subsamples did all five factors meet this criterion.

Individual Differences in Acquiescence Response Bias

The results of the previous analyses reveal that the Big Five factor structure only seems to replicate for respondents with higher educations. As hypothesized, one possible explanation for this finding is that the responses of the different subsamples are affected by an additional sources of variance, namely by differential tendencies for acquiescence response bias. To test this conjecture, an acquiescence index was computed for each respondent. The BFI-10 has two balanced keyed items for each of the Big Five factors. For example, Extraversion is assessed by the items “I see myself as someone who is reserved” and “I see myself as someone who is outgoing, sociable.” The responses to these items should be symmetrical about the answer scale’s middle category if an individual is responding consistently to the content of the two items. Acquiescence, in contrast, should result in a positive mean score of the two items if the answer scale is coded from +2 to −2, or a mean rating greater than 3 if the answer scale is coded from 5 to 1, indicating that agreement to the positive item is stronger than rejection of the negative item. Because there are five item pairs, we used the mean over all items to estimate each individual’s acquiescence response tendency.

Using this index, we tested whether less educated respondents demonstrated more acquiescence than persons with higher education. As shown in Table 6, this is indeed the case in both samples (F(2,2529) = 26.24, p < .001 and F(2,3288) = 13.50, p < .001, respectively). Moreover, the variance of the acquiescence indices is also significantly greater for lower educated persons than for persons with higher education (F(2,2529) = 16.91, p < .001 and F(2,3288) = 8.70, p < .001, respectively, by Levene’s test for equality of variances). Hence, acquiescence is not a universal or uniform response tendency for the lower educated, but rather it affects the scores of some individuals but not those of others.

Table 6.

Means and Standard Deviations of the Individual Means Across the Ten BFI-10 Items Separately for the Three Educational Groups

ISSP2003/2004
 ISSP2005/2006
Mean SD Mean SD
No or little formal education 3.25 .38 3.26 .36
Intermediate degree of formal education 3.22 .33 3.21 .33
High degrees of formal education 3.11 .28 3.18 .32
Open in a new tab

Factor Structures after Controlling for Acquiescence

An obvious hypothesis is that acquiescence tends to attenuate the patterns of correlations among the items that are based on the items’ content, so that statistically controlling for acquiescence should result in factors that better conform to the usual Big Five pattern. To test this hypothesis, we controlled for the individual mean responses by the use of ipsatized scores or within-person standardization (e.g. Ashton et al., 2004; Goldberg, 1990, 1992; McCrae, Herbst & Costa, 2001; Ten Berge, 1999). That is, for each respondent we subtracted the person’s mean across all ten items from each response and then divided these deviation scores by the individual’s standard deviation across all items. As described above, we also used the alternative and more conservative approach of controlling only for the individual mean responses (without a standardization by the individual standard deviation).

Using both types of data, we repeated the factor analyses for the two total samples and for the three subsamples with differing educational levels. For both types of data, in all samples and for each of the three educational levels, scree tests suggested extracting five factors. The five-factor solutions accounted for 69% to 76% of the variance. The varimax-rotated factor loadings for the ipsatized data are shown in Tables 7 and 8. Results for the mean-correction are highly similar and not reported here. Results show that for both methods of correction clear Big Five patterns emerged in all samples and in each of the three educational levels. All items loaded highly on their corresponding factors and few had substantial loadings on any other factor. More objectively, the congruence coefficients were now all above .90 -- mostly well above that -- both for the whole factor loading matrices and also for each individual factor.

Table 7.

Varimax rotated factor structures of the BFI-10 items (ipsatized data) for the total sample and for persons with no or little formal education, respectively.

Total Sample
 No or Little Formal Education
ISSP2003/2004 (N = 2567) ISSP2005/2006 (N = 3421) ISSP2003/2004 (N = 1063) ISSP2005/2006 (N = 1330)
I see myself as someone who … 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
… is reserved −.86 −.06 −.03 .03 .15 −.83 .02 −.03 .12 −.12 −.87 −.03 .01 .02 .11 −.81 −.01 .06 −.16 .15
… is outgoing, sociable .81 .06 .14 .08 .17 .79 −.10 .15 .15 −.04 .77 .11 .15 .08 .17 .77 .09 .18 −.08 .15
… is generally trusting .04 −.09 .09 .82 −.04 .11 .13 .11 −.05 −.87 .03 −.15 .03 .82 .04 .07 −.15 .08 −.84 −.03
… tends to find fault with others .00 −.13 .05 −.80 −.08 .28 .20 .15 −.11 .66 −.01 −.21 .03 −.80 −.02 .29 −.30 .16 .63 −.08
… does a thorough job .00 .04 .14 −.04 .73 −.12 −.01 .09 .78 .16 .05 .12 .19 .08 .61 −.14 .07 .05 .22 .79
… tends to be lazy −.02 .01 .12 −.08 −.82 −.16 .02 .08 −.80 .22 −.01 .07 .14 .03 −.89 −.18 .08 −.00 .33 −.75
… is relaxed, handles stress well −.06 .87 .05 .07 .03 −.09 −.86 .06 .05 −.05 −.06 .86 .07 .05 .08 −.11 .84 .08 −.03 −.03
… gets nervous easily −.19 −.81 .03 .03 −.02 −.22 .80 −.03 .03 −.03 −.21 −.79 .01 −.00 .02 −.27 −.79 −.01 −.03 −.03
… has an active imagination .38 .08 .68 −.03 .11 .31 −.14 .67 .08 .08 .43 .10 .63 −.12 .12 .42 .18 .59 .02 .11
… has few artistic interests .05 .04 −.94 −.06 .04 .05 −.03 −.93 .05 .06 .05 .02 −.94 −.06 .01 .05 .02 −.96 .00 .02
Vector congruence .94 .98 .96 .99 .97 .91 .97 .96 .98 .95 .92 .96 .95 .99 .96 .87 .94 .94 .91 .97
Matrix congruence .97 .95 .95 .93
Open in a new tab

Note. Highest loadings of each item set in bold; significant congruence coefficients underlined.

Table 8.

Varimax rotated factor structures of the BFI-10 items (ipsatized data) for persons with intermediate and high degrees of formal education, respectively.

Intermediate Level of Education
 High Level of Education
ISSP2003/2004 (N = 1026) ISSP2005/2006 (N = 1358) ISSP2003/2004 (N = 438) ISSP2005/2006 (N = 689)
I see myself as someone who … 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
… is reserved −.85 .09 −.01 −.04 .18 −.86 .04 −.03 −.06 .08 −.90 .06 .01 .10 .03 −.86 −.04 .02 .08 −.10
… is outgoing, sociable .82 −.02 .15 −.10 .16 .80 −.11 .15 .00 .13 .84 −.05 .19 .15 .05 .82 −.10 .17 .10 −.01
… is generally trusting .05 .03 .13 −.81 −.05 .12 .12 .12 −.85 −.06 .05 .09 .08 −.06 .83 .12 .09 .12 −.05 −.86
… tends to find fault with others −.01 .01 .06 .79 −.07 .21 .15 .14 .71 −.09 .04 .19 .03 −.11 −.79 .27 .16 .11 −.11 .70
… does a thorough job −.02 −.00 .14 .11 .77 −.03 .01 .04 .07 .76 −.00 .02 −.07 .79 −.06 −.16 .06 .04 .83 .12
… tends to be lazy −.02 .06 .12 .15 −.78 −.09 .03 .02 .10 −.82 −.04 .03 −.13 −.80 −.11 −.19 .15 .05 −.79 .19
… is relaxed, handles stress well −.06 −.88 .02 −.02 .00 −.04 −.87 .08 −.09 .05 −.03 −.86 .16 −.06 .14 −.08 −.85 .04 .14 −.03
… gets nervous easily −.17 .82 .04 −.03 −.06 −.19 .80 −.01 −.07 .04 −.14 .83 .13 −.06 .04 −.14 .82 −.02 .07 .02
… has an active imagination .34 −.03 .67 −.07 .08 .23 −.10 .71 .15 .11 .29 −.16 .73 .19 −.03 .25 −.13 .71 .01 .01
… has few artistic interests .08 −.05 −.93 .02 .04 .02 −.01 −.91 .13 .08 .04 −.10 −.90 .09 −.07 .08 −.08 −.91 .02 .03
Vector congruence .95 .99 .96 .98 .97 .95 .98 .97 .97 .98 .96 .97 .96 .96 .98 .92 .97 .97 .98 .97
Matrix congruence .97 .97 .97 .96
Open in a new tab

Note. Highest loadings of each item set in bold; significant congruence coefficients underlined.

Discussion

Because most Big Five questionnaires are validated in student samples, the present study investigated the extent to which the Big Five factor structure replicates in samples that are more heterogeneous with regard to the educational background of the respondents. Based on two large random samples that are representative of the German adult population, we only partially replicated the factor structure predicted by the Big Five for both samples. We then tested if this finding is related to the persons’ educational background. Splitting the samples into three subgroups based on the respondents’ educational background clearly revealed that for persons with no, low, or intermediate secondary education, the Big Five questionnaire did not result in the typical Big Five factor structure. However, for persons with higher secondary education (i.e., those with a general university entrance qualification or with a university degree), and thus for samples comparable to the previously mostly investigated student samples, the five-factor structure replicated clearly. These results suggest that factor structures appear to be highly sensitive to a person’s educational level.

Soto and colleagues (2008) recently showed that children and adolescents exhibit a greater variability in acquiescence response bias, which, as an additional source of variance, influences the psychometric properties of their personality ratings. Several earlier studies (e.g., Krosnick et al., 1996) suggested that individual differences in acquiescence bias are correlated with social status and education. Hence, we hypothesized that the failure to replicate the usual Big Five structure for persons with lower education is due to their increased likelihood for response bias. The data indeed revealed that acquiescence is higher for those respondents with lower education. Moreover, controlling for acquiescence clearly diminished the differences in the factorial structures between the educational groups and yielded five-factor structures for all educational levels that were clearly interpretable in terms of the Big Five.

The effects of acquiescence response bias can be explained in many different ways. One possibility is that the less educated are not as used to thinking abstractly and to considering the hypothetical but rather are tied more to the concrete and immediate (Flavell, Miller, & Miller, 1993; Toomela, 2000, 2003a, 2003b). These competencies are assumed to be related to a respondent’s likelihood to arrive at meaningful judgments that describe him- or herself in relatively abstract psychological terms. Hence, their judgments should be more affected by systematic response biases such as acquiescence (cf. Goldberg, 1963; Soto et al., 2008). Future researchers may investigate the extent to which the present findings with regard to educational biases can be further extended to other socio-demographical and/or psychological differences among respondents. Moreover, future studies should test the effects of controlling for acquiescence, either by the use of deviation scores or standard scores, on the construct validity of the measures.

In future research, we also need to investigate in more detail the causes of individual differences in response-scale usage among persons with different educational backgrounds. Specifically, we need to understand how respondents of different educational levels understand and answer personality items more generally. One approach could be to conduct cognitive interviews (e.g., Willis, 2005) in which the respondents think aloud while answering the items. Another method involves the comparison of different forms of items or rating scales to discover those formats that are least sensitive to response biases and thus to differences among persons of different educational levels. What we should not do in the future, however, is to ignore such effects and assume that questionnaires validated on undergraduates will also be appropriate for everyone else.

One limitation of the present study is that it is solely based on results of a single and extremely short Big Five instrument; future studies should aim to replicate the present findings with more heterogeneous and longer Big Five scales. An even more important limitation of the present research is that it is based on a measure that is completely balanced in its content keying across each of the five dimensions. The use of fully ipsatized or deviation scores will only be effective to the extent to which the item pool is so balanced. Otherwise, the removal of respondents’ mean scores will also remove substantive content that is confounded with acquiescent response bias. In unbalanced item pools, the most extreme being those that include only items all scored in the same direction of one or more content dimensions, it may be impossible to unconfound (and thus eliminate) individual differences in response bias.

In sum, our findings based on two large samples representative of the German adult population clearly showed that the five-factor structure is sensitive to educational effects and that these effects seem to be due to a more pronounced tendency for acquiescent responding in less well-educated respondents.

Acknowledgments

Funds for the second author were provided by Grant AG20048 from the National Institute for Aging, National Institutes of Health, U. S. Public Health Service.

Footnotes

1

This method of controlling for acquiescence assumes the existence of balanced pairs of true and false items per dimension.

2

Differences between the factor structures of the ISSP 2003/2004 and the ISSP 2005/2006 might be due to this exchanged Agreeableness item.

3

The similarity of two 5-dimensional factor solutions, X and Y, can be assessed by computing the congruence coefficient c = trace(XY′)/((trace(XX′)trace(YY′))1/2. Let X be the ideal Big5 factor loadings and Y the corresponding empirical varimax-rotated factor loadings of some sample of persons. To find the critical value of c for the nullest of all null hypotheses, we computed the first five principal components of the intercorrelations of 10 variables, each consisting of 100 cases drawn from a rectangular random distribution. Then, we removed meaningless differences of X and Y by an optimal permutation (out of 120 possibilities) and reflection (out of 5 possibilities for each permutation) of the columns of Y. That is, c is computed for X and YPR, where P is a permutation matrix and R a reflection matrix that maximize c. Repeating this simulation 1,000 times, we found that if one wants to be 99% certain that the congruence of X and YPR is “significant”, c should be greater than 0.61. Similar simulations where Y is fitted to X by the much wider class of orthogonal Procrustes transformations (Gower & Dijksterhuis, 2004) show that c should be greater than .78 to reject random congruency with at least 99% confidence. Yet, these statistical benchmarks are only weak lower bounds for deciding that Y replicates X, becauseLorenzo-Seva & Ten Berge (2006) have shown that seasoned factor analysts are likely to interpret X and Y as “fairly” similar only if c is at least .85. Assessments of “good” replications even require congruence values of at least .94. Hence, “this result should prevent congruence below .85 from being interpreted as indicative of any factor similarity at all” (Lorenzo-Seva & Ten Berge, 2006, p. 10).

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Allik J, McCrae RR. Escapable conclusions: Toomela (2003) and the universality of trait structure. Journal of Personality and Social Psychology. 2004;87:261–265. doi: 10.1037/0022-3514.87.2.261. [DOI] [PubMed] [Google Scholar]
  2. Allik J, Laidra K, Realo A, Pullmann H. Personality development from 12 to 18 years of age: changes in mean levels and structure of traits. European Journal of Personality. 2004;18:445–462. [Google Scholar]
  3. Ashton MC, Lee K, Perugini M, Szarota P, de Vries RE, Di Blas L, et al. A six-factor structure of personality-descriptive adjectives. Solutions from psycholoexical studies in seven languages. Journal of Personality and Social Psychology. 2004;86:356–366. doi: 10.1037/0022-3514.86.2.356. [DOI] [PubMed] [Google Scholar]
  4. Ayidiya SA, McClendon MJ. Response effects in mail surveys. Public Opinion Quarterly. 1990;54:229–247. [Google Scholar]
  5. Benet-Martínez V, John OP. Los cinco grandes across cultures and ethnic groups: Multitrait multimethod analyses of the Big Five in Spanish and English. Journal of Personality and Social Psychology. 1998;75:729–750. doi: 10.1037//0022-3514.75.3.729. [DOI] [PubMed] [Google Scholar]
  6. Costa PT, McCrae RR. NEOPI/FFI manual supplement. Odessa, FL: Psychological Assessment Resources; 1989. [Google Scholar]
  7. Costa PT, McCrae RR. Professional Manual. Odessa, FL: Psychological Assessment Resources; 1992. Revised NEO Personality Inventory (NEO PI-R) and NEO Five Factor Inventory. [Google Scholar]
  8. Drasgow F, Probst TA. The psychometrics of adaptation: Evaluating measurement equivalence across languages and cultures. In: Hambleton RK, Merenda PF, Spielberger CD, editors. Adapting educational and psychological tests for cross-cultural assessment . Hillsdale, NJ: Erlbaum; 2004. pp. 265–296. [Google Scholar]
  9. Flavell JH, Miller PH, Miller SA. Cognitive Developments. Engelwood Cliffs, New Jersey: Prentice-Hall; 1993. [Google Scholar]
  10. Goldberg LR. A model of item ambiguity in personality assessment. Educational and Psychological Measurement. 1963;23:467–492. [Google Scholar]
  11. Goldberg LR. An alternative “Description of personality”: The Big-Five factor structure. Journal of Personality and Social Psychology. 1990;59:1216–1229. doi: 10.1037//0022-3514.59.6.1216. [DOI] [PubMed] [Google Scholar]
  12. Goldberg LR. The development of markers for the Big-Five factor structure. Psychological Assessment. 1992;4:26–42. [Google Scholar]
  13. Gove WR, Geerken MR. Response bias in surveys of mental health: An empirical investigation. American Journal of Sociology. 1977;82:1289–1317. doi: 10.1086/226466. [DOI] [PubMed] [Google Scholar]
  14. Groves RM. Survey errors and survey costs. New York: John Wiley; 1989. [Google Scholar]
  15. Gower JC, Dijksterhuis GB. Procrustes problems. Oxford: 2004. [Google Scholar]
  16. Gudjonsson GH. The relationship between interrogative suggestibility and acquiescence: Empirical findings and theoretical implications. Personality and Individual Differences. 1986;7:195–199. [Google Scholar]
  17. Haarmann A, Scholz E, Wasmer M, Blohm M, Harkness J. Konzeption und Durchführung der Allgemeinen Bevölkerungsumfrage der Sozialwissenschaften“ (ALLBUS) 2004. [Concept and implementation of the German general social survey (ALLBUS)] ZUMA-Arbeitsbericht 2006 [Google Scholar]
  18. Jackson DN, Messick S. Content and style in personality assessment. Psychological Bulletin. 1958;55:243–252. doi: 10.1037/h0045996. [DOI] [PubMed] [Google Scholar]
  19. Javeline D. Response effects in polite cultures: A test of acquiescence in Kazakhstan. Public Opinion Quarterly. 1999;63:1–28. [Google Scholar]
  20. John OP, Donahue EM, Kentle RL. The Big Five Inventory- Versions 4a and 54. Berkeley, CA: University of California, Berkeley, Institute of Personality and Social Research; 1991. [Google Scholar]
  21. John OP, Srivastava S. The Big Five trait taxonomy: History, measurement, and theoretical perspectives. In: Pervin LA, John OP, editors. Handbook of personality: Theory and research. 2. New York: Guilford; 1999. pp. 102–138. [Google Scholar]
  22. Koch A. Wenn “mehr” nicht gleichbedeutend mit “besser” ist: Ausschöpfungsquoten und Stichprobenverzerrungen in allgemeinen Bevölkerungsumfragen. [When “more” isn’t “better”: Response rates and sample biases in general population surveys] ZUMA-Nachrichten. 1998;42:66–90. [Google Scholar]
  23. Körner A, Geyer M, Brähler E. Das NEO-Fünf-Faktoren Inventar (NEO-FFI). Validierung anhand einer deutschen Bevölkerungsstichprobe. [The NEO Five Factor Inventory (NEO-FFI). Validation on a German population representative sample] Diagnostica. 2002;48:19–27. [Google Scholar]
  24. Krosnick JA, Narayan S, Smith WR. Satisficing in surveys: Initial evidence. New Directions for Evaluation. 1996;70:29–44. [Google Scholar]
  25. Lang FR, Lüdtke O, Asendorpf J. Testgüte und psychometrische Äquivalenz der deutschen Version des Big Five Inventory (BFI) bei jungen, mittelalten und alten Erwachsenen. [Validity and psychometric equivalence of the German version of the Big Five Inventory in young, middle-aged and old adults] Diagnostica. 2001;47:111–112. [Google Scholar]
  26. Lenski GE, Leggett JC. Caste, class, and deference in the research interview. American Journal of Sociology. 1960;65:463–467. [Google Scholar]
  27. Lorenzo-Seva U, Jos MF, ten Berge Tucker’s congruence coefficient as a meaningful index of factor similarity. Methodology. 2006;2:57–64. [Google Scholar]
  28. McCrae RR, Herbst JH, Costa PT., Jr . Effects of acquiescence on personality factor structures. In: Riemann R, Spinath FM, Ostendorf F, editors. Personality an temperament: Genetics, evolution, and structure. Berlin: Pabst Science Publishers; 2001. pp. 217–231. [Google Scholar]
  29. McCrae RR, Zonderman AB, Costa PT, Jr, Bond MH, Paunonen SV. Evaluating replicability of factors in the Revised NEO Personality Inventory: Confirmatory factor analysis versus Procrustes rotation. Journal of Personality and Social Psychology. 1996;70:552–566. [Google Scholar]
  30. Mõttus R, Allik J, Realo A. Is the Big Five valid at all levels of education?. Poster presented at the 13th Biennal Meeting of the International Society for the Study of Individual Differences (ISSID).2007a. [Google Scholar]
  31. Mõttus R, Allik J, Pullmann H. Does personality vary across ability levels? A study using self and other ratings. Journal of Research in Personality. 2007b;41:155–170. [Google Scholar]
  32. Narayan S, Krosnick JA. Education moderates some response effects in attitude measurement. Public Opinion Quarterly. 1996;60:58–88. [Google Scholar]
  33. Rammstedt B. Die deutsche Version des Big Five Inventories (BFI): Übersetzung und Validierung eines Fragebogens zur Erfassung des Fünf-Faktoren-Modells der Persönlichkeit. Unveröffentlichte Diplomarbeit. Universität Bielefeld; Germany: 1997. [Google Scholar]
  34. Rammstedt B, John OP. Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. Journal of Research in Personality. 2007;41:203–212. [Google Scholar]
  35. Rammstedt B, Krebs D. Does response scale format affect the answering of personality scales? Assessing the Big Five dimensions of personality with different response scales in a dependent sample. European Journal of Psychological Assessment. 2007;23:32–38. [Google Scholar]
  36. Rolland JP, Parker W, Stumpf H. A psychometric examination of the French translation of the NEO-PI-R and NEO-FFI. Journal of Personality Assessment. 1998;71:269–291. [Google Scholar]
  37. Sigelman CK, Budd EC, Spanhel CL, Schoenrock CJ. When in doubt say yes: Acquiescence in interviews with mentally retarded persons. Mental Retardation. 1981;19:53–58. [PubMed] [Google Scholar]
  38. Soto CJ, John OP, Gosling SD, Potter J. The developmental psychometrics of Big-Five self-reports: Acquiescence, factor structure, coherence, and differentiation from ages 10 to 20. Journal of Personality and Social Psychology. 2008;94:718–737. doi: 10.1037/0022-3514.94.4.718. [DOI] [PubMed] [Google Scholar]
  39. Statistisches Bundesamt. Fachserie 1: Bevölkerung und Erwerbstätigkeit (Ergebnisse des Mikrozensus). [Series 1: Population and Occupation (Results of the Microcensus 2003)] Stuttgart: Metzler-Poeschel 2005 [Google Scholar]
  40. Statistisches Bundesamt. Ergebnisse des Mikrozensus 2005. Fachserie 1: Bevölkerung und Erwerbstätigkeit. [Results of the Microcensus 2005. Series 1: Population and Occupation] Wiesbaden 2007 [Google Scholar]
  41. Ten Berge JMF. A legitimate case of component analysis of ipsative measures, and partialling the mean as an alternative to ipsatization. Multivariate Behavioral Research. 1999;34:89–102. doi: 10.1207/s15327906mbr3401_4. [DOI] [PubMed] [Google Scholar]
  42. Tokar DM, Fischer AR, Snell AF, Harik-Williams N. Efficient assessment of the Five-Factor Model of personality: Structural validity analyses of the NEO-Five-Factor Inventory (Form S) Measurement and Evaluation in Counselling and Development. 1999;32:14–30. [Google Scholar]
  43. Toomela A. Stages of mental development: Where to look? Trames: A Journal of the Humanities in Social Sciences. 2000;4:21–52. [Google Scholar]
  44. Toomela A. Relationship between personality structure, structure of word meaning, and cognitive ability: A study of cultural mechanisms of personality. Journal of Personality and Social Psychology. 2003a;85:723–735. doi: 10.1037/0022-3514.85.4.723. [DOI] [PubMed] [Google Scholar]
  45. Toomela A. Development of symbol meaning and the emergence of the semiotically mediated mind. In: Toomela A, editor. Cultural guidance in the development of the human mind. Westport, CA: Aplex Publishing; 2003b. pp. 163–209. [Google Scholar]
  46. Vandenberg RJ, Lance CE. A review and synthesis of the measurement equivalence literature: suggestions, practices, and recommendations for organizational research. Organizational Research Methods. 2000;3:4–70. [Google Scholar]
  47. Wasmer M, Scholz E, Blohm M. Konzeption und Durchführung der Allgemeinen Bevolkerungsumfrage der Sozialwissenschaften“ (ALLBUS) 2006. [Concept and Implementation of the German general social survey (ALLBUS) 2006] ZUMA-Arbeitsbericht 2007 [Google Scholar]
  48. Willis GB. A tool for improving questionnaire design. Thousand Oaks: Sage Publications; 2005. Cognitive Interviewing. [Google Scholar]

RESOURCES