Measurement confounding affects the extent to which verbal IQ explains social gradients in mortality

Abstract

Background

IQ is thought to explain social gradients in mortality. IQ scores are based roughly equally on Verbal IQ (VIQ) and Performance IQ tests. VIQ tests, however, are suspected to confound true verbal ability with socioeconomic status (SES), raising the possibility that associations between SES and IQ scores might be overestimated. We examined, first, whether two of the most common types of VIQ tests exhibited differential item functioning (DIF) favouring persons of higher SES and/or majority race/ethnicity. Second, we assessed what impact, if any, this had on estimates of the extent to which VIQ explains social gradients in mortality.

Methods

Data from the General Social Survey-National Death Index cohort, a US population representative dataset, was used. Item response theory models queried social-factor DIF on the Thorndike Verbal Intelligence Scale and Wechsler Adult Intelligence Scales, Revised Similarities test. Cox models examined mortality associations among SES and VIQ scores corrected and uncorrected for DIF.

Results

When uncorrected for DIF, VIQ was correlated with income, education, occupational prestige and race, with correlation coefficients ranging between |0.12| and |0.43|. After correcting for DIF, correlations ranged from |0.06| to |0.16|. Uncorrected VIQ scores explained 11–40% of the Relative Index of Inequalities in mortality for social factors, while DIF-corrected scores explained 2–29%.

Conclusions

Two of the common forms of VIQ tests appear confound verbal intelligence with SES. Since these tests appear in most IQ batteries, circumspection may be warranted in estimating the amount of social inequalities in mortality attributable to IQ.

Socioeconomic status (SES) and cognitive ability are powerful predictors of health and longevity.¹ In fact, cognitive ability, as measured by the IQ, has been hypothesised to account for much of the SES-related health gradient.² Supporting this hypothesis, correlational studies suggest that those with stronger cognitive skills may be able to better understand medical instructions, navigate social bureaucracies, and avoid accidental death.^3–8 IQ also putatively correlates with markers of SES in the 0.4–0.55 range.⁹

IQ scores are themselves composed of two types of tests. ‘Performance IQ’ tests assess non-verbal reasoning and analytic ability. Tests of ‘Verbal IQ’ (VIQ) reflect language-based reasoning ability and knowledge.¹⁰ ‘True’ VIQ is presumed to reflect factors, such as the speed and facility of language acquisition, and abstract and symbolic manipulation of language to solve problems and attain goals.¹⁰ These skills assume different forms depending on the socioeconomic environment in which language is learned and reinforced, however.^11,12 Thus, language-based problem solving may vary markedly across socioeconomic strata.

In psychometric theory, VIQ ‘true scores’ refer to test scores putatively measuring the concept of ‘true VIQ.’¹³ Under common statistical assumptions, VIQ true scores can be separated from other factors potentially influencing performance on VIQ tests that fall outside the definition of verbal intelligence. Many such factors have been implicated in VIQ test performance, including higher education, middle class SES and majority culture.^11,12 VIQ tests scores, therefore, run the risk of mixing or confounding verbal intelligence true scores with educationally or socially acquired knowledge, class-differential verbal styles, academic motivation, standardised testing experience and other residue of social position.

More technically, this measurement confounding arises from violations of collapsibility and exchangeability.¹⁴ In other words, the association between the latent trait of ‘verbal intelligence’ (unobserved) and scores on a test of VIQ (observed) may not be collapsible across SES strata. Moreover, persons may not be exchangeable on non-intelligence attributes that affect response to items on the test (ie, measures of social standing). This type of systematic measurement error is called Differential Item Functioning (DIF).¹⁵ If it is present to a significant degree on a VIQ test, persons of lower SES may achieve artificially deflated VIQ test scores, driving the apparent association between SES and VIQ upward.¹⁶ As a result, the extent to which VIQ scores explain social gradients in mortality may appear larger than it actually is.

We examined whether two common forms of VIQ test exhibited DIF related to socioeconomic indicators (education, income, occupational prestige) and race in a US national sample. Although race/ethnicity may be less associated with social class in European countries, it is often considered a dimension of social stratification in the USA. We then compared SES correlations with VIQ scores, adjusted and unadjusted for DIF. Finally, we examined how much of the association between SES factors and all-cause mortality could be explained by VIQ scores, with and without correction for DIF. Our goal was to test a central premise of current research—that cognitive ability explains social patterns in mortality.

METHODS

Sample and design

We used data from the General Social Survey (GSS), an annual nationally representative sampling of US population social practices and attitudes. Conducted by the National Opinion Research Center at the University of Chicago, the GSS uses a multistage probability sampling of non-institutionalised adults age 18 years and over, with response rates from 70% to 82% in any given year,¹⁷ yielding demographically identical annual samples. The GSS records age, gender, race/ethnicity, respondent occupation, income and years of education on the basis of face-to-face interviews with subjects. The Gallup-Thorndike Test of Verbal Intelligence¹⁸ (hereafter Gallup-Thorndike) was administered during these interviews to one-third to one-half of the sample randomly selected during the years 1978, 1982, 1984, 1987–2000. We used data for 9381 persons with complete data for all variables of interest. Those lacking data (usually an SES indicator) were more likely female, younger, minority, and in worse self-rated health (p<0.001); the resulting analytic sample was still broadly similar to that of the USA in 2000.¹⁹ The second VIQ test, the Wechsler Adult Intelligence Scales—Revised Similarities Test (WAIS-R Similarities), was administered in 1996 and yielded an analytic sample of 2444, by design demographically comparable to the broader Gallup Thorndike sample.

Measures

Occupation was coded using the Socioeconomic Index (SEI), a continuous measure of occupational prestige, based on US Census information. Income was calibrated to 1990 US dollars. The Gallup-Thorndike¹⁸ originally comprised 20 items taken from the Institute for Educational Research (IER) Intelligence Scale CAVD.²⁰ In the GSS, 10 of these items were administered in person by an interviewer. Tests of vocabulary are presumed to assess word familiarity, and also (1) concept formation (without which the correct definition cannot be given) and (2) the ability to deduce meaning of unfamiliar words based on known roots or syllables, using answer choices provided.²¹

The second VIQ test, the WAIS-R Similarities test,²² presents persons with successively more difficult questions about how two different things are alike. For instance, an easy question might be ‘how are a fly and a mouse related?’ A completely correct response, such as ‘they are both animals’, receives two points. A response that is correct but does not capture the similarity at the most abstract level (eg, ‘they both have eyes’) receives one point. The WAIS-R manual contains detailed guidelines for scoring responses as 0, 1, or 2.²²

Vital status through 2008 was ascertained from the National Death Index. The validity of the National Death Index is typically high, with matching certainty arising from social security numbers and the additional identifiers in the GSS reaching 99.8%.²³ Further details on the GSS-National Death Index matching are available.¹⁷

Analysis

Occupational prestige, incopme, and education were scaled by Relative Index of Inequalities (RII). The RII scores the person of highest standing on a social dimension as 0, and the lowest as 1.^24,25 A 1 unit change in regression models, therefore, is interpretable as a relative risk, but a specific kind: the risk at the absolute top, relative to the absolute bottom, of a distribution.

Item Response Theory (IRT) analyses of the Gallup-Thorndike were conducted with the Rasch model²⁶ and WAIS-R similarities analyses used the graded response model.²⁷ The Rasch model is formally equivalent to a mixed-effect logistic regression treating test items as repeated measures within person,²⁸ and estimating the probability of success or ‘difficulty parameter’ for each item independently of an examinee’s standing on the latent trait (random effect). The graded response model is an extension for ordered responses analogous to the extension from a binary to ordered logistic model (we relaxed the proportional odds assumption). The online supplementary material provides technical details of these IRT models and DIF analysis.²⁹

Briefly, we examined DIF related to race, SEI, household income and education, as well as age and gender (which are correlated with SES) using interaction terms²⁸ in three increasingly stringent steps. In step 1, we screened social factor interaction terms separately for each item. In our second step, the model adjusted for all previously identified sources of DIF for a single item simultaneously. In the third stage, we adjusted for DIF factors across all items simultaneously. At each step, we retained those that were significant and met a DIF effect size threshold such that the item’s difficulty was 30% easier at one end of a sociodemographic dimension than at the other, irrespective of VIQ true score. Latent trait scores for each test were then estimated from IRT models with and without this final set of DIF interaction terms.

We examined impact of DIF on SES-VIQ associations via Pearson correlations between each SES factor and VIQ scores unadjusted and adjusted for DIF. We computed the absolute difference (r_unadj−r_adj), as well as relative difference (r_unadj/r_adj) between DIF adjusted and unadjusted score correlations.³⁰ We also estimated the association of DIF-corrected and uncorrected VIQ scores with mortality using Cox proportional hazards models with attained age as time scale and GSS baseline age as point of entry into the risk set,^31,32 fitting three models for each SES factor. Each model included gender as a covariate with time-varying hazards, based on preliminary proportionality analysis. The first model estimated the SES factor’s RII, or the HR for those at the most disadvantaged, versus advantaged end of the distribution. A second model then added VIQ scores unadjusted for DIF and computed the excess hazard explained by these scores as (HR_unadjusted−HR_adjusted)/(HR_unadjusted−1), with 95% CIs obtained via bootstrap (1000 replicates). A third model then controlled for DIF-adjusted VIQ scores, again computing the change in estimate.

RESULTS

Table 1 shows the sample demographics for the Gallup-Thorndike sample (left) and the WAIS-R similarities subset of that sample (right). With respect to VIQ tests, assumptions underlying IRT models appeared to be satisfactory.³³ For the Gallup-Thorndike, of the 40 interaction terms involving race, SEI, household income and education, 21 remained statistically significant and met the effect size criteria by the end of the three-stage screening. For the WAIS-R similarities test, of the 80 possible interactions, 14 remained significant and met the effect size threshold at the end of the third stage. Social factor DIF favoured white race and higher education, occupation and income. Additional age-related and gender-related DIF was observed on both tests, although the pattern did not consistently favour one gender or younger versus older persons. Online Supplementary table S1 lists the sources of DIF by item for each test. Social DIF seemed more apparent on vocabulary (9 out of 10 items) than similarities (5 out of 8 items).²⁹

Table 1.

Demographic composition of the analytic sample: 1978–2002 General Social Survey inked to the 2008 mortality via the National Death Index

	Mean/number	SD/percent of total sample
Age (years)	44.2	16.6
Gender
Male	6422	45.1%
Female	7817	54.9%
Race/ethnicity
White	11 894	83.5%
Black	1838	12.9%
Other race	507	3.6%
Socioeconomic status
Occupational prestige*	42.15	13.9
Income†	44, 455	43 916
Education (years)	13.0	3.0

^*The occupational Prestige score reflects the 1970 Socioeconomic Index for 1979–1989, and the 1980 Socioeconomic Index for 1990–2002 respondents.

^†Income is in year 2000 US dollars.

Table 2 reports the correlations between SES factors and VIQ test scores corrected and uncorrected for DIF. SES correlations with Gallup-Thorndike scores unadjusted for DIF were 0.16–0.33 larger in absolute magnitude, and 2.8–4.4 larger in relative magnitude, than unadjusted scores. For the WAIS-R Similarities, absolute differences in SES correlations ranged from 0.06 to 0.24 and relative differences from 2.0 to 2.6. DIF-adjusted correlations between SES factors and VIQ indicators fell outside the 95% CI of correlations with non-adjusted scores for all SES indicators on both tests.

Table 2.

Pearson correlations between SES, race, and verbal IQ test scores: 1978–2002 general social survey linked to the 2008 mortality via the National Death Index

	Latent trait scores, no DIF adjustment	Latent trait scores, DIF adjustment	Absolute difference	Relative difference
Gallup-Thorndike Verbal Intelligence test
Occupational prestige	0.32 (0.30 to 0.34)	0.11 (0.09 to 0.13)	0.21	2.91
Education	0.43 (0.41 to 0.45)	0.10 (0.08 to 0.12)	0.33	4.30
Income	0.25 (0.23 to 0.27)	0.09 (0.07 to 0.11)	0.14	2.78
Minority race/ethnicity	−0.21 (−0.19 to −0.23)	−0.05 (−0.03 to −0.07)	0.16	4.20
Wechsler Adult Intelligence Scales Revised Similarities test
Occupational Prestige	0.26 (0.23 to 0.32)	0.10 (0.06 to 0.14)	0.16	2.60
Education	0.40 (0.37 to 0.43)	0.16 (0.12 to 0.20)	0.24	2.50
Income	0.22 (0.18 to 0.26)	0.09 (0.05 to 0.13)	0.13	2.44
Minority race / ethnicity	−0.12 (−0.08 to −0.16)	−0.06 (−0.02 to −0.10)	0.06	2.00

Notes: Pearson correlations (r) with bootstrap 95% CIs. For minority race/ethnicity, positive correlations indicate higher VIQ scores among minorities, negative correlations indicate lower VIQ scores. Income in Year 2000 US dollars. Occupational Prestige is 1970 Socioeconomic Index for 1979–1989, 1980 SEI for 1990–2002 respondents. n=9381 for Gallup-Thorndike and 2444 for Similarities test.

DIF, Differential Item Functioning; VIQ, verbal IQ; SES, socioeconomic status.

Table 3 shows the RII as a HR for mortality for each social factor. Minority race exhibited non-proportional hazards (diminishing risk over the lifecourse), so estimates are presented at age 50 years. Table 3 also shows the change in estimate observed when controlling for latent trait scores adjusted and unadjusted for DIF. VIQ scores unadjusted for DIF accounted for smaller, but non-zero portions of social inequalities in mortality. For the Gallup-Thorndike, the change in estimate arising from corrected scores fell outside of the CI of that for uncorrected scores across three of four social factors. For the WAIS-R similarities test, the same pattern arose, but with wider CIs. Table 4 shows the RII for the WAIS-R similarities and Gallup-Thorndike scores with and without SES-corrected DIF. DIF-corrected scores showed smaller RIIs, with no appreciable difference in the proportion explained by SES. The latter quantity evidenced a very wide CI encompassing 0 in all cases. Sensitivity analyses revealed linearity in the log hazard for all factors, no VIQ social factor interactions or proportionality violations, nearly identical results excluding deaths within the first year, and comparable results with 2-parameter IRT models.

Table 3.

Social inequalities in mortality explained by biased and unbiased VIQ test scores

	HR	Excess risk explained by VIQ scores uncorrected for DIF	Excess risk explained by VIQ scores corrected for DIF
SES factors adjusted for Gallup-Thorndike test
Occupational prestige	1.66 (1.34 to 2.05)	0.29 (0.11 to 0.47)	0.07 (0.01 to 0.13)
Education	1.92 (1.47 to 2.52)	0.40 (0.14 to 0.66)	0.29 (0.11 to 0.48)
Income	2.05 (1.71 to 2.45)	0.15 (0.05 to 0.24)	0.04 (0.00 to 0.07)
Minority Race/ethnicity	1.74 (1.52 to 1.98)	0.11 (0.05 to 0.17)	0.02 (0.00 to 0.03)
SES factors adjusted for WAIS-R Similarities Test
Socioeconomic Index	1.94 (1.16 to 3.21)	0.13 (−0.27 to 0.72)	0.04 (−0.17 to 0.32)
Education	1.65 (0.90 to 3.01)	0.36 (−2.98 to 4.70)	0.09 (−1.30 to 1.52)
Income	2.34 (1.56 to 3.50)	0.04 (−0.23 to 0.27)	0.01 (−0.14 to 0.15)
Minority	1.44 (1.01 to 2.06)	0.09 (−0.26 to 0.59)	0.02 (−0.09 to 0.21)

Note. VIQ=verbal IQ. The HR is the Relative Index of Inequalities, or ratio of risk for those at the bottom of each distribution (ie, worst scoring or most disadvantaged) compared to those at the top. Percent of RII explained=excess risk explained by adjustment factor, computed as (HR_unadjusted−HR_adjusted)/(HR_unadjusted−1) with 95% CIs based on bootstrap percentiles over 1000 replicates of 1000 observations. Same n=9381 for Gallup Thorndike and 2444 for WAIS-R similarities test. Results from separate Cox proportional hazard models for each factor, with age as time scale, all adjusted for gender as time-varying covariate due to non-proportional hazard. Minority status is also time varying, with estimates presented at age 50.

Table 4.

VIQ Inequalities in Mortality Explained by SES Indicators

	VIQ tests adjusted for SES Factors
	Hazard ratio	Excess risk Explained by SES indicators
Gallup-Thorndike, uncorrected	1.49 (1.28 to 1.75)	0.73 (−0.05 to 1.02)
Gallup-Thorndike, DIF corrected	1.39 (1.20 to 1.61)	0.69 (−0.11 to 0.97)
Similarities, uncorrected	1.30 (0.94 to 1.79)	0.76 (−5.37 to 6.51)
Similarities, DIF corrected	1.17 (0.83 to 1.64)	0.79 (−6.2 to 9.73)

VIQ=verbal IQ. The HR is the Relative Index of Inequalities, or ratio of risk for those at the bottom of each distribution (ie, worst scoring or most disadvantaged) compared to those at the top. Percent of RII explained=excess risk explained by adjustment factor, computed as (HR_unadjusted−HR_adjusted)/(HR_unadjusted−1) with 95% CIs based on bootstrap percentiles over 1000 replicates of 1000 observations. Same n=9381 for Gallup Thorndike and 2444 for WAIS-R similarities test. Results from separate Cox proportional hazard models for each factor, with age as time scale, all adjusted for gender as time-varying covariate due to non-proportional hazard. Minority status is also time varying, with estimates presented at age 50 years. DIF, differential item functioning; SES, socioeconomic status.

DISCUSSION

Across two VIQ tests, we found DIF favouring persons of higher SES and/or majority race/ethnicity group. Correcting for this, DIF reduced correlations between VIQ scores and educational attainment, occupational status, income, as well VIQ differences between African–Americans and Caucasians. In turn, VIQ scores adjusted for DIF explained smaller amounts of social inequalities in mortality.

Some have argued that intelligence, rather than SES, is the fundamental cause of differentials in mortality.² This assertion is supported by many findings that cognitive ability test scores are substantial confounders of SES mortality risk.³⁴ Our findings suggest that DIF-corrected VIQ scores had slightly less association with SES and mortality than uncorrected ones, so a portion of the predictive power of VIQ may arise from indirect SES variance captured by VIQ test scores.

It is important to note, however, that small social differentials in VIQ still existed even when DIF was controlled. Accordingly, DIF-corrected VIQ scores continue to explain a modest portion of social gradients in mortality. This would suggest that VIQ is somehow involved in social inequalities in mortality, albeit to a smaller extent than has been presumed.

Environmental exposures³⁵ and early malnutrition³⁶ have documented effects on brain development and cognitive ability, and it is plausible, if not likely, that persons scoring lower on IQ tests, consequently, are challenged with respect to school performance, occupational advancement and earnings.³⁷ Thus, our data indeed suggest a legitimate—and probably reciprocal—association between VIQ and SES. Given the importance of this issue for policy, the critical question is not whether there is a link, but exactly how much measurement inaccuracy inflates our current estimates.

Specifically, DIF observed here may be explained by numerous factors affecting IQ test performance that are associated with social disadvantage. These include achievement motivation,³⁸ greater test performance anxiety and stress,^39,40 fear that poor test scores will be used to perpetuate stereotypes about class and intelligence,^41,42 lack of familiarity with test content among participants from lower SES and/or racial/ethnic minority subcultures,¹¹ different norms for, or uncertainty in, approaching test problems,^43,44 use of different dialects, distrust of examiners administering the test,^45,46 less familiarity with testing,⁴⁷ a lower reading level,^48,49 and poorer test-taking skills.⁵⁰ The difficulty of disentangling verbal intelligence from factors relating to culture and academic achievement has been reported for some time.¹⁰ Nevertheless, the fairness of IQ tests across SES is justified, in part, by reports that IQ tests with unknown degrees of SES DIF predict SES outcomes.³⁷ Such justifications may require reconsideration if VIQ tests confound Verbal IQ and SES to a non-trivial degree.

Our results must be interpreted with a balanced understanding of strengths and limitations. First, while these considerations suggest that VIQ tests capture educational and other SES variance, an important parallel argument has been offered: years of education, perhaps the most common index of SES, might actually measure some form(s) of intelligence, because cognitive abilities are generally required to achieve higher levels of education.⁵¹ From this viewpoint, adjusting any type of VIQ score for education-related DIF corrects for an IQ proxy and, thus, is an overcorrection. However, since ‘years of education’ is not a multi-item test score, IRT analyses cannot examine the issue. One future solution may be to use multi-item tests of academic achievement as a measure of education amenable to traditional IRT approaches and, thus, potentially separable from various forms of IQ. Second, VIQ is just one of two components of general IQ scores. Tests of the other component, Performance IQ, have been suggested by many,^52,53 but not all,^11,12 to avoid mixing SES with cognitive ability measurement. In this regard, many in the cognitive epidemiology community have begun to focus on measures of Performance IQ, including tests of reaction time or processing speed, as the key cognitive abilities predictive of mortality.⁸

Third, we only examine two common tests of VIQ. Although we did not study other tests, these two tests correlate highly with other VIQ tests (ie, 0.7 to 0.8),^10,54 and with general IQ scores.^10,21 Thus, we suspect that other VIQ tests, and general IQ scores, may be susceptible to some extent to this phenomenon. However, these results may or may not generalise to noncognitive psychological tests, such as personality measures, which may also be vulnerable to DIF and deserve study in their own right. It is also important to remember that SES is multidimensional, that some dimensions of SES might be more vulnerable to DIF than others, that different dimensions of SES may have differential associations with mortality, and these associations may vary at different points in the lifespan.

Although our analysis addresses these concerns for three common indicators of SES measured once, the use of other indicators would be helpful, such as the quality of education received or family social position. Longitudinal studies could examine the extent to which cognitive abilities at various points in the lifespan mediate prior SES-related health risks. Performance IQ, and/or tests based on theories of multiple intelligences,⁵⁵ may contribute better to our understanding of the inter-relationships between class, intelligence and health.

Ultimately, most IQ batteries used in epidemiologic study include vocabulary and/or similarities in VIQ tests. Thus, the behaviour of these tests will be transmitted to general or composite IQ scores, upon which many conclusions are based. If the other tests in the battery do not evidence social-factor DIF, overestimation of IQ-SES associations will be more attenuated. However, the number of other tests in the battery exhibiting similar DIF will dictate the degree of overestimation, and this is an unknown. Our findings thus constitute a ‘proof of principle’ suggesting care in interpreting data on IQ and social gradients in mortality.

What is already known on this subject.

• ▸General IQ scores are thought to partially explain social gradients in mortality.
• ▸General IQ scores are composed of Performance IQ, and Verbal IQ (VIQ) tests.
• ▸VIQ tests are suspected to confound true cognitive ability with socioeconomic status (SES).
• ▸Measurement error may lead to overestimates of the extent to IQ explains social inequalities in mortality.

What this study adds.

• ▸Two common types of VIQ tests exhibit differential item functioning favouring persons of higher SES in a nationally representative US cohort.
• ▸High correlations between VIQ scores and SES are inflated due to differential item functioning.
• ▸Correction for differential item functioning reduces the explanatory role of IQ in social inequalities in mortality.