Health Literacy, Education, and Internal Consistency of Psychological Scales

Abstract

Background:

Measurement error might lead to biased estimates, causing ineffective interventions and service delivery. Identifying measurement error of health-related instruments helps develop accurate assessment of health-related constructs.

Objective:

We compared the internal consistency of eight psychological scales used in health research in groups with adequate versus limited health literacy and in groups with higher versus lower education.

Methods:

Participants (N = 1,005) from a nationally representative internet panel completed eight self-report scales: (1) information avoidance, (2) cognitive causation, (3) unpredictability, (4) perceived severity, (5) time orientation, (6) internal health locus of control, (7) need for cognition, and (8) social desirability. The first four assess beliefs about diabetes and colon cancer. We used the Newest Vital Sign to categorize participants' health literacy (limited vs. adequate). We also categorized participants' education (high school or less vs. more than high school). We compared the Cronbach's alpha for each psychological scale between groups with different health literacy and education levels using the Feldt test.

Key Results:

Among all the 13 subscales, scale internal consistency was significantly lower among people with limited health literacy than those with adequate health literacy for five subscales: information avoidance for colon cancer (0.80 vs. 0.88), unpredictability of diabetes (0.84 vs. 0.88), perceived severity for diabetes (0.66 vs. 0.75), need for cognition (0.63 vs. 0.82), and social desirability (0.52 vs. 0.68). Internal consistency was significantly lower among people who had a high school education or less than among those with more than a high school education for four scales: perceived severity of diabetes (0.70 vs. 0.75), present orientation (0.60 vs. 0.66), need for cognition (0.73 vs. 0.80), and social desirability (0.61 vs. 0.70).

Conclusions:

Several psychological instruments demonstrated significantly lower internal consistency when used in a sample with limited health literacy or education. To advance health disparities research, we need to develop new scales with alternative conceptualizations of the constructs to produce a measure that is reliable among multiple populations. [HLRP: Health Literacy Research and Practice. 2021;5(3):e244–e255.]

Plain Language Summary:

We compared the internal consistency of several psychological scales in groups with adequate versus limited health literacy and higher versus lower education. For several scales, internal consistency was significantly lower among (1) people with limited health literacy compared those who have adequate health literacy and/or (2) people who had a high school education or less compared to those with more than a high school education.

Psychological measurement instruments can be used to better understand psychological phenomena, explicate the psychological processes that make an intervention successful (or not), help researchers test hypotheses, identify target populations through conducting needs assessments, and screen people for clinical services (Sturm & Ash, 2005). However, to maximize their utility for advancing theory and improving interventions, psychological measurement instruments must demonstrate that they produce scores that are valid and reliable across samples. Importantly, reliability is a necessary (although not itself sufficient) criterion for validity (Crocker & Algina, 1986; Onwuegbuzie & Daniel, 2002).

Internal consistency is one way of assessing reliability (Henson & Thompson, 2002). A measure yielding scores with low internal consistency will add error variance to a statistical model and, therefore, could lead to incorrect conclusions that a construct and outcome are more weakly related to each other than is actually the case (Reinhardt, 1991). This may also lead to the erroneous inclusion or exclusion of constructs in intervention models or of intervention components that target the latent construct, thereby inhibiting intervention effects in behavioral intervention trials or leading to inaccurate behavioral theories.

Individual Differences in Internal Consistency in Psychological Measurement Instruments

Scale internal consistency is not stable across populations with varying characteristics (Henson et al., 2001). However, many psychological measures are still used with participants who are dramatically different (in terms of demographics, health literacy, and other critical factors) from the sample within which the scale was originally developed. These differences in the characteristics of subsequent populations may affect the internal consistency of test scores across groups.

Many instruments have been developed with samples of college students. Traditionally, these students have had higher socioeconomic status, literacy skills, and have been less demographically diverse than the general population (Hanel & Vione, 2016). Instruments developed in samples of college students may be interpreted differently or have less personal relevance when used in samples of non-college students, which might reduce the scale's internal consistency (Shepperd et al., 2016).

There is evidence that internal consistencies for psychological instruments differ across diverse groups of study participants who vary according to language, cultural background, education, and reading skills (Gjersing et al., 2010; Shepperd et al., 2016; Taras et al., 2009). For example, the internal consistency for three key psychological measures—the behavioral inhibition scale/behavioral activation scale (BIS/BAS), the regulatory focus questionnaire (RFQ), and the need for cognition scale (NCS)—differed between groups with high and low education (Shepperd et al., 2016). The BIS/BAS, RFQ, and NCS were all originally developed using college students (Carver & White, 1994; Cohen, 1957; Higgins et al., 2001). Shepperd et al. (2016) found that these three measures all have lower internal consistency for people with a high school education or less, and higher internal consistency for people with more than a high school education. They attribute lower internal consistency among people with lower education to their having lower literacy skills and being infrequent readers.

Limited internal consistency in the BIS/BAS, RFQ, and NCS suggests that other scales used in health behavior research may have similar limitations. For example, avoidance of health information (hereafter “information avoidance”), cognitive causation (i.e., the belief that thinking about a health problem will cause it to occur), beliefs that disease onset is unpredictable, perceived severity of disease, time orientation (i.e., the extent to which people think about the future or present), internal health locus of control (i.e., the extent to which people believe that their health is in their control), and social desirability (i.e., the tendency to answer questions in a way one thinks others would approve of) have all been used in health research. However, to our knowledge, with the exception of NCS, the potential for differential reliability of these scales as a function of health literacy or education has not been previously examined.

Health Literacy

Health literacy could also affect internal consistency. Health literacy represents the ability to “obtain, process, and understand basic health information and services needed to make appropriate health decisions” (Kindig et al., 2004). Reading, writing, and numeracy skills are essential components of health literacy (Parker et al., 1995). Besides education, other factors may influence people's health literacy, including living in poverty, race/ethnicity, age, and disability (Kutner et al., 2006). Thus, comparing groups with different health literacy levels contributes novel findings to the literature.

Survey items that ask about health are potentially affected by people's ability to process and understand health information. Therefore, it is likely that an instrument that has good internal consistency among people with adequate health literacy may have poor internal consistency among those with limited health literacy. Given the high levels of limited health literacy in the United States (U.S. Department of Health and Human Services, 2008), understanding these potential distinctions in the internal consistency of some psychological scales will be practically and methodologically useful for basic social and health psychology research and applied public health promotion research. For example, identifying the psychological scales that have poor internal consistency among people with limited health literacy would lead to scale revision and construct reconceptualization to reduce measurement error so that future research would generate more accurate and less biased results.

Although researchers have studied measurement invariance by age and gender of 15 psychological measures, including NCS (Hussey & Hughes, 2020), they have not examined invariance by education or health literacy. Researchers also determined that people with limited health literacy may respond differently to purportedly validated scales than those with adequate health literacy (Taple et al., 2019), but none of the eight scales identified in the previous section have been evaluated to determine whether there are differences in their internal consistency according to health literacy.

Objective and Hypotheses

In the present study, we examined whether the internal consistency of eight psychological scales used in health research differed by health literacy or education. We contribute to the literature in three key ways: (1) using a nationally representative sample (vs. a convenience sample in Shepperd et al., 2016), (2) comparing groups with different health literacy levels as well as education levels (vs. only education), and (3) examining the internal consistency of several scales used in health research that were not included in studies by Shepperd et al. (2016) or Hussey & Hughes (2020). We hypothesize that the scores of people with lower education and lower health literacy will be less internally consistent than the scores of people with higher education and higher health literacy.

Methods

The study was approved by the University at Buffalo Institutional Review Board. This article presents a secondary analysis of data collected for a larger study designed to examine possible psychological mechanisms underlying not knowing one's risk for common diseases (Orom et al., 2018).

Sample

GfK, a market research firm with an academic research arm, conducted recruitment and data collection from May to June 2016. GfK maintained a standing representative panel (KnowledgePanel) of 55,000 people. For this study, GfK sent email invitations to 1,818 KnowledgePanel members, and 1,033 (56.8%) of them completed the survey. Responses for 26 participants were dropped because they met two or more of the following criteria for inattentive responding: (1) completed the survey in less than 5.5 minutes (one-fourth of the median time of 22 minutes), (2) marked identical responses or straight-lined for at least one-half of the eight question grids, (3) failed both of the survey validation items (asking participants to select somewhat agree for one item and somewhat disagree for the other item), and (4) gave different answers to a repeated factual question about their health insurance status. Two additional participants were excluded due to having prevalent diabetes and colorectal cancer. Thus, we included a final sample of 1,005 participants in our data analyses.

Measure

We examined the internal consistency of eight psychological instruments that were included in the original study design (Orom et al., 2018; Waters et al., 2018). Several of the scales had subscales. See Table 1 for information about the Flesch-Kincaid Reading Level (FKRL) for each subscale, as well as the population(s) in which it was developed.

Table 1.

The Eight Psychological Scales Compared in this Study

Scale	Population in Which the Scale Was Developed	Flesch-Kincaid Reading Level
Information Avoidance Scale (Howell & Shepperd, 2016)	Undergraduates, high school students, U.S. adults, African Americans, and adult women
Diabetes		7.5
1. I feel like I have enough information to know my risk of getting diabetes.
2. I would rather not know about diabetes.
3. I would prefer to avoid learning about diabetes.
4. Even if it will upset me, I want to know about diabetes.
5. I want to know about diabetes.
6. I can think of situations in which I would rather not know about diabetes.
7. It is important to know about diabetes.
8. I want to know about diabetes immediately.
Colon cancer		7.2
1. I feel like I have enough information to know my risk of getting colon cancer.
2. I would rather not know about colon cancer.
3. I would prefer to avoid learning about colon cancer.
4. Even if it will upset me, I want to know about colon cancer.
5. I want to know about colon cancer.
6. I can think of situations in which I would rather not know about colon cancer.
7. It is important to know about colon cancer.
8. I want to know about colon cancer immediately.
Cognitive Causation Scale (Hay et al., 2014)	Undergraduate psychology students, community men, immigrants, African Americans, and Black Caribbeans
Diabetes		8.1
1. If I think too hard about the possibility of getting diabetes, I could get it.
2. If I don't believe I will get diabetes, I won't.
3. Negative thoughts about getting diabetes might make me get it.
4. Considering that I could get diabetes might bring on bad luck.
5. Too much thought about diabetes risk could encourage the disease.
6. Thinking that I am likely to get diabetes may give me diabetes.
7. In general, if a person thinks about the possibility of getting diabetes, they are more likely to get it.
Colon cancer		8.0
1. If I think too hard about the possibility of getting colon cancer, I could get it.
2. If I don't believe I will get colon cancer, I won't.
3. Negative thoughts about getting colon cancer might make me get it.
4. Considering that I could get colon cancer might bring on bad luck.
5. Too much thought about colon cancer risk could encourage the disease.
6. Thinking that I am likely to get colon cancer may give me colon cancer.
7. In general, if a person thinks about the possibility of getting colon cancer, they are more likely to get it.
Unpredictability of Cancer Scale (Hay et al., 2014)	Undergraduate psychology students, community men, immigrants, African Americans, and Black Caribbeans
Diabetes		6.5
1. Anybody can get diabetes, no matter what they do.
2. Diabetes can strike anyone at any time.
3. You never know who is going to get diabetes.
Colon cancer		6.3
1. Anybody can get colon cancer, no matter what they do.
2. Colon cancer can strike anyone at any time.
3. You never know who is going to get colon cancer.
Perceived Severity Scale (Moss-Morris et al., 2002)	People from seven illness groups (asthma, diabetes, rheumatoid arthritis, chronic pain, acute pain, myocardial infarction, and multiple sclerosis) in New Zealand and an HIV patient group in the United Kingdom
Diabetes		7.1
1. Diabetes is a serious condition.
2. If I had diabetes, it would have major consequences on my life.
3. If I had diabetes, it would not have much effect on my life.
4. If I had diabetes, it would have serious financial consequences.
5. If I had diabetes, it would cause difficulties for those who are close to me.
Colon cancer		7.0
1. Colon cancer is a serious condition.
2. If I had colon cancer, it would have major consequences on my life.
3. If I had colon cancer, it would not have much effect on my life.
4. If I had colon cancer, it would have serious financial consequences.
5. If I had colon cancer, it would cause difficulties for those who are close to me.
Time Orientation Scale (Lukwago et al., 2001)	African American women from urban housing communities with low income
Present orientation		6.1
1. My day-to-day is too busy to think about the future.
2. There's no sense in thinking about the future before it gets there.
3. What happens to me in the future is out of my control.
Future orientation		5.6
1. I have a plan for what I want to do in the next 5 years of my life.
2. The choices I have made in life clearly show that I think about the future.
3. I often think about how my actions today will affect my health when I am older.
Internal Health Locus of Control Scale (Wallston et al., 1978)	A sample of 90% White people and 74% having at least some college education	4.1
1. If I get sick, it is my own behavior which determines how soon I get well again.
2. I am in control of my health.
3. When I get sick, I am to blame.
4. The main thing which affects my health is what I myself do.
5. If I take care of myself, I can avoid illness.
6. If I take the right actions, I can stay healthy.
Need for Cognition Scale (Sherrard & Czaja, 1999)	Primarily undergraduate university and technical institute students	6.9
1. I like to have the responsibility of handling a situation that requires a lot of thinking.
2. Thinking is not my idea of fun.
3. I would rather do something that requires little thought than something that is sure to challenge my thinking abilities.
4. I only think as hard as I have to.
5. I really enjoy a task that involves coming up with new solutions to problems.
6. Learning new ways to think doesn't excite me very much.
7. I prefer my life to be filled with puzzles that I must solve.
Social Desirability Scale (Paulhus, 1991)	College students	6.2
1. There have been occasions when I have taken advantage of someone.
2. I always obey laws, even if I'm unlikely to get caught.
3. I never swear.
4. I sometimes drive faster than the speed limit.
5. I have done things that I don't tell other people about.
6. I never take things that don't belong to me.
7. I have taken sick-leave from work or school even though I wasn't really sick.
8. I have never damaged a library book or store merchandise without reporting it.

The Newest Vital Sign (NVS) is a measure of health literacy that contains six questions to test participants' understanding of information on a mock-up nutrition label for ice cream (Weiss et al., 2005). Participants received 1 point for each correct answer. A missing response was considered incorrect and received a score of 0. The NVS score was categorized as follows: high likelihood of limited health literacy (0–1 correct); possibility of limited health literacy (2–3 correct); almost always adequate health literacy (4–6 correct) (Weiss et al., 2005). In our sample, only 64 (6%) participants scored in the lowest category (0–1). Thus, we treated health literacy as a binary variable, dividing health literacy as limited (NVS 0–3) or adequate (NVS 4–6). This approach has been used in previous studies (Ghaddar et al., 2012; Griffey et al., 2014; Hudon et al., 2012; Protheroe et al., 2017). The NVS was selected because it could be used in the context of a relatively brief, self-administered online survey and yields reliable and valid scores among various populations across different age ranges, races/ethnicities, and health conditions (Shealy & Threatt, 2016; Weiss et al., 2005).

Demographic variables included sex, age, race/ethnicity (non-Hispanic White, non-Hispanic Black, Hispanic, and race/ethnicity unknown), and education (less than high school, high school, some college, Bachelor's degree or higher).

Data Analysis

For each measure, we computed internal consistency using Cronbach's alpha overall, by health literacy group, and by education level. To compare alphas across health literacy and education levels, we use the Feldt test (Feldt, 1969). The Feldt test can be used to compare alphas across two groups, where the statistic W, is the ratio of (1-alpha2) to (1-alpha1) and follows an F distribution with degrees of freedom v1=N1–1 and v2=N2–1. The sample of 1,000 participants exceeds the minimum requirements for the Feldt test (Feldt & Kim, 2006). We collapsed the education variable into two levels (high school and below vs. above high school) because the Feldt test cannot be used to compare alphas across more than two groups. See Table A for alphas across the four education levels.

Table A.

Cronbach's Alpha Across Four Education Levels

Instrument	Education Level
	Less than High School	High School	Some College	Bachelor's Degree or Higher
Information avoidance diabetes	0.88	0.85	0.78	0.85
Information avoidance colon cancer	0.84	0.87	0.83	0.87
Cognitive causation diabetes	0.96	0.97	0.96	0.96
Cognitive causation colon cancer	0.97	0.96	0.96	0.97
Unpredictability diabetes	0.87	0.88	0.87	0.85
Unpredictability colon cancer	0.90	0.89	0.89	0.84
Perceived severity diabetes	0.73	0.69	0.72	0.77
Perceived severity colon cancer	0.74	0.70	0.74	0.74
Present orientation	0.62	0.59	0.68	0.61
Future orientation	0.59	0.63	0.62	0.62
Internal health locus of control	0.84	0.78	0.77	0.77
Need for cognition	0.76	0.72	0.77	0.81
Social desirability	0.65	0.61	0.69	0.66

Results

About 20% of our sample had limited health literacy (n = 206) and about 37% (n = 372) of the sample had completed high school or less. Just 14% (n = 136) had both limited health literacy and low formal education, and 56% (n = 563) had adequate health literacy and high formal education. However, among people with low education, 37% had limited health literacy; among people with high education, only 11% had limited health literacy. Our dichotomous variables of health literacy and education were correlated (phi = 0.31, p < .001). See Table 2 for detailed information about sample characteristics.

Table 2.

Sample Characteristics (N = 1,005)

Characteristic	n	%
Sex
Male	484	48
Female	521	52
Age, years
18–24	86	9
25–34	153	15
35–44	135	13
45–54	175	17
55–64	218	22
65+	238	24
Race/ethnicity
White, not Hispanic	741	74
Black, not Hispanic	98	10
Hispanic	97	9
Race/ethnicity unkown	69	7
Education
Less than high school	77	8
High school	295	29
Some college	283	28
Bachelor's degree or higher	350	35
Health literacy
Limited	206	20
Adequate	799	80

Health Literacy

As shown in Table 3, compared to people with adequate health literacy, those with limited health literacy received scores with statistically significant lower internal consistency on 5 of the 13 subscales: information avoidance for colon cancer (0.80 vs. 0.88, p < .001), unpredictability for diabetes (0.84 vs. 0.88, p = .01), perceived severity for diabetes (0.66 vs. 0.75, p = .003), need for cognition (0.63 vs. 0.82, p < .001), and social desirability (0.52 vs. 0.68, p < .001). In contrast, compared to people with adequate health literacy, those with limited health literacy had higher internal consistency for present orientation (0.69 vs. 0.61, p = .01) and locus of control (0.82 vs. 0.77, p = .03). There were no significant differences in internal consistency for any of the other scales.

Table 3.

Cronbach's Alpha for the Scales

Instrument (Number of Items)	Entire Sample	Limited HL	Adequate HL	p	High School Education or Less	High School Education or More	p
Information avoidance diabetes (8)	0.84	0.85	0.85	.44	0.86	0.83	.04
Information avoidance colon cancer (8)	0.86	0.83	0.88	.002	0.86	0.86	.38
Cognitive causation diabetes (5)	0.97	0.96	0.96	.12	0.97	0.96	.06
Cognitive causation colon cancer (5)	0.97	0.97	0.96	.32	0.96	0.97	.06
Unpredictability diabetes (3)	0.87	0.82	0.88	< .001	0.88	0.86	.83
Unpredictability colon cancer (3)	0.87	0.87	0.87	.48	0.90	0.86	.001
Perceived severity diabetes (5)	0.73	0.69	0.75	.04	0.70	0.75	.03
Perceived severity colon cancer (5)	0.73	0.74	0.70	.12	0.71	0.74	.09
Present orientation (3)	0.64	0.70	0.61	.01	0.60	0.66	.04
Future orientation (3)	0.68	0.64	0.69	.10	0.65	0.69	.09
Internal health locus of control (6)	0.78	0.81	0.77	.07	0.80	0.77	.08
Need for cognition (7)	0.80	0.66	0.82	< .001	0.73	0.80	< .001
Social sesirability (8)	0.65	0.53	0.68	.001	0.61	0.70	.04

Note. p values are based on the Feldt test comparing alphas across the two levels of HL or education. HL = health literacy.

Education

Compared to people with education above high school, those who only completed high school or less yielded significantly lower internal consistency on 4 of the 13 subscales: perceived severity for diabetes (0.70 vs. 0.75, p = .03), present orientation (0.60 vs. 0.66, p = .04), need for cognition (0.73 vs. 0.80, p < .001), and social desirability (0.61 vs. 0.70, p = .04). In contrast, internal consistency was greater for those who completed high school or less than for those with more than a high school education on 2 of the 13 subscales scales: information avoidance for diabetes (0.86 vs. 0.83, p = .04) and unpredictability for colon cancer (0.90 vs. 0.86, p = .001). There were no other significant differences in internal consistency for any of the other scales.

Discussion

Our hypothesis received mixed support across three key findings. First, consistent with our hypothesis, internal consistency for three subscales was significantly lower among people with lower education and/or limited health literacy compared to people with higher education and adequate health literacy. Specifically, the need for cognition and perceived severity of diabetes subscales had poor internal consistency among people with limited health literacy. The internal consistency for the social desirability and the two time-orientation subscales were relatively low across all groups compared to other scales.

Second, despite a small number of statistically significant differences between groups, internal consistency remained relatively high regardless of health literacy and educational attainment for eight of the subscales: information avoidance (diabetes and colon cancer), cognitive causation (diabetes and colon cancer), unpredictability (diabetes and colon cancer), perceived severity (colon cancer), and internal health locus of control. Shepperd et al. (2016) also found significant differences in the internal consistency of the need for cognition scale between people with more and less formal education. Our findings for the other scales included in this study are novel.

Possible Explanations for the Results

High readability levels might be one explanation for our results. However, exploratory examination of the FKRL of the social desirability, need for cognition, and perceived severity of diabetes scales all indicated that someone who could read at the sixth-grade level should be able to understand the items (Stockmeyer, 2009). The FKRL also indicated that the time orientation subscales could be read by someone with a sixth-grade reading level, and its internal consistency was nearly universally unacceptable. Therefore, we conclude that readability at the sixth-grade level did not contribute to the low internal consistency of those instruments.

Another explanation could be that the construct validity of the instruments is stronger for groups that were involved in their development. That is, items measuring social desirability, need for cognition, and perceived severity of diabetes might hold different meaning and/or relevance for people with high versus low education and with adequate versus limited health literacy. For example, the version of the “need for cognition” scale that we administered was developed using college students majoring in arts, business, or social sciences (Sherrard & Czaja, 1999). The social desirability instrument was also developed using college students (Paulhus, 1991). College students are different from the general population across a wide variety of characteristics and experiences (Hanel & Vione, 2016), which could lead them to think about the items differently from people with no college experience. This may help explain why the need for cognition scale had significantly higher internal consistency among people with high education and adequate health literacy than among those with low education and limited health literacy. Although the internal consistency for the time-orientation scales was unacceptable overall and by health literacy and education, they were acceptable among African Americans (alpha = 0.77 and 0.74 for future and present orientation, respectively). The scale was developed in African American women from urban housing communities with low-income (Lukwago et al., 2001).

The information avoidance, cognitive causation, and unpredictability scales were developed and validated with people from a variety of backgrounds, including undergraduate college students, people with low socioeconomic status (Hay et al., 2014; Howell & Shepperd, 2016), people residing in rural areas (Howell & Shepperd, 2016), and patients from an urban primary care clinic (Hay et al., 2014). Thus, it is less surprising that these scales showed good internal consistency across health literacy and education levels. The perceived severity scale was developed with people from seven illness groups in New Zealand and the United Kingdom (Moss-Morris et al., 2002). The internal health locus of control scale was developed among a sample composed of 90% White people and with 74% having at least some college education (Wallston et al., 1978). Future research should investigate whether unacceptable internal consistency of some scales among people with limited health literacy, education, or both is due to differing levels of meaningfulness and relevance of the items for certain populations. One way to address these problems would be to develop new scales with alternative conceptualizations of the constructs through conducting cognitive interviews, particularly among populations with low education and health literacy.

Although the cognitive causation scales for diabetes and colon cancer were skewed (Table B) and associated with health literacy (Table C), the alphas for these two scales did not significantly differ by health literacy level. Furthermore, they exhibited high internal consistency among those with adequate and limited health literacy (0.96-0.97). None of the other scales exhibited strong skewness. Thus, floor or ceiling effects are unlikely to have influenced our findings.

Table B.

Score Distributions

Instrument	Possible Range	Observed Range	Mean	Median	SD
Information avoidance diabetes	1–4	1–4	1.95	2	0.59
Information avoidance colon cancer	1–4	1–4	1.96	2	0.62
Cognitive causation diabetes	1–4	1–4	1.37	1	0.62
Cognitive causation colon cancer	1–4	1–4	1.40	1	0.64
Unpredictability diabetes	1–4	1–4	2.87	3	0.75
Unpredictability colon cancer	1–4	1–4	3.19	3	0.67
Perceived severity diabetes	1–4	1–4	2.97	3	0.58
Perceived severity colon cancer	1–4	1–4	3.32	3.4	0.56
Present orientation	3–12	3–12	5.95	6	1.74
Future orientation	4–16	4–16	10.99	11	2.23
Internal health locus of control	6–24	6–24	17.29	17	2.77
Need for cognition	4–28	7–28	19.93	20	3.60
Social desirability	0–8	0–8	3.11	3	1.85

Table C.

Correlation Between Newest Vital Sign and Scale Scores

Instrument	r	p
Information avoidance diabetes	−0.13	< .001
Information avoidance colon cancer	−0.13	< .001
Cognitive causation diabetes	−0.30	< .001
Cognitive causation colon cancer	−0.28	< .001
Unpredictability diabetes	−0.11	< .001
Unpredictability colon cancer	0.06	.067
Perceived severity diabetes	0.13	< .001
Perceived severity colon cancer	0.29	< .001
Present orientation	−0.25	< .001
Future orientation	0.03	.392
Internal health locus of control	0.02	.466
Need for cognition	0.24	< .001
Social desirability	−0.03	.316

Note. Newest Vital Sign ranges from 0 to 6.

Study Limitations

First, although we can say that several instruments had unacceptable internal consistency, either in specific demographic groups or in the overall sample, we cannot draw conclusions about the instruments' overall validity or reliability. Second, several instruments have multiple versions with different numbers of items, and we only tested one version. If an instrument has more items, it tends to have higher internal consistency (Cortina, 1993). Therefore, it remains unknown whether the low internal consistency problem is unique for that version or generalizes to other versions. Third, we used the NVS to measure health literacy; however, there are many other health literacy instruments such as the Test of Functional Health Literacy in Adults (Parkert et al., 1995), the Rapid Estimate of Adult Literacy in Medicine (Haun et al., 2014), the 14-item Health Literacy Scale (Suka et al., 2013), and the European Health Literacy Survey (Sørensen et al., 2015). The results might be slightly different if we assess health literacy using other measures because each measure may assess different skills related to health literacy (Haun et al., 2014). Last, we administered NVS by computer whereas the original NVS was developed as a one-on-one interviewer-administered instrument; however, recent studies indicated that NVS can be administered by computer (Mansfield et al., 2018; Weiss, 2018).

Conclusions

We compared the internal consistency of eight psychological instruments used in health research in participants with adequate and limited health literacy as well as higher and lower education. We conclude that the social desirability and the need for cognition instruments may not accurately measure their target constructs in groups with limited education or health literacy. These findings demonstrate the need for development of new scales in vulnerable populations.

Researchers should be mindful that scales with acceptable internal consistency in their sample as a whole may have unacceptable internal consistency in some sample subsets. Such undesirable variability in internal consistency could undermine instruments' ability to detect phenomena that do, in fact, exist in nature. Considering the importance of psychological instruments for research and practice in psychology and public health (Sturm & Ash, 2005), and ensuring that measures of psychological constructs have acceptable internal consistency among those with limited formal education and health literacy may increase the applicability of research and practice to those groups and thereby alleviate, or at least prevent the exacerbation of, health disparities (Ramírez et al., 2005).