Comparing the performance of the STOFHLA and NVS among and between English and Spanish speakers

Abstract

Given the growing body of evidence demonstrating the significant implications of health literacy on a myriad of outcomes, researchers continue to incorporate health literacy metrics in studies. With this proliferation in measurement of health literacy in research, it has become increasingly important to understand how various health literacy tools perform in specific populations. Our objective was to compare the performance of two widely used tests, the Short Test of Functional Health Literacy in Adults (STOFHLA) and the Newest Vital Sign (NVS) among and between a sample of English and Spanish-speaking patients. Adults (N=402) age 50–75 participating in a trial to promote colorectal cancer screening completed in-person interviews which included both measures of health literacy. In the full sample, the tests were moderately correlated (r=0.69, p<0.0001); however, there was a stronger correlation among those completing the test in Spanish (r=0.83) as compared to English (r=0.58, p<0.0001). English speakers more often were categorized as having adequate literacy by the STOFHLA as compared to the NVS, whereas Spanish speakers scored consistently low on both instruments. These findings indicate that the categorization of participants into levels of literacy is likely to vary, depending on whether the NVS or STOFHLA is used for assessment, a factor which researchers should be aware of when selecting literacy assessments.

Introduction

Over the past three decades, health literacy research has yielded a significant body of evidence demonstrating that individuals with lower literacy and numeracy skills (‘limited health literacy’) exhibit a variety of poor health outcomes. Such outcomes range from less health knowledge to worse clinical outcomes, higher rates of hospitalization and greater mortality risk (DeWalt, Berkman, Sheridan, Lohr, & Pignone, 2004; Institute of Medicine [IOM], 2004; Berkman, Sheridan, Donahue, Halpern, & Crotty, 2011). Even as the measurement of health literacy in the clinical setting remains controversial (Paasche-Orlow & Wolf, 2008), measurement of health literacy skills in health services and outcomes research often has been identified as essential. Such measurement has identified disparities that warrant follow-up and remediation (Clement, Ibrahim, Crichton, Wolf, & Rowlands, 2009; Pignone, DeWalt, Sheridan, Berkman, & Lohr, 2005; Hasnain-Wynia & Wolf, 2010). Consequently, health literacy is frequently measured in research studies (Cameron, Wolf & Baker, 2011); numerous tools have been proposed and validated to measure this construct.

Increasingly, research studies are calling for the inclusion of diverse populations, including non-English speaking participants, necessitating the use of health literacy tools that are available in multiple languages. The Newest Vital Sign (NVS) was developed and promoted early on as a possible research and clinical measure of health literacy (Weiss et al., 2005). Its potential benefits include that it is brief (less than 2 minutes to administer) and is available in English and Spanish (Weiss et al., 2005). Early validation studies suggested the NVS discriminates patients as well if not even better than the most popular assessment – the short version of the Test of Functional Health Literacy in Adults (STOFHLA). The STOFHLA is also available and validated in English and Spanish, but takes three times longer to administer (Baker, Williams, Parker, Gazmararian, & Nurss, 1999; Osborn et al., 2007; Wolf et al., 2012).

With the continuing interest in and measurement of health literacy in research, it becomes increasingly important to understand how the tools to measure this construct perform. Each tool has limitations, either due to variable psychometric testing, use of cutoff thresholds, or predictive validity post-development in certain studies (Wolf et al., 2012; McCormack et al., 2010). Although both the NVS and STOFHLA have been validated in English and Spanish (Weiss et al., 2005; Baker et al., 1999), it is less known if individuals are categorized consistently by these measurements into the same levels of literacy, regardless of language. This knowledge could be significant and informative for research purposes, given that health literacy at its foundation overlaps with language and cultural factors. Recognizing the increasing diversity in trials that include a measurement of health literacy, the objective of this study is to compare the performance of the widely used STOFHLA and the alternative NVS tool both among and between English and Spanish-speaking patients.

Methods

Study sample

Participants were English- or Spanish-speaking patients between the ages of 50–75 who were seeking care at one of eight federally qualified health centers (FQHCs) or one academic medical center, and who consented to participate in a trial designed to promote colorectal cancer (CRC) screening. These health centers serve a predominately urban, low-income, and African American and Hispanic/Latino population. Additional inclusion criteria were that patients were registered for a non-acute physician appointment, were not up to date with CRC screening, and had been seen at the clinic at least twice (including visit on day of enrollment) in the previous two years. Per requirements of the larger trial, patients were excluded if they were non-English or Spanish speakers; were unable to view study materials (e.g., blind, deaf, too ill); were up to date with CRC screening; had personal or family history of CRC, polyps or inflammatory bowel disease; if physician deemed patient ineligible for screening (e.g., patient had active cancer); or saw a non-study physician. Exclusion based on up-to-date CRC screening and personal or family history was based both on patient self-report and confirmed by chart review.

Data and protocol

This study was conducted from September 2010 to January 2013. Same-day, in-person interviews were conducted by trained research assistants with all consented patients prior to and immediately after their scheduled medical appointment. Patients chose whether they preferred to complete the interviews in English or Spanish. The study interview included questions on socio-demographic information, items assessing knowledge, attitudes and self-efficacy regarding CRC and CRC screening as well as two validated measures of health literacy: the STOFHLA and the NVS. Institutional Review Boards at the Northwestern University Feinberg School of Medicine and the University of Illinois Hospital & Health Sciences System approved this study.

Demographics

The primary socio-demographic measures for this study were self-reported patient age, sex, ethnicity, years of education, health insurance status, and household income.

Health literacy assessments

STOFHLA

The STOFHLA was administered to patients prior to their medical appointment. The reading comprehension portion of the test includes 36 items to measure a patient’s ability to read and understand common medical scenarios via use of the Cloze procedure, where reading passages are missing every fifth to seventh word and participants must choose the correct replacement from a multiple choice list. Four items comprise the numeracy portion of the test to measure one’s ability to read and understand numbers. Scores on the STOFHLA range from 0–100 and are divided into three categories of health literacy: inadequate (0–53), marginal (54–66), and adequate (67–100) (Baker et al., 1999).

NVS

The NVS was administered to patients immediately after their medical appointment. Respondents reviewed a full page “Nutrition Facts” label from a pint of ice cream in the language of their interview and were asked the instrument’s six label-related questions (Weiss et al., 2005). The research assistant recorded and scored patient responses on a separate score sheet, giving one point for each correct answer. Numeric scores on the NVS range from 0–6 and are divided into three categories of health literacy risk: suggests high likelihood of limited literacy (0–1), indicates the possibility of limited literacy (2–3), and almost always indicates adequate literacy (4–6).

Data analyses

All analyses were performed using SAS 9.2 (SAS Institute, Inc., Cary, NC). Participant characteristics among English- and Spanish-speakers were compared using chi-square or t-tests as appropriate. McNemar’s test was used to assess the agreement of health literacy categorization between the instruments in either English or Spanish. Spearman correlation coefficients were used for bivariate correlation analysis to examine the relationship between numeric scores on the health literacy instruments (STOFHLA and NVS). Weighted kappa statistics were used to find the strength of agreement between the three parallel scoring categories of the STOFHLA and the NVS. Specifically, the “adequate” category of the STOFHLA was compared to the “almost always indicates adequate literacy” of the NVS, “marginal” to “indicates the possibility of limited literacy,” and “inadequate” to “suggests high likelihood of limited literacy.” We use the terms inadequate, marginal, and adequate literacy to refer to the three literacy categories of both tests. An asymptotic Z test was used to compare kappa values. Multi-factor analysis of variance was used to assess the relationship between literacy and language, adjusting for education level.

Results

Sample characteristics

Among 402 patients, the mean age was 57.7 (SD = 6.1), the majority of whom were women (74.4%). Hispanic/Latinos accounted for 40.1% of the sample; over one-third (36.6%) of patients completed the literacy tests in Spanish. Although there were no differences in the age or sex of participants completing the assessments in English or Spanish, there were significant differences identified in education, insurance, and income. Those completing the assessments in Spanish were less educated and were less likely to have insurance than those completing the assessments in English. Over three-fourths of Spanish-speakers reported 0–6 years of education, while the majority of English-speakers reported 7–12 years of education. There were significantly more participants in the Spanish group with unspecified income (not sure, did not know, refused). Table 1 depicts study participant demographic characteristics, effect sizes, and corresponding test scores by language.

Table 1.

Demographic characteristics and test scores, by respondent language (N = 402)

	Total (N = 402)	Spanish (n=147)	English (n=255)	p-value	Effect Size
Demographics
Age, mean (± SD)	57.7 (6.1)	58.0 (6.5)	57.6 (5.8)	0.53	0.07*
Female, n (%)	299 (74.4)	105 (71.4)	194 (76.1)	0.30	0.051**
Hispanic, n (%)	161 (40.1)	147 (100.0)	14 (5.5)	<0.0001	0.924**
Education, n (%)
0–6 years	120 (29.9)	113 (76.9)	7 (2.7)	<0.0001	0.798**
7–12 years	185 (46.0)	25 (17.0)	160 (62.8)
13 or more years	94 (23.4)	6 (4.1)	88 (34.5)
Missing	3 (0.7)	3 (2.0)	0 (0.0)
Insured, n (%)	284 (70.6)	71 (48.3)	213 (83.5)	<0.0001	0.373**
Household Income, n (%)
<$20,000/year	262 (65.2)	86 (58.5)	176 (69.0)	<0.0001	0.305**
$20,001 – $40,000/year	59 (14.7)	19 (12.9)	40 (15.7)
$40,001 – $60,000/year	17 (4.2)	1 (0.7)	16 (6.3)
>$60,001/year	4 (1.0)	0 (0.0)	4 (1.6)
Unspecified	60 (14.9)	41 (27.9)	19 (7.4)
Test Scores, mean (SD)
STOFHLA	56.8 (35.3)	31.1 (34.0)	71.7 (26.5)	<0.0001	1.38*
NVS	1.5 (1.7)	1.0 (1.5)	1.7 (1.7)	<0.0001	0.44*

^*Mean difference divided by pooled standard deviation

^**Square root of chi square statistic divided by the number of cells in contingency table

STOFHLA and NVS

Categorization of health literacy by instrument and language

As shown in Tables 2 and 3, the majority (51.7%) of the overall sample was categorized as having adequate literacy by the STOFHLA, whereas the NVS categorized only 13.9% as having adequate literacy. When viewed by language, the difference between categorizations on the two literacy tests was most remarkable among English-speaking participants: 21.6% of them were categorized as having inadequate literacy by the STOFHLA, as compared to 54.1% by the NVS (p<0.0001). Scores for Spanish-speakers appeared to be more consistent between the two literacy tests, particularly at the lower ends of the spectrum: 73.5% were identified as having inadequate literacy by the STOFHLA, 70.1% by the NVS, p=0.07 (Tables 2 and 3).

Table 2.

STOFHLA health literacy categorization, by respondent language

		STOFHLA Literacy Classification
		Inadequate (0–16) n (%)	Marginal (17–22) n (%)	Adequate (23–36) n (%)
Language	English + Spanish	163 (40.6)	31 (7.7)	208 (51.7)
	English	55 (21.6)	21 (8.2)	179 (70.2)
	Spanish	108 (73.5)	10 (6.8)	29 (19.7)

Table 3.

NVS health literacy categorization, by respondent language

		NVS Literacy Classification
		Inadequate (0–1) n (%)	Marginal (2–3) n (%)	Adequate (4–6) n (%)
Language	English + Spanish	241 (60.0)	105 (26.1)	56 (13.9)
	English	138 (54.1)	74 (29.0)	43 (16.9)
	Spanish	103 (70.1)	31 (21.1)	13 (8.8)

Relationship of numeric scores by language

Overall, participant scores on the STOFHLA and the NVS were highly correlated (r = 0.69, p<0.0001). Although significant for both groups, there was a stronger correlation among those completing the tests in Spanish (r=0.83), compared English (r=0.58, p<0.0001). Overall, the numeric STOFHLA scores appear to have a bimodal distribution (Figure 1); however, when assessed by language, distinct distributions are apparent. Among English speakers, STOFHLA scores are negatively skewed, whereas among Spanish speakers the scores are positively skewed, reflecting significantly lower numeric scores for those completing the test in Spanish (Table 1, see also Figures 2a and c). The overall distribution of numeric NVS scores is positively skewed (Figure 1), with the majority of both English and Spanish speakers scoring less than two (see Table 1; see also Figures 2b and d). Thus, when assessed using the STOFHLA, English speakers received relatively high numeric scores, yet their NVS scores were much lower. Spanish speakers, however, scored consistently low on both literacy assessments.

Figure 1.

Overall distribution of STOFHLA, NVS scores (N=402)

Figure 2.

Distribution of STOFHLA and NVS scores, by language

Relationship of health literacy categories by language

An agreement analysis revealed moderate agreement between the three STOFHLA literacy categories and the three comparable NVS categories (weighted kappa = 0.34, 95% CI 0.29 – 0.40). Among English speakers, agreement was lower (weighted kappa = 0.22, 95% CI 0.16 – 0.28) than for those completing the tests in Spanish [(kappa = 0.55, 95% CI 0.44 – 0.67), p<0.0001].

Relationship of Literacy Scores and Language

We observed a significant effect of participant language on both the STOFHLA (F _(1,400) = 252.8, p <0.0001) and the NVS (F _{(1, 400)} = 18.1, p <0.0001). As demographic characteristics also revealed a significant difference in education among English- and Spanish-speakers, we sought to assess the independent effect of language on numeric literacy scores, while controlling for education. Although we observed a significant effect of education on numeric literacy scores (both STOFHLA and NVS) in these analyses, the significant effect for language remained even when controlling for education [F _{(1, 395)} = 5.9, p <0.019 for STOFHLA; F _{(1, 395)} = 17.9, p <0.0001 for NVS].

Discussion

We found that depending upon the tool used in its measurement, study participants were identified as having varying health literacy levels. Our findings suggest that despite moderate correlations between the STOFHLA and the NVS, when viewed independently by language, health literacy classification among English-speakers is more dissimilar between the tests than among Spanish-speakers. Conversely for Spanish-speakers, health literacy classification was consistent, even when adjusting for education. These findings indicate that the categorization of participants into levels of literacy is likely to vary, depending on whether the NVS or STOFHLA is used. For example, if only the NVS was used to measure health literacy in our sample, our outcomes would suggest that there were no differences in health literacy between the English and Spanish-speaking participants. However, based on participants’ scores on the STOFHLA, results would indicate significant differences in health literacy between the English- and Spanish-speakers.

The significant inconsistencies observed between the STOFHLA and NVS identification of individuals with limited health literacy among English-speakers in this study echo concerns previously raised (Kirk et al., 2012; Patel et al., 2011). Such misidentification, or inconsistent identification, may place these individuals at risk for poor health outcomes. These findings highlight the implications for research studies using measures of health literacy, where depending on the health literacy measurement used, individuals may be classified differently across the health literacy spectrum.

Previous studies report wide ranges of values for correlations between various versions of the TOFHLA and the NVS, ranging from 0.49 to 0.62; most of these studies focused on English-speakers only (Osborn et al., 2007; Barber et al., 2009; Kirk et al., 2012; Patel et al., 2011; Wolf et al., 2012; Miser, Jeppesen, & Wallace, 2013). Weiss et al. (2005) identified a stronger correlation among the full version of the TOFHLA and the NVS for those completing tests in English (r=0.59) than in Spanish (r=0.49). Our results, which diverge from previous research, indicate a stronger correlation among the STOFHLA and NVS for those completing them in Spanish than in English. To our knowledge, such a high correlation between the numeric scores on the STOFHLA and NVS among Spanish-speakers has not been observed previously. Although we did observe an independent effect of language on health literacy, regardless of test, there was also a significant effect of education on health literacy, which has been seen in numerous other studies (Kirk et al., 2012; Patel et al., 2011; Barber et al., 2009; Aguirre, Ebrahim, & Shea, 2005).

While the STOFHLA and the NVS are moderately to strongly correlated based on our research and previous studies, they likely measure complimentary but somewhat divergent aspects of health literacy. The STOFHLA adapted and contextualized a common approach used to assess reading fluency (the Cloze procedure), whereas the NVS melds both reading comprehension and numeracy skills, which are both required to successfully understand an everyday healthcare task (a nutrition label).

Given that Spanish and English-speakers performed differently across the two tests, even when adjusting for education, there may be something else driving these differences, perhaps a cultural component. For instance, Spanish-speakers may be less familiar with reading the nutrition label used in the NVS (format). Furthermore, the STOFHLA is a very U.S.-centric measure with references to Medicaid, a program with which Spanish-speakers may be unfamiliar (content). These test attributes may be contributing to poor performance among Spanish-speakers across both assessments, leading to the observed lack of variance across assessments. Conversely, the difference seen in the English-speakers may be explained by the fact that the STOFHLA is measuring reading comprehension whereas the NVS measures numeracy. Indeed, the NVS is widely considered the more difficult assessment (Morrison, Schapira, Hoffmann, Brousseau, 2014; Dunn-Navarra et al., 2012; Kirk et al., 2012; Barber et al., 2009; Osborn et al., 2007).

There are several limitations to our research. Our results may not be generalizable to a population with greater variation in age or income. Our age range was restricted due to the requirements of the larger trial from which these data arose. In addition, the majority of our sample included patients seeking care at FQHCs, which serve the uninsured and underserved; our results may not generalize to populations with higher income levels. Another limitation is that the order of the administration of the STOFHLA and the NVS was not varied given the structure of the study interview. Participants always completed the STOFHLA first, at the end of the pre-test, which occurred immediately prior to the physician visit. It is possible that participants experienced some fatigue prior to completing the NVS, as it was administered at the end of the post-test which occurred after their physician visit. However, our results are consistent with previous studies also reporting lower NVS scores as compared to TOFHLA and STOFHLA scores (Dunn-Navarra, Stockwell, Meyer, & Larson, 2012; Sahm, Wolf, Curtis, & McCarthy, 2012; Miser et al., 2013).

Our results demonstrate that researchers should be aware of probable differences based on participant language when choosing a health literacy assessment in the design of their study. Health literacy measures may perform differently across languages; we must understand what implications such differences may have for any given research study. If we are expecting that these measures, which have been validated in English and Spanish, will perform the same when administered to individuals in both languages, we may well be mistaken. Further, we endorse choosing a health literacy tool to reflect the specific tasks and skills required of an individual in a particular health setting. For instance, if one is testing an intervention related to CRC screening, where a patient may have a colonoscopy, the reading-comprehension focused STOFHLA may be the best test to assess the literacy level as preparation for the colonoscopy involves following a series of written instructions. By contrast, a participant enrolled in a diabetes-related study whose self-management involves extensive numeracy skills to interpret glucose readings, administer medications, and follow dietary guidelines may benefit the most from the NVS or other diabetes-specific numeracy tests. These tests may better assess her ability to adequately carry out numeracy skills used in diabetes management (Huizinga, et al., 2008). Consequently, we recommend that specific health literacy tools be chosen not solely based on individual time constraints or language, but rather on which construct (numeracy, reading comprehension or problem-solving) a patient needs to utilize most within a specific setting to achieve positive health outcomes.