Short, Subjective Measures of Numeracy and General Health Literacy in an Adult Emergency Department Setting

Abstract

Objectives

To evaluate the reliability and validity of brief subjective measures of numeracy and general health literacy in the adult emergency department setting.

Methods

A convenience sample of adult emergency department patients completed subjective measures of general health literacy (Short Literacy Screening questions, SLS) and numeracy (Subjective Numeracy Scale, SNS). These patients also completed two objective tests of literacy (the Short Test of Functional Health Literacy in Adults, S-TOFHLA; and the Rapid Estimate of Adult Literacy in Medicine, REALM) and an objective test of numeracy (WRAT4).

Internal reliability of the subjective measures was assessed using Cronbach’s alpha. Construct validity of the subjective measures was assessed by correlating them against the S-TOFHLA, REALM, and WRAT4, using Spearman’s rank correlation coefficients, receiver operating characteristics (ROC) curves, and hierarchical, multiple linear regression with adjustment for patient age, gender, race, and education.

Results

The median age of the 207 patients surveyed was 46 (interquartile range 32, 59); twenty-seven percent were African American. Sixty-one percent of patients reported their highest level of education was high school or below. As measured by the S-TOFHLA and REALM, most patients had adequate literacy levels (89% and 80%, respectively), while 44% of patients had below average numeracy skills on the WRAT4. Median SLS was 14 (IQR 12, 15) on a scale of 3 to 15; median SNS was 36 (IQR 30, 42) on a scale of 6 to 48.

The SLS and SNS had good internal reliability, with Cronbach’s alphas of 0.74 and 0.82, respectively. The SLS Spearman’s rank order correlation coefficient was 0.33 (95% confidence interval 0.20, 0.45) for the S-TOFHLA, with a standardized beta coefficient of 0.36 (p<0.05) after adjustment for patient demographics. The SLS correlation coefficient was 0.26 (95% CI 0.13, 0.38) for the REALM, with a standardized beta coefficient of 0.38 (p<0.05) after adjustment for patient demographics. The area under the curve (AUC) for the SLS ROC curves was 0.74 (95% CI 0.68, 0.80) when compared to the S-TOFHLA and 0.72 (95% CI 0.65, 0.78) when compared to the REALM. The SNS predicted numeracy well, with a correlation coefficient of 0.57 (95% CI 0.47,0.65) for the WRAT4, a standardized beta coefficient of 0.30(p<0.05) after adjustment for patient demographics, and an ROC AUC of 0.77 (95% CI of 0.70, 0.82).

Conclusions

The SNS and SLS are reliable, valid tests that can be used to rapidly estimate general health literacy and numeracy skill levels in adult emergency department patients. Continuing work is needed to establish their ability to predict clinical outcomes.

Introduction

Health literacy is the degree to which individuals can obtain, process, and understand basic health information and services needed to make appropriate health decisions.¹ In 2003, the National Assessment of Adult Literacy found that approximately 90 million Americans, or nearly 36% of the adult population, had basic or below basic literacy skills; approximately 110 million American adults (44%) had basic or below basic quantitative literacy, or numeracy, skills.² Multiple studies conducted in outpatient clinics have shown that patients with lower literacy and numeracy skills are more likely to have less understanding of health information, have poor health behaviors, require hospital admission, and experience worse clinical outcomes.^3–14 Patients with lower literacy are also less likely to seek preventative health services and are more likely to use the emergency department for their care.^15–17

Identifying patients with low health literacy and numeracy levels can be difficult. Medical providers often overestimate their patients’ literacy and numeracy skills.^18–20 Health literacy is not reliably predicted by educational level, age, or gender alone; numeracy skills in particular vary widely even in highly-educated populations.^21,22 Patients rarely volunteer information regarding their health literacy or numeracy skills; unfortunately, clinicians may approach informed consent and even therapeutic options differently based on assumptions of patient health literacy and disease knowledge.^23–25 While several tests that measure numeracy and general health literacy are available, many of these tests are relatively time-consuming and labor-intensive to administer, and they may be intimidating to some patients. Administration and interpretation of these tests can be difficult to integrate into the clinical setting, and the best use of their results has yet to be determined.

There is need for reliable, valid measures of numeracy and health literacy that can be easily administered and used in the emergency department setting.²⁶ Literacy and numeracy skills can have a significant impact on a patient’s ability to manage his or her chronic diseases and are likely to affect acute disease management. The barriers to measuring numeracy and literacy that exist in outpatient clinic settings are compounded in the emergency department by lack of long-term relationships and an even more rapid pace of evaluation.

As an alternative to lengthy, objective tests, two brief, subjective measures of literacy and numeracy have been developed: the 3-item Short Literacy Survey (SLS)^27–29 and the 8-item Subjective Numeracy Scale (SNS).^30,31 Patients prefer subjective tests (which focus on self-reported confidence or preferences) and indicate they are more likely to take these tests, compared with objective tests that require mathematical calculations or test content knowledge.³⁰ However, subjective measures of numeracy and literacy have not been validated or used extensively in the emergency department setting. Their ability to predict clinical outcomes such as emergency department recidivism, hospital readmission, and death have also not been evaluated, though there may be an association between the SLS and clinical outcomes in heart failure patients.³²

The aim of this study was to evaluate whether subjective measures of numeracy and general health literacy, administered in the emergency department setting, would correlate with previously validated measures of literacy and numeracy, and whether these subjective measures of literacy and numeracy explain significant, unique variance in objective measures of literacy and numeracy above and beyond patient demographics.

Methods

Study Methods

This study was conducted in an urban level 1 adult emergency department with an annual census of approximately 55,000. Between June 2009 and April 2010, patients were enrolled in the study between 8 am and 9 pm as a convenience sample. Those who were critically ill or did not speak English were excluded. Patients who consented to participate were given five research-assistant administered tests of numeracy and general health literacy in the following order: the SLS and SNS, followed by the literacy subscale of the S-TOFHLA, the short REALM, and the WRAT4 mathematical subtest. Age, gender, and self-reported insurance status and education level were collected. All data was managed using REDCap electronic data capture tools hosted at our institution.³³ This study was approved by the local institutional review board.

Objective Measures of General Health Literacy and Numeracy

The S-TOFHLA is one of the most widely used tests to measure general health literacy.²⁶ Designed for use in outpatient clinics, it consists of a general health literacy subscale and a health numeracy subscale.^34–36 We administered only the literacy portion, which consists of two prose passages and has a time limit of seven minutes. This test uses the Cloze procedure to measure literacy, in which the patient completes phrases missing every fifth to seventh word. The S-TOFHLA yields a continuous score from 0 to 36, which in many studies is categorized into “inadequate literacy” (0–16), “marginal literacy” (17–22), and “adequate literacy” (23–36).⁶ “Adequate literacy” approximates a 9^th grade reading level.

Another widely used tool for measuring health literacy is the REALM, in which the patient is asked to read 66 words out loud. The total of correctly pronounced words is summed to generate a score between 0 and 66. For ease of comparisons across tests, prior researchers have grouped REALM scores to approximate S-TOFHLA score categories: “inadequate literacy” (REALM score 0–44); “marginal literacy” (REALM score 45–60); and “adequate literacy” (REALM score 61–66).²⁸

The wide range achievement test (WRAT4) mathematical subtest consists of 40 computational questions of increasing difficulty that are to be completed within a 15-minute time limit. WRAT4 scores are normalized for age, with the population mean set at a score of 100, and a standard deviation of 15 points.³⁷

Subjective Tests of Health Literacy and Numeracy

The SLS consists of three questions, each with a five-point Likert response scale, that are read aloud to patients and which focus on self-reported literacy levels and needs.²⁸ For the purposes of this study, the three items were summed into a total score after reverse-coding the “confident with forms” item. The higher the summed score, the greater the patients’ subjective assessment of his or her general health literacy skills. Recent research indicates that low literacy, as measured by the SLS, may be associated with worse clinical outcomes in heart failure patients.³²

The SNS consists of eight written questions, each with a six-point Likert response scale. Four questions (“fractions,” “percentages,” “tip,” “shirt”) address perceived arithmetic abilities; four questions (“newspaper,” “words,” “weather,” and “numerical useful”) address preferences regarding the use of numbers.^30,31 As with the SLS, the scores were summed after reverse coding the “weather” item; a larger score indicates a higher subjective rating of numeracy abilities and preferences.

Analysis

Internal consistency of the subjective numeracy and literacy scales was evaluated using Cronbach’s alpha. Concurrent validity of the SLS and SNS was evaluated using Spearman’s rank order correlation with the REALM, S-TOFHLA, and the WRAT4 mathematical subtest. Using the S-TOFHLA, REALM, and WRAT4 as standards for comparison, receiver operating characteristics (ROC) curves were calculated for each of the subjective tests. Similar to prior analyses,²⁸ “adequate” literacy levels, as defined by the S-TOFHLA and REALM, and “average” or “above average,” as defined by the WRAT4 mathematical subtest, were used as the comparison standards. Hierarchical, multiple linear regression was used to evaluate the contribution of the SLS and SNS to the prediction of the objective measures of numeracy and literacy, after adjustment for age, gender, race, and education level. Education was treated as an ordinal variable, with five categories (elementary school or grades one through five; middle school, or grades six through eight; high school, or grades nine through twelve; any college; and postgraduate education).

Unlike more traditional experimental study designs, validation study sample size estimates are based on psychometric statistics and expert recommendations that five to 10 subjects be included for each survey item and covariate, up to approximately 300 individuals.³⁸ Based on the 11 total items in the SLS and SNS, as well as the four covariates of age, gender, race, and education, a minimum of 150 subjects was required.

Analyses were conducted using STATA/IC 11.1 (Stata Statistical Software: Release 11.1, College Station, TX, Stata Corporation, 2010).

Results

A total of 209 patients consented to participate. Two patients were unable to complete all the tests after enrolling; one became too ill, and the other became confused. These patients were excluded from the analysis, leaving a sample size of 207.

Baseline demographic data of the patients included in the analysis are presented in Table 1. Approximately 45% of participants were female, and 27% were African American; eight percent had completed less than high school, and 38% had completed some college. Approximately 36% of patients were on state or federal insurance, 54% had private insurance, and 10% had no insurance.

Table 1.

Patient Characteristics

Demographic (n=207)		Percentages* (Frequency)
Age, years	Median 46 (IQR 32,59)
Gender	Male	55 (113/207)
	Female	45 (94/207)
Race	African American	27 (55/207)
	Caucasian	67 (140/207)
	Other	12 (12/207)
Insurance Status	Private/Group	54 (111/207)
	Medicare	19 (38/207)
	Federal	17 (36/207)
	Self Pay	10 (22/207)
Highest Level of Education	Elementary (1–5)	2 (4/207)
	Middle School (6–8)	6 (13/207)
	High School (9–12)	53 (110/207)
	Any College	27 (56/207)
	Postgraduate	11 (24/207)
Employment Status	Employed	54 (112/207)
	Retired	15 (31/207)
	Unemployed	17 (36/207)
	Disability	14 (28/207)
ESI category**	1	0 (0/207)
	2	60 (125/207)
	3	38 (79/207)
	4	1 (3/207)
	5	0 (0/207)

^*May not sum precisely due to rounding

^**Estimated Severity of Illness = measure of illness acuity;

Scale 1–5, 1=most acutely ill, 5=least acutely ill

Of the 207 subjects, approximately 89% had adequate literacy, or approximately a 9^th grade reading level, when measured by the S-TOFHLA (Table 2). Eighty percent of the subjects had adequate literacy as measured by the REALM. The SLS was similarly skewed, with a median score of 14 (interquartile range 12, 15) on a scale of 3 to 15. Adequate numeracy skills, or at least an “average” WRAT4 score (approximately an 8^th grade numeracy level), was found in only 56% of subjects; 36% of subjects had below average or low numeracy skills. On a possible scale of 6 to 48, the median SNS score was 36 (IQR 30, 42). Scatterplots of the SLS compared against the S-TOFHLA and REALM, as well as the SNS compared against the WRAT4 mathematical subtest, can be found in the Figure 2.

Table 2.

Performance on Literacy and Numeracy Scales

Scale (n=207)		Median (IQR)*
S-TOFHLA		35 (31,36)
Inadequate	7% (14/207)
Marginal	4% (9/207)
Adequate	89% (184/207)
REALM		65 (63, 66)
Inadequate	9% (19/207)
Marginal	11% (23/207)
Adequate	80% (165/207)
WRAT4		92 (81, 106)
>Below Ave	44% (90/207)
Average	36% (75/207)
>Above Ave	20% (42/207)
SLS (summed)		14 (12, 15)
SNS (summed)		36 (30, 42)

^*Median (Interquartile Range)

Figure 2. Scatter Plots of Subjective Measures of Numeracy and Literacy vs Objective Measures.

A: SLS vs S-TOFLHA

B: SLS vs REALM

C: SLS vs WRAT4 Math Subtest

Cronbach’s alpha for the three SLS questions is 0.74. The Spearman’s rank correlations in Table 3 show that the summed SLS correlates significantly with objective measures of numeracy and health literacy as well as with educational level and the summed SNS. For the SLS, the area under the curve (AUC) for the nonparametric ROC curve, using the S-TOFHLA as the comparison standard, is 0.74 (95% CI 0.68, 0.80) (Figure 1). The AUC for the nonparametric ROC curve using the REALM as the comparison standard is 0.72 (0.65, 0.78).

Table 3.

Spearman’s Rank Correlation Coefficients (n = 207)^*

	S-TOFHLA	REALM	WRAT4	Education Level
S-TOFHLA	1	--	--	0.29 (0.16, 0.41)
REALM	0.56 (0.46, 0.65)	1	0.49 (0.37, 0.38)	0.36 (0.24, 0.48)
WRAT4	0.54 (0.43, 0.63)	0.49 (0.37, 0.58)	1	0.52 (0.41, 0.62)
SLS	0.33 (0.20, 0.45)	0.26 (0.13, 0.38)	0.26 (0.13, 0.38)	0.25 (0.12, 0.37)
SNS	0.36 (0.23,0. 47)	0.36 (0.23, 0.47)	0.57 (0.47, 0.65)	0.48 (0.37, 0.58)

^*correlation coefficient (95% Confidence Interval); all correlations are statistically significant, P<0.001

Figure 1. ROC Curves and AUC for Subjective Measures vs Objective Measures.

A: ROC and AUC for SLS, vs S-TOFHLA 0.74 (95% CI 0.68, 0.80)

B: ROC and AUC for SLS vs REALM 0.72 (95% CI 0.65, 0.78)

C: ROC and AUC for SNS vs WRAT4 0.77 (95% CI 0.70, 0.82)

Cronbach’s alpha for the eight SLS questions is 0.82. The Spearman’s rank correlations in Table 3 show that the summed SNS correlates significantly with objective measures of numeracy and health literacy, the summed SLS, and educational level. For the summed SNS, the AUC for the nonparametric ROC curve, using the WRAT4 as the comparison standard, is 0.77 (95% CI 0.70, 0.82) (Figure 1).

In a series of hierarchical regression analyses, the objective measures of numeracy and health literacy were regressed against SLS and SNS, after adjusting for age, gender, race, and self-reported education level (Table 4). After adjustment, the SLS and SNS together uniquely account for an additional 15.1% of the variance of the S-TOFHLA, 18.1% of the variance of the REALM, and 8.6% of the variance of the WRAT-4 (p<0.05). Moreover, the SLS appropriately explains more of the variance in the two objective literacy measures (36% for the S-TOFHLA and 38% for the REALM, p<0.05), while the SNS appropriately explains more of the variance in the objective numeracy measure (30% for the WRAT4, p<0.05).

Table 4.

Standardized regression weights from multivariate analyses predicting objective measures of literacy and numeracy.⁺

	Model 1: STOFHLA as Outcome	Model 2: REALM as Outcome	Model 3: WRAT4 as Outcome
Education	0.14*	0.03	0.27*
Age	−0.26*	−0.08	−0.01
Gender (Male)	−0.08	−0.15*	0.03
Race (Non-White)	−0.11	−0.15*	−0.24*
Literacy (SLS)	0.36*	0.38*	0.09
Numeracy (SNS)	0.11	0.16*	0.30*
ΔR2++	0.151*	0.181*	0.086*
Total R2	0.323*	0.285*	0.393*

⁺Adjusted for age, gender, race, and education. All regression weights are for the final regression equation

⁺⁺Change in R² due to entering SLS and SNS on step 2 of the hierarchical analysis

^*p < 0.05

Discussion

To our knowledge, this is the first study to evaluate the use of subjective measures of literacy and numeracy in an emergency department setting. In the past, emergency department literacy or numeracy studies generally focused on content, or disease-specific knowledge, making comparisons among studies very difficult.^{16,17,39–50} The lack of a simple, concise bedside tool has been a significant barrier to identifying and measuring individual patient numeracy and general health literacy in the emergency department setting.

Our findings indicate that the SLS correlates well with the S-TOFHLA, which measures general health literacy, and moderately well with the REALM, which is a more broad measure of literacy. Previously published data for outpatient SLS validation reveals similar ROC curves,^27–29,51 though correlations were not included. Previously published correlation coefficients for the SNS are also similar to our findings.^30,31,52

The S-TOFHLA and REALM are comparison standards, not gold standard measures of literacy. In fact, the Spearman’s rank order correlation coefficient for the S-TOFHLA and REALM is 0.80.³⁶ It is likely that the S-TOFHLA and REALM measure different aspects of literacy, and the SLS and SNS measure other additional aspects of the overall construct of general health literacy. Lower correlation coefficients may not detract from a tool’s ability to predict important clinical outcomes; heart failure patients identified as having low literacy by the SLS may have higher mortality.³² Interestingly, correlation of the SLS with the REALM is similar to its correlation with the WRAT4, which was designed to measure only numeracy. The SNS correlates well with the WRAT4, and although the SNS correlates significantly with measures of literacy, those correlations are lower than the association of the SNS with numeracy, as would be expected.

Confirming prior research,⁵³ we found that self-reported education level does predict, to a moderate degree, numeracy and general health literacy; however, using the SLS and SNS together increases the variance explained in the objective measures considerably. The best estimate of a patient’s numeracy and general health literacy skills can be obtained by combining self-reported education level with the SLS and SNS. The optimal interpretation and use of the SLS and SNS are not addressed in this study. As noted in previous research, the optimal cut point of a screening test in a particular setting depends on the test accuracy, prevalence of inadequate health literacy, costs of testing and false-positive classification, and benefits of identifying true positives.²⁷ Use of the SLS and SNS can be tailored to multiple research and clinical settings.

In these analyses, the full eight-item version of the SNS was used. It may not be necessary, however, to use all eight items. Previous work suggested the potential use of a five-item version of the SNS.³¹ Exploratory analysis of our data confirms that the five-item SNS functions nearly as well as the eight-item version, and an even briefer, three-item SNS (“fractions,” “shirt,” and “useful”) may function as well as the five- or eight-item survey. Future work will be needed to confirm these findings.

The currently available validated tests used to measure numeracy and general health literacy, the S-TOFHLA, REALM, and WRAT4 mathematical subtest, are cumbersome, time-consuming to administer and take, and generally poorly received by patients. Subjective measures of numeracy and literacy, on the other hand, are easily administered at the bedside, require only a few minutes, and are preferred by patients. Combining the SNS and the full SLS would require patients to answer eleven questions and would take most patients fewer than three minutes to complete. If the SNS can indeed be reduced to three items, patients with low numeracy and health literacy could be identified in fewer than two minutes of survey time. The use of these brief, subjective tests may help direct physicians as they counsel patients regarding risks associated with procedures and health behaviors. These tests would also allow for real-time identification of patients who could benefit from interventions to improve their understanding and increase their disease self-management skill-set. For example, rather than providing the patient with high school-level reading discharge instructions, the patient may be given an alternative set of discharge instructions consisting of illustrations.

However, it is as yet unclear how much literacy and numeracy contribute to clinical outcomes such as ED recidivism, admission rates, and resource utilization, particularly for specific diseases such as asthma, renal failure, and heart failure. Future work will be necessary to establish the predictive validity of the SLS and SNS—that is, how well these tests, above and beyond patient demographics, predict clinical outcomes

Limitations

Subjective estimates of numeracy and literacy are prone to the errors associated with any self-reports or estimates of behaviors. Patients may be unaware of their lack of ability, or they may overestimate their abilities due to perceived social-desirability. This may, in part, explain the skewed nature of the self-reported literacy scores, although it is more likely that the SLS is subject to the same ceiling effects as the S-TOFHLA and REALM: most patients meet the criteria for adequate literacy, thus resulting in a skewed distribution of literacy scores, with relatively small variance. This ceiling effect, with its relatively small variance, may partly explain the relatively low correlation between the SLS and the S-TOFHLA and REALM. Numeracy, which has less of a social desirability bias, is more normally distributed. Given the wide range in mathematical abilities, it is likely that the subjective and objective numeracy tests measure a true increase in variation in the general adult emergency department population.

Patients enrolled in this study were drawn from a convenience sample, and it is unknown how many patients were approached who then declined, or why these patients declined. However, the demographics of the patients who participated in this study are similar to the overall demographics of the patients seen in our emergency department, making these findings more likely to be applicable to the general adult emergency department population. Participants were not formally screened for dementia or delirium; it is possible that mild, undetected cognitive impairments could have influenced the findings, although it would be reasonable to assume that unintentional inclusion of delirious or confused patients would result in lower numeracy and/or literacy scores.

Conclusions

Our findings suggest that subjective measures can be used as brief, valid estimates of numeracy and general health literacy in the adult emergency department setting. Combined, the SLS and SNS predict objective numeracy and general health literacy better than either alone, and above and beyond self-reported education level. In the future, they may be used to help identify patients with very low numeracy and health literacy skills for targeted, personalized intervention. Their ability to predict clinical outcomes such as ED recidivism, hospitalization, and mortality requires further investigation.