Feasibility and Diagnostic Accuracy of Brief Health Literacy and Numeracy Screening Instruments in an Urban Emergency Department

Abstract

Objectives

To evaluate the diagnostic accuracy of five health literacy screening instruments in emergency department (ED) patients: the Rapid Evaluation of Adult Literacy in Medicine-Revised (REALM-R), the Newest Vital Sign (NVS), Single Item Literacy Screens (SILS), health numeracy, and physician gestalt. A secondary objective was to evaluate the feasibility of these instruments as measured by administration time, time on task, and interruptions during test administration.

Methods

This was a prospective observational cross-sectional study of a convenience sampling of adult patients presenting during March 2011 and February 2012 to one urban university-affiliated ED. Subjects were consenting non-critically ill, English-speaking patients over the age of 18 years without aphasia, dementia, mental retardation, or inability to communicate. The diagnostic test characteristics of the REALM-R, NVS, SILS, health numeracy, and physician gestalt were quantitatively assessed by using the short Test of Functional Health Literacy in Adults (S-TOHFLA). A score of 22 or less was the criterion standard for limited health literacy (LHL).

Results

Four hundred thirty-five participants were enrolled, with mean age of 45 years (SD ±15.7 years) and 18% had less than a high school education. As defined by an S-TOHFLA score of 22 or less, the prevalence of LHL was 23.9%. In contrast, the NVS, REALM-R, and physician gestalt identified 64.8%, 48.5%, and 35% of participants as LHL, respectively. A normal NVS screen was the most useful test to exclude LHL, with a negative likelihood ratio of 0.04 (95% CI = 0.01 to 0.17). When abnormal, none of the screening instruments, including physician gestalt, significantly increased the post-test probability of LHL. The NVS and REALM-R require 3 and 5 minutes less time to administer than the S-TOHFLA. Administration of the REALM-R is associated with less test interruptions.

Conclusions

One-quarter of these ED patients had marginal or inadequate health literacy. Among the brief screening instruments evaluated, a normal Newest Vital Sign result accurately reduced the probability of limited health literacy, although it will identify two-thirds of ED patients as high-risk for limited health literacy. None of the brief screening instruments significantly increases the probability of limited health literacy when abnormal.

INTRODUCTION

Health literacy is defined by the Institute of Medicine (IOM) as “the degree to which individuals can obtain, process, and understand basic health information and services needed to make appropriate health decisions.”¹ Limited health literacy (LHL) is widely recognized as a major determinant of important health outcomes, and is estimated by the IOM to cost $73 billion annually.² According to the American Medical Association, poor health literacy is “a stronger predictor of a person’s health than age, income, employment status, education level, and race.”^3-6 Inadequate health literacy is associated with poorer health status, less knowledge about chronic disease self-management, lower rates of medication adherence, and higher rates of acute health care utilization in patients with chronic diseases, as well as with increased hospitalization rates and mortality.^1,5,7-9 The IOM has ranked addressing LHL among the first quartile of research priorities. Estimates of the number of adults with LHL vary greatly depending upon the population and screening instrument. The best available estimate from a nationally representative sample is that nearly half of all American adults can be categorized as having LHL,¹⁰ although this estimate was generated using an instrument that is not publicly available. Other estimates of health literacy in patient populations use various publicly available screening instruments.^11-17

The effect of health literacy is due to an interaction between patients’ health literacy skills and the demands that the health context, such as the emergency department (ED), place on these skills.¹ Discrepancies observed between patients’ reading abilities and ED discharge materials were noted in the emergency medicine literature nearly 20 years ago.^18-20 Recently, the importance of assessing patients’ health literacy skills in the ED has garnered increasing attention, with estimates of the prevalence of LHL ranging from 10.5% to 88% depending on the screening instrument used and the geographic locale.^11-17 Adequate health literacy can potentially influence multiple aspects of ED care, including the ability to comprehend and incorporate verbal information provided by health care personnel, understand numeric risk information, comprehend written materials and forms, give informed consent, access health care services, and adhere to post-discharge follow-up and medication recommendations.¹ Despite recent interest in this area, there is a paucity of ED-based research investigating the association between LHL and health care outcomes.¹⁷

A major barrier to high-quality studies of health literacy in the ED is the lack of health literacy measures validated for use in busy clinical settings like the ED. Most studies evaluating the diagnostic accuracy of screening instruments have not been ED-based, but rather have been performed in controlled clinical research environments where factors often accompanying ED visits, such as anxiety, time pressures, acute illness, noise, and interruptions are not present.^21-32 It is known that a tool that has been validated for use in one environment may not be valid in others.^33,34

A related gap in knowledge and potential barrier to translation of research findings into clinical practice relates to the feasibility of use of health literacy instruments for routine screening in the ED. We are not aware of any published data regarding the feasibility of use of these instruments for screening in the ED.

The primary objectives of this study were: 1) to evaluate and compare the diagnostic accuracy of five brief objective and subjective screening instruments for LHL in urban ED patients, and 2) to evaluate the feasibility of these instruments based upon time burden of administration and number of interruptions relative to performance. A secondary objective of this study was to assess the accuracy of emergency physician (EP) gestalt for LHL.

METHODS

Study Design

This was a prospective, cross-sectional study using convenience sampling. We sought to minimize design-related bias by using the Standards for Reporting of Diagnostic Accuracy (STARD) criteria, which consist of a 25-item checklist that serves to ensure consistent and reproducible quality in the design, conduct, and reporting of diagnostic trials.^35,36 Hospital institutional review board approval was granted and written informed consent was obtained from all subjects prior to enrollment.

Study Setting and Population

This study was conducted in an urban academic ED with over 97,000 total annual visits. Trained research assistants recruited study participants from the ED between March 1, 2011 and February 29, 2012 at different times of the day on different days of the week. All ED patients aged at least18 years were prospectively identified for enrollment by review of the electronic medical record dashboard by research assistants. Exclusion criteria included undue patient distress as judged by the attending physician, altered mental status, aphasia, mental handicap, previously diagnosed dementia or insurmountable communication barrier as judged by family or the screener, non-English speaking, sexual assault victims, acute psychiatric illness, or corrected visual acuity worse than 20/100 using both eyes. Exclusion criteria were meant to minimize confounding variables that could bias estimates of diagnostic accuracy of health literacy screening instruments.

Study Protocol

Deidentified age, race, and sex data were recorded for patients declining to participate for comparison with consenting patients. For each eligible and consenting patient, a research assistant administered all health literacy tests during the patient’s ED visit. All screeners received standardized training on the administration of screening instruments, consisting of an in-person presentation, review of a pre-recorded training presentation, practice sessions administering the instruments to mock patients, and monitored screening of the first patient. The screeners were paid research assistants and medical students. Training included specific instruction to avoid language that might lead to feelings of shame or embarrassment among patients with LHL.³⁷ Screeners read participants standardized instructions for each of the health literacy screening instruments and recorded the start and stop times for all screening tests other than the single-item literacy tests (SILS). Instructions were given that screening should not interfere with patient care. Stop and restart times were also recorded for interruptions that occurred during screening tests such as nursing care, laboratory or imaging studies, etc. When present during screening, family members and friends were asked not to assist with responses. The order of presentation of screening instruments was varied based on even or odd days of the week, aimed at altering whether the Short Test of Functional Health Literacy in Adults (S-TOFHLA), which was the lengthiest test, was presented first or last. The screeners discretely recorded all participant responses. Demographic data elements were collected during the interview and from the electronic medical record. Attending EPs were not informed of the health literacy-screening test results. After enrollment of the first 112 patients, the study protocol was amended to solicit from either the resident or attending treating physician, their subjective estimate of the patient’s level of health literacy in advance of testing (see the data collection instrument in Data Supplement 1). This change did not alter the patient component of the protocol. All data were entered into a Microsoft Access database (Microsoft Corp., Redmond, WA).

Screening Tools

The screening tools administered included the abbreviated S-TOFHLA,^23,24 which was used as the criterion standard; the Rapid Evaluation of Adult Literacy in Medicine – Revised (REALM-R);²⁵ the Newest Vital Sign (NVS);²⁷ and three SILS²⁹ that were considered as single items, and as an index. The thresholds for LHL for the S-TOFHLA,^23,24 REALM-R,^21,22 and NVS^27,28 are based on published scoring rules for the instruments. The cutoff for the SILS was not previously established in the literature. We derived the threshold for the SILS based on the value that optimized both sensitivity and specificity in our population. We also evaluated physician gestalt. We describe each of these screening instruments in detail in Data Supplement 2. In addition, Data Supplement 3 illustrates the REALM-R and Data Supplement 4 demonstrates the NVS.

Outcome Measures

The criterion standard for LHL was an abbreviated S-TOFHLA score ≤ 22, which is the sum of inadequate health literacy (S-TOFHLA score 0 to 16) and marginal health literacy (S-TOFHLA score 17 to 22). The primary outcome measures in the evaluation of diagnostic accuracy consisted of 1) test characteristics for each instrument, including positive likelihood ratios (LR+) and negative likelihood ratios (LR-) compared with the S-TOFHLA criterion standard, and 2) the correlation of each test with the S-TOFHLA. The primary outcome measures in the evaluation of feasibility were 1) the time on average that it took to administer the “time on test,” (i.e., total time minus time during interruptions), and 2) the number and duration of interruptions during administration of the tests.

Depending on the goals of LHL detection, it may be alternately preferable to use tests that optimize sensitivity versus specificity. Because of this, in determining which tests are the most feasible, we defined feasibility based on maximizing measures of ease and efficiency (time on test and interruptions) while achieving the best performance, using LR+ (when the emphasis is on identifying LHL) and the LR- (when the emphasis is on excluding LHL). Although the risk of interruptions naturally increases with an increasing time of test administration, we considered interruptions in addition to time, because interruptions are felt to lead to worse task performance that could affect accuracy.³⁸

Data Analysis

Analysis was conducted according to STARD criteria with SAS (version 9.2, SAS Institute, Cary, NC).³⁶ Demographic data were compared between excluded and enrolled patient groups using t-tests for normally distributed continuous variables, and chi-square for categorical variables. Each instrument was dichotomized with commonly used cutoffs in order to report LR+ and LR- to interpret clinically with individual patients. A summary SILS score was computed by assigning a value of “5” for the response indicating the least difficulty using or interpreting medical information independently, and a value of “1” for the response indicating the most difficulty. For example, for the first SILS question inquiring how often patients have somebody else read hospital materials, an “always” response was scored a “1” and a “never” response was scored a “5.” The summed SILS score ranged from 3 to 15. The SILS demonstrated reasonable internal consistency reliability. The standardized Cronbach’s alpha for the three-item SILS measure was 0.78.

Standard operating characteristics of diagnostic tests were computed for each health literacy screening instrument, including sensitivity, specificity, LRs, receiver operating characteristic (ROC) curves with area under the ROC curve (AUC), and 95% confidence intervals (CI). We compared the AUCs for statistically significant differences.³⁹ To evaluate the validity of each instrument, we examined Spearman’s correlation coefficient against the abbreviated S-TOFHLA. Verifying a sensitivity of 90% with 5% range of error, and a baseline prevalence of LHL of 40%,¹⁷ would require 346 subjects to be enrolled with complete data collection. Assuming 19% of subjects would have incomplete data collection based on our previous experience, we planned to enroll 430 subjects.⁴⁰

Time was analyzed in increments of whole minutes. Times of less than one minute were rounded up to the nearest whole minute; this was felt to best reflect the level of precision reliably achievable. Mean and median total test time and interruption data were calculated for each individual screening tool.

RESULTS

We approached 588 patients, excluded 142, and enrolled 446 patients (Figure 1). We excluded from analysis 11 patients who did not complete the abbreviated S-TOFHLA, leaving 435 subjects in this analysis. The mean age of patients was 45 years (SD ±15.7 years), and 18% had less than a high school education. Additional demographics are summarized in Table 1. The age, sex, and race of enrolled patients did not differ significantly from those of excluded patients or the 93,476 total patient visits to the ED in 2011.

Figure 1.

Enrollment Flow Diagram *11 patients excluded after enrollment due to missing S-TOFHLA scores. S-TOFHLA = Short Test of Functional Health Literacy in Adults.

Table 1.

Patient Demographics (N=435)

Demographics	n (%)
Female	241 (55.5)
Race/ethnicity
White	133 (30.8)
Black	292 (67.6)
Asian	1 (0.2)
Other	6 (1.4)
Education Level Attained
Less than high school	77 (17.7)
High school	218 (50.1)
Some college or higher	140 (32.2)
Primary Insurance
Private	146 (33.3)
Self-pay	115 (26.4)
Medicaid	89 (20.5)
Medicare	73 (16.8)
Private + Medicare	2 (0.5)
Other insurance	10 (2.3)
Age in yrs, mean (±SD)	45 (±15.7)

Diagnostic Accuracy

As summarized in Table 2, 23.9% of patients had LHL as identified by the S-TOFHLA (13.1% with inadequate and 10.8% with marginal functional health literacy). The REALM-R identified 48.5% of patients as having LHL, and the NVS identified 64.8% as having LHL (29.7% with a high likelihood of limited literacy, and another 35.1% as possible limited literacy). Physician gestalt identified 35.0% of patients as having LHL, including 8.1% with inadequate health literacy and an additional 26.9% with marginal health literacy.

Table 2.

Health Literacy Interpretations (N=435)

Instrument	Number of Participants (%)
Abbreviated S-TOFHLA
Inadequate health literacy (score 0-16)	57 (13.1)
Marginal health literacy (score 17-22)	47 (10.8)
Adequate health literacy (score 23-36)	331 (76.1)
REALM-R, n=433
Lower health literacy (≤6 items correct)	220 (48.5)
Higher health literacy (>6 items correct)	223 (51.5)
NVS, n=428
High likelihood of limited literacy (0-1 items correct)	127 (29.7)
Possible limited literacy (2-3 items correct)	150 (35.1)
Adequate literacy (4-6 items correct)	151 (35.3)
Numeracy score
0	79 (18.2)
1	171 (39.3)
2	118 (27.1)
3	49 (11.3)
4	18 (4.1)
Summed SILS, n=433
3	3 (0.7)
4	5 (1.2)
5	4 (0.7)
6	4 (0.9)
7	5 (1.2)
8	11 (2.5)
9	19 (4.4)
10	16 (3.7)
11	36 (8.3)
12	49 (11.3)
13	54 (12.5)
14	79 (18.2)
15	149 (34.4)
Physician gestalt, n=309
Inadequate	25 (8.1)
Marginal	83 (26.9)
Adequate	207 (65.1)

S-TOFHLA = Short Test of Functional Health Literacy in Adults; REALM-R = Rapid Estimate of Adult Literacy in Medicine-Revised; NVS = Newest Vital Sign; SILS = Single Item Literacy Screens

Table 3 demonstrates the relative performance of the screening tools tested. The NVS had the best LR- to exclude LHL at 0.04 (95% CI = 0.01 to 0.17). The LR+ estimates ranged from 1.61 to 2.78. Although the summed SILS scores at a cutoff of ≤ 12 only had a LR+ 2.78 (95% CI = 1.92 to 4.02), some individual SILS question responses would significantly increase the probability of LHL. Specifically, a response of “always” need help reading (LR+ 7.42, 95% CI = 1.13 to 48.89), “not at all” on confidence with medical forms (LR+ 9.93, 95% CI = 1.9 to 52.03), and “poor” on ability to read (LR+ 16.56, 95% CI = 2.01 to 136.46) would each significantly increase the probability of LHL in an individual patient. Two-by-two contingency tables for each instrument are reproduced in Data Supplement 5. Unaided EP impression of health literacy was inadequate to significantly increase or decrease the probability of LHL. AUCs are presented in Figure 2, demonstrating that the distributions for NVS and REALM-R were widely overlapping.

Table 3.

Diagnostic Test Characteristics of NVS, REALM-R, SILS, and Numeracy Tests (N=435)

Instrument	Sensitivity % (95% CI)	Specificity % (95% CI)	Positive Likelihood Ratio (95% CI)	Negative Likelihood Ratio (95% CI)
NVS ≤ 3, n=428	98.0 (93.1-99.8)	45.7 (40.3-51.3)	1.81 (1.51-2.17)	0.04 (0.01-0.17)
REALM-R ≤ 6, n=433	80.8 (73.2-88.3)	61.7 (56.5-67.0)	2.14 (1.64-2.80)	0.30 (0.19-0.46)
Numeracy ≤ 1	80.8 (73.2-88.3)	50.2 (44.8-55.5)	1.61 (1.30-2.01)	0.38 (0.24-0.60)
SILS ≤ 12, n=433	68.0 (59.0-77.07)	75.5 (70.8-80.1)	2.78 (1.92-4.02)	0.42 (0.31-0.57)
Help Reading, n=433
1 – Always	9.7 (4.0-15.4)	98.8 (96.9-99.7)	7.42 (1.13-48.89)	0.92 (0.86-0.98)
2 – Often	16.5 (9.3-23.7)	97.0 (95.1-98.8)	5.30 (1.59-17.64)	0.87 (0.79-0.95)
3 – Sometimes	42.7 (33.2-52.3)	85.5 (81.7-89.3)	2.81 (1.65-4.78)	0.69 (0.58-0.83)
4 – Rarely	49.5 (39.9-59.2)	66.1 (61.0-71.2)	1.49 (1.06-2.08)	0.75 (0.59-0.96)
5 – Never	100	0.00	1.00	--
Dichotomized*	42.7 (33.2-52.3)	85.5 (81.7-89.3)	2.81 (1.65-4.78)	0.69 (0.58-0.81)
Medical Form Help
1 – Not at all	14.4 (7.7-21.2)	98.5 (97.2-99.8)	9.93 (1.90-52.03)	0.86 (0.79-0.94)
2 – A little bit	30.8 (21.9-39.6)	94.6 (92.1-97.0)	5.69 (2.38-13.59)	0.73 (0.64-0.84)
3 – Somewhat	54.8 (45.2-64.4)	80.4 (76.1-84.6)	2.78 (1.80-4.29)	0.57 (0.45-0.72)
4 – Quite a bit	74.0 (65.6-82.5)	55.9 (50.5-61.2)	1.69 (1.32-2.17)	0.45 (0.31-0.66)
5 – Extremely	1.00	0.00	1.00	--
Dichotomized*	54.8 (45.2-64.4)	80.4 (76.1-84.6)	2.78 (1.80-4.29)	0.57 (0.45-0.72)
Ability To Read
1 – Very poor or terrible	6.7 (1.9-11.6)	100.0 (98.9-100.0)
2 – Poor	14.4 (7.7-21.2)	99.1 (97.4-99.8)	16.56 (2.01-136.46)	0.86 (0.79-0.93)
3 – Okay	40.4 (31.0-49.8)	95.5 (93.2-97.7)	8.58 (3.36-21.87)	0.64 (0.55-0.76)
4 – Good	72.1 (63.5-80.7)	75.2 (70.6-80.0)	2.83 (1.97-4.08)	0.40 (0.29-0.55)
5 – Excellent or very good	1.00	0.00	1.00	--
Dichotomized*	40.4 (31.0-49.8)	95.5 (92.2-97.7)	8.58 (3.36-21.87)	0.64 (0.55-0.76)
Physician gestalt, n=309: inadequate or marginally adequate	52.7 (41.3-64.1)	70.6 (64.8-76.5)	1.72 (1.20-2.47)	0.70 (0.55-0.89)

^*Dichotomization used the following groupings: help reading (always/often as adequate, other responses as inadequate), medical form help (not at all/a little bit as adequate, other responses as inadequate), ability to read (excellent/very good/good as adequate, other responses as inadequate).

REALM-R = Rapid Estimate of Adult Literacy in Medicine-Revised; NVS = Newest Vital Sign; SILS = Single Item Literacy Screens

Figure 2.

ROC curve. NVS = Newest Vital Sign; REALM-R = Rapid Evaluation of Adult Literacy in Medicine-Revised; ROC = receiver operating characteristic; SILS = Single Item Literacy Screens.

The NVS and REALM-R had the highest correlations with the abbreviated S-TOFHLA (Spearman correlation 0.602 and 0.540 respectively, p < 0.001). The SILS (correlation range 0.239-0.485, p < 0.001) and physician gestalt (correlation 0.285, p < 0.001) were weakly correlated with the S-TOFHLA.

Feasibility

The S-TOHFLA had the longest time administration burden (time on test), followed by the NVS, which took on average 2.9 minutes less than the S-TOFHLA, and then the REALM-R, which took on average 5.0 minutes less than the S-TOFHLA and almost a third of the time of the NVS (Table 4). The REALM-R had the fewest interruptions, with 0.5% of tests being interrupted compared to the NVS and S-TOHFLA, which were interrupted 6.0% and 13.1% of the time, respectively. On average, patients with lower health literacy levels took longer to complete tests than those with adequate health literacy levels.

Table 4.

Feasibility Assessments

Health Literacy	S-TOFHLA (n=434)				REALM-R (n=433)				NVS (n=428)
	Total Time n=434	% Interrupted n=435	Average Length of Interruption n=57	Time on Test n=433	Total Time n=433	% Interrupted n=433	Average Length of Interruption n=2	Time on Test n=433	Total Time n=428	% Interrupted n=428	Average Length of Interruption n=26	Time on Test n=429
Total	6.55 (7)	13.1	3.77 (2)	6.07 (6)	1.06 (1)	0.46	1.5 (1.5)	1.06 (1)	3.31 (3)	5.98	2.85 (2)	3.13 (3)
Adequate	6.29 (6)	14.20	3.53 (2)	5.79 (6)	1.03 (1)	0.45	2.00 (2)	1.03 (1)	3.07 (2)	5.96	3.11 (2)	2.89 (2)
Possible/Limited‡	7.40 (7)	9.62	5.00 (1)	6.98 (7)	1.09 (1)	0.48	1.00 (1)	1.09 (1)	3.43 (3)	6.14	2.71 (2)	3.27 (3)

Total time = total time for the test in minutes; Time on test = (Total time – Interrupted time)

^‡For REALM-R, only have inadequate and adequate health literacy

Data are reported as mean (median) unless otherwise noted

S-TOFHLA = Short Test of Functional Health Literacy in Adults; REALM-R = Rapid Estimate of Adult Literacy in Medicine-Revised; NVS = Newest Vital Sign

DISCUSSION

Limited health literacy is a relatively silent epidemic in today’s ED, threatening patient safety and with profound implications including access to quality care.^41,42 LHL also potentially affects informed consent, prudent layperson policies, shared decision-making, and the consideration of results of prior studies that did not stratify by health literacy, among other areas. Examining levels of health literacy remains a challenge for researchers, policy-makers, and emergency providers. Screening tools developed for use in clinical settings have been studied in the ED, but none of these studies measured health literacy using an instrument validated to do so.^18-20,43-46 For any instrument used routinely in the ED setting, rather than just for research, simplicity and efficiency in training and administration are critical for adoption and reliability, particularly as these are included alongside other regulatory screening demands and patient care needs.⁴⁷ We envision a test that can be administered routinely to quickly and accurately assess a patient’s health literacy level without disrupting patient flow. It should be noted that routine screening in clinical settings is an area of current controversy among health literacy experts, in that clinical environments are presently not geared to tailor therapies or interventions based on determination of health literacy level.⁴⁸

Though feasibility of health literacy screening tools has not been studied in the ED, validation studies in the ED are not entirely missing. Recently, McNaughton et al. published an ED-based observational investigation assessing the diagnostic accuracy of the REALM-R, S-TOFHLA, Wide Range Achievement Test 4, and subjective literacy screen questions, reporting on the validity of these instruments.³² McNaughten et al. provided a needed first step in assessment of these tools in the ED. However, the study did not report on sensitivity, specificity, or LRs, and did not use the STARD criteria in the conduct or reporting of the study, threatening its internal validity.³⁶ That study’s primarily white demographic also limits the external validity of its findings to similar populations until additional validation trials are available. Compared to that of McNaughton et al., our population had a higher proportion of LHL, and the ROC AUC for the REALM-R was higher in our study (0.80 versus 0.72). Our population sample had a lower education level and was more ethnically diverse, reflective of many urban ED settings. In our sample, we obtained similar estimates of internal reliability (i.e. Cronbach’s alpha) to that of McNaughton et al. for the screening instruments evaluated with significantly higher correlations.

Using the S-TOFHLA as the criterion standard, our results demonstrate a prevalence of LHL in our population varying from approximately 24% to 65% depending on the screening test used. The wide range of estimates of LHL between instruments is probably related to the specific health literacy domains tested by each instrument.⁴⁹ The NVS resulted in the highest prevalence estimate of LHL. This tool tests the health literacy domains of document literacy and numeracy in addition to print literacy, while the REALM-R and S-TOFHLA are more based on word pronunciation, reading comprehension, and print literacy.⁵⁰ Based on these differential performance results, health literacy tasks that require basic numeracy skills may therefore be particularly problematic for this patient population, suggesting the possible importance of screening for limited numeracy.⁵¹ Furthermore, the abbreviated S-TOFHLA is an imperfect criterion standard. It does not assess all domains of the health literacy construct, was originally validated on the population of one clinic, and did not use available published diagnostic accuracy methods.^24,36,49,50 Nonetheless, it is perhaps the most widely accepted criterion standard for distinguishing LHL from adequate health literacy, commonly used in studies of this nature, and we are not aware of a better approach for assessment that is publicly available.³¹ In our study, however, the abbreviated S-TOFHLA was also the least feasible screening tool for routine use, taking on average nearly six minutes to administer and incurring the most interruptions.

The NVS was highly correlated with the abbreviated S-TOFHLA, and was the most accurate screening instrument to rule out LHL with a LR- of 0.04. The NVS did not fare as well on feasibility, with an average “time on test” of 3.13 minutes. Incidentally, our data provide the first estimates of sensitivity and specificity for the NVS in an ED population. When stratified by health literacy level, for both the S-TOFHLA and the NVS, patients in the lowest categories had times on test that were shorter than those in the marginal or possible categories were. It is not clear if this is might be because those in the lowest categories provided answers quickly without as much deliberation.

The REALM-R was highly correlated with the abbreviated S-TOFHLA, but using the S-TOFHLA as the criterion standard identified almost half of patients as LHL. The REALM-R LR+ of 2.14 (95% CI = 1.64 to 2.80) is among the better values but still fails to achieve a significant discriminatory level. The LR- of 0.30 for the REALM-R (95% CI = 0.19 to 0.46) is also respectable. Although we did not specifically study training or test for ease of use, on its face, the REALM-R is simpler to administer compared to the S-TOHFLA and NVS. The REALM-R had the shortest time burden and fewest interruptions of the tests for which we measured time of administration, while still achieving excellent sensitivity for identifying LHL, making it the most feasible of these tests.

The ability to exclude LHL with the composite SILS, using a threshold score of 12, was roughly equivalent to that for the individual SILS items and the REALM-R, with a LR- of 0.42. The summed SILS score had a clinically inconsequential, but superior LR+ (2.78; 95% CI = 1.92 to 4.02), although the CIs were overlapping with the LR+ confidence intervals for the other screening instruments. The single question “How would you rate your ability to read?” performed competitively with the three other screening tools in identifying LHL. Although we did not measure time for completion of the SILS, anecdotally the single questions consistently yielded patient response times less than one minute.

It is known that physicians in non-ED settings overestimate their patients’ health literacy,^52,53 but this has never been assessed among EPs. Our findings were consistent with an accumulating body of evidence that physician gestalt is an inaccurate estimate for LHL.^52,53 Physician gestalt cannot exclude inadequate health literacy (LR- of 0.70, 95% CI = 0.55 to 0.89), and physician gestalt for identifying LHL had a LR+ of 1.72 (95% CI = 1.20 to 2.47), which is in the same range as the tested health literacy screening instruments except for some of the SILS. However, using the guide that LR+ exceeding 10 are optimal diagnostic or screening tests, none of the screening instruments, including physician gestalt, is sufficiently accurate to diagnose LHL without further testing.⁵⁴ Another interpretation of our data is that physician gestalt is as accurate as any of the currently available validated health literacy tests and does not require additional time for testing. The development of a more comprehensive measure of health literacy that is feasible for administration in clinical settings would further advance this area of inquiry.

Several high-yield research opportunities for assessing and intervening on health literacy exist. One important construct is understanding the optimal personnel, situation, and time during the ED course of care in which health literacy screening should occur. Another important question for future research is whether screening patients for LHL allows clinicians to target interventions and improve post-ED health outcomes. Another research question is understanding the most meaningful definition of LHL among ED populations. Less resource-intense and inexpensive interventions justify the broadest definition of LHL, whereas more resource-intense and expensive interventions favor a more narrow definition of LHL.

Interventions such as “teachback” for discharge instructions have been proposed in the context of “universal precautions” for LHL. In this case, screening is foregone and the intervention is applied to all.⁵⁵ Although few would argue that discharge instructions should be more clear and understandable, there are little data evaluating the effectiveness, reliability, or time effect of teachback, particularly as a ‘universal precaution.’^56,57 Moreover, discharge instructions are but one area where identifying those with LHL may be important. Those interventions that are more costly and resource-intense, such as individualized counseling, or use of discharge materials that may benefit from additional instruction, such as pictorial representations, may warrant a more restricted selection for those found to be at highest risk. Research is needed to examine whether different types of interventions targeting patients with LHL, defined broadly or narrowly, improve comprehension of information, and ultimately, outcomes for ED patients. Assuming that screening is felt to be of value, use of a validated instrument is of particular importance to insure that a construct one desires to measure is in fact addressed by the tool being used, and that this can be employed reliably.

LIMITATIONS

This was a single-center observational trial and excluded several groups of previously described patients, limiting the external validity in dissimilar populations. Specifically, we cannot extrapolate the estimates of diagnostic accuracy for the health literacy screening instruments to patients with undue distress, sexual assault victims, acute psychiatric illness, altered mental status, aphasia, mental or visual handicap, dementia, non-English speaking individuals, or those with communication barriers. We did not assess these populations and each instrument could be more or less accurate in these groups. Although we selected the most commonly used tools to evaluate, there are several other brief screening instruments in lesser use that we did not evaluate, including the Medical Term Recognition Test.³⁰ Even though we did not assess inter- or intra-rater reliability of these screening instruments, the ability to do so when the interval between tests is short as it would be in ED settings is largely limited by test-retest phenomena.⁵⁸ To our knowledge, the reliability of the health literacy screening instruments has yet to be described in any research or clinical setting. Although convenience sampling includes the potential for selection bias and spectrum bias, which can falsely increase sensitivity and specificity,⁵⁹ nearly all ED-based health literacy studies, including the only other validation study, have used convenience sampling. Ours is the only study that compares enrolled and declining patients along with the general ED population, demonstrating lack of differences among these populations in terms of basic demographics.

Contrary to STARD recommendations to blind the outcome assessors to the new test(s) being evaluated and vice versa, our screeners collected both the index tests and the abbreviated S-TOFHLA criterion standard. This increases the risk for incorporation bias, which can also falsely increase sensitivity and specificity.^59,60 However, both the index tests and the criterion standard were not scored real-time. Instead, responses were entered into a database that then computed the score, which was used to provide the health literacy category for each patient. These methods minimize but do not completely eliminate the potential for incorporation bias.

CONCLUSIONS

In an urban ED population, the prevalence of patients with limited health literacy as determined by the abbreviated S-TOFHLA was 23.9%, and ranged as high as 64.8% when using the NVS. When compared to this criterion standard, the NVS performed best at excluding limited health literacy. However, none of the instruments that we evaluated significantly increased the post-test probability of limited health literacy. The S-TOHFLA and the NVS are less feasible for use in the ED, taking on average approximately six and three minutes respectively to administer, and incurring frequent interruptions. The REALM-R was the most feasible of the tools for which we measured administration time, and this test performed reasonably well in reducing the probability of limited health literacy. If the SILS questions can be administered in approximately one minute, then this is the most feasible of the instruments with the best performance for identifying lower health literacy. Physician gestalt does not accurately identify or exclude limited health literacy. Selection of the optimal screening tool should consider these diagnostic test characteristics, including likelihood ratios, in conjunction with the goals and objectives that are intended by limited health literacy screening efforts.