A systematic review of instruments to measure health literacy of patients in emergency departments

Abstract

Objectives

Knowledge of patient's health literacy (HL) in the emergency department (ED) can facilitate care delivery and reduce poor health outcomes. This systematic review investigates HL measurement instruments used in the ED and their psychometric properties, accuracy in detecting limited HL, and feasibility.

Methods

We searched in five biomedical databases for studies published between 1990 and January 2021, evaluating HL measurement instruments tested in the ED on internal consistency, criterion validity, diagnostic accuracy, or feasibility. Reviewers screened studies for relevance and assessed methodologic quality with published criteria. Data were synthesized around study and instrument characteristics and outcomes of interest.

Results

Of the 2,376 references screened, seven met our inclusion criteria. Studied instruments varied in objective (n = 5) and subjective (n = 6) measurement of HL skills, and in HL constructs measured. The Brief Health Literacy Screen (BHLS) and the Subjective Numeracy Scale demonstrate acceptable and good internal consistency across studies. None of the instruments perform consistently well on criterion validity. The Rapid Estimate of Adult Literacy in Medicine–Revised and the Newest Vital Sign, both objective tests with short administration times, demonstrate good accuracy in one study with high risk of bias. The BHLS, a short subjective measure, shows moderate accuracy across studies including one with low risk of bias.

Conclusions

Several short instruments seem valid in measuring HL and accurate in detecting limited HL among ED patients, each with its practical advantages and disadvantages and specific measurement of HL. Additional research is necessary to develop a robust evidence base supporting these instruments.

INTRODUCTION

Health literacy (HL) is the individuals’ capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions. ¹ Important HL skills include reading and writing ability and numeracy skills. HL is an important determinant for health behavior, including planning and adjusting lifestyle, participation in medical decision making, treatment adherence, and recognizing when and how to access health care services. ² , ³

Limited HL is increasingly perceived as a global public health concern. ⁴ , ⁵ Levels of HL have been surveyed in industrialized countries such as the United States, Canada, Australia, and in the European Union (EU), with the prevalence of limited HL varying from 29% to 62%. ⁵ , ⁶ , ⁷ Limited HL has been associated with a wide range of adverse health effects, including worse self‐management skills, ⁸ , ⁹ greater risk of hospitalization, ED visits and lack of preventative care, worse health status, and lower quality of life. ¹⁰ The prevalence of limited HL in the emergency department (ED) is wide ranging across studies but generally high with estimates up to 88% depending on the visitor type and on the measurement instruments used. ¹¹ , ¹² Among ED patients, limited HL is associated with worse health status, higher number of health care utilization such as ED recidivism, and higher risk of death. ¹¹ , ¹² , ¹³ , ¹⁴ Although the explanatory mechanisms underlying the relations between limited HL and adverse outcomes are rather complex, many of the poor outcomes associated with limited HL may be caused or exacerbated by inadequacies in clinician–patient communication. ¹¹ , ¹⁵ , ¹⁶ Not recognizing low literacy among ED visitors by clinicians can lead to suboptimal patient involvement in and receipt of care. Available ED reading materials and standard information by provided clinicians are often too complex for this patient group, thereby increasing the risk of patients being un‐ or misinformed. ¹¹ , ¹⁶ Moreover, clinicians may approach treatment options differently than patients based on their assumptions of patient's HL and disease knowledge. ¹⁶ , ¹⁷ Timely recognition of limited HL in the ED can be a first important step in overcoming these inadequacies and the negative outcomes associated with limited HL.

Over the past decade, there has been a growing effort in developing HL measurement instruments aimed at anticipating on limited HL cases. ¹⁸ , ¹⁹ , ²⁰ Moreover, the 2004 Institute of Medicine report on HL recommended that HL assessment should be part of health care information systems to facilitate large‐scale studies of the effects of HL as well as the evaluation of interventions targeting limited HL. ¹ However, there is no actual overview and critical appraisal of HL measurement instruments used in the ED setting. Alqudah et al. ²¹ reviewed HL measurement instruments in the ED, but their review consisted of publications until 2011 on specific instruments using word recognition procedures with demonstrated concurrent validity and a maximum administration time of 5 min.

An actual and more comprehensive overview of HL measurement instruments studied in the ED informs clinicians on available instruments and may contribute to the identification of an instrument that is favorable for use in their ED. Therefore, our aim was to systematically review scientific literature on instruments used in the ED and their psychometric properties, accuracy in detecting limited HL, and feasibility.

METHODS

We planned and reported this systematic review in accordance with the reporting guidance provided in the Preferred Reporting Items for Systematic Review and Meta‐Analysis (PRISMA). ²² The protocol of this review was established a priori and registered on the International Prospective Register of Systematic Reviews (PROSPERO) website with ID CRD42020174997.

Data sources and strategy

We searched for articles published between January 1, 1990, and March 18, 2020, in the following databases: PubMed (including MEDLINE), Cumulative Index to Nursing and Allied Health Literature, Cochrane Library, EMBASE, and PsychInfo. Due to the COVID‐19 pandemic, study activities were postponed. An additional search was therefore performed to find relevant articles published between March 19, 2020, and January 11, 2021. Search strategies (Appendix S1) comprised a combination of key search terms related to the concepts of “emergency department,” “health literacy,” and “measurement tools.” Specific HL screenings instruments identified in previous literature studies as the criterion or reference standard were also included in the search strategies, namely, the Test of Functional Health Literacy among Adults (TOFHLA), the Rapid Estimate of Adult Literacy in Medicine (REALM), and the Wide Range Achievement Test (WRAT). ²³ Additional relevant articles were searched for by manually checking the reference lists of eligible articles and review articles.

Eligibility criteria and study selection

A fourth‐year medical student (JC) and a senior health scientist (GH) independently assessed inclusion eligibility of the retrieved references. References were included if they: 1) were published full text and with an abstract in English; 2) evaluated one or more instruments aimed at screening patient's HL in the ED; and 3) reported data on one or a combination of the following outcomes: the instruments’ internal consistency, criterion validity (assessed by the correlation of the instrument with the short or extended version of the TOFHLA, the REALM, or the WRAT), diagnostic accuracy (i.e., its ability to discriminate between patients with and without limited HL), or feasibility. Conference abstracts and publications without original data were excluded from the analysis. After initial screening of the titles and abstracts, both reviewers read the full texts of included articles and screened these for eligibility. Discrepancies were discussed and taken to a third person (YS) if no agreement could be reached.

Data extraction

Data were extracted using a standardized form that assessed study characteristics (e.g., country, study setting, population, sample size), instrument description, reference methods and results. Data regarding internal consistency included Cronbach's alpha values. Data regarding criterion validity included correlation coefficients (Pearson's r, Spearman's rho, Kendall's tau depending on type of data). Data regarding diagnostic accuracy included: area under the curve (AUC) scores as the derived summary measure for diagnostic accuracy, sensitivity, and specificity scores and related 95% confidence intervals (CIs). Data regarding feasibility included: the mean total administration time (AT), the mean time on test (TOT), the proportion of administrations with interruptions (PI), and the mean length of interruptions (TOI) per test. Data were extracted by JC and reviewed for completeness and accuracy by GH. Discrepancies were resolved by discussion.

Assessment of study quality

Methodologic quality was assessed independently by JC and GH. Disagreements were resolved by consensus or by involving YS as required. The methodologic quality and applicability of diagnostic accuracy studies was assessed using the Quality Assessment of Diagnostic Accuracy Studies tool (QUADAS‐2). ²⁴ This tool assesses the risk of bias within four domains: patient selection, index test, reference standard, and flow and timing (Table S1). Risk of bias was assessed for each domain by answering signaling questions with "yes," "no," or "unclear" to assist judgments. Concerns regarding applicability were also determined for the first three domains. Risk of bias and applicability concerns per domain were rated as "high," "low," or "unclear." If, within one domain, all signaling questions were answered "yes" then risk of bias for that domain was judged "low." If one or more signaling questions were answered "no" then risk of bias was judged "high." Studies were overall judged "low risk of bias" or "low concerns regarding applicability" if risk of bias and applicability concerns were scored "low" on all domains relating to either bias or applicability. Studies were judged “at risk of bias” or as having “concerns regarding applicability” if a study was judged “high” or “unclear” on one or more domains. ²⁴ Box H of the COnsensus‐based Standards for the selection of health status Measurement INstruments (COSMIN) was used to assess the quality of studies reporting criterion validity. Overall quality was determined by taking the lowest rating of any item in the box (i.e., the “worst score counts” principle). ²⁵ Inter‐rater agreement was calculated for the scores on the QUADAS‐2 signaling questions combined and for the scores on the COSMIN checklist items by between‐group kappa agreement, using the assessments from each reviewer before resolution of disagreements. Publication bias was not assessed because of the small numbers of studies for any given instrument. Moreover, methods to detect publication bias in studies assessing diagnostic accuracy data are considered unreliable. ²⁶ , ²⁷

Data synthesis and analysis

Data were organized in tabular form to describe study characteristics and quality, instrument characteristics, comparators, and outcomes of interest. Descriptive statistics were used to summarize psychometric outcome data and compare them against set criteria. For internal consistency, we used Cronbach's alpha cutoffs: >0.9 excellent, >0.8 good, >0.7 acceptable, >0.6 questionable, >0.5 poor, and <0.5 unacceptable. ²⁸ For criterion validity, we used correlation coefficient cutoffs: >0.7 high, 0.5–0.7 moderate, and <0.5 low. ²⁹ For diagnostic accuracy, we used AUC score cutoffs: >0.8 good, 0.6–0.8 moderate, and <0.6 poor. ³⁰ Instruments were also categorized based on their mode of measurement into objective measurement of HL derived by one or more direct tests of skills and subjective measurement of HL by individuals’ self‐report of perceived skills. ¹⁹ Heterogeneity in clinical instruments and outcome reporting limited our ability to conduct a meta‐analysis. Instead, we conducted a descriptive analysis of the psychometric, diagnostic, and feasibility results of each study.

RESULTS

Search results

Our initial search identified 2,145 records. The additional search identified 231 records resulting in a total of 2,376 records. After exclusion of duplicates, 1,578 records were screened by title and abstract. Seventy‐one full‐text articles were retrieved and reviewed, of which 64 were excluded. Most excluded articles (n = 46) did not report data on our outcomes of interest. Other articles were excluded because instruments were not evaluated in the ED, a full‐text copy was not available and content turned out to be a conference abstract. No additional relevant articles were found from the reference lists of the articles that were reviewed in full‐text and from review articles. Consequently, the final set comprised seven unique published studies that underwent full‐text extraction (Figure 1).

FIGURE 1.

Flow chart of the study selection process

Study characteristics

Characteristics of the seven included studies are summarized in Tables 1 and 2. All studies were published between 2011 and 2020 and performed in the United States. The vast majority were single center, ³¹ , ³² , ³³ , ³⁴ , ³⁶ prospective observational cohort studies, ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁷ conducted in urban academic EDs. ³¹ , ³² , ³⁴ , ³⁶ One study was conducted in four urban academic EDs. ³⁷ Another study was performed in one pediatric ED. ³³ One study described a secondary analysis of prospectively collected cohort data from multiple EDs in the United States. ³⁵ Study participants consisted of non–critically ill patients, mostly (older) adults. One study focused on the caregivers of children aged 12 years and younger. ³³ All studies consisted of English‐speaking participants. Three studies also included Spanish‐speaking participants. ³³ , ³⁶ , ³⁷ Sample sizes varied from 202 to 2,770 participants. The included studies made efforts to minimize the impact of confounding effects on instruments’ validity and diagnostic accuracy, mostly by excluding patients with specific mental, cognitive or physical conditions, or impairments that are known to impede an accurate measurement of HL. Two studies described counterbalanced testing of instruments to reduce bias due to test fatigue. ³² , ³³ One study stratified instrument correlation scores by lower and higher educational level. ³³  Three studies evaluated the internal consistency of instruments. ³¹ , ³² , ³⁵ Five reported data on criterion validity. ³¹ , ³² , ³³ , ³⁴ , ³⁵ Two studies evaluated the diagnostic accuracy ³² , ³⁷ and the feasibility of instruments. ³² , ³⁶ Most of the included studies (n = 6; 86%) evaluated multiple instruments ³¹ , ³² , ³⁴ , ³⁵ , ³⁶ , ³⁷ or different instrument versions (short and extended). ³² , ³⁴ , ³⁵ , ³⁷

TABLE 1.

Characteristics of the included studies

First author, year	Setting (country)	Design	Population					Sampling	Excluded
				Size	Age (years), mean (±SD)	% Female	% Race
McNaughton, 2011 31	Single urban academic ED (United States)	Prospective observational cohort	Adults	207	46 a	55	68 W; 27 B; 6 O	Convenience	Critically ill; non‐English speakers
Carpenter, 2014 32	Single urban academic ED (United States)	Prospective observational cohort	Adults	435	45 (±16)	55	68 B; 31 W; <1 A; 1 O	Convenience	Critically ill; distressed; altered mental status; aphasia; mentally or visually impaired
Morrison, 2014 33	Single (sub)urban pediatric ED (United States)	Prospective observational cohort	Caregivers of nonchildren (≤12 years)	501	32 a	85	47 W; 37 B; 10 H; 5 O	Random	Non‐English or Spanish speakers; distressed child; child presented for maltreatment or non‐accidental trauma; <5th grade reading level
Kiechle, 2015 34	Single suburban ED (United States)	Prospective observational cohort	Adults	400	38 (±14) a	58	63 W; 30 B; 5 H; 3 O	Sequential block enrollment (based on education levels)	Critically ill; non‐English speakers; decisionally or visually impaired; intoxicated
McNaughton, 2015 35	Multiple EDs (United States)	Secondary analysis of six prospective study cohorts	Adults	207	NR	NR	NR	Convenience	NR
McGuinness, 2020 36	Single urban academic ED (United States)	Prospective observational cohort	Older adults (55–90 years)	104 b ; 98 c	68 (NR) b ; 69 (NR) c	51 b ; 52 c	NR	Convenience	Critically ill; non‐English or Spanish speakers; distressed; altered mental status, reading, speech or cognitive disability
Merchant, 2020 37	Four urban academic EDs (United States)	Prospective observational cohort	Adults (18–64 years)	2,770	44 (±12)	61	56 W; 43 B; 1 O	Random	Critically ill; non‐English or Spanish speakers; intoxicated, physically, mentally or cognitively impaired, <2nd grade reading level

Abbreviations: A, Asian; B, Black or Afro American; H, Hispanic or Latino; NR, not reported; O, other; W, White or Caucasian.

^a Median score.

^b Completed the NVS.

^c Completed the SAHL.

TABLE 2.

Tested instruments, outcomes of interest, and methodologic quality per study

First author, year	Tested instruments	Language	Outcomes				QUADAS‐2		COSMIN Box H
			IC	CV	DA	F	RB	AC
McNaughton, 2011 31	BHLS; SNS‐8	English	√	√	√		+	−	Fair
Carpenter, 2014 32	REALM‐R; NVS; SILS questions a ; BHLS	English	√	√	√	√	+	−	Fair
Morrison, 2014 33	NVS	English; Spanish		√			NA	NA	Fair
Kiechle, 2015 34	S‐TOFHLA; NVS; SILS questions a ; BHLS; REALM‐R; METER	English		√			NA	NA	Poor
McNaughton, 2015 35	SNS‐3; SNS‐8	English	√	√			NA	NA	Fair
McGuinness, 2020 36	NVS; SAHL	English; Spanish				√	NA	NA	NA
Merchant, 2020 37	SILS questions a ; BHLS	English; Spanish			√		−	−	NA

Abbreviations: BHLS, Brief Health Literacy Screen; CV, Criterion validity; COSMIN, COnsensus‐based Standards for the selection of health status Measurement Instruments; DA, diagnostic accuracy; F, feasibility; IC, internal consistency; METER, Medical Term Recognition Test; NA, not applicable; NVS, Newest Vital Sign; QUADAS‐2, Quality Assessment of Diagnostic Accuracy Studies; REALM‐R, Rapid Estimate of Adult Literacy in Medicine‐Revised; SNS, Short Numeracy Scale; SILS, Single Item Literacy Screener; S‐TOHFLA, Short Test of Functional Health Literacy among Adults. +, At risk of bias, concerns regarding applicability; −, low risk of bias, low concerns regarding applicability.

^a All three SILS questions: i.e., 1) "How often do you have someone (like a family member, friend, hospital or clinic worker, a caregiver, or anyone else) help you read materials given to you by the hospital, clinic, or your health care provider?" 2) "How confident are you in filling out medical forms by yourself?" 3) "How often do you have problems learning about your medical condition or health because of difficulty reading and understanding written information given to you by the hospital, clinic, or your health care provider?"

Study quality

The summarized methodologic quality of each study is presented in Table 2. Details on the study quality are provided in Tables S1 and S2. Inter‐rater agreement for the scores on the QUADAS‐2 signaling questions was high with a kappa score of 0.89. The percentage agreement between both raters for scores per risk of bias domain varied between 80 and 100. Inter‐rater agreement for the scores on the COSMIN checklist items was high with a kappa score of 0.84. None of the three diagnostic studies assessed with the QUADAS‐2 ³¹ , ³² , ³⁷ scored "low risk of bias" on all four domains. All three showed high risk of bias in the patient selection: i.e., patients were sampled consecutively ³¹ , ³² and inappropriate exclusions were not avoided because second‐grade reading level patients were excluded, which may have influenced accuracy findings. ³⁷ Only one study showed sufficient information to determine appropriate conduct and interpretation of the index tests. ³⁷ In all three studies the reference standard used may have introduced bias as the standards used are not the undisputable gold standard (i.e., estimates of test accuracy are not based on the assumption that the standard is 100% sensitive and specific disagreements between the reference standard and index test result from incorrect classification by the index test). Furthermore, in one study screeners collected scores on both the index tests and the reference standard, thereby increasing the risk of incorporation bias, which can also falsely increase sensitivity and specificity. ³² All three showed appropriate flow and timing of the index test and reference standard. Applicability concerns were considered low for all three studies. Of the five studies assessed with the COSMIN Box H, ³¹ , ³² , ³³ , ³⁴ , ³⁵ four scored "fair," ³¹ , ³² , ³³ , ³⁵ and one scored "poor." ³⁴

Instruments

Characteristics

Table 2 provides an overview of the studied HL instruments. In total, 11 unique instruments were evaluated on either internal consistency, criterion validity, diagnostic accuracy, and/or feasibility. Five instruments used direct tests to assess individuals’ HL skills (objective measurement) by letting them solve tasks dealing with print literacy, numeracy, or oral literacy. The Short Test of Functional Health Literacy in Adults (S‐TOFHLA) includes a condensed 36‐item version of the TOFHLA testing reading comprehension. ³⁴ The Rapid Estimate of Adult Literacy in Medicine‐Revised (REALM‐R) is a shortened version of the REALM, which tests individuals’ pronunciation of eight medical words (e.g., anemia and osteoporosis). ³² , ³⁴  The Newest Vital Sign (NVS) consists of a fictitious ice cream nutrition label that is handed to the patient, as the interviewer asks six accompanying questions to assess literacy and numeracy skills. ³² , ³⁴ , ³⁶ The Medical Term Recognition Test (METER) contains a list of 40 medical words mixed in with nonwords. The patient is asked to identify the real words. ³⁴ The Short Assessment of Health Literacy (SAHL) includes 18 interviewer‐administered items designed to assess patients’ ability to read and understand common medical terms. Each item contains a medical term printed in boldface and two association words (i.e., the key and the distracter). Correct answers are determined by both correct pronunciation and accurate association. ³⁶ , ³⁸

Six instruments used the elicitation of self‐reported perceived skills in print literacy and numeracy. The three Single Item Literacy Screener (SILS) consist of one question (5‐point Likert scale) to assess individuals’ self‐perceived 1) need for help in reading hospital materials, 2) confidence in filling out medical forms, and 3) difficulty understanding written information in trying to learn more about a medical condition. ³² , ³⁴ , ³⁷ The Brief Health Literacy Screen (BHLS) are the three SILS‐questions combined in one instrument. ³¹ , ³² , ³⁴ , ³⁷ The two Subjective Numeracy Scale (SNS) versions are measures of individuals’ perceived ability to perform various mathematical tasks and preference for the use of numerical versus prose information. The SNS‐8 consists of eight questions (6‐point Likert scale) asking individuals to assess their numeracy skills in different contexts and their preferences for the presentation of numerical and probabilistic information. ³¹ , ³⁵ The SNS‐3 is a condensed version of the SNS‐8 with two questions on numeracy skills and one on subject preference. ³⁵ Thresholds for detecting limited HL were provided for each of the studied instruments based on previously published scoring rules for the instrument. All instruments were administered in English. In three studies, the NVS, the SAHL, the BHLS, and the three SILS questions were also administered in Spanish. ³³ , ³⁶ , ³⁷

Internal consistency

Three instruments were tested on internal consistency (Table 3). The BHLS demonstrated acceptable internal consistency with Cronbach's alpha scores of 0.74 and 0.78. ³¹ , ³² The same applied to the SNS‐3 with a reported alpha of 0.78. ³⁵ The SNS‐8 showed good internal consistency with alpha scores of 0.82 and 0.83 reported in two studies by McNaughton et al. ³¹ , ³⁵

TABLE 3.

Characteristics and psychometric properties of studied HL measurement instruments, categorized by mode of measurement

Instrument	Constructs measured	Items	Cronbach's alpha	Correlation coefficient			Diagnostic accuracy for detecting LHL
				With S‐TOFHLA	With REALM‐R	With WRAT‐4	AUC (95% CI)	SE (95% CI)	SP (95% CI)
Objective measurement approach (n = 5)
S‐TOFHLA	Close‐type comprehension	36	NR	NR	0.56 [34]	NR	NR	NR	NR
REALM‐R	Recognition and pronunciation of medical words	8	NR	0.54 [32]; 0.56 [34]	NR	NR	0.80 (0.73–0.86) g [32]	80.8 (73.2–88.3) g [32]	61.7 (56.5–67.0) g [32]
NVS	Reading and comprehension of nutrition label	6	NR	0.60 [32]; 0.62 [34]; 0.45 [33]; 0.32 a [33]; 0.47 b [33]	0.57 [34]	NR	0.83 (0.78–0.87) g [32]	98.0 (93.1–99.8) g [32]	45.7 (40.3–51.3) g [32]
METER	Recognition of medical words	80	NR	0.53 [34]	0.73 [34]	NR	NR	NR	NR
SAHL	Reading and comprehension of medical words	18	NR	NR	NR	NR	NR	NR	NR
Subjective measurement approach (n = 6)
SILS‐help with reading	Screening question c	1	NA	0.42 [34]	0.38 [34]	NR	0.59 (0.56–0.62) h [37]; 0.58 (0.56–0.61) i [37]	42.7 (33.2–52.3) g [32]	85.5 (81.7–89.3) g [32]
SILS‐confident with forms	Screening question d	1	NA	0.39 [34]	0.40 [34]	NR	0.62 (0.59–0.65) h [37]; 0.60 (0.57–0.63) i [37]	54.8 (45.2–64.4) g [32]	80.4 (76.1–84.6) g [32]
SILS‐understanding information	Screening question e	1	NA	0.40 [34]	0.43 [34]	NR	0.63 (0.60–0.66) h [37]; 0.59 (0.56–0.62) i [37]	40.4 (31.0–49.8) g [32]	95.5 (92.2–97.7) g [32]
BHLS	Reading and comprehension of medical information	3	0.74 [31]; 0.78 [32]	0.33 [31]; 0.24–0.49 [32]; 0.41 [34]	0.26 [31]; 0.44 [34]	0.26 d [31]	0.74 (0.62–0.87) g [31]; 0.72 (0.62–0.81) j [31]; 0.77 (0.70–0.83) k [31]; 0.77 (0.70–0.83) g [32]; 0.66 (0.63–0.70) h [37]; 0.62 (0.59–0.64) i [37]	68.0 (59.0–77.1) g [32]	75.5 (70.8–80.1) g [32]
SNS‐8	Self‐reported numeracy abilities and preferences	8	0.82 [31]; 0.83 [35]	0.36 [31]; 0.40 [35]	0.36 [31]; 0.37 [35]	0.57 f [31]; 0.59 [35]	0.77 (0.70–0.82) k [31]	NR	NR
SNS‐3	Self‐reported numeracy abilities and preferences	3	0.78 [35]	0.38 [35]	0.38 [35]	0.59 [35]	NR	NR	NR

Abbreviations: AUC, area under the curve; BHLS, Brief Health Literacy Screen; HL, health literacy; METER, Medical Term Recognition Test; NA, not applicable; NVS, Newest Vital Sign; NR, not reported; SE, sensitivity; SAHL, Short Assessment of Health Literacy; SILS, Single Item Literacy Screener; SNS, Short Numeracy Scale; SP, specificity; S‐TOHFLA, Short Test of Functional Health Literacy among Adults; REALM‐R, Rapid Estimate of Adult Literacy in Medicine‐Revised; WRAT, Wide Range Achievement Test.

^a Group with higher educational attainment.

^b Group with lower educational attainment.

^c “How often do you have someone (like a family member, friend, hospital or clinic worker, a caregiver, or anyone else) help you read materials given to you by the hospital, clinic, or your health care provider?"

^d “How confident are you in filling out medical forms by yourself?"

^e “How often do you have problems learning about your medical condition or health because of difficulty reading and understanding written information given to you by the hospital, clinic, or your health care provider?"

^f 40‐item subtest of the WRAT‐4 to objectively measure mathematic skills.

Reference methods:

^g S‐TOFHLA.

^h SAHL‐English version.

ⁱ SAHL‐Spanish version.

^j REALM‐R.

^k WRAT‐4 mathematical subtest.

Criterion validity

Ten instruments were tested on validity against one or more reference standards (Table 3). Only the METER showed a high correlation with the REALM‐R (r = 0.73). ³⁴ This may be explained by the fact that both instruments use the same medical words in testing literacy. ³⁴ The METER showed moderate validity (r = 0.53) with the S‐TOFHLA as the reference standard. ³⁴ The S‐TOFHLA showed moderate validity against the REALM‐R as reference standard and vice versa. ³² , ³⁴ The validity of the NVS was poor to moderate with correlation coefficients varying from 0.45 to 0.62 against the S‐TOFHLA and the REALM‐R as reference standards. The SILS questions showed poor validity against three reference standards (i.e., the S‐TOFHLA, the REALM‐R, and the WRAT‐4 mathematical subtest), both individually (r = 0.38–0.43) ³⁴ and as questions combined in the BHLS (r = 0.24–0.49). ³¹ , ³² , ³⁴ For the SNS‐8 and SNS‐3, the correlations with S‐TOFHLA and REALM‐R were poor (r = 0.36–0.40). ³¹ , ³⁵ In contrast, both instruments showed moderate correlation with the WRAT‐4, which included a substantial items on numeracy like the SNS (r = 0.57 and r = 0.59). ³¹ , ³⁵

Diagnostic accuracy

Accuracy in detecting limited HL was tested for seven instruments (Table 3). The REALM‐R and NVS demonstrated good accuracy using the S‐TOFHLA as reference standard. AUC values for the REALM‐R and the NVS were 0.80 (95% CI = 0.73–0.86) and 0.83 (95% CI = 0.78–0.87), respectively. ³² The REALM‐R provides reasonable sensitivity and specificity for the ED setting with detecting 81% of patients with limited HL and correctly reporting 62% of patients with adequate HL. The NVS appears to be highly sensitive (98%), but less specific (46%). ³² The BHLS showed moderate diagnostic accuracy against various reference methods (i.e., the S‐TOFHLA, the REALM‐R, the WRAT‐4, and the SAHL) with AUC values ranging between 0.62 (95% CI 0.59–0.64) and 0.77 (95% CI = 0.70–0.83). ³¹ , ³² , ³⁷ Diagnostic accuracy of the three separate SILS questions, both in the English and in Spanish version, were poor when using the SAHL as the reference standard. AUC values ranged between 0.58 (95% CI = 0.56–0.61) and 0.63 (95% CI = 0.60–0.66). ³⁷ Finally, the SNS‐8 demonstrated moderate accuracy in detecting limited HL against the WRAT‐4 with an AUC of 0.77 (95% CI = 0.70–0.82). ³¹

Feasibility

Four instruments were evaluated on feasibility (Table 4). The REALM‐R demonstrated the shortest mean administration time (1.06 min). ³² The NVS and the SAHL show similar ease of use with regard to time of administration (means range between 3.31 and 3.57 min). ³² , ³⁶ Interruptions during administration were minimal for all three instruments (<6.1%), particularly for the REALM‐R (0.5%). Compared to the other tested instruments, the S‐TOFHLA had the longest administration time (mean = 6.55 min) and the highest percentage (13.1) of interruptions during test performance. ³²

TABLE 4.

HL measurement instruments tested on feasibility

Instrument	First author (year)	Sample (n)	Mean time of administration (min) a	Time on test (min/s) b	% Interrupted	Mean time of interruptions (min/s)
S‐TOFHLA	Carpenter (2014) 32	434	6.55 min	6.07 min	13.1	3.77 min
NVS	Carpenter (2014) 32	428	3.31 min	3.13 min	6.0	2.85 min
	McGuinness (2020) 36	104	3.57 min c	NR	4.8	5.54 s
SAHL	McGuinness (2020) 36	98	3.45 min c	NR	6.1	4.96 s
REALM‐R	Carpenter (2014) 32	433	1.06 min	1.06 min	0.5	1.50 min

Abbreviations: HL, health literacy; NVS, Newest Vital Sign; REALM‐R, Rapid Estimate of Adult Literacy in Medicine‐Revised; S‐TOFHLA, Short Test of Functional Health Literacy in Adults; SAHL, Short Assessment of Health Literacy.

^a Total time for the test.

^b Total time – interrupted time.

^c Originally reported in seconds and converted to minutes.

DISCUSSION

To our knowledge, this is the first comprehensive review of HL measurement instruments tested in the ED. Although a substantial number of instruments have been recently developed and tested in various health care settings and across different populations, ¹⁸ , ¹⁹ , ²⁰ , ²⁴ the evaluation of such instruments in the ED setting remains scarce. Only seven studies, all performed in the United States, fulfilled our inclusion criteria. Nevertheless, our findings provide a valuable overview that could help ED professionals in selecting the most appropriate HL measurement instrument for clinical and scientific purposes. ¹

With regard to psychometrics, the following conclusions can be drawn. First, the BHLS and the SNS (short and extended) have good evidence for reliability in measuring HL in the ED as they demonstrated acceptable and good internal consistency across studies. The internal reliability of a HL measure may be high in other nonacute health care settings, but this outcome could be different when measuring patient's HL level under different circumstances like in the ED. Second, none of the studied HL instruments performed consistently well on criterion validity. Most instruments demonstrated poor or moderate validity against different reference standards. The question may arise whether the true HL status of ED patients can be captured well enough by a relatively simple screening tool often measuring individuals’ self‐perceived HL abilities that are sensitive for bias. ¹⁹ , ³⁹ Another explanation may be found in inappropriate validation criteria used by the studies, because selected reference methods may have measured a different domain of the multidimensional concept of HL than the studied HL instrument itself (e.g., reading ability and word comprehension versus numeracy skills). ¹⁹ , ⁴⁰ Third, a limited number of instruments were tested on accuracy in detecting limited HL among ED visitors with mixed results. The REALM‐R and the NVS showed good diagnostic accuracy (especially high sensitivity) against one reference standard. However, these performance outcomes originate from a single study performed in one ED with a high risk of bias and, therefore, should be interpreted with caution. This also applies to the SNS‐8 showing moderate accuracy. The BHLS showed moderate accuracy in detecting limited HL and better performance compared to each of the poorly performing SILS questions. Evidence for the diagnostic performance of the BHLS is strengthened by the fact that accuracy was tested against different reference standards across multiple studies including > 3,000 adult participants. One of these studies was a multicenter study with low risk of bias.

Apart from psychometric properties, previous reviews dealing with HL measurement emphasized that the choice to use a particular HL instrument also depends on practical considerations related to administering the instrument and the advantages and disadvantages of using an objective versus subjective measurement mode. ¹⁸ , ¹⁹ Although the BHLS and the SNS‐8 showed moderate accuracy in detecting limited HL and were not tested on feasibility in the ED, both measures seem to have several practical benefits. First, they appear to incur minimal administration time, which is important for ED professionals with limited available time for screening HL levels. Second, both measurements are based on self‐reported answers that do not require in‐person testing by trained staff using prepared materials. ²³ Third, the self‐perceived assessment of HL by both instruments involves less cognitive effort and a reduced risk for shame or stigma compared to objective tests like the REALM‐R and the NVS. ¹⁹ , ⁴¹ , ⁴² In contrast, the major benefit of the REALM‐R and the NVS is their direct and objective measurement of the individuals’ skill based on empirically grounded data. ¹⁹ Unlike the BHLS and the SNS‐8, these objective tests are not prone to bias associated with self‐reports (e.g., patients overestimating their abilities due to perceived social desirability). ³¹ Moreover, both the REALM‐R and the NVS demonstrated short administration times in the ED that correspond with previous evaluations of the instruments in other health care settings. ²³ As previous reviews on HL measurement instruments already argued for the general population, ¹⁹ the choice for a particular instrument in the ED finally depends on the specific HL domain(s) that one wants to measure and respond to in the ED. ¹⁹ The BHLS, the REALM‐R, and the NVS share a common focus on measuring individuals’ reading ability and comprehension of health information while the SNS‐8 specifically assesses numeracy skills and preferences.

This review may guide clinicians and policy‐makers in their efforts to identify patients with limited HL as a primary step to improve patient–provider communication in the ED and ultimately health outcomes for this vulnerable population. Once limited HL is identified, tailored strategies can be used to improve information comprehension and to facilitate shared decision making in the ED as highlighted in the proceedings of the 2016 Academic Emergency Medicine Consensus Conference on Shared Decision Making in the Emergency Department. ⁴³ Although strong evidence for one or more accurate and feasible HL instruments remains limited and findings only apply for English‐ and Spanish‐speaking populations, our systematic review informs ED clinicians and policy‐makers by presenting currently available instruments, along with their advantages and disadvantages, that can be used for the assessment of HL in daily practice. They should take these above‐mentioned considerations into account when selecting a measurement instrument as a mean to overcome the barriers to health care delivery and the negative outcomes associated with limited HL. Various interventions, albeit with limited evidence base in the ED setting, are available with the potential to overcome such barriers throughout a patient's ED visit. Interventions at the clinical–patient level include clear communication (e.g., slow down, use of plain language, lowering the level of detail, presenting essential information by itself or first), confirmation of understanding (e.g., teach‐back method), and reinforcement (e.g., combining verbal information with illustrations). ¹⁵ , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ Interventions at the system–patient level include clear educational materials at the appropriate literacy level, visual aids, clear medication labeling, and shame‐free clinical environments. ¹⁵ , ⁴⁴ , ⁴⁶

LIMITATIONS

Our systematic review has several limitations. First, the heterogeneity of studied instruments measuring different domains of the multidimensional concept of HL and the variety of reference standards used make it difficult to compare instruments’ validity and diagnostic performance in the ED. Second, the evidence provided on criterion validity and diagnostic performance is based on a limited number of studies with poor methodologic quality and should therefore be interpreted with caution. Third, all included studies were performed in the United States and involved instruments that were mainly tested in a single urban ED among English‐speaking patients. These aspects may limit the external validity of instruments and the extrapolation of estimates of validity and diagnostic accuracy to ED settings elsewhere serving populations with dissimilar sociodemographic characteristics (e.g., non–English speakers). Moreover, in most of the included studies patients were not formally screened on potentially confounding characteristics (e.g., dementia or undue distress). Therefore, mild cognitive dysfunctions may have been undetected and influenced the findings. ⁴⁸ Finally, as with any systematic review, selection bias is possible. Although we conducted an extensive search of electronic literature, the search was limited by peer‐reviewed full‐text publications with an abstract in English language only.

CONCLUSION

This review highlights the existence of several short and simple instruments that appear valid in measuring health literacy levels and accurate in detecting limited health literacy among ED patients. These instruments differ in the mode of measurement, each with its practical advantages and disadvantages, and in the measurement of health literacy domains. Unfortunately, the low number of included studies and their methodologic limitations hinder the demonstration of robust evidence supporting one or more instruments. In the context of the widespread problem of limited health literacy and the ED as a first point of contact for many patients where they receive important health information, our findings call for more research to develop a robust evidence base for rapid and easy‐to‐administer health literacy measurement instruments that are psychometrically and diagnostically sound regardless of language spoken. ³⁷ Future research may benefit from the following considerations. First, a better alignment of instruments with definitions of health literacy and tested against a corresponding criterion standard would facilitate a more meaningful comparison of instruments. Second, future testing of instruments following the Standards for Reporting Diagnostic Accuracy (STARD) ⁴⁹ and in accordance with the QUADAS‐2 criteria would improve determining instrument's diagnostic accuracy with limited risk of bias. Third, the use of larger study samples including non‐English speakers across multiple and different types of EDs, also outside the United States, could improve the instruments’ external validity. Findings of this systematic review should encourage and guide health care professionals and scientists further in their efforts to detect ED patients with limited health literacy and to facilitate their involvement in and receipt of optimal health care.

CONFLICT OF INTEREST

The authors have no potential conflicts to disclose.

AUTHOR CONTRIBUTIONS

Gijs Hesselink and Yvonne Schoon conceived and designed the study. Gijs Hesselink and Joey Cheng were responsible for data collection. Gijs Hesselink and Joey Cheng analyzed and interpreted the data. Gijs Hesselink and Joey Cheng drafted the paper, which was critically revised for important intellectual content by Yvonne Schoon. Gijs Hesselink takes responsibility for the paper as a whole.

Supporting information

Data Supplement S1. Supplemental material.

Data Supplement S2. Supplemental material.

Data Supplement S3. Supplemental material.

Hesselink G, Cheng J, Schoon Y. A systematic review of instruments to measure health literacy of patients in emergency departments. Acad Emerg Med. 2022;29:890–901. doi: 10.1111/acem.14428