Health literacy screening of geriatric monolingual Spanish-speaking patients using single-item literacy screening questions and education

Abstract

Objective

We describe the performance of Single Item Literacy Screener (SILS) questions, and educational attainment, as screening for inadequate health literacy (IHL) in older monolingual Spanish speakers.

Design

We used a cross-sectional design, interviewing participants once at the time of their arrival for a clinic appointment.

Setting

We conducted this study in primary care and geriatrics clinics in an urban US safety-net hospital.

Method

We conducted in-person interviews with older monolingual Spanish-speaking diabetes patients, comparing responses to three SILS questions, and education, to shortened Test of Functional Health Literacy in Adults (sTOFHLA) scores. We calculated sensitivities, specificities and areas under the receiving–operator characteristic (AUROC) curve.

Results

We interviewed 160 patients – 134 (84%) had IHL by sTOFHLA scores. The best performing SILS question, ‘How confident are you filling out medical forms by yourself?’ had an AUROC curve of 0.82 (95% CI 0.75–0.89). Using this question’s most stringent cut-off, sensitivity was 0.93 (95% CI 0.89–0.97); specificity was 0.27 (95% CI 0.20–0.34). The other two SILS questions had AUROC curves less than 0.50. The educational achievement AUROC curve was 0.88 (95% CI 0.78–0.97); using an education cut-off of six years or less had a specificity to 0.81 (95% CI 0.75–0.87) and sensitivity of 0.83 (95% CI 0.77–0.89).

Conclusion

Clinicians and investigators considering using single items as screeners for IHL in older US monolingual Spanish speakers should either use the ‘confidence with forms’ SILS, being aware of its specificity limitations, or a single question assessing educational achievement.

Introduction

Older Spanish speakers are the fastest growing demographic group within the US. Inadequate health literacy (IHL), shown in multiple studies to be associated with poor health outcomes,¹ is especially prevalent among older Spanish speakers. Among patients aged 60 years or more tested in their preferred language at two public hospitals, 77% of Spanish speakers had IHL, compared to 48% of English speakers.²

In order to target interventions mitigating the relationship between IHL and poor health outcomes among older US Spanish speakers, we need clinically useful screening tools to identify those with IHL. Although the Test of Functional Health Literacy in Adults (TOFHLA), generally considered the ‘gold standard’ for measuring health literacy, is useful for research, even its shortened form is too time consuming for use in time-pressured clinical settings. Further, it cannot be used in telephone surveys.³

To address this problem, investigators have developed and tested single questions to discern patients having a higher probability of having IHL.^3–6 Chew et al.^3,4 demonstrated that three Single Item Literacy Screening (SILS) questions are ‘best performers’ among English speakers. These questions are: (1) ‘How confident are you filling out medical forms by yourself?’ (extremely, quite a bit, somewhat, a little bit, not at all); (2) ‘How often do you have someone help you read hospital materials?’ (always, often, sometimes, occasionally, never); and (3) ‘How often do you have problems learning about your medical condition because of difficulty understanding written information?’ (always, often, sometimes, occasionally, never).^3,4

These SILS questions, however, have had limited testing in non-English speakers and there may be important cultural and linguistic differences affecting question interpretation and response.⁷ We found only a single study exploring potential differences between English and Spanish speakers with respect to the performance of the SILS questions. This study found the specificity of the SILS questions to be consistently lower in US Spanish speakers compared to English speakers.⁸ Before using these questions in clinical settings or research studies, we need to further determine their accuracy in identifying monolingual Spanish speakers with IHL. This paper describes the performance of these three SILS questions in older monolingual Spanish speakers, and compares their performance to using educational attainment as screening for IHL.

Methods

Participants and recruitment

We embedded this study into a screening protocol of monolingual Spanish-speaking patients with diabetes conducted at the Geriatrics and Internal Medicine clinics of Los Angeles County + University of Southern California (LAC+USC), a large county-funded safety-net hospital. We included patients who were age 65 years or more, had diabetes for at least one year, were patients in LAC+USC Geriatrics or Internal Medicine clinics for at least one year, spoke Spanish, spoke English less than ‘well’, had no evidence of psychosis or significant cognitive or physical impairment, had a corrected visual acuity of 20/100 and arrived for an appointment at the study clinics between 14 September 2009 and 15 July 2010. A trained, fully bi-lingual, bi-cultural research assistant (RA) implemented all study procedures. All study materials were translated to Spanish and back translated to English by a certified translator. The Human Subjects Committee of the RAND Corporation approved the study protocol.

We generated lists of patients with upcoming clinical appointments and determined, by electronic medical record (EMR) review, age, gender, primary language, history of and duration of diabetes, diagnosis of psychosis or moderate or severe cognitive impairment and time as a patient in the study clinics. At the time of their scheduled clinic visit, the RA approached patients meeting these study criteria and invited them to participate. We provided a sports bottle and $5 to those who accepted this invitation.

Procedures and measures

After the potential participant provided informed consent, the RA conducted a one-to-one interview to assess English proficiency and physical, cognitive and visual impairment. We determined English proficiency by self-report (‘Would you say you speak English very well, well, not well, or not at all?’). We assessed for physical impairment by asking the patient if s/he needed assistance in any of the five basic activities of daily living (ADLs) and undiagnosed cognitive impairment using the six-item screening tool developed by Callahan et al.⁹ We measured corrected visual acuity using a hand-held, number-based Snellen card; we provided reading glasses to those who needed them. In patients who could not read numbers, we used the card’s low-literacy symbol equivalents. Those who reported speaking English less than ‘well’, did not need assistance in any ADLs, scored four or better on the Callahan’s tool and had a corrected visual acuity of 20/100 or better were then asked to proceed to the literacy assessment. Those who did not meet one or more of these eligibility criteria were thanked, keeping the sports bottle and $5 in appreciation for their time.

We began the literacy assessment by asking the three SILS questions in random order. Then, we asked participants to read five words translated to Spanish from the Rapid Evaluation of Adult Literacy in Medicine (REALM) (Eye, Flu, Pill, Meals, Nerves). Those who could read at least one of the five words were then asked to complete the Spanish version of the 14-point font shortened Test of Functional Health Literacy in Adults (sTOFHLA). The prose section of the sTOFHLA consists of two passages of text, one giving instructions for getting an x-ray, another on procedures regarding Medicaid. In each sentence, one or two words have been omitted. For each omission, respondents choose from a list of four words the one that best ‘fits’ the sentence.¹⁰ For example, the initial sentence, in English, is ‘Your doctor has sent you to have a _____ x-ray’ and respondents choose between the words stomach, diabetes, stitches and germs as being the one that should go in the blank space.¹¹ After the literacy assessment we asked participants to self-report their educational achievement with the question, ‘How many years of schooling did you complete?’.

Analysis

As is standard, we used a sTOFHLA prose score of equal to or less than 16, out of a range of 0–36, to classify participants as having IHL. Participants who were unable to read at least one of the five REALM words, or if they indicated that they were unable to read the sTOFHLA when it was handed to them, were assigned a sTOFHLA score of zero. We calculated sensitivities and specificities at each potential cut-off point for each SILS question, as well as for educational achievement, compared to the sTOFHLA. Then, in order to obtain an overall assessment of the performance for each cut-off point or criterion, we plotted a receiving–operator characteristic (ROC) curve¹² and calculated the areas under the receiving–operator characteristic (AUROC) curve, using maximum likelihood estimation and assuming an underlying binormal distribution. In a ROC curve the true positive fraction (sensitivity) is plotted in function of the false positive fraction (1-specificity) for different cut-off points. Each point on the ROC curve represents a specific decision threshold. A test with perfect discrimination (no overlap in the two distributions) has a ROC plot that passes through the upper-left corner (100% sensitivity, 100% specificity). Therefore, the closer the ROC plot is to the upper-left corner, the higher the overall accuracy of the test.¹³ We also combined the questions into two- or three-question scales, with and without accounting for education level, calculated their AUROC curves, and compared them, using an alpha of 0.05, to the individual items.¹² All analyses were performed using Stata 10.0.

Results

As depicted in Figure 1, we identified 366 patients who met our EMR review criteria, 315 of whom arrived for an appointment during our study period. Fifty-one of those identified by EMRs did not arrive for appointment and 93 arrived when the RA was not available. Thus, 224 (61%) were invited to participate and 220 (98%) agreed. Of these, 60 (73%) spoke English well, had significant physical impairment, evidence of cognitive impairment or corrected visual acuity less than 20/100, and thus were ineligible. Our final sample was thus 160 participants.

Figure 1.

Flow of potential participants through enrolment. DM: diabetes; EMR: electronic medical record.

Table 1 shows our sample demographics. Participant ages ranged from 65 to 89 years, mean age was 72.0 (sd.5.7); 71% were female. Educational levels were, in general, low with 41 (26%) of participants reporting having never attended school, 99 (62%) had one to six years of schooling and 6 (4%) had 12 or more years of education. By the prose sections of the sTOFHLA, 84% had IHL. Thirty-one (19%) had a sTOFHLA score of zero, and 92 (58%) were able to answer less than one-fourth of the questions correct.

Table 1.

Sample demographics and literacy

Mean age, yrs (SD)	72.0 (5.7)
Female, #(%)	115 (71%)
Education, #(%)
0 years	41 (26%)
1–6 years	99 (62%)
7–11 years	14 (9%)
12+ years	6 (4%)
Inadequate health literacy (sTOFHLA ≤ 16), #(%)	134 (84%)

sTOFHLA: shortened Test of Functional Health Literacy in Adults.

Table 2 shows the sensitivities and specificities at each cut-off point, and AUROC curves, of each SILS question and educational attainment, for detecting IHL. ROC curve plots are shown in Figure 2. The best performing question, ‘How confident are you filling out medical forms by yourself?’ had an AUROC curve of 0.82 (95% CI 0.75–0.89). Using the most stringent cut-off point of any answer less than ‘extremely confident’ as a positive screener had a sensitivity of 0.93 (95% CI 0.89–0.97) for detecting IHL; however, specificity was 0.27 (95% CI 0.20–0.34). The specificity improved to 0.65 (95% CI 0.58–0.73) at the next cut-off point, any answer less than ‘quite a bit confident’, but its sensitivity point estimate was less at 0.75 (95% CI 0.69–0.82). The questions, ‘How often do you have problems learning about your medical condition because of difficulty understanding written information?’ and ‘How often do you have someone help you read hospital materials?’ had AUROC curves of less than 0.50.

Table 2.

Sensitivities, specificities and areas under the receiving–operator characteristic (AUROC) curves of Single Item Literacy Screener (SILS) questions and educational achievement

Cut-off point	Sensitivity-point estimate(95% CI)	Specificity-point estimate(95% CI)	AUROC curves point estimate(95% CI)
SILS question: ‘How confident are you filling out medical forms by yourself?’(‘¿Qué tan seguro/a se siente al tener que llenar formularios usted mismo/a?’)
< Extremely	1.00	0	0.83 (0.75–0.90)
≤ Quite a bit	0.94 (0.91–0.98)	0.27 (0.20–0.34)
≤ Somewhat	0.76 (0.70–0.83)	0.65 (0.58–0.73)
≤ A little bit	0.68 (0.61–0.75)	0.84 (0.79–0.90)
≤ Not at all	0.58 (0.51–0.66)	0.96 (0.93–0.99)
SILS question: ‘How often do you have problems learning about your medical condition because of difficulty understanding written information?’(‘¿Qué tan seguido tiene problemas aprendiendo acerca de su condición médica por causa de la dificultad a entender la información por escrito?’)
≥ Never	1.00	0	0.24 (0.13–0.35)
≥ Occasionally	0.67 (0.60–0.74)	0.04 (0.01–0.07)
≥ Sometimes	0.64 (0.58–0.72)	0.08 (0.04–0.12)
≥ Often	0.47 (0.39–0.55)	0.34 (0.27–0.42)
≥ Always	0.31 (0.23–0.38)	0.42 (0.35–0.50)
SILS question: ‘How often do you have someone help you read hospital materials?’(¿’Qué tan seguido tiene a alguien que le ayude a leer los materiales del hospital?’)
≥ Never	1.00	0	0.32 (0.20–0.43)
≥ Occasionally	0.54 (0.47–0.62)	0.08 (0.04–0.12)
≥ Sometimes	0.52 (0.45–0.68)	0.08 (0.04–0.12)
≥ Often	0.40 (0.33–0.48)	0.27 (0.20–0.34)
≥ Always	0.27 (0.20–0.34)	0.35 (0.27–0.42)
Education level: ‘How many years of schooling did you complete?’(‘¿Cuántos años de estudio completó?’)
12 or more years	1.00	0	0.88 (0.78–0.97)
≤11 years	0.99 (0.97–1.00)	0.15 (0.10–0.21)
≤6 years	0.93 (0.89–0.97)	0.42 (0.35–0.50)
≤0 years	0.31(0.23–0.38)	1.00

Figure 2.

Receiving–operator curves for each Single Item Literacy Screener question.

Educational achievement, using a cut-off of any number of years less than 12, was similarly sensitive (0.99, 95% CI 0.97–1.00), but non-specific (0.15, 95% CI 0.10–0.21) for detecting IHL. However, using a cut-off of less than or equal to six years of education improved specificity to 0.81 (95% CI 0.75–0.87) with a sensitivity of 0.83 (95% CI 0.77–0.89). The AUROC curve for educational achievement was 0.88 (95% CI 0.78–0.97). There was no statistically significant difference detected between the education and ‘confidence with forms’ AUROC curves.

The AUROC curve for combination of the SILS confidence question and education was not significantly different from the curves for the two individual items. The AUROC curves for the other scales combining the SILS with or without education were significantly inferior (p < 0.05) to them. (Data not shown.)

Discussion

Summary and practice implications

In summary, of the three SILS questions tested, the question, ‘How confident are you filling out medical forms by yourself?’ performs best in detecting IHL among older US patients who are monolingual Spanish speakers. However, although its sensitivity, when using its highest cut-off point is high, its specificity is relatively low. The other two SILS questions have AUROC curves less than 0.50, which indicates that they are not useful for screening older monolingual Spanish-speaking patients for IHL. Years of education using a cut-off point of six years performs well in this population and may be more specific than the ‘confidence with forms’ SILS question for detecting IHL. Combining the ‘confidence with forms’ SILS question with education does not improve performance over either item individually.

Although the finding that the ‘confidence with forms’ SILS question performs the best is similar to Chew et al.’s³ findings among younger English-speaking patients, the specificity of this question, in our study sample, appears to be inferior to that found by Chew et al.³ This replicates the findings of Sarkar et al.⁸ that the specificity of the SILS questions is lower in US Spanish speakers compared to English speakers. Further, our finding that the other two SILS questions have AUROC curves less than 0.50 differs from those of Chew et al.³ and Sarkar et al.⁸ who found that, while these questions had inferior AUROC curves to the ‘confidence with forms’ question, they were still above 0.50. Our population differed from that of Sarkar et al.⁸ in that we only included monolingual Spanish speakers. This difference suggests that while these items may perform acceptably in English-speaking and bi-lingual patients, they should not be used in older US monolingual Spanish-speaking patients to identify IHL.

When using the most stringent cut-off point for the question, ‘How confident are you filling out medical forms by yourself?’, sensitivity is high. Fewer than one out of 10 patients with IHL will be ‘missed’ with this question. However, with its low specificity, more than seven out of 10 patients with adequate or marginal health literacy will be misclassified. This misclassification may be acceptable if the question is being used as an initial screen, and a more specific confirmatory test (such as administering the sTOFHLA) is subsequently being administered to those who test positive. However, if the question is being used in isolation, the group identified as having IHL will also include a large number of people who do not have IHL, the proportion of which will be determined by the prevalence of IHL in the sampled population. If this procedure is being done in order to target an intervention for those with IHL, potentially scarce resources will be misappropriated. If the question is part of a telephone or large sample survey, analyses will be less likely to find differences between those who screen positive and negative for IHL based on this question and cut-off point.

Our findings further suggest that, in older US monolingual Spanish speakers with low educational achievement, using the ‘confidence with forms’ SILS question is no better than using a simple measure of educational achievement to identify persons having a higher likelihood of having IHL. In fact, although there is no statistically significant difference demonstrated, in our sample the point estimate for the AUROC curve for educational achievement is greater than that of the ‘confidence with forms’ question. This finding contrasts with Chew et al.’s⁴ testing of these SILS questions against educational achievement in VA patients where the trend was in the opposite direction. This contrast is explained by the lower specificity of the ‘confidence in forms’ SILS question found in our sample compared to Chew et al.’s.⁴

The low specificity observed for all three SILS questions tested may be the result of our questions not asking patients to distinguish between Spanish and English versions of forms or other written materials they may receive and their ability to read them. Although they are receiving healthcare in an environment where written materials are almost always translated into Spanish, for some, their answers may have reflected their difficulties when encountering written materials in English, not Spanish. Future studies should explore and test questions that specifically assess patients’ confidence with forms or written materials in their preferred language.

Limitations

There are other limitations to this study. Most notably, the sample was collected from a single site – an urban public hospital in Los Angeles – with very high prevalences of illiteracy and very low health literacy. Thus, results may not be generalizable to Spanish speakers in other settings and geographic locations. A second limitation is that we tested only the three questions that Chew et al.⁴ found to be ‘best performers’ in English speakers; other questions that were developed but did not perform well in English speakers were not tested. Future work might explore the performance of other questions in Spanish speakers.

Conclusions

This study makes a very important finding that the SILS question ‘How confident are you filling out medical forms by yourself?’ is sensitive, but not specific, for screening for IHL in older US monolingual Spanish speakers with low educational levels. In those with low educational levels, using educational achievement appears to be more specific, with preserved sensitivity. These performance characteristics should be taken into account when conducting literacy-related research, or implementing interventions to address literacy, in older monolingual Spanish speakers. More broadly, our finding that the SILS ‘confidence’ question appears to be less specific in our population than in English-speaking patients, suggests that the performance of SILS questions differs between populations with various linguistic and demographic characteristics. Therefore, this and other literacy screening items should be tested in other diverse populations. In the meantime, investigators and clinicians considering the use of single items as screeners for IHL in older monolingual Spanish speakers with low education levels should either use the ‘confidence with forms’ SILS, being aware of its specificity limitations, or a question assessing educational achievement.