Paper self-administered questionnaires (SAQs) are a standard interviewing technique for surveys of sensitive topics. Although SAQs afford respondents greater privacy than face-to-face interviews, they have several limitations. First, they require literate respondents. Second, they require forms literacy—the ability to select consistent responses, implement general survey instructions, and correctly follow branching or skip instructions. Survey respondents may have difficulty following skip instructions if they are not forms literate.
This brief report examines the relationship between assessed levels of medical literacy, respondent characteristics, and the quality of measurements made in the 1997/98 Baltimore Sexually Transmitted Disease (STD) and Behavior Survey.
METHODS
Baltimore STD and Behavior Survey
The Baltimore STD and Behavior Survey (BSBS) collected data on sensitive health behaviors, including alcohol and drug use, sexual behaviors, and sexually transmitted diseases, among a probability sample of 1014 adults aged 18 to 45 years residing in Baltimore, Md. For details on sampling design and survey procedures, see Turner et al.1,2 Participants were randomly assigned to 1 of 2 interview modes, audio computer-assisted selfinterview (ACASI) or computer-assisted personal interview (CAPI). Survey questions contained branching or skip patterns—conditional paths through the questionnaire based on answers to specific questions. Participants assigned to the ACASI mode completed the entire questionnaire on a laptop computer. Branching patterns in ACASI were automated, removing the need for the respondent to follow skip instructions. Participants assigned to CAPI were administered the majority of questions by a trained interviewer but completed 2 paper SAQs for the more sensitive behaviors.
Rapid Estimate of Adult Literacy in Medicine
Following completion of the questionnaire, interviewers administered the Rapid Estimate of Adult Literacy in Medicine (REALM) to all participants. The REALM was designed for use in public health settings to identify patients who may need special attention with health care instructions because of low literacy.3 The REALM measures a respondent’s ability to read and correctly pronounce 66 common medical terms. The total number of correctly pronounced words measures each respondent’s REALM score. Scores are collapsed into 4 reading grade range estimates: grade 3 and below, grades 4 through 6, grades 7 through 8, and grade 9 and above.
Paper Self-Administered Questionnaires
The first SAQ contained 10 questions about alcohol and illicit drug use. Respondents who indicated that they had used alcohol were asked 4 CAGE scale questions on alcohol-related problems.4 Respondents indicating that they had never used alcohol were instructed to skip to questions that assessed illicit drug use. The second SAQ contained 12 questions about same-sex attraction and sexual contact, masturbation, forced sexual intercourse, and paid sexual intercourse.5,6 This form also contained skip patterns based on whether or not the respondent had engaged in a particular behavior.
Types of Errors
We identified 3 types of errors made in completing the SAQs. Skip errors included instances in which the respondent did not properly follow the printed skip instructions. Although this type of error may indicate low forms literacy or simple inattention to instructions, it does not suggest that the respondent was unable to comprehend the questions. A second category, logically inconsistent answers, suggests that the respondent was not reading the question and was merely circling an answer at random, or that the respondent did not understand the question. The final category, other errors, included circling more than 1 answer, writing in answers not offered as responses, or writing that the question did not apply.
RESULTS
Of the 1014 adults who completed the survey, 992 also completed the REALM instrument. Table 1 ▶ presents population-based estimates of medical literacy for the adult population of Baltimore.7 Five percent of adults are estimated to have REALM-assessed literacy scored at the level of grade 3 or below, 6.6% in the grades 4 through 6 range, 16.3% in the grades 7 through 8 range, and 72.1% at the level of grade 9 or higher. Estimates vary significantly by race, self-reported education, income, and sex.
TABLE 1.
—Medical Literacy as Assessed by the Rapid Estimate of Adult Literacy in Medicine (REALM) in the 1997/98 Baltimore STD and Behavior Survey
| Grade Range Estimatesa | ||||||
| Characteristics | n (unweighted) | ≤ Grade 3 (n = 50), % | Grades 4–6 (n = 65), % | Grades 7–8 (n = 162), % | ≥ Grade 9 (n = 715), % | Pb |
| Totals | 992 | 5.0 | 6.6 | 16.3 | 72.1 | |
| Race/ethnicity | ||||||
| African American | 562 | 8.4 | 7.3 | 21.2 | 63.1 | < .001 |
| Non–African American | 415 | 2.5 | 5.1 | 11.0 | 81.4 | |
| Sex | ||||||
| Male | 406 | 8.5 | 6.8 | 18.2 | 66.4 | < .01 |
| Female | 586 | 4.2 | 6.1 | 17.3 | 72.4 | |
| Age, y | ||||||
| 18–25 | 266 | 7.1 | 4.6 | 21.5 | 66.9 | .822 |
| 26–35 | 465 | 6.1 | 7.0 | 16.6 | 70.3 | |
| 36–45 | 248 | 6.3 | 7.5 | 16.2 | 70.1 | |
| Self-reported education | ||||||
| Some high school or less | 221 | 15.6 | 15.0 | 30.4 | 38.9 | < .001 |
| Finished high school or equivalent | 297 | 5.7 | 7.2 | 22.6 | 64.5 | |
| Some college or 2-year degree | 285 | 3.7 | 3.4 | 10.6 | 82.3 | |
| Finished college or postgraduate degree | 173 | 0.4 | 0.5 | 5.0 | 94.2 | |
| Self-reported income, $ | ||||||
| ≤ 5000 | 109 | 18.4 | 14.6 | 19.8 | 47.2 | < .001 |
| 5000–10 000 | 101 | 9.6 | 14.9 | 24.4 | 51.1 | |
| 10 001–20 000 | 175 | 6.5 | 4.1 | 29.9 | 59.5 | |
| 20 001–30 000 | 187 | 2.0 | 9.1 | 14.7 | 74.2 | |
| 30 001–50 000 | 214 | 2.3 | 3.8 | 15.4 | 78.5 | |
| ≥ 50 000 | 143 | 2.0 | 1.5 | 4.3 | 92.2 | |
Notes. Sample ns are unweighted. Percentages are weighted to adjust for differing probabilities of selection and nonresponse to align our estimates with 1997 Census tabulations by race/ethnicity, age, and gender (Bureau of the Census, SU-98–1). Row percentages total to 100%.
aGrade range estimates based on number of correct responses out of a possible 66. 0–18: grade 3 or below; 19–44: grades 4 through 6; 45–60: grades 7 through 8; 61–66: grade 9 and above.
bP values based on χ2 test with statistical adjustment for complex sample design.
To assess the effect of medical literacy on our study measurements, we compared the error rates for completion of the SAQ by respondents’ scores on the REALM. Table 2 ▶ summarizes the error rates of the 485 participants who completed the alcohol SAQ and the 487 participants who completed the same-sex sexual intercourse and masturbation SAQ; all of these participants also completed the REALM. Skip errors were the most common type of error across grade range estimates. On the alcohol SAQ, the overall error rate decreased significantly as the literacy level increased (P < .0001). At the literacy range of grade 3 or lower, only 21.4% of respondents were able to complete the alcohol SAQ without errors, compared with 70.5% of those at the level of grade 9 and above. However, skip errors were detected among 25% of those with estimated literacy at grade 9 or above, although only 0.8% answered inconsistently.
TABLE 2.
—Error Rates for Completion of Self-Administered Questionnaires (SAQs), 1997/98 Baltimore STD and Behavior Survey
| Grade Range Estimatesa | |||||
| Characteristics | n (unweighted) | No. errors (%) | Skip error (%)c | Inconsistent (%)d | Other (%)e |
| Alcohol/drug use SAQb | |||||
| ≤ Grade 3 | 28 | 21.4 | 64.3 | 7.1 | 7.1 |
| Grades 4–6 | 32 | 40.6 | 34.4 | 12.5 | 12.5 |
| Grades 7–8 | 69 | 56.5 | 36.2 | 0.0 | 7.2 |
| ≥ Grade 9 | 356 | 70.5 | 25.0 | 0.8 | 3.7 |
| Same-sex sexual intercourse/masturbation SAQb | |||||
| ≤ Grade 3 | 28 | 21.4 | 60.7 | 17.9 | 0.0 |
| Grades 4–6 | 34 | 32.4 | 55.9 | 5.9 | 5.9 |
| Grades 7–8 | 70 | 38.6 | 51.4 | 7.1 | 2.9 |
| ≥ Grade 9 | 355 | 53.8 | 39.7 | 6.5 | 0.0 |
Notes. Sample Ns are unweighted. Unweighted row percentages total to 100%.
aBased on number of correct responses out of a possible 66. 0–18: ≤ grade 3; 19–44: grades 4–6; 45–60: grades 7–8; 61–66: ≥ grade 9. P < 0.001 for Mantel–Haenszel test for linear trend.
bA total of 495 participants completed the alcohol SAQ (292 females and 203 males) and a total of 497 participants completed the same-gender sex and masturbation SAQ (295 females and 202 males). For each SAQ, 10 participants did not complete the REALM; totals are therefore 485 and 487, respectively.
cRespondent did not follow skip instructions on paper form (e.g., if Q1 equals “no,” skip to Q4).
dRespondent answered in an inconsistent manner (e.g., answered no to having same-gender contact, answered that most recent contact was last week on subsequent question).
eRespondent left entire page blank or did not fill out paper form correctly (e.g., circled 2 answers, answer written in).
The layout of the same-sex sexual intercourse and masturbation SAQ was more complex than the alcohol SAQ. The questions assessing respondents’ experience with masturbation contained 2 different skip instructions originating from the same question. For respondents scoring at the level of grade 9 and above, only 53.8% completed the same-sex sexual intercourse and masturbation SAQ without error, whereas 39.7% made skip errors. The proportion of respondents providing logically inconsistent answers on this form was highest among those at the level of grade 3 or below (17.9%).
CONCLUSIONS
Our findings support a link between low literacy, as assessed by the REALM, and participants’ inability to accurately complete a paper SAQ. We estimate that 28% of Baltimore adults (aged 18 to 45 years) have a REALM-rated literacy at the level of grade 8 or less and that 12% are at the level of grade 6 or less. These results have important implications for the survey measurement of health and other behaviors.
Our data suggest that, although persons with low medical literacy will provide answers on paper self-administered forms, they may respond to questions that they do not completely understand. For example, we found that logically inconsistent answers on the alcohol SAQ were 8 times more likely in the lowest than in the highest medical literacy group (error rate: 7.1% vs 0.8%). A similar pattern was found for the same-sex sexual intercourse and masturbation SAQ. Logically inconsistent answers were found 2.5 times more often with respondents at the level of grade 3 and below (17.9%) than with respondents at the level of grade 9 or higher (6.5%). Measurements made with SAQs are also vulnerable to errors when respondents do not follow questionnaire skip instructions. These error rates increase with lower literacy. Nonetheless, we found that error rates on skip instructions were quite high even among persons whose REALM-assessed literacy tested at the grade 9 and above range (25% to 40%).
These findings provide important evidence for the potential benefits of audio computer-assisted self-interviewing technologies.8–12 ACASI does not require respondent literacy; the respondent listens to the recorded questions and the defined response categories through headphones. ACASI eliminates the requirement that respondents be forms literate by automatically skipping respondents to the next question that is appropriate for them. The trend in survey research to adopt computer-based technologies offers promise for reducing the errors associated with low literacy, thereby improving the quality of survey measurements.
Acknowledgments
Primary support for this research was provided by National Institutes of Health grant R01-HD31067 to C. F. Turner. Additional support was provided by the Research Triangle Institute and by grant R01-MH56318.
The authors would like to thank the members of the 1997/98 Baltimore STD and Behavior Survey team for their contributions to the study and the survey operations staff of the Research Triangle Institute for their fielding of the survey. We would also like to thank Dr. Terry Davis and her colleagues for the use of their literacy assessment instrument.
Human Participant Protection The protocol for this study was approved by institutional review boards at the Research Triangle Institute (RTI) and the Johns Hopkins Medical Institutions (JHMIs).
Peer Reviewed
A. A. Al-Tayyib analyzed the data and wrote the brief. S. M. Rogers supervised data analysis and contributed to the writing. J. N. Gribble supervised the initial conception of the data analysis plan. M. Villarroel duplicated the data analysis for quality control purposes. C. F. Turner conceived and designed the 1997/98 Baltimore Sexually Transmitted Diseases and Behavior Survey and contributed to the writing of the brief. All authors approved the final version of this brief.
References
- 1.Turner CF, Rogers SM, Miller HG, et al. Untreated gonococcal and chlamydial infection in a probability sample of adults. JAMA. 2002; 287:726–733. [DOI] [PubMed] [Google Scholar]
- 2.Turner, CF, Gribble JN, Al-Tayyib AA, Chromy JR. Falsification in epidemiologic surveys: detection and remediation. Paper presented at ORI-NIH-AAAS-AAMC-NSF Conference on Research Integrity, Bethesda, Md, November 18–20, 2000.
- 3.Davis TC, Long SW, Jackson RH, et al. Rapid estimate of adult literacy in medicine: a shortened screening instrument. Fam Med. 1993;25:391–395. [PubMed] [Google Scholar]
- 4.Mayfield D, McLeod G, Hall P. The CAGE questionnaire: validation of a new alcoholism screening instrument. Am J Psychiatry. 1974;131:1121–1123. [DOI] [PubMed] [Google Scholar]
- 5.Laumann EO, Gagnon JH, Michael RT, Michaels S. The Social Organization of Sexuality. Chicago, Ill: University of Chicago Press; 1994.
- 6.Johnson AM, Wadsworth J, Wellings K, Field J. Sexual Attitudes and Lifestyles. Oxford, England: Blackwell Scientific Publications; 1994.
- 7.US Bureau of the Census, Population Division, Population Estimates Program. Population Estimates for Cities With Populations of 100,000 and Greater; July 1, 1998. Washington, DC: US Bureau of the Census; 1999. Publication series SU-98–1.
- 8.Turner CF, Ku L, Rogers SM, Lindberg LD, Pleck JH, Sonenstein FL. Adolescent sexual behavior, drug use, and violence: increased reporting with computer survey technology. Science. 1998;280:867–873. [DOI] [PubMed] [Google Scholar]
- 9.O’Reilly JM, Hubbard ML, Lessler JT, Biemer PP, Turner CF. Audio and video computer-assisted selfinterviewing: preliminary tests of new technologies for data collection. Journal of Official Statistics. 1994;10:197–214. [PMC free article] [PubMed] [Google Scholar]
- 10.Turner CF, Forsyth BH, O’Reilly JM, et al. Automated self-interviewing and the survey measurement of sensitive behaviors. In: Couper MP, Baker RP, Bethlehem J, et al, eds. Computer-Assisted Survey Information Collection. New York, NY: John Wiley & Sons, Inc; 1998: 455–473.
- 11.Rogers SM, Gribble JN, Turner CF, Miller HG. Entretiens Autoadministré sur ordinateur et mesure des comportements sensibles [Computerized selfinterviewing and the measurement of sensitive behaviors]. Population. 1999;54(2):231–250. [Google Scholar]
- 12.Gribble JN, Miller HG, Rogers SM, Turner CF. Interview mode and measurement of sexual behaviors: methodological issues. J Sex Res. 1999;36:16–24. [DOI] [PMC free article] [PubMed] [Google Scholar]