Usability of FDA-Approved Medication Guides

Abstract

BACKGROUND

Medication guides are required documents to be distributed to patients in order to convey serious risks associated with certain prescribed medicines. Little is known about the effectiveness of this information to adequately inform patients on safe use.

OBJECTIVE

To examine the readability, suitability, and comprehensibility of medication guides, particularly for those with limited literacy.

DESIGN

Assessments of suitability and readability of 185 medication guides, and a sub-study examining change in suitability and readability from 2006 to 2010 among 32 of the medication guides (Study 1); ‘open book’ comprehension assessment of medication guides (Study 2).

SETTING

Two general internal medicine clinics in Chicago, IL.

PATIENTS

Four hundred and forty-nine adults seeking primary care services, ages 18–85.

MEASUREMENTS

For Study 1, the Suitability Assessment of Materials (SAM) and Lexile score for readability. For Study 2, a tailored comprehension assessment of content found in three representative medication guides.

RESULTS

The 185 analyzed medication guides were on average 1923 words (SD = 1022), with a mean reading level of 10–11th grade. Only one medication guide was deemed suitable in SAM analyses. None provided summaries or reviews, or framed the context first, while very few were rated as having made the purpose evident (8 %), or limited the scope of content (22 %). For Study 2, participants’ comprehension of medication guides was poor (M = 52.7 % correct responses, SD = 22.6). In multivariable analysis, low and marginal literacy were independently associated with poorer understanding (β = –14.3, 95 % CI –18.0 – –10.6, p < 0.001; low: β = –23.7, 95 % CI –28.3 – –19.0, p < 0.001).

CONCLUSION

Current medication guides are of little value to patients, as they are too complex and difficult to understand especially for individuals with limited literacy. Explicit guidance is offered for improving these print materials.

Electronic supplementary material

The online version of this article (doi:10.1007/s11606-012-2068-7) contains supplementary material, which is available to authorized users.

BACKGROUND

For more than a decade, the Food and Drug Administration (FDA) has required ‘medication guides’ (med guides) be issued to consumers for prescription medications viewed to possess “serious and significant public health concerns.^1,2” These industry-developed, FDA-approved print materials are distributed by pharmacies at the time of medication dispensing with the intent that they be read by patients prior to taking a prescribed drug. They provide specific dosing administration instructions that could prevent serious adverse effects associated with taking the medication, warn individuals about significant health risks that could affect one’s decision to take the medication, and underscore the importance of taking the prescribed medication to the patient’s health, and the need for proper adherence.

Med guides have become an essential part of risk evaluation and mitigation strategies (REMS).^3–6 REMS are FDA-required, detailed pharmaceutical company plans for directly communicating the safe use of and risks associated with a certain drug to both prescribers and consumers. Prior investigations have repeatedly found that physicians and pharmacists miss opportunities to counsel patients on appropriate use of prescribed medicines,^7–10 so med guides remain a frontline and often sole channel for conveying risk information to patients.

There has been increasing criticism of the efficacy of med guides to help patients, as a limited number of studies have suggested that there are problems with their format, clarity of content, and manner in which they are disseminated.^11–13 Despite a 1996 Department of Health & Human Services memorandum that established readability standards for health materials,¹⁴ we found in our earlier research that the majority of med guides are too complex and written at a reading grade level not suitable for the majority of patients to comprehend.¹² Consequently, only one in four patients reported attending to these materials.¹² Shrank and colleagues also found that a required med guide was never distributed along with a prescription in observed prescription fills at pharmacies nationwide.¹⁵ Few studies to date have assessed patients’ ability to comprehend information contained in med guides, especially those with limited literacy. As the FDA is now aggressively investigating potential improvements and expansion of the med guides program,^16–19 evidence we can provide detailing individuals’ comprehension of the information could serve as a valuable baseline for later program evaluations. We therefore sought to document the prevalence of misunderstanding med guides and provide a systematic evaluation of their current content and format. We conducted two complementary investigations: 1) a readability and suitability assessment of med guides, and 2) comprehension testing among primary care adult patients. We hypothesized that med guides would have an unacceptable level of reading difficulty and accessibility, and patients’ comprehension would be low; those with limited literacy would be at greatest risk for not being able to read and understand the materials successfully.

METHODS

Study 1: Readability and Suitability of Med Guides

We used an adapted version of the Suitability Assessment of Materials (SAM) instrument²⁰ to assess the suitability of med guides for use among lower literate adults. The SAM is the most commonly used tool for evaluating the appropriateness of written health care information and instructions for use among lower literate adults. Trained raters analyzed materials across 22 factors in six categories that may impact readability and ease-of-use (content, literacy demand, graphics, layout and typography, learning simulation/motivation, cultural appropriateness). Cultural appropriateness was not rated as med guides were created for general audiences. Since none had cover graphics, that factor was also dropped. The specific factors we evaluated are identified in the Figure 1. The original scoring method was to classify, by factor, whether the document was ‘not suitable’, ‘suitable’, or ‘superior’. We modified this rating to reflect whether the document met the minimal criteria (suitable vs. not suitable). Two trained reviewers independently rated each med guide. Inter-rater reliability was calculated based on individual item ratings, and found to be high (K = 0.78). A third reviewer was used in cases of discordance. Overall suitability of a document was determined based on whether the document was classified as ‘suitable’ on 70 % or more of the categories.

Figure 1.

Percent of med guides achieving adequate rating according to suitability assessment of materials (N = 227).

To determine the length and readability of med guides, we used the Professional Lexile Analyzer, which measures the level of difficulty and complexity of written passages.²¹ This process was previously used by the study team on med guides and prescription instructions.^12,22 Lexile scores have previously been found to be independently associated with patient medication comprehension. Scores ranging from below zero to 2000 were recorded for each document. Grade levels were then estimated based on Lexile scores for ‘typical readers’ in that grade.

Sample

In April 2010, 227 med guides were identified on the FDA’s Center for Drug Evaluation and Research (CDER) website. All of these were considered initially eligible for analysis. The content of each med guide’s FDA downloadable PDF file were transferred into text documents. Any information not intended for the patient was removed. A total of 25 med guides described non-steroidal anti-inflammatory drugs (NSAIDs), and 22 had identical content. Therefore, they were analyzed as a single document. The other three contained information specific to a unique medication and were analyzed separately. Similarly, 22 of 34 antidepressant medications had identical content and were analyzed as one, while 12 were analyzed as individual medications. As a result we analyzed a total of 185 documents.

Subset Analysis 2006–2010

We examined whether the suitability, readability, and length of 32 out of 40 med guides previously investigated in a 2006 study by this research team had improved. Fisher’s exact tests were run to compare 2006 and 2010 suitability item scores (suitable vs. not suitable), and student’s t tests performed on the total percent of SAM items deemed adequate, Lexile score, and word count.

Study 2: Comprehension Testing

Participants

Patients ages 18–85 years at two academically affiliated, general internal medicine clinics in Chicago serving diverse patient populations participated in the study (N = 449). Eligibility criteria included that they be English-speaking, and have no uncorrectable visual, hearing, or severe cognitive impairment (determined by a brief 6-item screener).²³ Patients were compensated $20 for their participation. Clinic staff approached consecutive patients arriving for scheduled appointments and discussed potential study recruitment (>80 % participation rate, both sites). Institutional Review Boards at Northwestern University and the University of Illinois at Chicago approved the study.

Materials

Three med guides were selected to be presented to participants based on their content, format, and frequency of exposure. We calculated the word count, readability via Lexile score, and frequency of annual prescriptions of the drugs described in all 227 med guides available on the FDA’s website as of May 2010. Of the 26 med guides within the general range of the mean criteria, one was randomly chosen per each of the three most common routes of administration: oral tablet, oral liquid solution, and injectable. Chosen med guides were for Ritalin (oral tablet), Morphine Sulfate (oral solution taken), and Aranesp (injectable).

Measurement

A series of questions were created based on specific content within each med guide to assess patients’ understanding. These items were categorized items as: 1) decision making prior to use, 2) general use and storage, and 3) side effects. Items reflected either the basic task of information retrieval (finding and comprehending information), or inference (applying information to specific contexts). Assessments for med guides for each medicine varied slightly; Ritalin had 12 questions with 33 possible correct answers, Morphine Sulfate had 15 questions with 32 correct points, and Aranesp had 12 items with 34 possible correct responses. Participants could score a possible 99 correct points (see Table 2 for actual items).

Table 2.

Differences in Total and Subcategory Comprehension Scores by Literacy Level

Score	Overall Score Mean (SD)	Mean (SD) Scores by Literacy Level			P value
		Low (n = 63)	Marginal (n = 103)	Adequate (n = 283)
Total (out of 99 pts)	52.7 (22.6)	24.9 (13.8)	37.8 (15.7)	64.3 (17.1)	<0.0001
Decision making prior to use (36 pts)	18.3 (9.3)	8.0 (5.1)	12.0 (6.0)	22.9 (7.7)	<0.0001
General use and storage (15 pts)	9.3 (3.0)	5.9 (2.3)	7.6 (2.2)	10.7 (2.5)	<0.0001
Side effects (48 pts)	25.1 (11.7)	11.0 (8.1)	18.3 (9.6)	30.7 (8.8)	<0.0001

Procedure

Participants were guided through an untimed, ‘open book’ assessment of their comprehension of the three med guides. They were first shown the med guide for Ritalin and instructed to look it over for two minutes, after which the interviewer began asking questions. Participants were explicitly told they could take their time in answering and refer to the med guide as needed while responding. Verbatim responses were recorded and coded as correct or incorrect based on a preset list of acceptable options per question (developed by the study team). This process was repeated for the other two med guides (Morphine Sulfate and Aranesp, respectively). Interviewers then asked a brief set of questions regarding prior experience with the medication, including whether they or a family member were currently taking or had ever taken the medication being discussed. Demographic and socioeconomic information was also collected. As prior studies have shown that low literacy impacts comprehension of written medication information, we administered the Rapid Estimate of Adult Literacy in Medicine (REALM).²⁴ This is the most common measure in health literacy research.

Analysis

Frequencies and/or means and standard deviations were calculated for each patient characteristic. REALM scores were categorized as inadequate, marginal, or adequate based on known thresholds. Chi-square and/or student t tests were used to examine the association between literacy and each of the patient variables. The primary outcome was the mean total comprehension score (range: 0–99) across all three med guides. We used ANOVA models to compare mean total scores across literacy levels, and per med guide content subcategory. A Generalized linear model was used for multivariable regression analysis examining predictors of comprehension of med guide content. Analyses were performed using STATA version 10 (College Station, TX).

RESULTS

Study 1: Suitability and Readability

The 185 analyzed med guides were on average 1923 words (SD = 1022, range 351–8526). Mean Lexile score was 1130 (SD = 126, range 860–1570), which is estimated to be at a 10th to 11th grade reading level. Only seven med guides (3.8 %) met Keystone recommendations for reading difficulty ≤ 8th grade.¹⁴

According to the SAM, med guides were scored as adequate on an average of 50.3 % of items. Only one med guide was deemed suitable overall by having 70 % or more of variables scored as adequate. The proportion of med guides that received suitable scores in each document category is shown in the Figure 1. None provided summaries or reviews, or framed the context first, while very few were rated as having made the purpose evident (8 %), or limited the scope of content (22 %). Among med guides that included illustrations, only a third (34 %) were identified as having accompanying captions.

When we compared the suitability and readability of the 32 med guides assessed now and in 2006 , there was no significant change in average word count (M = 2216.3, SD = 146.2 vs. M = 2182.5, SD = 151.0; p = 0.56) or readability (M = 1144.4, SD = 24.4 vs. M = 1193, SD = 31.4; p = 0.13). Improvement in readability translated to about a half grade reduction (from 11th to 10–11th grade). Compared to 2006 SAM assessments, seven med guides had previously been rated as suitable for providing summaries, having a limited scope and evident purpose, whereas the present assessment found them unsuitable on these categories. Improvements were noted for two med guides that now included relevant illustrations, and interaction was used for learning stimulation.

Study 2: Comprehension

Table 1 presents the demographic characteristics of the sample. The mean age of participants was 51.4 years, with two thirds being female; half were black, and more than a third white. A third of participants had less than a high school education, and 36.9 % had limited literacy skills (14.0 % inadequate, 22.9 % marginal). One third (34.3 %) of participants had previously heard of a med guide, while half had prior experience with at least one of the medications featured in the med guides. Older age, black race, less education, and lower income were all associated with limited literacy (p < 0.001).

Table 1.

Participant Demographic Characteristics Overall and by Literacy Level (N = 449)

Characteristic	Total (N = 449)	Literacy Level
		Low (n = 63)	Marginal (n = 103)	Adequate (n = 283)	P value*
Age, %					<0.0001
18–30 years	20.3	2.2	16.5	81.3
31–45 years	30.5	12.4	21.9	65.7
46–60 years	31.2	24.3	27.1	48.6
61–85 years	18.0	12.4	24.7	63.0
Female, %	64.4	14.2	21.5	64.4	0.60
Race, %					<0.0001
White	35.2	1.9	7.0	91.1
Black	53.5	23.8	34.6	41.7
Other	11.4	5.9	17.7	76.5
Education, %					<0.0001
≤ High School	34.1	32.7	36.6	30.7
Some College	24.5	8.2	27.3	64.6
≥ College Grad	41.4	2.2	9.1	88.7
Income, %					<0.0001
<$20,000	34.8	31.0	35.9	33.1
$20,000–$50,000	22.5	7.5	20.2	72.3
>$50,000	42.7	3.4	12.4	84.3
Heard of Med Guide, %					0.14
Yes	33.9	11.8	21.7	66.5
No	62.6	16.0	24.2	59.8
Don’t Know	3.6	0.0	12.5	87.5
Prior Experience, %					0.01
No experience	52.1	12.8	18.0	69.2
Experience with ≥ 1 Med Guides	47.9	15.4	28.4	56.3

*p value for χ² tests of associations between literacy skills and patient characteristics

The Online appendix presents item performance for each of the med guides examined. Mean comprehension total score and subcategory scores are detailed in Table 2, also by literacy level. In general, patients had a hard time comprehending the med guides, with a mean total score of 52.7 (SD = 22.6). This was also true for each subcategory score. Participants with lower literacy consistently demonstrated poorer understanding of all med guides content, with increasing, gradient trends noted in total and subcategory comprehension scores across low, marginal, and adequate literacy levels.

In multivariable analysis controlling for all demographic and socioeconomic variables, including literacy and prior experience with med guides, low and marginal literacy were independently associated with a poorer comprehension of med guides (β = –14.3, 95 % CI –18.0 – –10.6, p < 0.001; low: β = –23.7, 95 % CI –28.3 – –19.0, p < 0.001, Table 3). Other risk factors included older age, black race, and less education. Interactions with age, literacy, and other variables were tested and none were significant.

Table 3.

Multivariate Model of Predictors of Understanding Medication Guides

Variables	β (95% CI)	p-value
Literacy
Inadequate	–23.7 (–28.3 – –19.0)	<0.001
Marginal	–14.3 (–18.0 – –10.6)	<0.001
Adequate	—	—
Age
18–30 years	—	—
31–45 years	–2.4 (–6.3 – 1.6)	0.24
46–60 years	–5.9 (–10.4 – –1.4)	0.01
61–85 years	–12.2 (–17.0 – –7.4)	<0.001
Race
White	—	—
Black	–9.5 (–13.3 – –5.8)	<0.001
Other	–7.3 (–11.8 – –2.7)	0.002
Gender
Male	—	—
Female	2.1 (–0.83 – 5.0)	0.16
Education
≤ High School	–14.6 (–19.1 – –10.1)	<0.001
Some College	–4.1 (–8.1 – –0.1)	0.05
≥ College Grad	—	—
Income
<$20,000	–3.1 (–7.5 – 1.4)	0.18
$20,000–50,000	–1.7 (–5.5 – 2.0)	0.37
>$50,000	—	—
# Daily Rx Medicines
0	—	—
1–2	–0.3 (–4.1 – 3.4)	0.86
3+	–1.4 (–5.6 – 2.8)	0.52
Prior Experience *
Experience	–1.6 (–4.3 – 1.2)	0.28
No Experience	—	—

^*Prior experience defined as current use, past use, use by a family member and/or helping a family member use one or more of the medicines featured in the medication guides tested

DISCUSSION

Available med guides fall below the threshold of acceptable standards for patient print materials set by both professional societies and the federal government. Despite a nearly eight-fold increase in the past five years in the number of drugs required by the FDA to have a med guide (from 40 in 2006 to 305 as of September 2011),²⁵ little to no improvement has been made in their readability and accessibility. We confirmed the inadequacy of med guides in a representative sample of primary care patients, who had considerable difficulty comprehending basic information from print med guides even while viewing the material and given adequate time to respond. This is a serious concern, as med guides are frequently the only means that patients have for receiving critical information on how to safely use higher risk drugs. The extent to which patients across all literacy levels did not understand any aspect of the med guides demonstrates that they are too complex to be useful. This might explain why patients in prior studies reported not reviewing consumer medication information, and why the majority in our study had never heard of med guides.^12,26–28

The Plain Writing Act of 2010 mandates that federal agencies, including the FDA, communicate to the public in a clear, understandable manner free of unclarified jargon.²⁹ What this new federal law does not state is how to determine whether this has been achieved. The Department of Health and Human Services and experts in health literacy are often inconsistent when defining a threshold for readability of print documentation. Some recommend an 8th grade level or below,¹⁴ while others have sought targets as low as below a 4th grade reading level.³⁰ Adding to the confusion are more recent debates on the utility of reading formulas.¹⁷ Most experts agree that readability assessments are important within a more comprehensive evaluation of consumer materials. An operational set of standards for guiding government and industry in best practices for designing print materials like med guides, and thereby offering a means to assess these communications in the future, are urgently needed. The Agency for Healthcare Research and Quality (AHRQ) is now leading an effort to set these standards.³¹

As in previous studies examining comprehension of print health materials,³² we found that individuals with lower literacy were significantly less able to navigate and retrieve information, and make inferences to support safe and appropriate use of a medicine than those in higher literacy groups. Also, older age and less education were independently linked to poorer comprehension. These findings, when considering other failed sources of spoken communication and prescription labeling elucidated in previous research,^33–35 convey the urgency for evidence-based approaches to the re-design of med guides.

Our study has limitations. Study 2 participants were recruited from two diverse settings; however eligibility did not require current or prior experiences with the medicines discussed in our assessment. Patients may be more attentive to the task of reviewing the content and responding to our comprehension questions if they were actually taking these prescriptions. We also were only able to evaluate comprehension of three representative med guides. A prior study by our team found that readability of print material, as measured by Lexile score, was a strong predictor of comprehension.³⁴ With only three med guides, we could not adequately assess the relationship between Lexile score and readability. We also only included English-speaking patients. Future research should closely examine the availability and quality of med guides in other languages, as Limited English Proficiency and poor quality translations of health materials have previously been implicated as risk factors to inadequate understanding of medication regimens.^36–38

Given the attention med guides have received from the FDA, the Brookings Institution and the pharmaceutical industry in response to REMS, there is a current push to improve them and seek out a single-document solution for providing medication information. This could end the redundancy in, confusion about, and lack of awareness of med guides and other leaflets. Our findings strongly suggest a need for the med guide program’s revision. Future improvements might begin with evidence-based readability standards and an explanation of the purpose of med guides included in the material. Providing a summary that highlights ‘need-to-know’ content could be a way of limiting and layering information in such a way that patients can self-tailor the amount of knowledge they wish to obtain about their prescription. All options should be considered for informing patients at all literacy levels on how to safely and appropriately use medication.