Patient understanding of drug risks: an evaluation of medication guide assessments

Abstract

Purpose

When a Medication Guide (MG) is part of Risk Evaluation and Mitigation Strategy (REMS), manufacturers assess the effectiveness of MGs through patient surveys, which have not undergone systematic evaluation. We aimed to characterize knowledge rates from these patient surveys, describe their design and respondent characteristics, and explore predictors of acceptable knowledge rates.

Methods

We analyzed MG assessments submitted to the Food and Drug Administration from September 2008 through June 2012. We evaluated the prevalence of specific characteristics, and calculated knowledge rates, whereby we defined “acceptable knowledge” when ≥ 80% of respondents correctly answered questions about the primary drug risk. Univariate logistic models were used to investigate the predictors of acceptable knowledge rates.

Results

We analyzed the first completed MG assessment for each drug with a patient survey, resulting in 66 unique MG assessments. The mean knowledge rate was 63.8%, with 20 MG assessments (30.3%) achieving the 80% threshold. Compared to assessments that did not reach acceptable knowledge rates, those that did were more likely associated with additional REMS elements (e.g. Elements to Assure Safe Use or Communication Plans). Other factors, including mean age, reading or understanding the MG, and being offered or accepting counseling were not associated with knowledge rates. There was considerable variation in the design of MG assessments.

Conclusions

Most MG assessments did not reach the 80% knowledge threshold, but those associated with additional interventions were more likely to achieve it. Our study highlights the need to improve patient-directed information and the methods of assessing it.

INTRODUCTION

Patient-directed medication information is provided to patients when a prescription is dispensed, and can be in the form of manufacturer-produced, Food and Drug Administration (FDA)-approved patient labeling, such as Medication Guides or Patient Package Inserts, or in the form of third party generated Consumer Medication Information (CMI). Medication Guides are paper handouts that are provided to patients at the point of prescription dispensing to educate them about certain drug risks and safe drug use practices, in order to avoid serious adverse events. The general provisions and requirements for the content and distribution of Medication Guides are set forth in regulation 21 CFR 208.

In certain cases, a Medication Guide can be an element of a Risk Evaluation and Mitigation Strategy (REMS) (FD&C Act, 21 U.S.C. 355-1). The FDA can require a REMS when it determines that one is necessary for a specific drug or drug class.¹ A REMS can include other elements, such as a Communication Plan, which is directed toward healthcare professionals, and one or more Elements to Assure Safe Use (ETASU), which impose restrictions of varying degrees on the use of the drug. Generally, REMS also includes a timetable for submission of assessments to the FDA, required by 18 months, by 3 years, and in the 7^th year after REMS approval, or as otherwise determined by the FDA. From March 2008, when the REMS provisions took effect, until January 2011, the FDA approved over 150 Medication Guides as part of a REMS program and 108 of the REMS programs included only a Medication Guide element.

As outlined in FDA guidance,¹ not all Medication Guides are part of a REMS. When a Medication Guide is part of a REMS, the FDA requires manufacturers to conduct assessments of the effectiveness of the Medication Guide. To conduct this assessment, manufacturers develop and administer patient knowledge surveys, which focus primarily on patients’ understanding of key drug risks that are identified in the “most important information” section of the Medication Guide.² Unlike the Medication Guides, which are developed according to standards set forth in regulation, the patient knowledge surveys are not developed, administered, analyzed, or reported in a standardized manner. Because of this lack of standardization and because survey methodologies are not fully developed for this purpose, uncertainty exists regarding the ability of Medication Guide assessments to measure the effectiveness of Medication Guides.^3,4

Several studies have raised questions about the suitability of written patient information, including Medication Guides, to convey important drug risks.^5–9 Currently, only one study, which included an independent assessment of a single Medication Guide, reported good patient understanding of drug risk,¹⁰ though factors other than the Medication Guide itself may have contributed to the observed level of understanding.¹¹ To date, no studies have comprehensively evaluated the findings of patient knowledge surveys, examined the characteristics of these surveys, or assessed their ability to measure the effectiveness of Medication Guides. We systematically reviewed Medication Guide assessments submitted to FDA to characterize knowledge rates from the patient surveys and to describe design and respondent characteristics of Medication Guide assessments. In addition, we conducted an exploratory analysis on predictors of acceptable knowledge rates.

METHODS

Data source

We systematically searched FDA Center for Drug Evaluation and Research (CDER) records for REMS-related Medication Guide assessment reports that included patient surveys submitted by manufacturers. The study database included all reports received between September 2008 and June 2012. In the event of multiple sequential assessments for a given REMS, we used the first assessment submitted during our study period to eliminate possible correlation between sequential assessments. The study analysis was limited to the aggregate data contained in the reports; we did not have access to subject-level data.

Data extraction

We used a standardized protocol to promote reliable data extraction. From each eligible Medication Guide assessment, we extracted Medication Guide assessment information and summary-level patient demographics. Based on available literature, we identified demographic variables with reported associations with knowledge, such as age, race/ethnicity, and education.^5,12,13 Because we relied on aggregate data in the reports, in certain instances we approximated the mean age based on provided frequency distributions for age categories when the sample mean age itself was not provided. New users of a drug were defined as patients who reported 6 months or less medication usage. We defined prevalent users as those who reported more than 6 months of medication usage.

We assessed the following characteristics of Medication Guide assessments: the time in months passed between drug approval and submission of the Medication Guide assessment to CDER, if a pilot survey was completed before the assessment, if a patient survey reminder was sent, and if a completion incentive was offered, including type/amount of incentive. The Medication Guide assessment response rate was defined, in accordance with the standard of the American Association for Public Opinion Research, as the number of respondents who completed a Medication Guide assessment divided by the total number of respondents who were mailed (via paper or electronic) an invitation to participate in the assessment.¹⁴ The completion rate was calculated as the number of respondents who completed the Medication Guide assessment divided by the total number of respondents who returned the assessment. We further analyzed if the respondents reported receiving a Medication Guide, the length of the Medication Guide, and the number of questions included in the Medication Guide assessment.

The reading rate was calculated as the number of assessment respondents who reported having read the Medication Guide divided by the number of respondents who reported that they had received the Medication Guide. Similarly, we calculated the proportion of respondents who were offered counseling, among them the proportion who accepted counseling, and among them the proportion of patients who reported that they understood the provided counseling.

Knowledge rates of the primary drug risk were determined from the assessment questions that asked specifically about the primary risk associated with the drug, as identified in the Medication Guide. We calculated knowledge rates for each Medication Guide assessment from the number of respondents who correctly identified the primary drug risk, divided by the number of respondents who completed the patient knowledge survey. We then used a threshold of at least 80% of respondents correctly identifying the primary drug risk as evidence of an acceptable knowledge rate.² We also summarized additional measures of knowledge captured within the Medication Guide assessments. We categorized knowledge questions on appropriate action related to symptoms of the primary drug risk as “knowing when to seek help”. Additionally, knowledge questions that asked about information that should be shared with the provider were grouped into an “appropriate communication with prescriber” category.

Data synthesis and analysis

We conducted a descriptive analysis of the Medication Guides’ characteristics, the assessments’ design characteristics, the respondents’ characteristics, and knowledge rates. We explored possible associations between potential determinants of the knowledge of the primary drug risk and the knowledge rate of the primary drug risk. We compared differences in reaching the primary drug risk knowledge threshold of 80% using Student’s t-test or univariate linear regression, when appropriate. We used single-variable logistic regression to evaluate the relationship between intervention (i.e. REMS element) and outcome (i.e. assessments achieving the 80% mean knowledge threshold). The models were not weighted by sample size. This resulted in unequal representation of individual survey respondents, but equal representation of each Medication Guide assessment. Potential determinants were specified a priori. We conducted all statistical analyses in SAS 9.2 (SAS Institute, Cary, NC).

RESULTS

Of the 145 REMS assessments received by CDER between September 2008 and June 2012, 81 contained Medication Guide assessments for 66 unique drugs. We included the first Medication Guide assessment for each drug during our study period (n = 66, Fig. 1).

Figure 1.

Summary of assessments searched and selected

Medication Guide assessment characteristics

It was not standard practice to provide the physical handouts of the Medication Guides to the respondents at the time of the knowledge survey, or to require respondents to have ever read the Medication Guide prior to the knowledge survey. More than half of the Medication Guide assessments included in this study had been pilot tested (57.6%, n = 38) (Table 1). However, if such detail was provided, pilot testing typically focused on the clarity of the questions, and there was no indication that other aspects of validity or reliability of the assessment questions were tested.

Table 1.

Medication Guide assessments and responder characteristics^*

Variable	Variable level	N of MG (n = 66)	Percent/mean (SD)
Assessment characteristics
Time between REMS approval and first submission of MG assessment, months	6 months	5	7.6%
	12 months	11	16.7%
	18 months	44	66.7%
	24 months	3	4.6%
	36 months	3	4.6%
		66	71.2 (±64.8)
Pilot survey completed before assessment		38	57.6%
Survey follow-up reminder		24	36.4%
Completion incentive		47	71.2%
Amount of incentive ($)		47	33.8 ±22.9
Response rate		55	20.6% (±27.4%)
Completion rate		51	87.1% (±23.7%)
Reported patient receipt of a Medication Guide		60	80.6% (±12.1%)
Length of Medication Guide (pages)		66	5.2 (±2.2)
Length of Medication Guide assessment (# questions)		66	19.6 (±6.5)
Number of drug risk questions in the Medication Guide assessment		66	3.5 (±1.8)
Participant characteristics
Number of participants		66	230 (±174)
History of prescription use	New	32	39.7% (±26.3%)
	Prevalent		59.1% (±26.4%)
Age (years)		58	50.7 (±8.9)
Race/ethnicity	White	40	75.1% (±18.3%)
	Black		10.4% (±8.6%)
	Hispanic		6.7% (±4.9%)
	Asian		3.1% (±4.0%)
	Other		5.5% (±7.0%)
Language spoken at home	English	20	96.9% (±3.2%)
	Spanish		1.8% (±1.6%)
	Other		1.7% (±2.3%)
Education	<high school	50	5.5% (±4.9%)
	High school graduate		23.8% (±10.1%)
	Some college		34.0% (±8.4%)
	College graduate		27.2% (±11.7%)
	Post graduate		15.0% (±6.6%)

MG = Medication Guide, REMS = Risk Mitigation and Evaluation Strategy, SD = standard deviation

^*Means presented in table were not weighted by sample size.

Thirty-six percent of the assessments reported using a follow-up reminder for non-respondents to the patient knowledge surveys. Fifty-five of the 66 Medication Guide assessments (83.3%) reported response rate information, with a mean response rate of 20.6% (Table 1). All of the 66 assessments reported the total number of respondents for the patient knowledge surveys. The mean number of participants across the assessments was 230 with a range of 8 to 1000 participants.

An average of 80.6% of assessment respondents reported ever receiving a Medication Guide at some point in time. Of the respondents who received a Medication Guide, 87.0% indicated that they had read the Medication Guide, of whom 98.7% reported understanding the Medication Guide. Across 49 assessments that included questions on counseling, a mean of 56.6% of respondents reported being offered counseling by a healthcare provider (e.g. physician, nurse, or pharmacist). Of the respondents who reported being offered counseling, 73.3% claimed that they accepted counseling, of which 97.3% claimed that they understood counseling. It is important to note that most of the assessments did not report when the respondent received the Medication Guide, when the respondent was counseled on the drug, or on the time interval between reading the Medication Guide or receiving counseling and responding to the survey.

Respondent characteristics

Across all of the Medication Guide assessments, the majority (59.1%) of respondents reported being prevalent users (Table 1). The mean age of respondents across surveys was 50.7 years. In the pooled sample across all surveys, the distribution of race largely resembled the racial composition of the adult U.S. population according to the 2011 U.S. Census, which estimated the U.S. population as 72.0% White, 13.0% Black, 5.0% Asian, 10.0% Other, and 16.0% Hispanic.¹⁵ Overall, the education level of respondents was somewhat higher than that of the general U.S. population, with a cumulative average of 94.5% of respondents having at least high school education, compared to 85.0% in the U.S. adult population.¹⁶

Respondent knowledge

All 66 Medication Guide assessments included a primary drug risk knowledge question (or questions), which was answered correctly on average by 63.8% of respondents (Fig. 2). Among the Medication Guide assessments that reported questions relating to the additional knowledge topics, on average 77.7% of respondents knew when to seek help (n = 56) and 72.6% scored correctly on appropriate communication with the prescribers (n = 49). Across these 66 Medication Guide assessments, only 30.3% (n = 20) of the surveys met the study threshold of 80% of participants correctly identifying the primary drug risk (Fig. 3).

Figure 2.

Distribution of knowledge rate in Medical Guide assessments by REMS element

Figure 3.

Proportion of Medication Guide assessments, by REMS element, meeting the 80% threshold for respondents’ correct identification of the primary drug risk

Since REMS elements other than Medication Guides require different levels of intervention and patient involvement, we stratified our analysis by REMS element to explore further the mean correct knowledge rate and the proportion of assessments meeting the study threshold. For the 34 REMS that consisted only of a Medication Guide, the mean correct knowledge rate was 56.3% and 6 assessments (17.6%) met the 80% threshold. For the 21 REMS that consisted of both a Medication Guide and a Communication Plan, the mean correct knowledge rate was 64.0% and 7 (33.3%) met the 80% threshold. For the six REMS that included both a Medication Guide and ETASU, the mean correct knowledge rate was 89.8% and 5 (83.3%) met the 80% threshold. Five REMS included all three elements (Medication Guide, Communication Plan, and ETASU), with a mean correct knowledge rate of 82.7% and 2 met the 80% threshold (40.0%). Assessments that contained additional REMS elements had increased odds of achieving the 80% threshold when compared to assessments that only contained Medication Guides [(Medication Guide and Communication Plan, OR: 2.3, 95% CI: 0.7–8.3, p = 0.19); and (Medication Guide, Communication Plan, and ETASU, OR: 3.1, 95% CI: 0.4–22.9, p = 0.26)]; however, this finding was only statistically significant for assessments that consisted of a Medication Guide and ETASU (OR: 23.3, 95% CI: 2.3–237.6, p = 0.01).

Furthermore, we investigated if the number of drug risks identified in the Medication Guide influenced the chance of reaching the 80% knowledge threshold. Thirty-four (51.5%) of the Medication Guide assessments had more than one REMS-related drug risk identified in the Medication Guide with 20.6% reaching the 80% knowledge threshold for the primary drug risk, whereas 40.6% of assessments with only one REMS-related drug risk reached the threshold (OR: 0.4, 95% CI: 0.1–1.1, p = 0.08).

The respondent populations in the Medication Guide assessments that reached the 80% knowledge threshold differed little from assessment populations that did not reach this threshold with regard to mean age, self-reported reading of the Medication Guide, self-reported understanding of the Medication Guide, being offered counseling, and accepting counseling (Table 2). Fewer participants reported being a prevalent user of the prescribed drug in the assessments that met the 80% knowledge threshold compared to assessments where the threshold was not met, but this difference was not statistically significant.

Table 2.

Univariate analysis of determinates of Medication Guide assessment outcomes

Determinants*	Knowledge rate < 80%	Knowledge rate ≥ 80%	p-Value†
Age in years, mean	51.0	49.7	0.609
Prevalent user‡	63.0%	42.5%	0.088
Reading Medication Guide	87.4%	85.7%	0.672
Understanding Medication Guide	99.0%	98.0%	0.196
Offered counseling	55.8%	58.4%	0.682
Accepted counseling	73.1%	74.6%	0.850
Understanding counseling	98.8%	91.2%	0.324

^*Analysis based on self-report from respondents to patient knowledge survey.

^†P-value derived from Student’s t-test analyses using the knowledge rate threshold (above, below) as the two independent groups and the identified determinant as the interval dependent variable. We defined statistically significant with a p-value of <0.05.

^‡Use of drug for >6 months by respondent.

DISCUSSION

We reviewed Medication Guide assessments conducted as part of a REMS assessment for 66 drugs submitted to FDA during the period from September 2008 through June 2012. Unlike previous studies that attempted to investigate patient understanding of risk communication,^17,18 we were able to review all eligible assessments that drug manufacturers submitted to CDER with a patient survey over a period of almost 4 years. We found that patient understanding of the primary drug risk was low as measured by the Medication Guide assessments. In less than one third of the 66 assessments were 80% of the respondents able to identify the primary drug risk accurately. This finding suggests an overall limited patient understanding of the primary drug risk. Our findings are consistent with other research that suggests difficulty understanding written prescription drug information.^5,12,13

Among the variables available in the assessments, our analyses did not identify factors specific to the Medication Guides or to the respondent populations that independently predicted a knowledge rate for the primary drug risk of at least 80%. We found that, on a study level, several factors did not distinguish between assessments that met the 80% knowledge threshold and those that did not. Because these responses are self-reported and not otherwise validated, it is possible that they do not accurately reflect the actual level of receiving and understanding the Medication Guide and counseling. Alternatively, understanding of a Medication Guide and receipt of counseling in the past may not result in sustained knowledge of a drug’s risks. While subject-level analyses may be helpful in further understanding the relationship of these self-reported responses to individual knowledge level, the high proportion of respondents who reported reading and understanding the Medication Guide suggests that Medication Guides are not providing a sufficient degree of sustained knowledge of REMS-related drug risks.

We observed that assessments with multiple REMS-related drug risks identified with the Medication Guide had lower odds of reaching the 80% knowledge threshold. Thus, it is possible that a single REMS-related drug risk message in a Medication Guide is less complicated as compared to Medication Guides with multiple REMS-related drug risk messages. Furthermore, assessments of Medication Guides that also included an ETASU were more likely to reach the mean knowledge rate of 80%, though a small sample size limited formal inferences. Several factors may support this finding. First, it is possible that the additional REMS elements, and not the Medication Guide, lead to improved knowledge. The Communication Plan or ETASU may focus the attention of both patients and healthcare professionals on the specific risk that is the subject of both the REMS and the Medication Guide to a greater extent than occurs for drugs that have a Medication Guide only. Finally, it is also possible that the drugs associated with the additional REMS elements receive more media attention concerning the drug risk (e.g. risk is highly publicized so patients encounter drug knowledge through multiple outlets).

Our study had a number of limitations. The Medication Guide assessments were submitted to CDER as aggregate data, therefore limiting us to the analysis of “average effects”. Without subject-level data, the results should be interpreted with caution, especially in instances where we were unable to identify associations between the predictors and outcomes. Information about respondents’ characteristics such as disease history was limited and did not allow investigation of relationships between knowledge scores and prior experience or relevance of the risk message. Most of the assessments reported low response rates; therefore, the surveyed population may not be representative of the broader population of users. The limited number of assessments analyzed also restricted us to univariate analyses to investigate the determinants of the aforementioned outcomes.

Another limitation that we encountered is the lack of standardization across Medication Guide assessments. Although the majority of assessments reported important demographic and assessment information in a similar manner, the operationalization of knowledge varied considerably, as every survey measured knowledge of primary drug risk differently. Some surveys were more direct in the questioning (e.g. “Does drug X cause Y? A. Yes or B. No”), but others required respondents to choose the correct REMS-related drug risk (e.g. “What can drug X cause? A. Rash, B. Tendon rupture, C. Allergic reaction, D. Death”). The apparent lack of reliability (e.g. test–retest, alternate-form, or internal consistency reliability) and validity testing (i.e. content validity, criterion validity, or construct validity) done within the Medication Guide assessment surveys raises questions whether drug risk knowledge was measured accurately, as well as what type of knowledge is being measured by the assessments. Furthermore, several important predictors of knowledge and comprehension (e.g., the Test of Functional Health Literacy in Adults (TOFHLA),¹⁹ or the Rapid Estimate of Adult Literacy in Medicine (REALM)²⁰) were not collected or reported in the Medication Guide assessments and the manner in which the patient surveys were administered only allows for the testing of current drug knowledge.

One of the goals of providing patient-directed written prescription drug information is to reduce or minimize the occurrence of serious adverse events. However, the ability of Medication Guides to reduce the occurrence of serious adverse events has not been established. In order to isolate and quantify the precise impact of Medication Guides, randomized controlled trials are needed, though clinical trials would not necessarily explain why Medication Guides promote understanding in some patients and not others. Furthermore, a clinical trial in which newly exposed patients are randomized to receive or not receive a Medication Guide may raise ethical concerns because certain patients would be deprived of written information about their medication.

Our findings are consistent with those of other studies that have reported poor patient understanding and comprehension of Medication Guides.^5,21,22 Even if the methodological weaknesses of the Medication Guide assessments are considered, it is unlikely that these assessments have substantially underestimated patient understanding. Moreover, our study illustrates the need to improve the Medication Guide assessments, including psychometric testing of the assessment instrument, representative sampling of relevant patient populations, and comprehensive ascertainment of patient and Medication Guide characteristics in the assessment of knowledge determinants. The FDA sought stakeholder opinion at a 2012 workshop² on how to best address the limitations of Medication Guide assessments and continues to work to improve the Medication Guide assessments. Enhanced Medication Guide assessments will hopefully deliver guidance for development of more effective Medication Guides. Near-term efforts should thus focus on improving the quality of Medication Guides and Medication Guide assessments. As well as, future studies are needed that expand the Medication Guide assessments beyond the evaluation of knowledge toward the ability of Medication Guides to reduce the occurrence of serious adverse events.

CONCLUSION

We found that patient understanding of the primary drug risk was low as measured by the Medication Guide assessments, a finding with important public health implications. Most of the Medication Guide assessments, specifically those with only Medication Guide elements, did not reach an acceptable level of primary drug risk knowledge. However, when a Medication Guide was used in conjunction with an ETASU or a Communication Plan, or there was only one REMS-related drug risk message identified within the Medication Guide, the knowledge rate of the primary drug risk may be improved. Taken together with other studies that have demonstrated the limitations of current patient-directed written prescription drug information,^5,21,22 our results highlight the need to improve the information patients receive about their medications, as well as respective methods of assessing patient-directed information.

KEY POINTS.

• Patient understanding of the primary drug risk was low as measured by the Medication Guide assessments.

• Most Medication Guide assessments did not reach the 80% knowledge threshold, but those associated with additional interventions were more likely to achieve it.

• Our study highlights the need to improve patient-directed information and the methods of assessing it.