Readability of Patient and Health Care Professional Targeted Dietary Supplement Leaflets Used for Diabetes and Chronic Fatigue Syndrome

Abstract

Objectives

The purpose of this study was to assess readability of patient and health care professional targeted dietary supplement (DS) leaflets used for diabetes mellitus (DM) and chronic fatigue syndrome (CFS) with a novel measurement tool and Flesch-Kincaid Grade Level (FKGL).

Methods

Patient and professional leaflets for DS used to treat DM and CFS from the Natural Medicines Comprehensive Database (NMCD) and Natural Standard (NS) databases were evaluated. Leaflets were analyzed using FKGL and the author-developed health information readability analyzer (HIReA). HIReA integrates lexical, semantic, syntactic, cohesion, and style features and yields values of −1 (very hard) to 1 (very easy).

Results

Patient-targeted leaflets substantially exceeded the consensus readability level (6th grade) as assessed by both FKGL (grade 13.0767) and HIReA (−0.2360). Professional leaflets were similarly more difficult to read as scored by HIReA (−0.7065) and FKGL (grade 14.7429). Most and least difficult-to-read sections in patient leaflets (NS/NMCD) were Related Terms (−0.8863)/Other Names (−0.8146), and Safety Concerns (0.0821)/Scientific Evidence (0.0629), respectively. Overall, leaflets in NS (−0.5721) were more difficult to read than those in NMCD (−0.3704). These differences appeared to be less pronounced when FKGL was used to assess the readability, indicating its lack of preciseness.

Conclusions

Readability for patient targeted DS leaflets is far more difficult than recommended levels. HIReA is a more precise method to measure readability than FKGL. The disparity between targeted levels of readability and measured levels may contribute to a lack of understanding by patients, with a resulting negative impact on adherence and outcomes.

Introduction

Use of dietary supplements (DS) in the United States has been measured by a number of national surveys yielding prevalence rates ranging from nearly half of the adult population¹ to 73% as recently reported by reported Timbo et al.² In this latter survey, DS most commonly used by responders included Echinacea, fish oil, ginseng, and St. John's wort. Additionally, the literature has demonstrated that certain subgroups and those suffering from specific medical conditions are independent predictors for likelihood to take DS. Patients living with diseases such as diabetes mellitus (DM) are significantly more likely to use complementary and alternative medicine (CAM) than nondiabetics (p < 0.0001).³ Similarly, those patients suffering from conditions that are inadequately treated by conventional medicine, such as chronic fatigue syndrome (CFS), are more apt to try CAM than nonfatigued individuals (p = 0.0006).⁴

Health care professionals are well positioned to initiate dialogue with patients about their use of DS for these types of conditions. Counseling and education can also be supplemented with written information, such as patient-targeted DS leaflets. Written information may be a particularly helpful tool as it has demonstrated the ability to have a positive impact on consumers.⁵ In order to optimize the use of this type of information, it should be constructed with a readability level appropriate for the target audience. However, over 100 million Americans struggle with comprehending and utilizing health information.⁶ Predicated on this and other considerations, the general consensus is that patient education material should be targeted near a sixth-grade (United States) reading level.⁷

The most commonly used measurement tool to determine readability is the Flesch-Kincaid Grade Level (FKGL). Other readability measures include the Flesch Reading Ease Formula, Fry Readability Graph, Gunning's Fog Index, and Simplified Measure of Gobbledygook. However, the ease of use of tools such as the FKGL (i.e., it is even available as a tool in Microsoft Word) is also its shortcoming, since it only considers sentence length and syllables per word. These tools have been used to measure readability in a variety of patient-targeted information including diabetes-related documents,^8–10 and Internet-based information on herbs used for cancer.¹¹ However, despite guidance existing for information on herbal resources for patients, no study as yet has examined the readability of patient-targeted information leaflets on DS used for DM or CFS.¹²

A novel, sophisticated tool was developed by the authors to measure readability with the intent of integrating factors absent in tools such as the FKGL.

Objective

The purpose of this study was to analyze and compare the readability of patient and health care professional–targeted DS leaflets used for DM and CFS with a novel measurement tool and the FKGL.

Methods

Database and leaflet selection

A list of dietary supplements (e.g., herbs, nutraceuticals, botanicals) used to treat DM and CFS was compiled via a search of the literature including: MEDLINE,^® EMBASE, IPA, and AltHealthWatch. Tertiary databases that focus on dietary supplement information and contain individual leaflets targeted for both patients and health care professionals were sought and identified. Natural Medicines Comprehensive Database (NMCD) and Natural Standard (NS) were two online databases that fit these criteria. To be included in the evaluation and comparison, DS leaflets for each condition had to be available in both databases as well as for both categories (i.e., patient and professional targeted). If any DS did not appear in both NMCD and NS, or did not provide both professional and patient-targeted leaflets, it was excluded from the initial list that was generated by the literature search. Hence, content from the Natural Standard/Harvard Medical School Monographs as well as the Flash Cards were similarly excluded. Patient and professional leaflets for supplements used in treating DM that satisfied the criteria were obtained and included: β-glucan, bitter melon, cinnamon, Gymnema sylvestre, panax ginseng, and stevia. Leaflets for supplements used in treating CFS were also procured and included: acetyl-l-carnitine, dehydroepiandrosterone, evening primrose, fish oil, melatonin, and S-adenosyl methionine (SAMe).

Readability tool creation and comparison

Unlike many readability tools, which only examine aspects such as syllables per word, sentence length or word overlap with pre-compiled easy word lists, the author-developed health information readability analyzer (HIReA) is built upon previously identified concepts relevant to readability including text features (i.e., number of characters/word, number of sentences/paragraph, number of words/sentence), syntactic aspects (i.e., parts of speech [POS] extracted; POS categories were noun, verb, pronoun, proper noun, particle, article, determiner, symbol, punctuation, possessive, preposition, adverb, and adjective), semantic features (i.e., average term and concept familiarity scores), cohesion (i.e., number of overlapping concepts in adjacent sentences), and white space ratio (the textual density and formatting criteria). The HIReA outputs a value of −1 (very hard) to 1 (very easy) as a measure of a document's readability, whereas the FKGL strictly provides grade levels. An earlier stage of development and testing of the HIReA has been detailed previously.¹³ Very easy texts (HIReA score of 1) can be understood by people with basic literacy (4th–6th-grade level), and very difficult texts are usually only comprehensible by domain experts. A dietary supplement leaflet with a HIReA score less than zero (i.e., a negative score) would require the reader to have some college education. Each DS leaflet was analyzed using the HIReA and the FKGL. Comparisons between professional and patient-targeted leaflets were made, and assessments of each leaflet subsection were also conducted.

Results

HIReA and FKGL ratings overview

Audience, disease type, and source

The professional and patient leaflets from each database were assessed for both disease states using each readability measurement tool for a total of 48 evaluations. DS leaflets were evaluated and compared by categorizing according to audience, disease type, and source; scores for HIReA and FKGL are detailed in Table 1. HIReA revealed that professional-targeted leaflets were substantially more difficult to read (−0.7065) than the patient targeted (−0.2360) leaflets for the 48 documents. A similar but smaller difference in readability was seen between DM (−0.5498) and CFS (−0.3927) documents. Similarly, we found overall that leaflets in NS (−0.5721) were more difficult to read than those in NMCD (−0.3704). It is notable that these differences appear to be less pronounced when FKGL was used to assess the readability, indicating its lack of preciseness (Table 1).

Table 1.

Readability by Target Audience, Disease Type, and Source

	HIReA score				Flesch-Kincaid Grade Level
Category	Average	Q1	Median	Q3	Average	Q1	Median	Q3
PAT	−0.2360	−0.3775	−0.1950	−0.1075	13.0767	11.7733	13.1209	14.3377
PRO	−0.7065	−0.8300	−0.7150	−0.5825	14.7429	13.8954	14.6539	15.7234
CFS	−0.3927	−0.7450	−0.4400	−0.1075	13.8615	13.1912	14.0517	15.0259
Diabetes	−0.5498	−0.7550	−0.5675	−0.3450	13.9581	12.9230	14.2515	15.0392
NMCD	−0.3704	−0.5775	−0.4400	−0.1913	12.9950	11.7733	12.9180	14.1643
NS	−0.5721	−0.8150	−0.7125	−0.3138	14.8246	14.1413	14.5532	15.5848

HIReA, health information readability analyzer; PAT, patient, PRO, professional; Q1, first quartile; Q3, third quartile; CFS, chronic fatigue syndrome; NMCD, Natural Medicine Comprehensive Database; NS, Natural Standard; 24 documents for each subcategory.

Readability of NMCD versus NS

The DS leaflets from NMCD and NS were further assessed for readability by HIReA and FKGL by their component parts. Patient-targeted leaflets were less difficult to read according to both measures in NMCD (HIReA− 0.1367, FKGL 11.8687) compared to NS (HIReA − 0.3354, FKGL 14.2847). Likewise, professional-targeted leaflets were harder to read in NS (HIReA − 0.8088, FKGL 15.3645) versus those in NMCD (HIReA − 0.6042, FKGL 14.1213). The overall average readability for both NMCD and NS patient-targeted leaflets was less than zero, indicating college level difficulty. However, notable sectional differences for the patient leaflets between NMCD and NS were observed and are detailed below.

Section analysis

Each leaflet consisted of several sections. All documents from each source, and those intended for a particular audience type, had the same subsections. The readability of these sections was also analyzed individually. Analysis by section, using HIReA, revealed that the most difficult-to-read section in patient leaflets for NS was Related Terms (−0.8863), which corresponded to the same type of section for NMCD but was titled Other Names (−0.8146). The easiest section to read in the patient leaflets for NMCD was Safety Concerns (0.0821) and for NS was Scientific Evidence (0.0629). Within each patient leaflet, half (3/6) of the sections for NMCD were within an acceptable readability range, whereas only 1 section of 7 in NS earned an acceptable readability score. Tables 2 and 3 outline all relevant descriptive statistics for each section of the leaflets found in NMCD and NS along with the corresponding FKGL grades.

Table 2.

Patient and Professional Leaflets by Section (NMCD)

	HIReA			FKGL
Section name	Average	Median	Q1, Q3	Average	Median	Q1, Q3
Patient leaflet
Other namesa	−0.8146	−0.9300	−0.97, −0.64	19.6907	19.6339	17.81, 22.00
What is it	−0.0542	−0.1250	−0.16, −0.07	5.7187	5.4029	3.36, 7.55
Is it effective	−0.1388	−0.1375	−0.17, −0.09	14.1399	14.0728	12.89, 14.61
How does it work	−0.0554	−0.0850	−0.11, 0.00	8.1066	8.1064	7.05, 9.11
Safety concernsb	0.0821	0.0025	−0.06, 0.13	8.0168	8.0629	7.65, 8.43
Interactions	−0.1438	−0.1150	−0.22, −0.02	12.7072	12.4176	11.41, 13.77
Professional leaflet
Also known as	−0.7933	−0.8875	−0.95, −0.67	19.0737	19.1932	13.88, 23.58
Scientific name	−0.5004	−0.4900	−0.78, −0.23	22.7578	24.3502	19.68, 27.46
People use this for	−0.5892	−0.6900	−0.81, −0.30	18.2302	18.2151	15.47, 20.26
Safetya	−0.9321	−0.9450	−0.98, −0.91	9.6650	9.8426	8.96, 10.42
Effectiveness	−0.7129	−0.7475	−0.89, −0.58	13.2878	13.6454	12.05, 14.56
Mechanism of action	−0.8779	−0.9250	−0.94, −0.85	15.0343	14.8974	14.13, 15.35
Adverse reactions	−0.7717	−0.8975	−0.93, −0.75	13.1188	13.3519	12.35, 14.61
Interactionsb	−0.4963	−0.5850	−0.63, −0.44	16.5631	16.7775	15.30, 17.71
Dosage/administration	−0.6383	−0.7450	−0.79, −0.49	13.2847	13.2726	11.80, 13.95
Editor's comments	−0.6058	−0.5650	−0.86, −0.48	13.1986	12.8081	11.55, 14.99

HIReA, health information readability analyzer; NMCD, Natural Medicine Comprehensive Database; Q1, first quartile; Q3, third quartile; FKGL, Flesch-Kincaid Grade Level.

^aIndicates hardest section.

^bIndicates least difficult section; evaluated 12 documents each for professional leaflets and patient leaflets.

Table 3.

Patient and Professional Leaflets by Section (NS)

	HIReA			FKGL
Section name	Average	Median	Q1, Q3	Average	Median	Q1, Q3
Patient leaflet
Related termsa	−0.8863	−0.9375	−0.98, −0.85	28.9894	28.2567	23.48, 33.68
Synonyms	−0.5846	−0.5800	−0.77, −0.45	14.7009	14.6260	13.61, 15.38
Scientific evidenceb	0.0629	0.0175	−0.03, 0.08	13.0968	12.8017	12.68, 13.80
Tradition/theory	−0.5075	−0.5325	−0.68, −0.41	24.1679	21.7670	20.15, 29.72
Dosing	−0.3983	−0.3575	−0.64, −0.19	12.4706	12.6503	11.87, 13.02
Safety	−0.3246	−0.2875	−0.45, −0.20	13.3046	13.2601	12.77, 14.01
Interactions	−0.2458	−0.2475	−0.43, −0.07	13.5564	13.1288	12.38, 14.41
Professional leaflet
Synonymsa	−0.9488	−0.9525	−0.97, −0.94	29.4332	30.0594	24.98, 31.09
Effectiveness	−0.8763	−0.9075	−0.94, −0.86	18.4383	17.6172	17.11, 18.56
Dosing/toxicology	−0.5242	−0.4350	−0.88, −0.27	13.6618	14.1258	12.36, 14.68
Precautions	−0.6879	−0.6950	−0.82, −0.58	14.4580	14.8468	13.91, 15.07
Interactions	−0.6925	−0.6800	−0.85, −0.52	16.0690	16.2432	14.69, 17.15
Mechanism of action	−0.9296	−0.9525	−0.98, −0.92	16.6438	16.7223	15.15, 18.06
Historyb	−0.4446	−0.3850	−0.57, −0.24	12.9608	12.6659	12.09, 13.80
Evidence discussion	−0.8013	−0.8725	−0.93, −0.73	14.5151	14.5379	14.00, 15.04
Products studied	−0.5604	−0.5800	−0.93, −0.38	13.6061	12.9109	10.90, 15.69

NS, Natural Standard; HIReA, health information readability analyzer; FKGL, Flesch-Kincaid Grade Level; Q1, first quartile; Q3, third quartile.

^aIndicates hardest section.

^bIndicates least difficult section; evaluated 12 documents each for professional leaflets and patient leaflets.

Dimensional analysis

As noted previously, HIReA measures a document's readability along five dimensions: lexical, syntactic, semantic, cohesion, and stylistic. Table 4 lists the average readability for the leaflet categories along each of these dimensions (scale and interpretation similar to that of overall readability). Overall, semantic (or vocabulary) difficulty appears to be the most outstanding issue. In patient-oriented materials, the lack of cohesion is another substantial problem. It can also be observed that irrespective of the criterion of classification, a category found to be harder is more difficult to read along more than half of the measured dimensions. For instance, a measure of success was considered to be four of the five dimensions measured as being easier after converting the professional to patient-targeted leaflets (Table 4).

Table 4.

HIReA Readability by Dimension

Category	Lexical	Syntactic	Semantic	Cohesion	Style
PAT	−0.0571	−0.1962	−0.4469	−0.6119	0.2800
PRO	−0.5038	−0.5329	−0.9767	−0.1225	−0.5335
CFS	−0.3593	−0.2929	−0.5565	−0.2169	−0.1713
Diabetes	−0.2014	−0.4362	−0.8671	−0.5175	−0.0823
NS	−0.3908	−0.3433	−0.7738	−0.3004	−0.3923
NMCD	−0.1700	−0.3858	−0.6498	−0.4340	0.1387

HIReA, health information readability analyzer; PAT, patient, PRO, professional; CFS, chronic fatigue syndrome; NS, Natural Standard; NMCD, Natural Medicine Comprehensive Database; 24 documents for each subcategory.

Discussion

Dietary supplements such as herbals are often used by patients suffering from conditions such as DM and CFS. Unfortunately, many patients equate natural with safe,^14,15 and not all health care providers have adequate training and knowledge to counsel patients about these products.^16–18 One method to improve counseling may be the provision of consumer-targeted leaflets about these products to their patients. Since the general consensus is that patient education materials should be prepared at a sixth-grade level (United States), it is important to know whether these leaflets are reaching or approaching that target of readability.

This study found that overall scores for DS patient leaflets used for DM and CFS fell short of the desirable readability target. This is despite the fact that the leaflets are from databases generally recognized as the closest to DS informational “gold standards.”¹⁹ This is likely due to the manner in which the “Consumer Information and Education Leaflets” are produced by NMCD and the “Patient Handouts” are created by NS. For NMCD, the text that populates its patient leaflets is directly drawn from their professional monographs. The patient leaflets contain a much smaller volume of content than professional leaflets and certain steps were taken to tailor the document to patients (e.g., converting headings within the leaflet from “Mechanism of Action” to “How does it work?”). However, the content largely resembles that of the source material, which accounts for the undesirable readability scores in the patient leaflets. For NS, the patient leaflet is actually the exact same document as their Bottom Line Monograph, which was created as “brief evidence-based reviews” for patients and professionals.²⁰ NS's interchangeable Bottom Line Monograph/Patient Handout is essentially an abbreviated version of their comprehensive Professional Monograph. Thus, rather than originating as a document targeted to patients, it is really an abridged version more suitable for a busy health care practitioner who only wants the aptly titled “bottom line” for the DS. This reduction in the amount of content rather than tailoring of content to patients accounts for the difficult readability as assessed by both HIReA and FKGL. It is important to note that NS also offers their Bottom Line Monographs in Spanish, but the readability of those leaflets was not assessed in this study. The readability was better in NMCD versus NS patient leaflets, but both were so difficult to read based on the overall score as to render the point moot and the leaflets of suspect use for many patients. However, some promise was shown as half of the key sections in NMCD patient leaflets were at or close to target readability. Additionally, one corresponding section for each patient leaflet (i.e., Other Names for NMCD and Related Terms for NS) was so difficult to read, and of arguable importance, that eliminating it substantially reduced the difficulty in readability.

The concern in producing and advertising education materials as suitable for patients is that health care practitioners may be tempted to give them to patients, as if they were medication leaflets accompanying a prescription. Similarly, institutions or individuals in decision-making capacities may elect to seek institutional subscriptions to these materials, believing they are providing a service to either their member health care professionals or to their consumer members. Based on the results of this study, the decision-making process may need to be altered. If practitioners use these leaflets as a substitute for counseling or direct their patients to them as a reference, the difficult readability may exacerbate a lack of understanding and ultimately negatively impact outcomes. Of course, health care practitioners should only be using these leaflets as a supplementary tool in efforts to counsel and educate their patients, but misconceptions about their utility could prove detrimental. These patient leaflets could still have a role—especially if their readability was improved—but should be used within the framework of the six steps of counseling on DS as outlined by Ashar and Rowland-Seymour.²¹

The dimensional analysis of the documents reveals that improving the readability of the leaflets requires improvement on all aspects of the document, with an emphasis on semantic difficulty. While all the documents can be made significantly more consumer friendly by simplifying the vocabulary, this may not be sufficient to achieve the target readability level; the lexical and syntactic constructs need to be concurrently improved without loss in cohesion.

In this study, we used both FKGL and the author-developed HIReA. FKGL appears to underestimate the difficulty of some documents. For instance, the readability of professional-targeted content was estimated to be roughly college level by FKGL (e.g., 14th or 15th grade), whereas according to the more precise HIReA (-0.7065) they would generally require some graduate school training to comprehend. An additional advantage of HIReA is that it allows us to perform more detailed analysis of readability and identify problematic aspects. Both NS and NMCD, in spite of using different methods to tailor leaflets to patients, have ultimately used professional-targeted leaflets as the starting point for patient-targeted leaflets. This approach may be inherently flawed, and it remains unclear whether its readability can be modified to an acceptable level. Treating creation of patient-targeted leaflets as an independent task may be more beneficial. The difference in the readability between leaflets related to CFS and DM may also indicate that it is harder to create readable documents for certain disease domains owing to their intrinsic complexity and nature of the information that needs to be conveyed.

Limitations

The chief limitation of this study is that the DS patient leaflets for DM and CFS examined only represent a small subset of the total number of leaflets in these databases. It is possible, though very unlikely, that these leaflets were substantially atypical in comparison to the readability of other leaflets. Even assuming variation between DS, conditions, and databases, the fact that every patient leaflet evaluated in this study would require college education and some domain expertise (based on the scores) largely ameliorates concerns about the representativeness of this admittedly limited sample. Also, while the reliability of the readability scores is high for both tools when used to examine the entire leaflet, some of the individual sections are not long enough to make them ideal candidates. As such, the readability values reported in this study should be considered to be more reliable at the leaflet level than in the sectional analysis. Finally, because this is an observational study and cannot determine causality, it is unknown whether improving the readability of the DS patient leaflets in these databases would result in improvements in patient adherence or outcomes.

Future directions

Our study develops a foundation for the use of this tool to assess readability of health-related text; however, it is but one step along the research continuum in this area. The next logical step in the transformation to translational research may very well be to develop complementary user readability evaluation studies.²² Subjects could be recruited and stratified by educational/reading levels to correlate findings with HIReA along with assessing user understanding of study materials via a set of evaluative questions. This process, in turn, could advance the results toward a true endpoint and further cement the utility and applicability of HIReA for its designed purpose.

Conclusions

Differences in readability were seen between professional- and patient-targeted herbal leaflets. Overall readability for patient-targeted leaflets is far more difficult than recommended levels. The HIReA is a more precise method to measure readability than tools such as the FKGL. The disparity between targeted levels of readability and measured levels may contribute to a lack of understanding by patients, with a resulting negative impact on adherence and outcomes. Health care practitioners and institutional decision makers should examine the readability of patient leaflets prior to adopting, recommending, or implementing their use for educational and counseling purposes.

Acknowledgments

This work is supported in part by National Institute of Health grants R01 LM07222, and R01 DK75837. Results of this study were presented in poster form at the 43rd American Society of Health-System Pharmacists Midyear Clinical Meeting in Orlando, FL.

Disclosure Statement

Dr. Clauson serves as a volunteer reviewer for Therapeutic Research Faculty, which produces NMCD. Dr. Zeng-Treitler and Mr. Kandula are affiliated with Harvard Medical School, but have no connection to the Natural Standard/Harvard Medical School Monographs.