Readability of Patient-oriented Online Dermatology Resources

Abstract

Background: Supplemental educational reading material is of no value to patients if it is not read and comprehended. Objective: Using standardized research tools, online patient education materials were comparatively assessed for readability and length in words to identify the strengths and weaknesses of widely utilized sources. Methods: Three sources of patient-education material on the internet (WebMD.com, Wikipedia.org, and MedicineOnline.com) were compared with materials produced by the American Academy of Dermatology for readability utilizing Flesch-Kincaid Grade Level and Flesch Reading Ease Scale. Automated word counts were used to determine the length of each educational piece. Results: The information presented in American Academy of Dermatology electronic pamphlets on the internet is significantly harder to comprehend than MedicineOnline.com, but easier than Wikipedia.org. The latter site proved significantly harder to comprehend than all other sources. The American Academy of Dermatology electronic pamphlets and MedicineOnline.com materials were the most concise, averaging 1,200 words or less, although this was not a statistically significant difference in length compared to other online patient-education resources. No single source of online patient-education material demonstrates optimal features with regard to each of these parameters. Limitations: Only 15 topic areas in the four most commonly accessed sources of patient information were analyzed in this study. Conclusion: No single source of commonly used internet patient-education material demonstrates optimal features with regard to readability, length, and presence of photographic illustrations. These educational materials should target a length of 1,200 words, be illustrated with clinical images, and readability should correspond with the national average reading level.

Health literacy has been defined as, “The degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions.”¹ The educational component of the dermatological encounter is of critical importance in fulfilling this capacity. Unfortunately, time restraints, patient anxiety, and disparities in educational level of the patient and physician impact patient education in a negative manner. Educational materials are designed to reinforce and supplement the information provided during the patient visit and may serve to increase patient adherence.² Utilizing these tools is associated with improved patient adherence and completion of colorectal screening tests³ and improved asthma management.⁴ In order to improve health literacy, it is of great importance to provide the patient with readable and comprehensible information. In addition, physician recommendation of a reliable website reduces the likelihood that patients will be influenced by inaccurate information.⁵

Readability, the ease with which written materials are read, is an essential factor in assessing the utility of a patient-oriented information resource.⁶ Increasing the readability of dermatological education materials, judged by scientific indices, has been associated with improved overall patient comprehension.^7,8 The Flesch-Kincaid Grade Level (FKGL) approximates the readability of a passage of text to a United States grade level.⁹ The FKGL, however, accurately measures reading materials only in the 3^rd to 12^th grade range.¹⁰ The Flesch Reading Ease Scale (FRES) grades readability on a 0-to-100 scale rather than on an educational grade scale.¹¹ Documents with higher numbers are more easily read. Scores ranging from 70 to 100 are considered “fairly easy” to “very easy” to read, respectively. Scores from 60 to 70 are considered to be of normal or standard difficulty (Table 1).¹²

Table 1.

Interpretation of Flesch Reading Ease scores

READING EASE SCORE	DESCRIPTION OF STYLE	TYPICAL MAGAZINE	SYLLABLES PER 100 WORDS	AVERAGE SENTENCE LENGTH
00–30	Very Difficult	Scientific	192	29
30–50	Difficult	Academic	167	25
50–60	Fairly Difficult	Quality	155	21
60–70	Standard	Digests	147	17
70–80	Fairly Easy	Slick Fiction	139	14
80–90	Easy	Pulp Fiction	131	11
90–100	Very Easy	Comics	123	8

Adapted from Flesch R. A new readability yardstick. J Appl Psychol. 1948;32:221–233.

The average reading level in the United States is between the eighth and ninth grade level,^10,13 and 36 percent of all Americans fall into the basic or below basic health literacy levels.¹⁴ A 2003 study by the National Center for Educational Statistics showed that patients with basic and below basic levels of health literacy were more likely to gather information from books, pamphlets, newspapers, and magazines rather than the internet/web sources. This demographic, however, may rapidly change in the future. The percentage of homes with Internet connections in the United States has increased to 62 percent, up from 18 percent in 1997.¹⁵

This study is designed to provide the evidence needed to make recommendations regarding online resources. The readability and length of the American Academy of Dermatology (AAD) online pamphlets, WebMD.com (WebMD), Wikipedia.org (Wikipedia), and Medicine Online.com (MedicineOnline) were assessed. The length of the article was included as an important characteristic because length impacts the readability of educational materials. Even motivated patients have a finite amount of time to read information and a limited attention span.¹⁶

Materials and Methods

The AAD educational pamphlets cover 62 topics and have been available free of charge online for the past 10 years. Eleven of the 15 topics included in the study were chosen from the list of 20 “top-selling” AAD pamphlets (AAD Internal Report, Pamphlet Editorial Board, March 2009). These pamphlets include Seborrheic Keratoses, Actinic Keratoses, Seborrheic Dermatitis, Eczema, Basal Cell Carcinoma, Acne, Psoriasis, Squamous Cell Carcinoma, Molluscum Contagiosum, Malignant Melanoma, and Tinea Versicolor. Four topics (Herpes Simplex, Herpes Zoster, Genital Warts, and Alopecia Areata) were chosen because they represent a spectrum of common and treatable dermatological disorders that may be particularly distressing to the patient due to their social ramifications.

The internet sources for this study were chosen using a Google search (“Online Medical Information”) in April 2009. Approximately 107 million pages were identified. The top five sources of information (WebMD.com, Medicinenet.com, The Online Merck Manual Library, Emedicine.medscape.com, and MedicineOnline.com) were considered for the study. Medicinenet was eliminated because it is a part of the WebMD network and contains similar information, although it is accessed less frequently on the web.^17,18 The Online Merck Manual Library and Emedicine.com were eliminated because they are references for medical professionals rather than patients.

Wikipedia.org is an online encyclopedia that may be edited by anyone, regardless of his or her qualifications.¹⁹ It was chosen for inclusion in the study because a Google search of the 15 chosen topics returned Wikipedia articles within the top five results for every topic, and as the top article (the “I'm Feeling Lucky” article) for 5 of the 15 topics. In addition, a study in the American Journal of Informatics concluded that Wikipedia is a highly utilized source of health information on the internet.²⁰

The entire body of text from each of the 15 topics was entered into Microsoft Word to evaluate the FKGL, the FRE score, and word count. The FKGL equation uses the number of words in a sentence and the quantity of syllables per word to estimate the American grade level education required for comprehension of a passage of written text. Using this formula, lower calculated results correlate with higher readability.⁹ Flesch-Kincaid Grade Level = 0.39 (words/sentence) + 11.8 (syllables/word) − 15.59. The Flesch Reading Ease formula similarly uses the number of syllables per 100-word block and the number of words per sentence to calculate the readability of a block of text. Unique to this formula, higher calculated results imply more readable text.^11,12 Flesch Reading Ease = 206.835 − 0.846 (word length) − 1.015 (sentence length). All related text for each source was utilized without references, unrelated information, self-assessment tools, or links to third-party sites.

The presence or absence of clinical photographs was recorded. There was no attempt made to judge the accuracy of material presented in these websites that would have required a committee of experts to grade the content of each educational piece. No human subjects were used in this research and, therefore, the Helsinki Protocol does not apply.

The three variables (FRE, FKGL, and word count) were analyzed according to source using SPSS version 17.0.1. The sources were then compared using a One-way Analysis of Variance (ANOVA) test with a Post-Hoc Scheffe test.

Prior to analyzing the data, the authors established “ideal” criteria for the readability of the resources. As the national readability has been stated to be between the 8^th to 9^th grade level,^10,13 the authors set the level of acceptable readability to be less than or equal to a FKGL of 8.9 and greater than or equal to 60.0 for the FRE.

Results

The FKGL, FRE, and word count for each topic are sum-marized in Table 2. Average and median statistics for each website are summarized in Table 3. All tested materials fell between 6.90 and 12.00 on the Flesch-Kincaid Grade Level (Figure 1). Con-sidering all 58 individual educational materials, 23 had a FKGL below the established 8.9 criteria of FKGL. These included 3 of 23 (13%) of AAD Internet pamphlets, 8 of 23 (35%) found on WebMD, and 12 of 23 (52%) on MedicineOnline. None of the Wikipedia articles was below the ninth grade level, and 11 of 15 Wikipedia articles exceeded 12.0 on the FKGL.

Table 2A.

Readability and length of selected online sources

AAD PAMPHLETS
TOPIC	FLESCH-KINCAID GRADE LEVEL	FLESCH READING EASE	WORD COUNT
Acne	9.4	52.0	989
Actinic keratosis	7.2	66.1	876
Alopecia areata	9.0	55.1	846
Basal cell carcinoma	8.9	55.6	999
Eczema	11.1	41.1	1015
Genital warts	9.6	50.4	655
Herpes simplex	9.5	55.3	1321
Herpes zoster	9.6	52.4	934
Melanoma	9.8	48.6	1132
Molluscum contagiosum	9.4	52.3	696
Psoriasis	11.5	39.5	1881
Seborrheic dermatitis	11.1	43.1	498
Seborrheic keratosis	7.7	61.1	795
Squamous cell carcinoma	9.8	50.7	4971
Tinea versicolor	10.6	50.9	523
AAD Average	9.6	51.6	1208.7
AAD Median	9.6	52.0	934

WebMD
TOPIC	FLESCH-KINCAID GRADE LEVEL	FLESCH READING EASE	WORD COUNT
Acne	7.1	64.8	3194
Actinic keratosis	9.2	54.0	679
Alopecia areata	8.4	59.1	925
Basal cell carcinoma	10.9	47.2	1021
Eczema	10.7	44.2	2069
Genital warts	8.5	61.1	4273
Herpes simplex	9.0	58.7	21097
Herpes zoster	8.5	58.1	3793
Melanoma	9.0	54.7	5673
Molluscum contagiosum	8.4	56.0	1722
Psoriasis	8.9	53.6	5511
Seborrheic dermatitis	12.0	26.8	315
Seborrheic keratosis	8.5	56.0	1242
Squamous cell carcinoma	11.4	41.5	2719
Tinea versicolor	8.8	57.0	1833
WebMD AVERAGE	9.3	52.9	3737.7
webMD MEDIAN	8.9	56.0	2069

Table 3.

Summary of findings

INFORMATION SOURCES	READABILITY	PHOTOGRAPHIC ILLUSTRATION	LENGTH	PHYSICIAN REVIEW
AAD	+	+++	+	+++
WebMD	+	−	−	+
Wikipedia	−	+	−	−
MedicineOnline	+++	−	+	+

− = undesirable; + = reasonable; +++ = exceptional

Figure 1.

Mean and median Flesch-Kincaid Grade Level for patient-education websites

When examining the entire body of educational materials, only 8 of 58 (14%) of the articles reached a FRE of 60.0. Of these, two were AAD pamphlets, two were from WebMD, and four were from MedicineOnline. No articles from Wikipedia exceeded a FRE of 60.0 (Figure 2).

Figure 2.

Mean and median Flesch Reading Ease for patient-education websites

One-way ANOVA for the four sources across the FRE and FKGL measures showed that there was at least one significant difference for all compared measures among the sources (p<0.01) (Table 3). The Scheffe test, which allowed a further delineation of these significant differences, produced the most striking finding. In all three measures, the AAD pamphlets, the WebMD re-sources, and the information on MedicineOnline were not sig-nificantly different from one another, with just one exception: Medicine Online materials demonstrated sig-nificantly better readability scores (p=0.02) regarding the FKGL (8.1) than the AAD pamphlets (9.6) and WebMD (9.3). MedicineOnline's FRE score, although better (58.2), proved to be not statistically more readable than the AAD pamphlets (51.6) and WebMD (52.9).

Wikipedia had the worst readability with respect to both FKGL (11.8) and FRE (35.5). All of these values indicate significantly worse readability than the AAD Pamphlets, WebMD, and Medicine Online (p<0.01). The median results for each of the information sources had a similar profile (Table 2). Interestingly, the AAD pamphlets had neither the best nor the worst readability according to any included parameter.

Table 2b.

Readability and length of selected online sources

WIKIPEDIA
TOPIC	FLESCH-KINCAID GRADE LEVEL	FLESCH READING EASE	WORD COUNT
Acne	12	35.3	5762
Actinic keratosis	12	40.8	643
Alopecia areata	11.1	47.5	1311
Basal cell carcinoma	12	34.9	2439
Eczema	12	31.8	4576
Genital warts	11.7	42.4	1173
Herpes simplex	12	29.7	5481
Herpes zoster	12	34.2	3243
Melanoma	12	31.6	5194
Molluscum contagiosum	10.4	48.2	1831
Psoriasis	12	28.6	5991
Seborrheic dermatitis	12	27.9	671
Seborrheic keratosis	11.4	38.7	616
Squamous cell carcinoma	12	30.6	1550
Tinea versicolor	12	29.6	740
WIKIPEDIA AVERAGE	11.8	35.5	2748.1
WIKIPEDIA MEDIAN	12	34	1831

MedicineOnline
TOPIC	FLESCH-KINCAID GRADE LEVEL	FLESCH READING EASE	WORD COUNT
Acne	8.6	56.1	1099
Actinic keratosis	7.9	57.9	3651
Alopecia areata*	Null	Null	Null
Basal cell carcinoma	6.9	67.5	767
Eczema	8.9	53.3	787
Genital warts**	8.6	57.9	1074
Herpes simplex	8	60	1118
Herpes zoster	7.6	60.9	754
Melanoma	8.3	57.7	1574
Molluscum contagiosum	8.9	52.7	491
Psoriasis	9	51.3	987
Seborrheic dermatitis	7.3	59.6	388
Seborrheic keratosis	7.4	61.1	390
Squamous cell carcinoma*	Null	Null	Null
Tinea versicolor	7.7	60.8	532
MEDICINEONLINE AVERAGE	8.1	58.2	1047.1
MEDICINEONLINE MEDIAN	8	58	787

^*Topic unavailable

^**Article only had information pertaining to women

WebMD and Wikipedia had the longest individual articles. The average length of a WebMD article was 3,737 words, approximately three times longer than the average AAD pamphlet (1,208 words). Wikipedia entries averaged 2,748 words, approximately two times longer than the AAD articles. MedicineOnline had the shortest entries with an average length of only 1,047 words (Figure 3). While One-way ANOVA testing yielded a statistically significant difference between the groups (p=0.023), this difference was primarily due to the highly variable lengths of the individual articles. The Scheffe test demonstrated that the differences between the individual groups did not reach statistical significance. The online AAD pamphlets are liberally illustrated with clinical photographs. Wikipedia uses clinical images sparingly, and MedicineOnline and WebMD have no clinical photographs.

Figure 3.

Mean and median word counts for patient-education websites

Discussion

The internet has revolutionized patient education. The availability of millions of pages of material is, however, both a benefit and a detriment. Medical misinformation is disappointingly common on the internet. Many postings represent opinion with no peer review. On the other hand, Wikipedia is usually not considered an academic resource, yet a 2005 study in Nature stated that the accuracy of the articles in Wikipedia rivals that of Encyclopedia Britannica.²¹ Accepting the fact that educated patients can best manage their own care, it is the physician's duty to guide their patients to appropriate resources for further reading at home. In addition, guiding patients to proper information sources can potentially decrease medical liability.²²

AAD pamphlets are written by dermatologists and edited by a professional staff. They make liberal use of excellent quality, topic-specific photographs. Photographs may reinforce the patient's awareness, understanding, and recall of information presented in the pamphlets.²³

Unfortunately, AAD pamphlets did not score as highly on readability indices as would be ideal. The average Flesch-Kincaid Grade Level for the pamphlets was 9.6, indicating a difficulty that exceeds the national reading average. They are also significantly more difficult to read than the MedicineOnline sources. Only three of the AAD pamphlets were assessed at or below the ideal 8.9 FKGL. On the other hand, they were concisely written (average 1,208 words; median 934 words).

MedicineOnline had the best overall readability indices. Twelve of the 13 available articles had readability of less than 8.9 on the FKGL, a significant advantage over AAD materials. Writing is concise, averaging near 1,000 words. MedicineOnline, however, had some significant shortcomings. The website contains complete information for only 12 of the common dermatological disease topics studied and no clinical images are included. No information was found on two of the topics (squamous cell carcinoma, alopecia areata) and information specific only to women was available for the genital warts topic. This website also contains multiple dead links.

WebMD articles tended to have better readability indices than the AAD pamphlets; however, these findings were not statistically significant. Of the 15 articles on WebMD, eight were assessed at or below the 8.9 Flesch-Kincaid Grade Level. These articles were significantly easier to read than Wikipedia resources with respect to all readability parameters. Information was available pertaining to each of the 15 AAD topics. WebMD has a panel of physician editors who routinely review the material posted on the website,²⁴ but it is not clear if dermatologists review all dermatology-related material. WebMD articles also had the highest word count (average 3,737 words; median 2,069 words), requiring 2 to 3 times as long to read this information when compared to the AAD materials. Differences in the length of these articles, however, did not reach statistical significance. No clinical images are included.

Wikipedia is a well-known source of information on virtually any topic. Unfortunately, it had the worst readability indices of all of the studied resources, decreasing the utility of this resource for less-educated patients. The FKGL of the Wikipedia text (mean 11.8; median 12.0) suggests high school completion would be required to adequately comprehend this material. In addition, many of the FKGL for Wikipedia exceeded the maximum value of the index, 12.0. The collaborative authorship of Wikipedia articles by the general public without professional editing provides an explanation for the lack of attention to targeting the eighth grade reading level that is most appropriate. Clinical images are used inconsistently.

Conclusion

Considering the expansion of internet-based patient-education materials, it is critical that these sites be scrutinized with scientific rigor. The authors' overall assessment of these educational materials is presented in Table 3. There is room for improvement for all of the resources included in this study. Each site should consider rewriting materials aiming for FKGL less than or equal to 8.9 (average national reading level) and FRE scores between 60 and 70. This goal is attainable for the AAD, as MedicineOnline and WebMD have demonstrated the ability to draft these resources in more readable formats. This goal will be more difficult to attain for Wikipedia unless professional editors are introduced. Wikipedia, in particular, needs to clearly define portals with more readable text and shorter articles specifically for patient education. These articles should be separated from professionally developed educational materials in order to gain the authors' recommendation for patient education. In addition, many physicians will never be comfortable with sources that do not perform a professional, independent, peer-review process of all web material.

As always, patients should be advised to “ask their doctor” to clarify any information they read, whether it be from a pamphlet, on the internet, or via any other form of communication.