Readability of Online Patient Education Materials From the AAOS Web Site

Abstract

One of the goals of the American Academy of Orthopaedic Surgeons (AAOS) is to disseminate patient education materials that suit the readability skills of the patient population. According to standard guidelines from healthcare organizations, the readability of patient education materials should be no higher than the sixth-grade level. We hypothesized the readability level of patient education materials available on the AAOS Web site would be higher than the recommended grade level, regardless when the material was available online. Readability scores of all articles from the AAOS Internet-based patient information Web site, “Your Orthopaedic Connection,” were determined using the Flesch-Kincaid grade formula. The mean Flesch-Kincaid grade level of the 426 unique articles was 10.43. Only 10 (2%) of the articles had the recommended readability level of sixth grade or lower. The readability of the articles did not change with time. Our findings suggest the majority of the patient education materials available on the AAOS Web site had readability scores that may be too difficult for comprehension by a substantial portion of the patient population.

Introduction

In 2000, the American Academy of Orthopaedic Surgeons (AAOS) introduced the Internet-based patient education database, “Your Orthopaedic Connection [3].” Every day more than 35,000 people visit this Web site to access education materials on diverse orthopaedic conditions [2] (oral communication, Jim Ogale, AAOS Web site staff, June 6, 2007). The goal of this Web site is to enhance patient-physician communication by providing validated and up-to-date information about various orthopaedic conditions in a way that is “sensitive to diversity and readability and to strengthen the bond between physicians and patients [24].”

Readability of a text is the reading comprehension level a person must have to understand the written material and is an important determinant of a person’s ability to comprehend health information [1, 14, 38]. The Flesch-Kincaid (FK) grade formula is one of the most commonly used tools to assess the readability of written materials in terms of the academic grade [1, 14]. A higher FK grade level of a text indicates a greater level of difficulty to read and comprehend the material and thus requires more advanced reading skills than would be required of a text with a lower FK grade level. Organizations like the National Institutes of Health, the National Work Group on Cancer and Health, and the American Medical Association have recommended the readability of patient education materials should be no higher than the sixth-grade level [15, 34, 35]. The average readability of the US adult population is at the eight-grade level [18]. However, several studies suggest the readability of patient education materials in most medical specialties is beyond these recommended levels [1, 14, 18, 34]. We recently assessed the readability scores of patient education materials that were available at the AAOS and Pediatric Orthopaedic Society of North America (POSNA) Web sites, pertaining exclusively to pediatric orthopaedic conditions [5]. Only 2% of the articles available in 2007 had a FK readability score of the sixth grade level or less. It is unknown whether there is a trend toward greater readability on these two sites, but if the readability of articles remains high over a period of time, it would suggest lack of awareness of the importance of readability in the orthopaedic community.

Based on the cited reports of relatively high reading levels of patient education material, we hypothesized the readability level of the majority of patient education materials available on the AAOS Web site would be higher than the recommended FK grade level of sixth grade. We then asked whether there was a trend toward lower readability scores of articles available on the Web site with time. Finally, we asked whether the variability of the readability level of articles would differ between subject categories in the patient education Web site.

Materials and Methods

We searched the patient education database, “Your Orthopaedic Connection,” of the AAOS Web site [3] during April 2007 and downloaded all 663 patient education articles. Materials had been created between September 1999 and July 2006. We excluded articles written in a language other than English. We noted the subject category of each article and the date of the last update and saved the text as a separate Microsoft^® Word^® (Microsoft Corp, Redmond, WA) document. The text was copied as plain text format to avoid HTML tags [30]. Any information related to Web page navigation, copyright notices, postal addresses, phone numbers, uniform resource locaters (URLs), disclaimers, date stamps, author information, citations, references, feedback questionnaires, or hyperlinks were omitted. Followup editing was performed as recommended by Flesch and others [19, 21] by removing decimal points from numbers and colons and semicolons in sentences. The FK grade formula is best suited for text arranged in a paragraph as opposed to list format [32]. A running passage containing a minimum of 35 words was defined as a paragraph. To get the most representative sample, similar to methods adopted by other researchers, the three longest paragraphs of each patient education article were selected [1, 17]. Our Institutional Review Board granted a waiver for the study.

There were 663 articles grouped under 21 subject categories in the AAOS patient information Web site, “Your Orthopaedic Connection [3].” Forty of these articles were in Spanish and thus excluded from the study. Of the remaining 623 articles, 573 articles in 19 subject categories met the study’s inclusion criteria. Of the 573 articles, 147 were listed in more than one subject category, resulting in 426 unique articles.

The FK grade level of readability was assessed using Microsoft^® Office Word^® software (2003 Service Pack 2; Microsoft Corp). The built-in tool to measure readability is disabled by default, and the user has to enable it by sequentially selecting the commands “Tools,” “Options,” “Spelling and Grammar,” and then enabling the option “Show readability statistics” followed by clicking on the icon for “Spelling and Grammar” from the tool bar. The underlying formula for determining the FK grade level is as follows [21, 27]: (0.39 × average number of words per sentence) + (11.8 × average number of syllables per word) – 15.59.

The same individual (SUK) calculated all FK grade levels. Interobserver reliability was assessed by calculating the intraclass correlation coefficient using 30 randomly selected articles that were graded by another individual (SB). An intraclass correlation coefficient of 0 to 0.24 reflects poor correlation; 0.25 to 0.49, low; 0.50 to 0.69, fair; 0.7 to 0.89, good; and 0.9 to 1.0, excellent [33]. The intraclass correlation coefficient for assessing FK grade level was 0.96, indicating excellent interobserver reliability.

The mean and 95% confidence interval values of the FK grade level were calculated. Using descriptive statistics and analysis of variance, the readability grade scores of articles grouped under different subject categories were analyzed. A two-sample t test was used to compare the FK grade for articles in each subject category against the rest of the articles. The Pearson correlation coefficient (r) was calculated to study the relationship of the FK grade of articles with the date when the latest version of the article was available online. Statistical analysis was performed using the SAS^® software (Version 9.1; SAS Institute Inc, Cary, NC).

Results

The majority (98%) of the 426 unique articles on the patient education Web site had readability scores that were higher than the sixth-grade level. Only 10 articles (2%) had the recommended readability level of the sixth-grade level or less [15, 34, 35]. The mean FK grade level of the 426 articles was 10.4 (95% confidence interval, 10.2–10.6). Moreover, 85% of the articles had readability above the eighth-grade level (Fig. 1).

Fig. 1.

The distribution of FK readability grades of 426 unique articles that were available at the AAOS patient education Web site, “Your Orthopaedic Connection,” [3] is illustrated.

The readability level did not change with time (r = 0.0003) (Fig. 2). Thus the readability of articles remained high throughout the entire period.

Fig. 2.

A scatterplot displays the lack of relationship of the FK grade of the online articles with the date when the latest version of the article was available online. Pearson correlation coefficient r = 0.0003 (n = 426).

Although none of the 19 subcategories of articles achieved a mean FK grade level of sixth or lower (Table 1), we found substantial variation in the readability scores among the patient education articles in different subcategories. The FK grades of articles in the Hand and Foot sections were lower (p = 0.001 and p = 0.028, respectively), indicating easier readability, whereas the FK grades of articles in the General Information and Patient Stories categories were higher (p = 0.006 and p < 0.001, respectively) than the remaining articles.

Table 1.

Flesch-Kincaid grade level of AAOS patient education articles (n = 573) in 19 subject categories

Subject categories	Number of articles	Flesch-Kincaid grade level		p Value
		Mean	95% confidence interval
Arm	13	9.8	9.0–10.7	0.346
Arthritis	22	10.8	10.2–11.3	0.146
Children	48	10.1	9.7–10.5	0.191
Foot*	41	9.9	9.4–10.3	0.028
General information**	61	10.9	10.5–11.2	0.006
Hand*	32	9.7	9.3–10.1	0.001
Hip	26	10.4	9.7–11.0	0.99
Injury prevention	64	10.0	9.3–10.7	0.23
Joint replacement	28	10.8	10.0–11.7	0.212
Knee	34	10.9	10.4–11.5	0.101
Neck	7	9.9	8.9–10.9	0.549
Osteoporosis	11	10.9	9.2–12.6	0.373
Patient stories**	33	11.9	11.4–12.4	< 0.001
Patient-centered care	15	11.3	9.8–12.7	0.086
Shoulder	21	10.2	9.5–11.0	0.78
Sports/exercise	68	9.8	9.0–10.5	0.083
Spine	22	10.0	9.1–10.9	0.421
Tumors	12	9.6	8.7–10.4	0.172
Women’s health	15	10.9	9.8–12.0	0.307

p value refers to the t test comparing each subject category with the remaining articles; *mean FK grade level was significantly lower than the remaining articles; **mean FK grade significantly higher than the remaining articles.

Discussion

There is increasing concern regarding the disparity between the readability levels of patient education materials available online and the reading skills of the target population [1, 14, 18]. Based on our review of the literature and our recent report dealing exclusively with the readability scores of online pediatric orthopaedic materials [5], the readability of medical information is higher than generally recommended. Based on this literature we hypothesized the readability level of the majority of patient education materials available on the AAOS Web site, “Your Orthopaedic Connection [3]” would be higher than the recommended FK grade level of sixth grade. We then asked whether there was a trend toward lower readability scores of articles available on the Web site with time. Finally, we asked whether the variability of the readability level of articles would differ between subject categories in the patient education Web site.

We note several limitations in our study. First, there is an inherent weakness in the assessment of readability of health-related text using the FK grade level because this tool relies solely on the number of syllables in a word and the number of words in a sentence [19]; the number of syllables may not accurately reflect reading level. In the field of medicine, the unfamiliarity of nonprofessionals to medical terms, even if they are short, such as “lupus,” “physis,” and “colon,” can lead to underestimation of the reading skills required to fully comprehend medical literature with the use of the FK grading system [23]. Although comprehension of a given material may be enhanced by the addition of illustrative figures, improved layout, and appropriate use of font size and color [34], the FK grading tool does not assess these features. Other instruments such as the Suitability Assessment of Materials (SAM), a relatively new tool, can assess these factors to measure the comprehensibility of patient education materials [7, 18, 25, 37]. However, SAM is not as validated as the FK grade score, is more time-consuming [18], and is “inherently subjective [37].” Furthermore, the scoring system by SAM is not based on grades and thus is incompatible with the recommendations by healthcare organizations [18]. Second, we did not directly assess the reading skills of our patient population. There are many validated tools available to assess the reading skills of a given population. These include the Wide Range Achievement Test, Rapid Estimate of Adult Literacy in Medicine, and the Test of Functional Health Literacy in Adults [10]. However, in the context of the current study, because the patient education materials available through the AAOS Web site are available in the public domain and do not have a definite target population, trying to match the reading level of these materials to the reading skills of our patient population may be irrelevant. Third, we excluded articles that were available in a language other than English. Given that 41% of the people who are visiting the AAOS Web site are from countries where English is not the primary language, the need to make patient education articles easier to read is even more imperative [2]. Fourth, we did not assess the entire text of the articles on the Web site, but only the longest three paragraphs. We presume these would reflect the entire article. Finally, we limited our assessment of readability of orthopaedic patient education materials to one Web site. Patients and their caretakers may access more than one Web site to gain additional insight into their orthopaedic ailments. Nevertheless, we believe our study is relevant because orthopaedic surgeons often direct their patients to the AAOS Web site to find accurate, peer-reviewed, and up-to-date information. The Web site reach as measured by number of daily unique visitors of more than 35,000 adds further credibility to our sample selection [2] (oral communication, Jim Ogale, AAOS Web site staff, June 6, 2007).

Numerous authors report patients seeking orthopaedic care extensively use the Internet as a resource for patient education [8, 9, 13, 28]. Access to the Internet and its use to obtain health information are increasing globally at a rapid pace. One study reported three of every four patients attending an orthopaedic clinic have access to the Internet [13]. The Pew Research Center report found, in 2006 alone, more than 100 million people in the United States searched the Internet to find health information about diseases from which they or their relatives and friends suffer [20]. However, patient education materials available on the Internet may not have the appropriate reading level for the average person. A survey conducted in a pediatric orthopaedic outpatient clinic reported 2/3 of the respondents believed the health information available on the Internet was “too technical [4].” However, the authors did not assess the issue of readability. Our findings highlight the need for orthopaedic surgeons and educators to recognize the concept of readability while preparing and reviewing health education materials for their patients. Although only 2% of the English articles on the AAOS Web site had readability at the sixth-grade level or lower, creating musculoskeletal education articles with easier readability for patients seems attainable.

A recent national adult literacy survey found 40 million people in the adult US population have literacy skills equivalent to less than the fifth-grade level and another 50 million have reading skills between the sixth- and eighth-grade levels [29, 36]. Reading grade level is distinct from the last academic grade achieved in schools and colleges. Patients read approximately five grades lower than their highest attained academic grade [26, 31]. The reason for this difference between academic grade and reading level may be multifactorial and possibly related to a flaw in the education system and the fact that a substantial portion of the patient population belong to the lower socioeconomic status [18]. Health literacy is the best predictor of an individual’s health status [38]. It is defined as the “degree to which individuals have the capacity to obtain, process and understand health information and services needed to make appropriate health decisions [29, 36].” Low health literacy is associated with poor health status, increased hospitalization rate, poor compliance to treatment, missed appointments, and increased healthcare expenditure [6, 10, 18, 34, 36].

The improved access to the Internet is making the online population increasingly similar to the general population [20]. The fact that 33% of the “online health seekers” had only a high school diploma or less education [20] adds further proof to this assumption. Recently, a national survey found 20% of adults with “below basic health literacy’” are getting their health information from the Internet [29]. Some studies also suggest even people with good literacy skills prefer materials written in simpler format and low grade level [16, 37]. Furthermore, surgeons often are using patient education handouts printed from Web pages for their patients, and thus the readability level of Internet-based health information materials should serve the needs of all segments of society [23]. Although the data regarding readability skills of Internet users are lacking, most researchers use readability standards of the general patient population to assess the readability of online materials [11, 12, 16, 22, 23, 30, 37].

Readability of patient education materials can be assessed readily using the FK grade formula, with a relatively simple set of keystroke instructions and software available on most personal computers. The FK grade formula originally was developed for the US military in the early 1970s [27]. Since then, this instrument has been extensively validated and researched [14, 21, 31]. The advent of computers and software capable of automating the calculation has made the formula relatively simple, quick, and intuitive [14, 21, 31] and within the reach of almost every healthcare worker. As established in our study, the FK grade formula has very high interobserver reliability. Our findings support the hypothesis that the readability of patient education materials in the AAOS patient education database would be higher than the recommended level. In addition, we found the readability scores of the text did not improve with time. These findings may reflect the lack of awareness regarding the concept of readability in the orthopaedic community. Moreover, in a recent study [5], despite using a smaller and distinct set of online patient education materials dealing exclusively with pediatric orthopaedic conditions, we arrived at a similar conclusion with only 2% of the articles having readability scores of the sixth grade level or less. The method used in that study [5] also was different, in that the entire text was subject to the FK formula as opposed to assessing the readability score based on the longest three paragraphs, as in the current study.

In addition to the FK grade level available on the Microsoft^® Office software, there are other software packages, such as Corel^® WordPerfect^® Office X3 (Corel Corp, Ottawa, Ontario, Canada), Readability Calculations (Micro Power and Light Co, Dallas, TX), Readability Studio 1.1 (Oleander Solutions, Vandalia, OH), and InText (Social Science Consulting, Rudolstadt, Germany) [35], that have readability assessment tools. However, we believe the readability score of a given text should not be the sole criterion to develop patient education materials. The “Living word vocabulary” contains approximately 43,000 words arranged in various grade levels of complexity and can be used to substitute difficult words [15, 31]. To allow improved comprehension by a larger segment of the society, simpler words can replace complex medical jargon. A list of lay terms that can be used instead of medical terms can be found at the following Web sites: http://uuhsc.utah.edu/pated/authors/substitute2.html and http://plainlanguage.gov/howto/wordsuggestions/index.cfm [15]. Detailed instructions on making more comprehensible patient education literature are available through the American Medical Association [38].

Ensuring patients receive education materials they can understand is the responsibility of physicians, professional organizations, and healthcare institutions [15]. Our findings suggest a substantial portion of the patient education articles available at the patient education library of the AAOS Web site have readability scores that are higher than the sixth-grade level. These findings suggest the online material presented may be too difficult for comprehension by a substantial portion of the patient population. To enhance the patient-physician dialogue, orthopaedic educators should attempt to keep the readability level of the patient education materials at a FK grade level of sixth grade or lower. Such measures will help accomplish the goal of the AAOS Web site of “improving the communication between orthopaedic surgeons and their patients” [24] and positively influence the health outcome of our patients.