Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2011 Apr 1.
Published in final edited form as: Plast Reconstr Surg. 2010 Apr;125(4):1271–1278. doi: 10.1097/PRS.0b013e3181d0ab9e

Readability of ASPS and ASAPS Educational Websites: An Analysis of Consumer Impact

Oluseyi Aliu 1, Kevin C Chung 2
PMCID: PMC2845740  NIHMSID: NIHMS172906  PMID: 20072085

Abstract

Background

Patients use the internet to educate themselves about health related topics and learning about plastic surgery is a common activity for the enthusiastic consumers in the United States. How to educate the consumers regarding plastic surgical procedures is a continued concern for plastic surgeons when faced with the growing portion of the American population having relatively low healthcare literacy. The usefulness of health related education materials on the internet depends largely on their comprehensibility and understandability for all who visit the websites.

Methods

We studied the readability of patient education materials related to common plastic surgery procedures from the ASPS and ASAPS websites and compared them with materials on similar topics from 10 popular health information providing sites.

Results

We found that all analyzed documents on the ASPS and ASAPS websites targeted to the consumers were rated to be more difficult than the recommended reading grade level for most American adults and these documents were consistently amongst the most difficult to read when compared to the other health information websites.

Conclusion

The internet is an increasingly popular avenue for patients to educate themselves about plastic surgery procedures. Patient education materials provided on ASPS and ASAPS websites should be written at recommended reading grade levels to ensure it is readable and comprehensible to their audience.

Keywords: readability, internet, literacy, plastic surgery, websites

Internet use has increased substantially over the past decade with approximately 220 million Americans (greater than 70% of the population) being internet users.1 It has been postulated that roughly 8 in 10 of American internet users have sought information on health related topics on the internet.2 Studies on select patient populations have shown a consistent propensity amongst patients to seek health-related information on the internet.3-6 The search and acquisition of health-related information on the internet is impactful, as the majority of patients using this medium report that medical information found on the internet influences their medical decisions.7

Health literacy is an important consideration in healthcare because it directly affects the ability of patients to make informed decisions, follow health maintenance regimens and treatment-related instructions in navigating the healthcare systems. Health literacy is a significant factor in predicting health outcomes, and people with low literacy tend to have low patient compliance, increased complication rates, and ultimately incur greater healthcare costs.10-18 In extensive literacy assessments, nearly half of American adults are classified as being low or limited in basic language and health literacy19-20. Low literacy hinders the usefulness of the internet for the purpose of health-related patient education. The level or number of years of formal education attained poorly correlates with health literacy as studies show that reading grade level of the average American adult is an average of 4 grades below the highest level of education attained.10,21 Therefore, materials written for the purposes of health education should be written between 6th to 8th reading grade levels.10,15,22-23

The specific aim of this study is to evaluate the readability of patient education materials on the American Society of Plastic Surgeons (ASPS) and American Society of Aesthetic Plastic Surgeons (ASAPS) websites. Secondly, we compared the readability of patient education materials for common plastic surgery procedures in these two websites with the most consulted health information websites as listed by Consumer Reports WebWatch.24 We hypothesize that (1) patient education materials from the ASPS and ASAPS websites are more difficult to read and comprehend than information on the same topics obtained from the other websites, and (2) patient education materials from other websites are more likely to fall in the recommended 6th to 8th reading grade level range than materials from the ASPS and ASAPS websites.

Materials and Methods

From a 2009 ASPS report, we acquired information on 15 common plastic surgery procedures in the calendar year of 2008, including cosmetic, reconstructive and minimally invasive procedures for both men and women (Table 1).25 We searched the patient education section of the ASPS and ASAPS websites for materials on these 15 plastic surgery procedures. These procedures were also researched on the 10 most consulted websites for health care information on the internet as listed by Consumer Reports WebWatch (Table 2).24 One of the websites in the top 10 was eliminated, drugs.com, because it provided pharmaceutical information only, the 11th website on the list was chosen to make the number of analyzed websites 10 in all.

Table 1.

Analyzed plastic surgery procedures and their popularity

Procedure No. performed in 2008 (in thousands)
Breast augmentation 307
Rhinoplasty 279
Liposuction 245
Blepharoplasty 221
Abdominoplasty 121
Botox 5,014
Fillers (Hyaluronic Acid) 1,109
Chemical peel 1,048
Laser hair removal 891
Microdermabrasion 841
Laser resurfacing 400
Breast reduction (women) 88
Breast reduction (men) 17
Hair transplantation 17
Open in a new tab

Source: American Society of Plastic Surgeons (ASPS). The National Clearing House of Plastic Surgery Statistics.

Table 2.

Most popular websites for health information, by traffic.

  1. National Institutes of Health

  2. WebMD

  3. MSN Health & Fitness

  4. About Health

  5. MedicineNet.com

  6. Yahoo! Health

  7. MayoClinic.com

  8. RealAge

  9. AOL Health

  10. Drugs.com*

  11. QualityHealth.com

  12. Aetna InteliHealth

  13. KidsHealth

  14. Healthology

  15. RxList

  16. Everyday Health

  17. MedHelp.org

  18. Prevention.com

  19. eMedicineHealth

  20. familydoctor.org

Open in a new tab

Source: Consumer reports Webwatch/Healthratings.org.

*

Not included in analysis as it provided only pharmaceutical information.

Analyzed websites in bold type.

After identifying patient education materials on the select 15 topics on all 12 websites (ASAPS, ASPS and the top 10 Consumer Reports listed sites), they were copied and pasted onto Microsoft word (Microsoft Office 2007, Redman, WA) documents. We omitted tables, diagrams, illustrations and their captions were omitted.

The patient education materials were in clearly delineated sections of the ASPS and ASAPS websites, so a site search was unnecessary. The other 10 sites treat broad range of non-plastic surgery. Hence we began our search on these site's main pages by typing in medical terminology or colloquialisms associated with the procedures we were investigating. We selected the first article that discussed the subject even if there were more comprehensive articles down the list. We also did not go beyond the first set of returned article pages from our search. The rationale for these were to establish consistency in search practices and secondly, internet use pattern studies suggest that internet users generally spend a relatively finite time online when in search of information.26

We used three programs to analyze the readability of each documents. The Flesch-Kincaid (FK) model, a tool included on Microsoft office word software (version 2007; Microsoft Corp, Redman, WA). This tool was activated by clicking on the “show readability statistics” from the “spelling and grammar” option under the tools tab. The FK model uses the formula; grade level = 0.39 (words per sentence) + 11.8 (syllables per word) -15.59. It provides reading grade level scores between 5th to 12th grade levels. Hence, documents that are more difficult than the 12th grade reading level are reported as the 12th grade reading level. The FK model has been widely used for analysis of health education materials.27-28

The second tool used is the SMOG (Simple Measure of Gobbledygook) index, which estimates the number of years of education required to comprehend a piece of writing. The SMOG index uses the formula; grade level = 1.043 (square root of 30 (number of polysyllables/number of sentences)) + 3.1291. We used a web-based program implementing this formula. The SMOG index, recommended by the National Cancer Institute, is widely used to estimate the readability of health messages.29-30 Grade levels reported by this index correlate highly (0.985) with the actual years of education attained by test subjects who have 100% comprehension of test materials.31 The grade level results from the SMOG index are generally higher than those obtained from the same documents with the FK-model because the SMOG index employs stricter criteria for readability and is based on 100% comprehension.32

A third measure of readability, the Flesch Reading Ease (FRE) is also included in the Microsoft Office Word program and is calculated alongside the FK model. The FRE formula is: score (0-100) = 206.835 − 1.015 (total words/total sentences) − 84.6 (total syllables/words). A score on the FRE between 90-100 implies easy comprehension by an 11-year-old student, a score of 60-70 indicates average 13-15 year-old students can understand the presented material and a score of 0-30 is comprehended by college level students.

All texts were modified to prepare them for analysis according to stipulations given in the literature to avoid underestimation of difficulty. They were condensed into single paragraphs to avoid effects from paragraph breaks. Decimals, abbreviations, dashes, colons, semicolons, parenthesis, and slashes were removed from texts to avoid distorting sentence length. Copyright notices, disclaimers, author names and information, drug trade names and trade names of products such as injectables were also removed.33-34

Averages with standard deviations from the FK-model and SMOG tests and scores from the FRE test were calculated for each website. Averages were first compared between the ASAPS and ASPS websites. The ASAPS and ASPS averages were combined and treated as the control and then compared to the other ten websites individually. We excluded one of the 15 procedures, intense pulsed light (IPL), from the study because none of the websites had a dedicated textual composition for education related to it.

Statistical analysis

Several epidemiological studies show that reading ability is normally distributed in the population.35-39 Hence, comparisons of the calculated averages of the readability indices were compared using the unpaired Student's t-test with a significance level set at p < 0.05.

Results

A total 14 topics were analyzed on 12 websites. Approximately 150 articles were analyzed with three readability measurements. Only three websites had patient education materials on all 14 topics. The average number of topics treated by each website was 12.6. Articles varied in length from 55 words to 2787 words. There was no correlation found between article length and readability.

FK-model analysis

Patient education materials from the ASPS and ASAPS websites scored a mean reading grade level of 11.9 ± 0.2 and 12.0 ± 0.3 respectively. The range of average reading grade levels across all 12 websites was from 8.9 ± 0.9 to 12.0 ± 0.3. We compared averages from each of the 10 popular health related websites to averages from ASPS and ASAPS websites respectively and this showed that the patient education materials from the ASPS and ASAPS websites were more difficult to read than materials from all other websites and significantly so than nine of the 10 websites (Table 3). There was no significant difference between the average readability scores from the ASPS and ASAPS websites (p = 0.41).

Table 3.

Average grade level (FK-Model, SMOG index) and scores (FRE) of analyzed websites

FK-Model (RGL)* FRE Scores SMOG index (RGL) *
ASPS 11.9±0.2 34.6±7.1 14.5±1.3
ASAPS 12.0±0.3 35.3±5.4 14.3±1.1
Medline 10.0±1.5 53.0±9.2 12.8±1.5
WebMD 10.8±0.8 45.5±6.8 12.4±0.6
MSNhealth & Fitness 10.7±1.0 48.0±5.0 12.4±0.7
Abouthealth 10.6±1.5 49.5±10.1 13.3±1.6
MedicineNet.com 9.9±1.0 51.8±8.0 12.8±0.8
Yahoo! Health 8.9±0.9 59.2±4.4 11.9±0.7
Mayoclinic.com 9.5±1.4 52.7±8.6 12.2±1.2
Realage 11.5±0.8 40.9±11.2 14.9±2.0
AOLhealth 9.9±1.1 51.5±5.8 12.7±1.2
Qualityhealth.com 10.4±1.1 43.8±13.8 13.2±2.0
Open in a new tab

Comparison of the difficulty of ASPS and ASAPS (shaded) patient education information to that of 10 popular websites. Bold type shows ASPS/ASAPS significantly more difficult than corresponding website (p <0.05). SMOG index, recommended by National Cancer Institute for health education documents, has more stringent criteria and hence is judged to be more appropriate for readability analysis.

*

Reading grade level

FRE score analysis

The FRE analysis gives a reading grade level result, a score between 0 and 100 indicating the ease of reading and comprehending a document, the lower the number, the more difficult to read the document. The average readability scores from the ASPS and ASAPS websites were 34.6 ± 7.1 and 35.3 ± 5.4 respectively. There was no significant difference between the two websites, p = 0.77. The range of FRE score averages across all 12 websites is between 59.2 ± 4.4 and 35.3 ± 5.4. We compared FRE score averages from each of the 10 popular health related websites to the ASPS and ASAPS averages, it showed that the patient education materials from the ASPS and ASAPS websites were more difficult to read than materials from all other websites and were significantly more difficult to read than materials from eight of the 10 websites (Table 3).

SMOG index analysis

Like the FK-model, the SMOG index reports readability as estimations of reading grade levels. Reading grade levels were uniformly higher than those reported by the FK-model on the same articles. The range of reading grade level averages across all 12 websites are between 11.9 ± 0.7 and 14.9 ± 2.0. Although the ASPS and ASAPS websites made up 2 of the 3 highest averages, realage.com contained the most difficult articles to read according to this analysis. As with the FK-model and the FRE analyses, there was no significant difference between the averages of the ASPS and ASAPS websites, p = 0.66. However, ASPS and ASAPS websites contained significantly more difficult articles than seven of the 10 other websites (Table 3).

Discussion

Internet popularity has grown globally and the majority of Americans (79%) are engaged on line. 1,40 Amongst its myriad applications, search for information including health-related information is one of commonest uses of the internet today, with 8 in 10 internet users haven searched for health information. 2,41 Sixty-nine percent of people using the internet for health education issues do not discuss the information found with their physicians; additionally, 60% believe the information from internet sources is equal to or better than physician provided information.3,7 Unfortunately, several studies on internet-acquired health-related information have suggested deficiencies in the quality of information with regards to accuracy, comprehensiveness and relevance of content found. 42-49 This finding has also been borne out in plastic surgery literature. 50-51

Literacy assessments in the United States have existed since the 1930s, and these studies have included health literacy that is defined as the measure of an individual's ability to read, understand and apply health-related information in making decisions and following treatment instruction. 52 Patients with low literacy are 52% more likely to need hospital admissions. 53 Literacy is the best predictor of health status, more than income, employment status, age, education level, racial or ethnic group. 8-9,17-18 Patients with low literacy understand their chronic diseases less and engage in more risky behaviors as well.17-18, 54-55 These studies report that approximately half of American adults have low health literacy.19-20 Even though majority of the adult population has achieved a high school education, the average reading level of the American adult is between the 8th and 9th grade level. 56

Given the internet's popularity, physicians in different organization have taken advantage of it for dissemination of information regarding patient education and consumer service. However, if health information provided on the internet by physicians is to be useful, the comprehensibility of the materials has to be ensured. Reading grade level contributes immensely to the comprehensibility of documents12. This project was undertaken to evaluate the readability of patient education materials on the American Society of Plastic Surgeons (ASPS) and the American Society of Aesthetic Plastic Surgery (ASAPS) websites. Furthermore, we compared the readability of materials found on the ASAPS and ASPS websites with information on the same topics gathered from the 10 most popular websites for health-related information as rated by a credible source for consumer-related data.24

The importance of ensuring the readability of patient education materials on the ASPS and ASAPS websites is borne out of the assumption that the relevance, comprehensiveness and accuracy of information provided these websites is guaranteed. This is in contrast with general web-based health information that several studies have suggested to be unreliable in quality. 42-51 Of the 150 articles that were analyzed in this study, only 5 articles, which equates to 3% of articles, achieved between 6th and 8th grade reading level and none scored below the 6th grade reading level with the FK-model analysis. Using the stricter criteria of the SMOG index analysis, none of the articles achieved the 6th to 8th reading grade level. Additionally, the ASPS and ASAPS patient education materials were consistently on average the most difficult to read by using all three methods of analysis and some articles even reaching a reading grade level of 17, which translates to beyond college level reading. The logical inference from these findings is that majority, if not all, of the information provided for patient education is not comprehensible to the majority of patients who may have limited literacy capability.

ASPS and ASAPS websites can be made more accessible to the general public. There are resources available to achieve this goal including materials provided by federal institutions including the Centers for Disease Control and Prevention (Simply Put)23, Center for Medicare Education (Writing Easy-to-Read Materials)57, National Library of Medicine (Understanding Health Literacy and its Barriers)58, National Cancer Institute (Clear and Simple: Developing Effective Print Materials for Low-Literate Readers)59, state agencies like the Washington state Office of Health Promotion (Guidelines to Developing Easy-to-Read Health Education Materials)60, and private organizations like the Society for Technical Communication, Usability Special Interest Group (Readability Research)61.

We recommend improving readability of patient education materials on the ASAPS and ASPS websites by paying attention to the message content (text), formatting, and use of adjunctive educational materials such as illustrations. There is evidence that some of these efforts have been considered and addressed to different degrees by both websites. A good example of this is the use of bold type to represent emphasized points and subheadings that are designed as questions to focus the reader's attention on a “chunk” of the subject matter at a time. Additionally, each sub-topic of the subject matter is treated independently on its own page with cues to the next sub-topic at the end of the text, which is also equivalent to chunking. ASPS website uses bulleted lists to enhance the readability of a document. Both websites employ illustrations and other visuals to elaborate on technical details and represent potential results that can also improve the comprehension of a document.

However, when we analyzed the quantifiable contributors to readability, the potential for vast improvement on both ASPS and ASAPS websites is evident. Word choice is particularly germane in composing medical education materials. It is recommended that alternative words are used for complex words, medical terms, and acronyms.23 Medical terms and phrases such as “lipoplasty”, “capsular contracture”, “magnetic resonance imaging”, “irregular pigmentation” were not uncommon words on both websites. Although the ASPS website provided a list of definitions at the end of documents, this was on a separate page; definitions should be included in context.23 Additionally, sentences of 8-10 words are recommended to enhance readability.23 The average sentence length for patient education documents on the ASPS website was 20.3 ± 2.0 and 18.5 ± 4.0 for the ASAPS websites. To draw an inference, this will very likely make it difficult for many patients to read and comprehend these documents.

Although Friedman's and Hoffman-Goetz's discussion was aimed specifically at the development of cancer information websites, we believe it is also applicable to websites for patient directed medical information in general. They recommend presenting materials for different subpopulations, pretesting for readability and comprehension, engaging website designers, health writers and graphic illustrators to focus on clarity, plain language and appropriate illustrations.33 Links to scientific details should be provided for patients with sophisticated reading skills because this could enhance their comprehension.33 The American Cancer Society (www.cancer.org) is one example of an organization with keen attention paid to readability and comprehensibility of health education materials they produce. Editors and writers are specifically encouraged to use plain language; topics are presented at several reading grade and peer-counseling programs have also been implemented.12

Our search patterns on the internet, especially on the 10 websites that were compared to the ASPS and ASAPS websites were not based on observed or actual patient search patterns. We made assumptions about documents that patients would choose for their enlightenment. We did standardize the search patterns to minimize the bias in results and our analysis. Furthermore, we did not directly test the comprehension of any cohort of the patient education materials as part of the study; instead, we used readability formulas to predict the difficulty patients would have with the documents. In spite of these limitations, readability formulas are reliable at giving an idea about the skill required to read and comprehend a document as they are calibrated against more laborious and detailed readability surveys. Hence, they can and should be used in the process of preparation of patient education materials for the internet to ensure usefulness of these documents to their intended audience.

The ASPS and ASAPS have taken advantage of the growing popularity of the internet, especially the propensity patients have shown to search for health-related information on this modern portal of knowledge dissemination. It is imperative that the information provided be useful for the target audience. Readability analysis shows that patient education materials on ASPS and ASAPS websites are presented at reading grade levels too high to comprehend by most American adults. Pre-testing patient education materials amongst the target audience to evaluate for readability comprehensibility is warranted with appropriate changes made to ensure ease of reading. Many resources are available to direct this effort.

Acknowledgments

Supported in part by a Midcareer Investigator Award in Patient-Oriented Research (K24 AR053120) from the National Institute of Arthritis and Musculoskeletal and Skin Diseases (to Dr. Kevin C. Chung).

Footnotes

None of the authors has a financial interest in any of the products, devices, or drugs mentioned in this manuscript.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  • 1.Barboza D. New York Times; New York: [June 15 2008]. China Surpasses U.S. in Number of intersnet Users. Available: http://www.nytimes.com/2008/07/26/business/worldbusiness/26internet.html. [Google Scholar]
  • 2.Fox S. Pew Internet and American Life Project. Washington, DC: [July 31, 2008]. Eight in ten internet users have looked for health information online, with increased interest in diet, fitness, drugs, health insurance, experimental treatments, and particular doctors and hospitals. Available: http://www.pewinternet.org/PPF/r/156/report_display.asp. [Google Scholar]
  • 3.Diaz JA, Griffith RA, Ng JJ, Reinert SE, Friedmann PD, Moulton AW. Patients' use of the Internet for medical information. J Gen Intern Med. 2002;17:180. doi: 10.1046/j.1525-1497.2002.10603.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Beall MS, III, Beall MS, Jr, Greenfield ML, Biermann JS. Patient Internet use in a community outpatient orthopaedic practice. Iowa Orthop J. 2002;22:103. [PMC free article] [PubMed] [Google Scholar]
  • 5.Krempec J, Hall J, Biermann JS. Internet use by patients in orthopaedic surgery. Iowa Orthop J. 2003;23:80. [PMC free article] [PubMed] [Google Scholar]
  • 6.Wolf JM, Ritter M, Weiss AP, Akelman E. Access and use of the internet in a hand surgery population. Hand Surg. 2004;9:29. doi: 10.1142/s0218810404002121. [DOI] [PubMed] [Google Scholar]
  • 7.Fox S. Pew Internet and American Life Project. Washington, DC: [June 15, 2009]. Most internet users start at a search engine when looking for health information online. Available: http://www.pewinternet.org/PPF/r/132/press_release.asp. [Google Scholar]
  • 8.Weiss BD, Hart G, McGee DL, D'Estelle S. Health status of illiterate adults: relation between literacy and health status among persons with low literacy skills. J Am Board Fam Pract. 1992;5:257. [PubMed] [Google Scholar]
  • 9.Baker DW, Parker RM, Williams MV, Clark WS, Nurss J. The relationship of patient reading ability to self-reported health and use of health services. Am J Public Health. 1997;87:1027. doi: 10.2105/ajph.87.6.1027. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Weiss BD. Health literacy: A manual for clinicians. Chicago, IL: 2003. [June 15, 2009]. Available: http://www.ihconline.org/toolkits/HealthLiteracy/AMAHealthLiteracyManualClinicians.pdf. [Google Scholar]
  • 11.Weiss BD, Blanchard JS, McGee DL, Hart G, Warren B, Burgoon M, et al. Illiteracy among Medicaid recipients and its relationship to health care costs. J Health Care Poor Underserved. 1994;5:99. doi: 10.1353/hpu.2010.0272. [DOI] [PubMed] [Google Scholar]
  • 12.Merriman B, Ades T, Seffrin JR. Health Literacy in the Information Age: Communicating Cancer Information to Patients and Families. CA Cancer J Clin. 2002;52:130. doi: 10.3322/canjclin.52.3.130. [DOI] [PubMed] [Google Scholar]
  • 13.Friedland RB. National Academy on an Aging Society. Washington DC: 1998. Understanding Health Literacy: New Estimates of the Costs of Inadequate Health Literacy. [Google Scholar]
  • 14.Center for Health Care Strategies. Lawrenceville, NJ: 1998. Fact Sheet: Low Health Literacy Skills Increase Annual Health Care Expenditures by $73 Billion. [Google Scholar]
  • 15.Doak CC, Doak LG, Root JH. Teaching patients with low literacy skills. 2nd. Philadelphia: JB Lippincott; 1996. pp. 1–9.pp. 44–49.pp. 73–89. [Google Scholar]
  • 16.Baker DW, Parker RM, Williams MV, Clark WS, Nurss J. The relationship of patient reading ability to self-reported health and use of health services. Am J Public Health. 1997;87:1027. doi: 10.2105/ajph.87.6.1027. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Williams MV, Baker DW, Honig EG, Lee TM, Nowlan A. Inadequate literacy is a barrier to asthma knowledge and self-care. Chest. 1998;114:1008. doi: 10.1378/chest.114.4.1008. [DOI] [PubMed] [Google Scholar]
  • 18.Williams MV, Baker DW, Parker RM, Nurss JR. Relationship of functional health literacy to patients' knowledge of their chronic disease: a study of patients with hypertension and diabetes. Arch Intern Med. 1998;158:166. doi: 10.1001/archinte.158.2.166. [DOI] [PubMed] [Google Scholar]
  • 19.Davis TC, Williams MV, Marin MA, et al. Health Literacy and Cancer Communication. CA Cancer J Clin. 2002;52:134. doi: 10.3322/canjclin.52.3.134. [DOI] [PubMed] [Google Scholar]
  • 20.Kirsch I, Jungeblut A, Jenkins L, Kolstad A. Adult literacy in America: a first look at the findings of the National Adult Literacy Survey. [June 15, 2009];US Department of Education. Available: http://nces.ed.gov/pubs93/93275.pdf.
  • 21.Taylor W. Cloze procedure: a new tool for measuring readability. Journalism Q. 1953;30:415. [Google Scholar]
  • 22.National Institutes of Health. Bethesda, MD: 2007. [July 20, 2009]. How to write easy to read health materials. Available: http://www.nlm.nih.gov/medlineplus/etr.html. [Google Scholar]
  • 23.Centers for Disease Control. Atlanta, GA: 1999. [July 20, 2009]. Scientific and technical information: simply put. Available: http://www.cdc.gov/od/oc/simpput.pdf. [Google Scholar]
  • 24.Consumer Reports WebWatch and the Health Improvement Institute (HII) Healthratings.org Revisits 20 Most-Trafficked Health Information Sites. [June 15 2009]; Available: http://www.consumerwebwatch.org/dynamic/health-report-infosite-ratings.cfm.
  • 25.2009 Report of 2008 Statistics: National Clearinghouse of Plastic Surgery Statistics. [June 1 2009];American Society of Plastic Surgeons. Available: http://www.plasticsurgery.org/Media/Press_Kits/Procedural_Statistics.html.
  • 26.Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA. 2001;285:2612. doi: 10.1001/jama.285.20.2612. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Klare GR. The measurement of readability. Vol. 1. Ames: Iowa State University Press; 1963. pp. 56–61.pp. 111–167. [Google Scholar]
  • 28.DuBay WH. Impact Information. Costa Mesa: [June 15, 2009]. The principles of readability. Available: http://www.impact-information.com/impactinfo/readability02.pdf. [Google Scholar]
  • 29.Walsh TM, Volsko TA. Readability assessment of internet-based consumer health information. Respir Care. 2008;53:1310. [PubMed] [Google Scholar]
  • 30.Readability testing in cancer communications: methods, examples, and resources for improving the readability of cancer message and materials. Department of Health Education and Welfare, Public Health Service, National Institutes of health, National Cancer Institute; Bethesda, MD.: 1979. [Google Scholar]
  • 31.McLaughlin GH. SMOG grading—A new readability formula. Journal of Reading. 1969;12:639. [Google Scholar]
  • 32.Ley P, Florio T. The use of readability formulas in healthcare. Psychology, Health and Medicine. 1996;1:7. [Google Scholar]
  • 33.Friedman DB, Hoffman-Goetz L. A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav. 2006;33:352. doi: 10.1177/1090198105277329. [DOI] [PubMed] [Google Scholar]
  • 34.Hochhauser M. Flesch-Kincaid in Microsoft Word is flawed. [July 20, 2009];CA Cancer J Clin. Available: http://caonline.amcancersoc.org/cgi/eletters/52/3/130.
  • 35.Rodgers B. The identification and prevalence of specific reading retardation. Br J Educ Psychol. 1983;51:369. doi: 10.1111/j.2044-8279.1983.tb02570.x. [DOI] [PubMed] [Google Scholar]
  • 36.Share DL, McGee R, McKenzie D, Williams S, Silva PA. Further evidence relating to the distinction between specific reading retardation and general reading backwardness. Br J Dev Psychol. 1987;5:35. [Google Scholar]
  • 37.Shaywitz SE, Escobar M, Shaywitz BA, Fletcher JM, Makuch R. Evidence that dyslexia may represent the lower tail of a normal distribution of reading ability. N Engl J Med. 1992;326:145. doi: 10.1056/NEJM199201163260301. [DOI] [PubMed] [Google Scholar]
  • 38.Silva PA, McGee R, Williams S. Some characteristics of 9-year-old boys with general reading backwardness or specific reading retardation. J Child Psychol Psychiatry. 1985;26:407. doi: 10.1111/j.1469-7610.1985.tb01942.x. [DOI] [PubMed] [Google Scholar]
  • 39.van der Wissel A, Zegers FE. Reading retardation revisited. Br J Dev Psychol. 1985;3:3. [Google Scholar]
  • 40.Coffman KG, Odlyzko AM. Growth of the Internet. [July 20 2009];AT and T Research Labs. 2001 Available: www.dtc.umn.edu/∼odlyzko/doc/oft.internet.growth.pdf.
  • 41.Mcullan M. Patients using the Internet to obtain health information: how this affects the patient-health professional relationship. Patient Educ Couns. 2006;63:24. doi: 10.1016/j.pec.2005.10.006. [DOI] [PubMed] [Google Scholar]
  • 42.Beredjiklian PK, Bozentka DJ, Steinberg DR, Bernstein J. Evaluating the source and content of orthopaedic information on the Internet: the case of carpal tunnel syndrome. J Bone Joint Surg. 2000;82A:1540. doi: 10.2106/00004623-200011000-00004. [DOI] [PubMed] [Google Scholar]
  • 43.Biermann JS, Golladay GJ, Greenfield ML, Baker LH. Evaluation of cancer information on the Internet. Cancer. 2000;86:381. [PubMed] [Google Scholar]
  • 44.Davison K. The quality of dietary information on the World Wide Web. Clin Perform Qual Health Care. 1997;5:64. [PubMed] [Google Scholar]
  • 45.Griffiths KM, Christensen H. Quality of web based information on treatment of depression: cross-sectional survey. BMJ. 2000;321:1511. doi: 10.1136/bmj.321.7275.1511. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the World Wide Web: systematic survey of advice on managing fever in children at home. BMJ. 1997;314:1875. doi: 10.1136/bmj.314.7098.1875. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Jiang YL. Quality evaluation of orthodontic information on the World Wide Web. Am J Orthod Dentofacial Orthop. 2000;118:4. doi: 10.1067/mod.2000.104492. [DOI] [PubMed] [Google Scholar]
  • 48.McClung HJ, Murray HD, Heitlinger LA. The Internet as a source for current patient information. Pediatrics. 1998;101:1. doi: 10.1542/peds.101.6.e2. [DOI] [PubMed] [Google Scholar]
  • 49.Soot LC, Moneta GL, Edwards JM. Vascular surgery and the internet: a poor source of patient oriented information. J Vasc Surg. 1999;30:84. doi: 10.1016/s0741-5214(99)70179-5. [DOI] [PubMed] [Google Scholar]
  • 50.Jejurikar SS, Rovak JM, Kuzon WM, Chung KC, Kotsis SV, Cederna PS. Evaluation of plastic surgery information on the internet. Ann Plast Surg. 2002;49:460. doi: 10.1097/00000637-200211000-00003. [DOI] [PubMed] [Google Scholar]
  • 51.Gordon JB, Barot LR, Fahey AL, Matthews MS. The internet as a source of information on breast augmentation. Plast Reconstr Surg. 2001;107:171. doi: 10.1097/00006534-200101000-00028. [DOI] [PubMed] [Google Scholar]
  • 52.Office of Disease Prevention and Health Promotion, US Department of Health and Human Services. Washington DC: 2000. [August 2 2009]. Healthy people 2010: Understanding and improving Health. Available: http://www.healthypeople.gov/Document/pdf/uih/2010uih.pdf. [Google Scholar]
  • 53.Baker DW, Gazmararian JA, Williams MV, Scott T, Parker RM, Green D, Ren J, Peel J. Functional health literacy and the risk of hospital admission among Medicare managed care enrollees. Am J Public Health. 2002;92:1278. doi: 10.2105/ajph.92.8.1278. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Davis TC, Byrd RS, Arnold CL, Auinger P, Bocchini JA., Jr Low literacy and violence among adolescents in a summer sports program. J Adolesc Health. 1999;24:403. doi: 10.1016/s1054-139x(98)00148-7. [DOI] [PubMed] [Google Scholar]
  • 55.Arnold CL, Davis TC, Berkel HJ, Jackson RH, Nandy I, London S. Smoking status, reading level, and knowledge of tobacco effects among low-income pregnant women. Prev Med. 2001;32:313. doi: 10.1006/pmed.2000.0815. [DOI] [PubMed] [Google Scholar]
  • 56.Doak LG, Doak CC, Meade CD. Strategies to improve cancer education materials. Oncol Nurs Forum. 1996;23:1305. [PubMed] [Google Scholar]
  • 57.Writing easy to read materials. Issue Brief Cent Medicare Educ. 2000;1:1. [PubMed] [Google Scholar]
  • 58.Zorn M, Allen PM, Horowitz MA. Current Bibliographies in Medicine. Bethesda, MD.: 2004. [August 2 2009]. Understanding health literacy and its barriers. Available: http://www.nlm.nih.gov/pubs/cbm/healthliteracybarriers.html. [Google Scholar]
  • 59.National Cancer Institute, National Institutes of Health. Bethesda, MD.: 2003. [August 2 2009]. Clear and simple: Developing Effective Print Materials for Low-Literate Readers. Available: http://www.nci.nih.gov/cancerinformation/clearandsimple. [Google Scholar]
  • 60.Office of Health Promotion, Washington State Department of Health. Olympia, WA.: 2000. [August 2 2009]. Guidelines for developing easy to read health education materials. Available: http://www3.doh.wa.gov/here/howto/images/easy2.html. [Google Scholar]
  • 61.Readability Research. [August 2 2009];Society for Technical Communication, Usability Special Interest Group. Available: http://www.stcsig.org/usability/topics/readability.html.

RESOURCES