Skip to main content
NCBI home page
Sage Choice logoLink to Sage Choice
. 2021 Aug 10;166(5):862–868. doi: 10.1177/01945998211033254

Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials

Joo Hyun Kim 1, Elysia Grose 2, Justine Philteos 2, David Forner 3,4, Christopher W Noel 2,4, Vincent Wu 2, Antoine Eskander 2,4,✉
PMCID: PMC9066686  PMID: 34372717

Abstract

Objective

Patient education materials across 3 national English otolaryngology–head and neck surgery (OHNS) societies: the American Academy of Otolaryngology–Head and Neck Surgery (AAO-HNS), the Canadian Society of Otolaryngology–Head and Neck Surgery (CSOHNS), and Ear, Nose, and Throat United Kingdom (ENT UK) were examined to determine whether they are written at a level suitable for patient comprehension.

Study Design

Cross-sectional study.

Setting

Online patient materials presented through OHNS national societies.

Methods

Readability was calculated using the Flesch-Kincaid Grade Level, Flesch-Kincaid Reading Ease Score, and Simple Measure of Gobbledygook Index. All public patient education materials available through the CSOHNS, AAO-HNS, and ENT UK websites were assessed. Patient education materials were grouped into categories by subspecialty.

Results

In total, 128 patient materials from the 3 societies were included in the study. All 3 societies required a minimum grade 9 reading comprehension level to understand their online materials. According to Flesch-Kincaid Grade Level, the CSOHNS required a significantly higher reading grade level to comprehend the materials presented when compared to AAO-HNS (11.3 vs 9.9; 95% CI, 0.5-2.4; P < .01) and ENT UK (11.3 vs 9.4; 95% CI, 0.9-2.9; P < .01). Patient education materials related to rhinology were the least readable among all 3 societies.

Conclusion

This study suggests that the reading level of the current patient materials presented through 3 national OHNS societies are written at a level that exceeds current recommendations. Promisingly, it highlights an improvement for the readability of patient materials presented through the AAO-HNS.

Keywords: readability, patient education, health literacy, otolaryngology

One in 5 patients obtain medical information about their disease or surgical procedure from the Internet prior to their appointment with an otolaryngologist–head and neck surgeon.1-3 It is estimated that approximately 54% of patients with head and neck cancer use the Internet to find information about their treatments.4-6 A single-institutional study exploring Internet use among otolaryngology–head and neck surgery (OHNS) patients concluded that approximately 61% of patients conducted an Internet search with regards to their condition before seeing a general otolaryngologist, 60% before a rhinologist, 44% before seeing a laryngologist, and 52% before seeing a neurotologist. 2 With patients becoming increasingly dependent on web-based health information,7,8 having online educational materials that can be understood by the patient population is of the utmost importance.

Health literacy, the degree to which an individual has the ability to understand and apply basic health information, has been established as one of the strongest predictors of overall health.9-12 Low health literacy is associated with poor health outcomes, including increased mortality rates and hospitalizations.13-15 This is of great concern since as high as 49% of Canadians,16,17 36% of Americans,18,19 and 59% of Europeans20,21 have low literacy and limited ability to understand the health information presented to them. Within OHNS, many studies have demonstrated that available online information has been deficient in readability and is not written at a level easily understood by the public.22-24 However, some of these materials were not necessarily intended for patient education specifically, and thus commonly used readability targets may have been unnecessary.

The American Medical Association (AMA) has recommended that medical information written for patients should aim for a reading level less than or equal to sixth grade as the average American reads at approximately an eighth-grade reading level. 25 Similarly, in Canada and the United Kingdom, it is estimated that 48% of Canadians have literacy levels that fall below a high school level, and 16% of adults in Northern Ireland and England read below the lowest level of reading proficiency, which equates to the estimated reading level of an individual 5 to 7 years of age.26,27 Previous studies have shown that OHNS web-based health information does not fulfill AMA recommendations.28-30 The Canadian Society of Otolaryngology–Head and Neck Surgery (CSOHNS), the American Academy of Otolaryngology–Head and Neck Surgery (AAO-HNS), and Ear, Nose, and Throat United Kingdom (ENT UK) have materials specifically targeted for patient education. However, the readability of CSOHNS and ENT UK materials has yet to be established, and the readability of the AAO-HNS was last assessed in 2016. 31 Herein, we aimed to analyze the readability of the patient resources made available through 3 of the largest English-speaking national societies, as physicians often turn their patients to these resources for further information.

Methods

All public patient education materials available through the CSOHNS, AAO-HNS, and ENT UK websites were assessed. Patient education materials were grouped into categories by subspecialty: otology, rhinology, head and neck oncologic surgery, and other. The other category included topics related to general OHNS, facial plastics, laryngology, and pediatric OHNS given the low number of patient education material in each individual category (see Supplementary Material 1 in the online version of the article). This study was exempted from the University of Toronto review board review because the educational material on the website is publicly accessible and does not involve patient records.

Readability Evaluation

The text from each webpage was pasted to Microsoft Word (Microsoft Corp) and removed of any formatting elements that might interfere with readability assessment (headings, symbols, author information, references, etc). The plain text from each webpage was then assessed using an online readability calculator (https://readable.com/), as demonstrated in other readability studies.29,31,32 This resource evaluates the text’s readability based on criteria such as the average number of syllables and sentence length and yields the following readability scores: Flesch-Kincaid Grade Level (FKGL), the Flesch-Kincaid Reading Ease (FRE) score, and Simple Measure of Gobbledygook Index (SMOG) reading grade. The formulas can be found in Supplementary Material 2 (in the online version of the article). The FKGL is the most validated readability formula used for analyzing medical texts, and both the FRE and FKGL have been widely used to assess readability in the OHNS literature.22,23,33-36 FKGL approximates the reading grade level necessary to comprehend the text (eg, a score of 6 corresponds to a sixth-grade reading level). The FRE determines the readability using a score from 0 to 100, with higher scores indicating a higher level of readability, as shown in Table 1 . The SMOG reading grade is a measure of readability that estimates the years of education needed to understand a piece of writing. 37 Materials with low FKGL, high FRE, and low SMOG scores are considered more readable.

Table 1.

Flesch-Kincaid Reading Ease (FRE) Readability Score and Interpretation.

FRE score Interpretation
90-100 Very easy
80-89 Easy
70-79 Fairly easy
60-69 Standard
50-59 Fairly difficult
30-49 Difficult
<30 Very confusing
Open in a new tab

Statistical Analyses

Categorical variables were reported using frequencies and proportions. Continuous variables were presented as mean and SD. Normalcy was assessed through the Shapiro-Wilk test and visual inspection of histogram and quantile-quantile plots. Comparison of continuous variables was performed using 1-way analysis of variance (ANOVA) with post hoc Tukey test. Comparisons of readability scores were made between CSOHNS, AAO-HNS, and ENT UK, along with different OHNS subspecialties (otology, rhinology, head and neck oncology, and other). A threshold of P < .05 was considered statistically significant. The effect sizes were reported using standardized mean differences (Cohen’s d). 38 The absolute value of the standardized mean differences was categorized as small (0.2-0.5), medium (0.5-0.8), or large (≥0.8). 39 Statistical analysis was performed using SPSS (v.26.0; SPSS, Inc).

Results

In total, 128 online patient materials from 3 different national societies were included in the study: 63 from the AAO-HNS, 20 from the CSOHNS, and 45 from ENT UK.

The mean FRE, FKGL, and SMOG scores across the 3 societies are outlined in Table 2 . The FRE scores corresponded to a reading interpretation of “fairly difficult” for ENT UK and AAO-HNS, whereas the FRE score for CSOHNS demonstrated a “difficult” interpretation. Furthermore, ENT UK had a significantly higher FRE score, compared to AAO-HNS (58.1 vs 50.6; d = 0.8, P < .01) and CSOHNS (58.1 vs 45.2; d = 1.3, P < .01). With regards to the SMOG analysis, CSOHNS was the least readable with a mean SMOG score corresponding to an undergraduate reading comprehension level (mean [SD], 13.5 [1.5]). ENT UK was the most readable according to the SMOG score and yet required a high school junior literacy on average (mean [SD], 11.8 [1.5]).

Table 2.

FRE, FKGL, and SMOG scores for American Academy of Otolaryngology–Head and Neck Surgery, Canadian Society of Otolaryngology–Head and Neck Surgery, and Ear, Nose, and Throat United Kingdom.

Society FRE, mean (SD) FKGL, mean (SD) SMOG, mean (SD)
American Academy of Otolaryngology–Head and Neck Surgery (n = 63) 50.6 (8.0) 9.9 (1.3) 12.2 (1.1)
Canadian Society of Otolaryngology–Head and Neck Surgery (n = 20) 45.2 (10.2) 11.3 (1.8) 13.5 (1.5)
Ear, Nose, and Throat United Kingdom (n = 45) 58.1 (9.9) 9.4 (1.8) 11.8 (1.5)
Open in a new tab

Abbreviations: FKGL, Flesch-Kincaid Grade Level; FRE, Flesch-Kincaid Reading Ease; SMOG, Simple Measure of Gobbledygook Index.

The FKGL grade-reading levels are outlined in Figure 1 . According to FKGL scores, the CSOHNS required a significantly higher reading grade level to comprehend the materials presented when compared to AAO-HNS (11.3 vs 9.9; d = 1.00, P < .01) and ENT UK (11.3 vs 9.4; d = 1.3, P < .01). All 3 societies required a minimum grade 9 reading comprehension level to understand their online materials, as evidenced by their FKGL scores.

Figure 1.

Figure 1.

Open in a new tab

Mean FKGL scores for American Academy of Otolaryngology–Head and Neck Surgery (AAO-HNS), Canadian Society of Otolaryngology–Head and Neck Surgery (CSOHNS), and Ear, Nose, and Throat United Kingdom (ENT UK). The dotted line signifies the recommended sixth-grade reading level.

The mean FRE, FKGL, and SMOG scores for the OHNS subspecialties presented through the 3 societies is shown in Table 3 . The patient materials on topics related to rhinology were the least readable between all 3 societies, as demonstrated in Figure 2 . The rhinology patient materials from the CSOHNS had higher FKGL scores and lower FRE scores when compared with all other subspecialties and societies. The materials that were the most readable from the CSOHNS and AAO-HNS were in head and neck oncology with mean (SD) FRE scores of 57.5 (10.4) and 52.6 (7.2), respectively. Similarly, across the 3 societies (AAO-HNS, CSOHNS, and ENTUK, respectively), patient materials for head and neck oncology were the most readable according to SMOG analysis—mean (SD) of 11.8 (0.9), 12.1 (1.3), and 11.6 (1.1). The least readable subspecialty material corresponded to the rhinology materials presented through CSOHNS with a mean (SD) SMOG score of 14.3 (0.9).

Table 3.

FRE, FKGL, and SMOG Scores per Subspecialty.

Society Subspecialty FRE, mean (SD) FKGL, mean (SD) SMOG, mean (SD)
American Academy of Otolaryngology–Head and Neck Surgery
 Otology (n = 16) 50.1 (8.1) 10.1 (1.2) 12.3 (1.1)
 Rhinology (n = 10) 43.9 (6.5) 11.0 (0.9) 13.2 (0.8)
 Head and neck oncology (n = 15) 52.6 (7.2) 9.4 (1.1) 11.8 (0.9)
 Other (n = 22) 52.5 (7.9) 9.6 (1.3) 12.0 (1.1)
Canadian Society of Otolaryngology–Head and Neck Surgery
 Otology (n = 3) 41.8 (7.0) 11.7 (1.3) 13.4 (0.9)
 Rhinology (n = 4) 41.5 (11.8) 12.4 (2.2) 14.3 (1.8)
 Head and neck oncology (n = 3) 57.5 (10.4) 9.2 (1.4) 12.1 (1.3)
 Other (n = 10) 44.0 (8.8) 11.4 (1.6) 13.7 (1.4)
Ear, Nose, and Throat United Kingdom
 Otology (n = 16) 55.2 (9.6) 10.0 (2.0) 12.3 (1.6)
 Rhinology (n = 11) 55.2 (10.1) 10.0 (1.8) 12.0 (1.6)
 Head and neck oncology (n = 9) 59.6 (7.5) 8.9 (1.3) 11.6 (1.1)
 Other (n = 9) 65.1 (9.7) 8.1 (1.4) 10.9 (1.2)
Open in a new tab

Abbreviations: FKGL, Flesch-Kincaid Grade Level; FRE, Flesch-Kincaid Reading Ease; SMOG, Simple Measure of Gobbledygook Index.

Figure 2.

Figure 2.

Open in a new tab

Mean Flesch-Kincaid Grade Level (FKGL) score per subspecialty across the societies. The solid line signifies the recommended sixth-grade reading level.

Discussion

In this study, it was demonstrated that the reading level of the current online patient materials presented through the 3 national OHNS societies was written at a level that exceeded current readability recommendations. The patient materials on topics related to rhinology were the least readable between all 3 societies.

Patient education is an integral component of patient-centered care and informed consent. Studies have shown that the use of online health information by patients can have a significant impact on the health care decision-making process.5,13,40,41 A study that set out to estimate the prevalence of poor reading skills in the OHNS outpatient population in the United Kingdom found that 28% of their patients had poor reading skills and were unable to comprehend medical leaflets. 42 Moreover, studies on the utilization of Internet videos and websites for OHNS procedures have concluded they are of poor quality and difficult for patients to comprehend, making national society websites the ideal source for trusted patient education material.43-46 This requires OHNS societies to provide accurate information, written at a comprehensible level, as these are viewed as trusted sources of information for patient education. 29

Our study concluded that neither AAO-HNS, CSOHNS, nor ENT UK had online patient materials written at a sixth-grade reading level as recommended by the AMA. Our current study has demonstrated that for each of these societies, across every subspecialty, the required reading grade level required is at a minimum of grade 11. Previous literature examining the materials presented through the AAO-HNS concluded that the online materials were written at a FKGL of 10.5 to 11.4. 31 This ignited change as it is well established in the literature that patient comprehension of medical information is integral to their involvement in treatment decisions and improves patient outcomes.47,48 Our study demonstrated improvement in the FKGL reading levels of the material of the AAO-HNS, as the FKGL reading level was 9.8—this is encouraging as it demonstrates a decrease in reading difficulty of 1 to 2 academic grades. Furthermore, the online materials presented through CSOHNS and ENT UK have not been previously analyzed for their degree of readability. This study found that both patient materials from ENT UK and CSOHNS were found to have significantly higher FRE, FKGL, and SMOG scores, respectively, when compared to AAO-HNS. These findings highlight that there still remains room for improvement to make educational material on national society websites more comprehensible for patients.

With regards to our analysis of OHNS subspecialties, rhinology was unanimously the least readable topic across the 3 societies with reading grade levels ranging from the 10th grade to the 12th grade. These findings are in keeping with previous studies analyzing the readability of patient materials presented through the American Rhinology Society. 49 One possible explanation for this is the inherent complexity embedded in rhinology, as concepts surrounding the location and conceptualization of the sinuses may be difficult for the average reader. Kasabwala et al 29 analyzed patient materials from the AAO-HNS in 2012 and found that materials related to conditions of the nose and mouth had the highest percentage of complex words when compared to other categories. Wong and Levi 31 in 2016 displayed methods of mitigating the medical jargon intrinsic to rhinology, including explaining the location and purposes of the sinuses before describing the symptoms of sinusitis.

It is estimated that 54% of patients with head and neck cancer use the Internet as a source of health information. 4 This study also found that patient materials related to head and neck oncology from the AAO-HNS and CSOHNS were the most readable at approximately a ninth-grade reading level. This demonstrates an improvement from the study done by Kasabwala et al 29 in 2012 as patient materials related to cancer from the AAO-HNS had an average FKGL score of 11.5. Although this is reassuring, studies have demonstrated that 12% to 47% of head and neck cancer survivors have inadequate health literacy41,48,50; as such, there remains room for improvement as many of these resources are written above the recommended sixth-grade reading level. A study analyzing the readability of online medical information looked at the online patient materials presented through the national societies for the numerous surgical subspecialties, and they compared the readability results to those presented through the AAO-HNS. They concluded that, similar to the AAO-HNS, the readability scores for the patient education materials from the various other surgical subspecialty websites were above the recommended sixth-grade level. 32 Many patients access online patient education materials preconsultation without having prior discussions with health care professionals to provide a foundational understanding of their condition 51 ; thus, patients should be encouraged and be given opportunities to bring in the materials that they have read independently for the physician to address any questions that have arisen from these resources.

As evidenced by the reading difficulty of the analyzed texts, writing production for conditions within OHNS is a difficult task. The results of this study highlight the need for improving the readability and the overall comprehension of patient education materials available on the websites of OHNS national societies. First, individuals with expertise in health literacy and patient education, such as specialized librarians, could be incorporated into the writing process to ensure readability.52,53 Patient engagement in the development of this material is critical, but many of the patients chosen for this task may have a higher level of education and may not reflect the target population. 54 As such, patients with a wide range of educational backgrounds should be engaged in the production process. In addition, materials should be put through a readability software prior to publication to objectively evaluate their reading difficulty. Some other suggestions for improving the readability of patient education materials include limiting the number of words in sentences to a maximum of 8 to 10 words, 55 using 1- to 2-syllable words, 56 and replacing medical jargon with simple terms where possible.57,58 Supplemental Material 3 (in the online version of the article) provides examples of these additional suggestions implemented to further improve the readability of current text found on the 3 OHNS societies analyzed. In addition, some ways authors can improve readability that are not necessarily captured with readability scores include (1) writing in an active voice59,60; (2) the strategic use of formatting, including highlighting or bolding important points 61 ; and (3) incorporating mixed media such as videos and images to improve the understanding of medical concepts. 62 These findings further necessitate the need to employ strategies that ensure that the patient information available online is easily understood by the patient population at large.

There are a few limitations to our study. It is difficult to determine whether the patient education materials included in this study were intended for patients to read in the absence of discussions with their health care provider. FRE and FKGL are readability formulas that calculate the number of syllables and words in a sentence, and these may not adequately account for nuanced complexity of the text, which could potentially bias our results toward a conservative or lower readability score. Furthermore, readability is only 1 aspect of the appropriateness of written information. Availability, format, style, and content are all important factors that, if not addressed, can act as barriers to patient access. In addition, these assessment tools do not take into account videos, diagrams, and other presentation elements that contribute to the quality and understandability of the online patient materials.

Conclusion

The findings of this study suggest that the reading level of the current online patient materials presented through 3 national OHNS societies (AAO-HNS, CSOHNS, and ENT UK) does not meet readability standards. Encouragingly, it highlights an improvement for the readability of patient materials presented through the AAO-HNS. Societies developing patient material should ensure that a professional writer and editor are included as part of the team to ensure readability as a key requirement. Ensuring resources are easily comprehensible has the potential to improve shared decision-making, patient satisfaction, and postoperative outcomes, all of which are important facets of patient-centered health care delivery. Thus, we hope the findings of this current study ignite revisions to the current patient materials presented through the AAO-HNS, CSOHNS, and ENT UK.

Supplemental Material

sj-docx-1-oto-10.1177_01945998211033254 – Supplemental material for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials
Click here for additional data file. (45.3KB, docx)

Supplemental material, sj-docx-1-oto-10.1177_01945998211033254 for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials by Joo Hyun Kim, Elysia Grose, Justine Philteos, David Forner, Christopher W. Noel, Vincent Wu and Antoine Eskander in Otolaryngology–Head and Neck Surgery

sj-docx-2-oto-10.1177_01945998211033254 – Supplemental material for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials
Click here for additional data file. (13KB, docx)

Supplemental material, sj-docx-2-oto-10.1177_01945998211033254 for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials by Joo Hyun Kim, Elysia Grose, Justine Philteos, David Forner, Christopher W. Noel, Vincent Wu and Antoine Eskander in Otolaryngology–Head and Neck Surgery

sj-docx-3-oto-10.1177_01945998211033254 – Supplemental material for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials
Click here for additional data file. (14.7KB, docx)

Supplemental material, sj-docx-3-oto-10.1177_01945998211033254 for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials by Joo Hyun Kim, Elysia Grose, Justine Philteos, David Forner, Christopher W. Noel, Vincent Wu and Antoine Eskander in Otolaryngology–Head and Neck Surgery

Footnotes

Author Contributions: Joo Hyun Kim, carried out the study, acquired the data set, drafted the manuscript, and read and approved the final manuscript and agree to be accountable for the author’s own contributions; Elysia Grose, carried out the study, analyzed the data set, drafted the manuscript, and read and approved the final manuscript and agree to be accountable for the author’s own contributions; Justine Philteos, participated in the design of the study, performed data analysis, drafted the manuscript, read and approved the final manuscript and agree to be accountable for the author’s own contributions; David Forner, contributed to study design, revised the manuscript, assisted with statistics, and read and approved the final manuscript and agree to be accountable for the author’s own contributions; Christopher W. Noel, contributed to study design, revised the manuscript, assisted with statistics, and read and approved the final manuscript and agree to be accountable for the author’s own contributions; Vincent Wu, contributed to study design, formulated study design, revised the manuscript, and read and approved the final manuscript and agree to be accountable for the author’s own contributions; Antoine Eskander, revised the manuscript and read and approved the final manuscript and agree to be accountable for the author’s own contributions.

Disclosures: Competing interests: None.

Sponsorships: None.

Funding source: None.

Supplemental Material: Additional supporting information is available in the online version of the article.

References

  • 1. Tassone P, Georgalas C, Patel NN, Appleby E, Kotecha B. Do otolaryngology out-patients use the internet prior to attending their appointment? J Laryngol Otol. 2004;118(1):34-38. [DOI] [PubMed] [Google Scholar]
  • 2. Pagedar NA, Schularick NM, Lee PC, Karnell LH. Health-related Internet use among otolaryngology patients. Ann Otol Rhinol Laryngol. 2018;127(8):551-557. [DOI] [PubMed] [Google Scholar]
  • 3. Ren C, Deng Z, Hong Z, Zhang W. Health information in the digital age: an empirical study of the perceived benefits and costs of seeking and using health information from online sources. Health Info Libr J. 2019;36(2):153-167. [DOI] [PubMed] [Google Scholar]
  • 4. Rogers SN, Rozek A, Aleyaasin N, Promod P, Lowe D. Internet use among head and neck cancer survivors in the North West of England. Br J Oral Maxillofac Surg. 2012;50(3):208-214. [DOI] [PubMed] [Google Scholar]
  • 5. Ziebland S, Chapple A, Dumelow C, Evans J, Prinjha S, Rozmovits L. How the Internet affects patients’ experience of cancer: a qualitative study. BMJ. 2004;328(7439):564. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Ni Riordain R, McCreary C. Head and neck cancer information on the Internet: type, accuracy and content. Oral Oncol. 2009;45(8):675-677. [DOI] [PubMed] [Google Scholar]
  • 7. Cline RJ, Haynes KM. Consumer health information seeking on the Internet: the state of the art. Health Educ Res. 2001;16(6):671-692. [DOI] [PubMed] [Google Scholar]
  • 8. Diaz JA, Griffith RA, Ng JJ, Reinert SE, Friedmann PD, Moulton AW. Patients’ use of the Internet for medical information. J Gen Intern Med. 2002;17(3):180-185. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Megwalu UC. Health literacy in patients with head and neck cancer: an understudied issue. JAMA Otolaryngol Head Neck Surg. 2017;143(7):645-646. [DOI] [PubMed] [Google Scholar]
  • 10. Baur C, Harris L, Squire E. The U.S. national action plan to improve health literacy: a model for positive organizational change. Stud Health Technol Inform. 2017;240:186-202. [PubMed] [Google Scholar]
  • 11. Bujnowska-Fedak MM, Wegierek P. The impact of online health information on patient health behaviours and making decisions concerning health. Int J Environ Res Public Health. 2020;17(3):880. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12. Bussey LG, Sillence E. The role of internet resources in health decision-making: a qualitative study. Digit Health. 2019;5:2055207619888073. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Wallace LS, Rogers ES, Weiss BD. Relationship between health literacy and health-related quality of life among Tennesseans. Tenn Med. 2008;101(5):35-39. [PubMed] [Google Scholar]
  • 14. Safeer RS, Keenan J. Health literacy: the gap between physicians and patients. Am Fam Physician. 2005;72(3):463-468. [PubMed] [Google Scholar]
  • 15. Bostock S, Steptoe A. Association between low functional health literacy and mortality in older adults: longitudinal cohort study. BMJ. 2012;344:e1602. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Rootman I, Ronson B. Literacy and health research in Canada: where have we been and where should we go? Can J Public Health. 2005;96(suppl 2):S62-S77. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17. O’Donnell J. Health literacy in Canada; a primer for students: Laurie Hoffman-Goetz, Lorie Donelle, and Rukhsana Ahmed : Canadian Scholars Press, Toronto, Ontario, Canada, 2014. J Cancer Educ. 2017;32(2):211-212. [DOI] [PubMed] [Google Scholar]
  • 18. Gazmararian JA, Curran JW, Parker RM, Bernhardt JM, DeBuono BA. Public health literacy in America: an ethical imperative. Am J Prev Med. 2005;28(3):317-322. [DOI] [PubMed] [Google Scholar]
  • 19. Cutilli CC, Bennett IM. Understanding the health literacy of America: results of the National Assessment of Adult Literacy. Orthop Nurs. 2009;28(1):27-32. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. Sorensen K, Brand H. Health literacy—a strategic asset for corporate social responsibility in Europe. J Health Commun. 2011;16(suppl 3):322-327. [DOI] [PubMed] [Google Scholar]
  • 21. van der Heide I, Uiters E, Sorensen K, et al. Health literacy in Europe: the development and validation of health literacy prediction models. Eur J Public Health. 2016;26(6):906-911. [DOI] [PubMed] [Google Scholar]
  • 22. Chi E, Jabbour N, Aaronson NL. Quality and readability of websites for patient information on tonsillectomy and sleep apnea. Int J Pediatr Otorhinolaryngol. 2017;98:1-3. [DOI] [PubMed] [Google Scholar]
  • 23. Yi GS, Hu A. Quality and readability of online information on in-office vocal fold injections. Ann Otol Rhinol Laryngol. 2020;129(3):294-300. [DOI] [PubMed] [Google Scholar]
  • 24. Seymour N, Lakhani R, Hartley B, Cochrane L, Jephson C. Cochlear implantation: an assessment of quality and readability of web-based information aimed at patients. Cochlear Implants Int. 2015;16(6):321-325. [DOI] [PubMed] [Google Scholar]
  • 25. Weiss BD. Health Literacy and Patient Safety: Help Patients Understand. American Medical Association; 2007. [Google Scholar]
  • 26. Harding C RE, Williams J, Peters M. The 2011 Skills for Life Survey: A Survey of Literacy, Numeracy, and ICT Levels in England. Department for Business Innovation and Skills (BIS); 2012. [Google Scholar]
  • 27. Statistics Canada. Skills in Canada: First Results from the Programme for the International Assessment of Adult Competencies (PIAAC). Statistics Canada; 2013. [Google Scholar]
  • 28. Svider PF, Agarwal N, Choudhry OJ, et al. Readability assessment of online patient education materials from academic otolaryngology–head and neck surgery departments. Am J Otolaryngol. 2013;34(1):31-35. [DOI] [PubMed] [Google Scholar]
  • 29. Kasabwala K, Agarwal N, Hansberry DR, Baredes S, Eloy JA. Readability assessment of patient education materials from the American Academy of Otolaryngology–Head and Neck Surgery Foundation. Otolaryngol Head Neck Surg. 2012;147(3):466-471. [DOI] [PubMed] [Google Scholar]
  • 30. Best J, Muzaffar J, Mitchell-Innes A. Quality of information available via the Internet for patients with head and neck cancer: are we improving? Eur Arch Otorhinolaryngol. 2015;272(11):3499-3505. [DOI] [PubMed] [Google Scholar]
  • 31. Wong K, Levi JR. Readability trends of online information by the American Academy of Otolaryngology–Head and Neck Surgery Foundation. Otolaryngol Head Neck Surg. 2017;156(1):96-102. [DOI] [PubMed] [Google Scholar]
  • 32. Hansberry DR, Agarwal N, Shah R, et al. Analysis of the readability of patient education materials from surgical subspecialties. Laryngoscope. 2014;124(2):405-412. [DOI] [PubMed] [Google Scholar]
  • 33. Kandula S, Zeng-Treitler Q. Creating a gold standard for the readability measurement of health texts. AMIA Annu Symp Proc. 2008;2008:353-357. [PMC free article] [PubMed] [Google Scholar]
  • 34. Williamson JM, Martin AG. Analysis of patient information leaflets provided by a district general hospital by the Flesch and Flesch-Kincaid method. Int J Clin Pract. 2010;64(13):1824-1831. [DOI] [PubMed] [Google Scholar]
  • 35. Grabeel KL, Russomanno J, Oelschlegel S, Tester E, Heidel RE. Computerized versus hand-scored health literacy tools: a comparison of Simple Measure of Gobbledygook (SMOG) and Flesch-Kincaid in printed patient education materials. J Med Libr Assoc. 2018;106(1):38-45. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36. O’Connell Ferster AP, Hu A. Evaluating the quality and readability of Internet information sources regarding the treatment of swallowing disorders. Ear Nose Throat J. 2017;96(3):128-138. [DOI] [PubMed] [Google Scholar]
  • 37. Brangan S. Development of SMOG-Cro readability formula for healthcare communication and patient education. Coll Antropol. 2015;39(1):11-20. [PubMed] [Google Scholar]
  • 38. Cohen J. Statistical Power Analysis for the Behavioral Science. 2nd ed. Lawrence Erlbaum; 1988. [Google Scholar]
  • 39. Sullivan GM, Feinn R. Using effect size—or why the P value is not enough. J Grad Med Educ. 2012;4(3):279-282. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40. Wilson CJ, Mitchelson AJ, Tzeng TH, et al. Caring for the surgically anxious patient: a review of the interventions and a guide to optimizing surgical outcomes. Am J Surg. 2016;212(1):151-159. [DOI] [PubMed] [Google Scholar]
  • 41. Koay K, Schofield P, Gough K, et al. Suboptimal health literacy in patients with lung cancer or head and neck cancer. Support Care Cancer. 2013;21(8):2237-2245. [DOI] [PubMed] [Google Scholar]
  • 42. Kubba H. Reading skills of otolaryngology outpatients: implications for information provision. J Laryngol Otol. 2000;114(9):694-696. [DOI] [PubMed] [Google Scholar]
  • 43. Kong KA, Hu A. Readability assessment of online tracheostomy care resources. Otolaryngol Head Neck Surg. 2015;152(2):272-278. [DOI] [PubMed] [Google Scholar]
  • 44. Wu V, Lee DJ, Vescan A, Lee JM. Evaluating YouTube as a source of patient information for functional endoscopic sinus surgery [published online October 6, 2020]. Ear Nose Throat J. [DOI] [PubMed] [Google Scholar]
  • 45. Balakrishnan V, Chandy Z, Hseih A, Bui TL, Verma SP. Readability and understandability of online vocal cord paralysis materials. Otolaryngol Head Neck Surg. 2016;154(3):460-464. [DOI] [PubMed] [Google Scholar]
  • 46. Levin M WV, Lee DJ, Cusimano MD, Lee JM. Validity and usefulness of YouTube videos related to endoscopic transsphenoidal surgery for patient information [published online January 19, 2021]. J Neurol Surg Pt B Skull Base. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47. Miller TA. Health literacy and adherence to medical treatment in chronic and acute illness: a meta-analysis. Patient Educ Couns. 2016;99(7):1079-1086. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48. Clarke N, Dunne S, Coffey L, et al. Health literacy impacts self-management, quality of life and fear of recurrence in head and neck cancer survivors [published online January 9, 2021]. J Cancer Surviv. [DOI] [PubMed] [Google Scholar]
  • 49. Kasabwala K, Misra P, Hansberry DR, et al. Readability assessment of the American Rhinologic Society patient education materials. Int Forum Allergy Rhinol. 2013;3(4):325-333. [DOI] [PubMed] [Google Scholar]
  • 50. Nilsen ML, Moskovitz J, Lyu L, et al. Health literacy: impact on quality of life in head and neck cancer survivors. Laryngoscope. 2020;130(10):2354-2359. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51. Clarke MA, Moore JL, Steege LM, et al. Health information needs, sources, and barriers of primary care patients to achieve patient-centered care: a literature review. Health Inform J. 2016;22(4):992-1016. [DOI] [PubMed] [Google Scholar]
  • 52. Blanck AW, Marshall C. Patient education materials from the layperson’s perspective: the importance of readability. J Nurses Staff Dev. 2011;27(2):E8-10; quiz E11-12. [DOI] [PubMed] [Google Scholar]
  • 53. Mayo HG, Pestonjee SF. Creating understandable health information. J Hosp Librarianship. 2004;4(4):113-118. [Google Scholar]
  • 54. Coulter A, Entwistle V, Gilbert D. Sharing decisions with patients: is the information good enough? BMJ. 1999;318(7179):318-322. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55. Mayeaux EJ, Jr, Murphy PW, Arnold C, Davis TC, Jackson RH, Sentell T. Improving patient education for patients with low literacy skills. Am Fam Physician. 1996;53(1):205-211. [PubMed] [Google Scholar]
  • 56. Bernier MJ. Developing and evaluating printed education materials: a prescriptive model for quality. Orthop Nurs. 1993;12(6):39-46. [DOI] [PubMed] [Google Scholar]
  • 57. Jackson RH, Davis TC, Bairnsfather LE, George RB, Crouch MA, Gault H. Patient reading ability: an overlooked problem in health care. South Med J. 1991;84(10):1172-1175. [DOI] [PubMed] [Google Scholar]
  • 58. Cherla DV, Sanghvi S, Choudhry OJ, Liu JK, Eloy JA. Readability assessment of Internet-based patient education materials related to endoscopic sinus surgery. Laryngoscope. 2012;122(8):1649-1654. [DOI] [PubMed] [Google Scholar]
  • 59. Albert T, Chadwick S. How readable are practice leaflets? BMJ. 1992;305(6864):1266-1268. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60. Manning D. Writing readable health messages. Public Health Rep. 1981;96(5):464-465. [PMC free article] [PubMed] [Google Scholar]
  • 61. Brooks DA. Techniques for teaching ED patients with low literacy skills. J Emerg Nurs. 1998;24(6):601-603. [DOI] [PubMed] [Google Scholar]
  • 62. Pentz RD, Lohani M, Hayban M, et al. Videos improve patient understanding of misunderstood chemotherapy terminology. Cancer. 2019;125(22):4011-4018. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

sj-docx-1-oto-10.1177_01945998211033254 – Supplemental material for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials
Click here for additional data file. (45.3KB, docx)

Supplemental material, sj-docx-1-oto-10.1177_01945998211033254 for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials by Joo Hyun Kim, Elysia Grose, Justine Philteos, David Forner, Christopher W. Noel, Vincent Wu and Antoine Eskander in Otolaryngology–Head and Neck Surgery

sj-docx-2-oto-10.1177_01945998211033254 – Supplemental material for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials
Click here for additional data file. (13KB, docx)

Supplemental material, sj-docx-2-oto-10.1177_01945998211033254 for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials by Joo Hyun Kim, Elysia Grose, Justine Philteos, David Forner, Christopher W. Noel, Vincent Wu and Antoine Eskander in Otolaryngology–Head and Neck Surgery

sj-docx-3-oto-10.1177_01945998211033254 – Supplemental material for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials
Click here for additional data file. (14.7KB, docx)

Supplemental material, sj-docx-3-oto-10.1177_01945998211033254 for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials by Joo Hyun Kim, Elysia Grose, Justine Philteos, David Forner, Christopher W. Noel, Vincent Wu and Antoine Eskander in Otolaryngology–Head and Neck Surgery

Articles from Otolaryngology--Head and Neck Surgery are provided here courtesy of SAGE Publications

RESOURCES