Readability of online patient education material for the novel coronavirus disease (COVID-19): a cross-sectional health literacy study

Abstract

Objectives

The internet has become one of the most important resources for the general population when searching for healthcare information. However, the information available is not always suitable for all readers because of its difficult readability. We sought to assess the readability of online information regarding the novel coronavirus disease (COVID-19) and establish whether they follow the patient educational information reading level recommendations.

Study design

This is a cross-sectional study.

Methods

We searched five key terms on Google and the first 30 results from each of the searches were considered for analysis. Five validated readability tests were utilized to establish the reading level for each article.

Results

Of the 150 gathered articles, 61 met the inclusion criteria and were evaluated. None (0%) of the articles met the recommended 5th to 6th grade reading level (of an 11-12-year-old). The mean readability scores were Flesch Reading Ease 44.14, Flesch-Kincaid Grade Level 12.04, Gunning-Fog Index 14.27, Simple Measure of Gobbledygook SMOG Index 10.71, and Coleman-Liau Index 12.69.

Conclusions

Online educational articles on COVID-19 provide information too difficult to read for the general population. The readability of articles regarding COVID-19 and other diseases needs to improve so that the general population may understand health information better and may respond adequately to protect themselves and limit the spread of infection.

Highlights

• •Online health information regarding COVID-19 is too difficult to read and understand.
• •Based on past research, health articles too difficult to understand may cause misinformation to spread and public panic.
• •The readability of COVID-19 needs to improve so that the general population may respond adequately to protect themselves and limit the spread of infection.

Introduction

The internet has grown to become one of the most popular resources for people in finding health information.¹ In the United States, up to 80% of all adult internet users have looked online to find information about several health issues.² 70% of health seekers express that the information that they have found has influenced them in their decision-making regarding the treatment of their disease.³ Although the information is readily accessible, it's utility may be variable depending on the readability of the information. The mean reading level of adults in the United States has been estimated to be equivalent to that of a 13- to 14-year old.⁴ Taking this into account, the American Medical Association (AMA) and United States Department of Health and Human Services (USDHHS) have recommended that patient educational information should not exceed the reading level of an 11- to 12-year old.⁵ However, numerous studies have shown that the reading difficulty of health articles online is much higher than recommended.6, 7, 8 To date, no readability analysis has been performed regarding the novel coronavirus disease (COVID-19).

COVID-19 has resulted in thousands of deaths worldwide and has resulted in more fatalities than the previous two coronavirus epidemics combined (i.e. SARS and MERS). As of March 11, 2020, the outbreak was recognized by the World Health Organization (WHO) as a pandemic.⁹ Thus, proper public education becomes critical so that patients may prevent and contain the infection. Because the internet is often the first source of information regarding health care for patients, it is critical to evaluate the readability level of the information.¹⁰ In this study, we sought to assess the readability of online information regarding COVID-19.

Methods

Search strategy

On March 13, 2020, the Google search engine was used to search for the following five key terms: “Coronavirus,” “COVID-2019,” “SARS CoV-2,” “2019-nCoV,” and “What is the coronavirus”. 7.39 × 10⁹, 4.59 × 10⁹, 1.43 × 10⁸, 1.02 × 10⁸, and 5.08 × 10⁹ search results were identified for each search, respectively. The first 30 results per key term were evaluated because 90% of internet users do not look past this number.¹¹

Inclusion and exclusion criteria

A total of 150 search results were considered for the readability analysis. Articles not in English, duplicates, newspaper articles (because they contained mostly political updates), biomedical journal articles, non–open-access articles (behind a paywall), and statistical websites were excluded.

Readability assessment

All articles were restructured into plain text and all irrelevant material was deleted, such as figures, their legends, images, and references. The analysis was performed using five readability formulas: Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), Gunning-Fog Index (GFI), Simple Measure of Gobbledygook (SMOG) Index, and Coleman-Liau Index (CLI).

The FRE and the FKGL are different formulas that use the average sentence length in words and average syllables per 100 words for the assessment.¹²

GFI uses the average sentence length and the number of words containing three or more syllables for the calculations. Polysyllabic words are excluded, including proper nouns, a combination of easy words (including hyphenated words), and polysyllable verbs whose third syllable is “es” or “ed”.¹³

SMOG Index is calculated by counting every polysyllabic word (containing three or more syllables) in sections containing 10 sentences each placed in the beginning, in the middle, and at the end of the text in question.¹⁴ In contrast to the other readability formulas, CLI does not take the number of syllables into account. Instead, it makes the assessment based on the average number of letters and sentences per 100 words.¹⁵

The FRE score determines the reading ease based on a scale from 0 to 100, where a lower score indicates a higher difficulty (0–30 is very difficult, 30–50 is difficult, 50–60 is fairly difficult, 60–70 is standard, 70–80 is fairly easy, 80–90 is easy, 90–100 is very easy).¹² The other four formulas, however, use a scale based on the educational level needed to understand the text. A score less than 6 is regarded as 6th grade reading level (11-12 years), a score of 7 as 7th grade reading level (12-13 years), a score of 8 as 8th grade (13-14 years), 9 as high school freshman (14-15 years), 10 as high school sophomore (15-16 years), 11 as high school junior (16-17 years), 12 as high school senior (17-18 years), 13 as college freshman (18-19 years), 14 as college sophomore (19-20 years), 15 as college junior (20-21 years), 16 as college senior (21-22 years), and 17 or above is regarded as college graduate (someone over the age of 22 years).

Results

Of the 150 articles, 61 were analyzed as they met the inclusion criteria. The mean reading level of the articles was equivalent to a high school senior/college freshman (17 to 18 year olds) (12.4 ± 2.1). The minimum score was 8.8, whereas the maximum score was 20.1. According to the mean FRE score, the articles are considered difficult to read (44 ± 11.5). In Table 1 , the mean scores of the readability formulas are presented. Table 2 shows the distribution of the scores.

Table 1.

Mean score for individual formulas used in the readability assessment of the websites.

Readability formula	Mean score	Standard deviation
Flesch Reading Ease (FRE)	44.14	11.46
Flesch-Kincaid Grade Level (FKGL)	12.04	2.67
Gunning-Fog Index (GFI)	14.27	2.84
Simple Measure of Gobbledygook (SMOG) Index	10.71	1.96
Coleman-Liau Index (CLI)	12.69	1.86

Table 2.

Distribution of scores within each individual formula.

Readability formula	Score	Number of websites
Flesch Reading Ease (FRE)	Easy (80–100)	0
	Average (60–79)	5
	Difficult (0–59)	56
Flesch-Kincaid Grade Level (FKGL)	Below 6	0
	6–10	13
	Above 10	48
Gunning-Fog Index (GFI)	Below 6	0
	6–10	1
	Above 10	60
Simple Measure of Gobbledygook (SMOG) Index	Below 6	0
	6–10	23
	Above 10	38
Coleman-Liau Index (CLI)	Below 6	0
	6–10	7
	Above 10	54

All articles were of at least a high school sophomore grade level (15-16 years old). According to the FRE, 91.8% of the articles were found difficult to read. On the FKGL scale, 78.68% of the articles had a readability index of a high school senior grade (17- to 18-year olds). Similarly, on GFI, SMOG, and CLI, the percentage of articles that were above the readability index of a high school senior (17- to 18-year olds) were 98.36%, 62.29%, and 88.52%, respectively.

Table 3 displays that the information provided from websites related to governments, hospitals and health organizations (such as WHO) are also not following the recommendations for educational material. All of the medical articles were written beyond the recommended 5th- to 6th-grade level (11- to 12-year olds).¹⁶

Table 3.

Mean readability score of websites related to governments, hospitals/clinics, or health institutions/organizations.

Readability formula	Mean score	Standard deviation
Flesch Reading Ease (FRE)	47.82	12.76
Flesch-Kincaid Grade Level (FKGL)	11.51	3.06
Gunning-Fog Index (GFI)	13.57	3.10
Simple Measure of Gobbledygook (SMOG) Index	10.17	2.16
Coleman-Liau Index (CLI)	12.65	1.82

Fig. 1 shows a comparison between the overall readability, government health organization readability, and hospital readability.

Fig. 1.

The mean score for each of the readability formulas used in the assessment of the overall readability (colored blue), government health organizations readability (colored red) and hospital readability (colored purple). Abbreviations: avg, average; gov, government health organizations; FKGL, Flesch-Kincaid Grade Level; FRE, Flesch Reading Ease; GFI, Gunning-Fog Index; GFS, Gunning-Fog Score; CLI, Coleman-Liau Index; SMOG, Simple Measure of Gobbledygook.

Discussion

Readability analysis

We found that online information about COVID-19 is too difficult for the general population to read and comprehend. None of the articles met the 5th- to 6th-grade reading level (11- to 12-year olds) recommended by the AMA and the USDHHS.^{5 , 16} Most articles (84%) were designated as too difficult to read, which makes it tough for the public to acquire understandable information regarding COVID-19. To reduce the spread of infection and, thus, reduce the burden on a country's healthcare system, a country's population needs access to understandable information online. We urge prominent organizations such as the WHO and the Centers for Disease Control and Prevention to make their online information friendlier to the general public. Easy readability of relevant medical information empowers individuals to take the right steps to protect themselves. This may reduce panic and anxiety especially in the midst of a pandemic. Our findings are novel because our article is the first to do a readability analysis on COVID-19. Morever, we conducted our readability analysis on COVID-19 during the pandemic itself to provide a contemporary and relevant view of the problem.

Context

Medical terminology is an essential factor affecting the readability of a text. Even if the person's educational level is high, long sentences and unfamiliar words can make the text challenging to read. If they are not accustomed to medical literature, misunderstanding and misinformation may occur and the patient may stop researching basic medical care. Therefore, information aiming to educate patients should be clear and understandable. A 2018 systematic review analyzing 157 readability studies found that the readability level of online health articles is incomprehensible for the public.⁸

Limitations

This study is limited by the constraints of a cross-sectional study. Because the material available on the internet is constantly increasing, the results of our study will only reflect what information was present for the public at this point in time. However, our study brings awareness to the problem; this may influence hospitals and government organizations to reduce the reading difficulty on their websites. Morevoer, this readability study is based on text only. Infographics and videos—which may often enhance the understanding of a text—were removed to perform the analysis. This was another limitation of this study.

Future directions

Previous studies have shown that the quality of health information online is biased, misleading, and poor.^{17 , 18} Thus, although we assessed the readability of COVID-19 health articles, the quality of these articles still needs to be evaluated. In addition, because YouTube has become a major source of patient information, the quality of information on this platform also needs to be assessed.19, 20, 21, 22 The role and effectiveness of online medical resources concerning telemedicine may also be explored.²³

We encourage websites to display COVID-19 infographics and videos as they may be a more friendly way of providing information to the public. Moreover, infographics are easy to potentially share on social media and may help spread the health information about the disease.

Public health implications

Considering Europe is now the epicenter of COVID-19 pandemic, it becomes critical that government sites such as ecdc.europa.eu (European Centre for Disease Prevention and Control) publish advisories in simple understandable terms. Studies show that these articles play a critical role in recommending health safety guidelines, reduce the burden on the healthcare system, and help healthcare workers prioritize managing the disease effectively.24, 25, 26, 27 An analysis of the 1918 influenza pandemic showed that early implementation of certain interventions (e.g. isolation policies, mask ordinances, and bans on public gatherings) reduced influenza transmission.²⁶ We assert that for these interventions to be effective, the public needs access to understandable online health information so that personal measures to contain and prevent the disease may be taken (e.g. by washing heads frequently, practicing respiratory hygiene, and seeking medical care early). Low health literacy has been associated with nonadherence to treatment plans and medical regimens, poor patient self-care, high healthcare costs, and increased risk of hospitalization and mortality.²⁸ Thus we hope that this article serves as a “call to action” for health authorities to provide more comprehensible reading material online.

Conclusion

Online information regarding COVID-19 is too difficult to read and understand as designated by the AMA and the USDHHS.⁵ This includes websites run by governments and health institutions such as the WHO. Because the internet now is one of our most popular sources of information, it is critical that people are provided with understandable information. Health articles too difficult to understand may cause misinformation to spread, public panic due to a lack of accessible information, and a greater burden on a country's healthcare system.

Author statements

Ethical approval

No ethical approval was required as all the data analyzed were publicly available.

Funding

None reported.

Competing interests

None reported.