Readability, content, and quality of COVID-19 patient education materials from academic medical centers in the United States

Abstract

Background

SARS-CoV-2 has spread rapidly resulted in a global pandemic and public health crisis. The internet is a frequently used resource for providing patient education materials (PEMs). The aim of this study was to evaluate the readability, content, and quality of web-based PEMs on COVID-19 from US academic medical centers.

Methods

The names of US medical schools were obtained from the Association of American Medical Colleges website (n = 145). Institutional, hospital, and heath encyclopedia websites associated with each schools’ medical center were identified using Google. Readability of COVID-19 PEMs was calculated using three validated indices: (1) Flesh-Kincaid Grade Level, (2) Simple Measure of Gobbledygook, and (3) Gunning Frequency of Gobbledygook. Content was evaluated using a scoring matrix based on materials available on the Center for Disease Control website. The Patient Education Material Assessment Tool for Print (PEMAT-P) was used to assess usability and actionability.

Results

A total of 141 (97%) PEMs met inclusion criteria and were analyzed for readability, content, and quality. The mean readability was above the recommended sixth grade reading level (P < .001). Content was variable across PEMs. The PEMAT-P scores reflected good understandability with a median score of 83% (IQR 75%-87%), while actionability was poor with a median score of 41% (IQR 40%-60%).

Conclusions

Despite availability of web-based PEMs for COVID-19, the readability was significantly higher than the National Institute of Health and US Department of Health and Human Services recommended sixth grade reading level and actionability of PEMs was low. It is critical to provide readable PEMs on COVID-19 to effectively disseminate accurate information and facilitate patients’ understanding of the virus, how it spreads, and how to protect themselves.

Background

The novel Coronavirus, SARS-CoV-2, and subsequent COVID-19 pandemic has led to a public health crisis.^{1 , 2} In the United States, the COVID-19 outbreak has had profound economic and social impact as a result of statewide quarantines, social distancing and public face mask mandates. One significant consequence has been the effect on the healthcare system, which as of September 2020 has seen over 6 million cases and 185,000 fatalities nationwide.³

Patients infected with SARS-CoV-2 can present with a myriad of symptoms, which can range from mild to severe. Typical symptoms include fever, cough, shortness of breath, and muscle pain.⁴ Severe complications have been reported in 33% of patients, including acute respiratory distress syndrome, septic shock and severe pneumonia.^{5 , 6} Currently, there is no treatment or approved vaccine against SARS-CoV-2 and cases worldwide continue to rise.

Our understanding of this infection has evolved and expanded since the start of the pandemic. However, there continues to be a number of uncertainties regarding the wide-ranging manifestations of the disease, it is appropriate treatment and management, and optimal ways of preventing transmission.7, 8, 9, 10 Additionally, there are many misconceptions among the general public and a need for reliable information for the public. The internet is a frequently used resource for providing patient education materials (PEM). Studies show that of the 78% of adults in the United States who have internet access, 83% of them utilize the internet to search for health information.¹¹ However, much of the health information available online is written above the sixth grade reading level recommended for PEM by the National Institute of Health and US Department of Health and Human Services.¹²

It is critical to maximize the readability of PEMs on COVID-19 to effectively disseminate accurate information. Increasing public knowledge about this infection can potentially improve health outcomes by teaching early identification of symptoms and effective methods for preventing the spread of disease. The primary aim of this study was to evaluate the readability, content, and quality of web-based PEMs on COVID-19 from US academic medical centers.

Materials and methods

This article was prepared in accordance with the guidelines for “Strengthening the Reporting of Observational Studies in Epidemiology” (STROBE).¹³

The names of US medical schools were obtained from the Association of American Medical Colleges website (n = 145). Institutional websites, hospital websites, and heath encyclopedia websites associated with each schools’ medical center were identified using the Google search engine. All websites were searched for PEMs using the following key words: coronavirus, COVID, COVID-19, and SARS-CoV-2. Search for PEMs occurred from June 17, 2020 through June 26, 2020. PEMs were excluded if they were written in languages other than English or information was only presented in graphic, pictoral, video, or tabular format. PEMs that were linked from nonacademic websites were also excluded. All PEMs were evaluated for readability, content, and quality.

Text from PEMs was copied into a plain text document using Microsoft Word and edited. Text unrelated to educational material, including advertisements, website URLs, copyright information, navigation tools, addresses, and citations, was removed as it may influence readability scores. Supplemental editing was performed as recommended.¹⁴ Readability of PEMs was calculated using three validated indices to determine the grade level necessary to understand the material: (1) Flesh-Kincaid Grade Level, (2) Simple Measure of Gobbledygook, and (3) Gunning Frequency of Gobbledygook (Gunning FOG).^{15 , 16} Each uses a unique formula to evaluate readability based on a combination of mean number of syllables per word, mean number of words per sentence, numbers of sentences and numbers of polysyllabic words.^{14 , 17 , 18} Each readability index reflects an estimated grade level that is required to read and comprehend a text without difficulty, therefore the lower the index, the easier the text is to read. A score of 6 in any of the indices reflects a sixth grade reading level.

The content of each PEM was evaluated using a scoring matrix developed by the authors (JK, EL, PT, WG, CE) based on COVID-19 educational materials found on the CDC website on June 10, 2020.¹⁹ The matrix consisted of 67 items organized into 10 domains: (1) definition of coronavirus, (2) signs and symptoms of the illness, (3) testing, (4) disease spread, (5) primary prevention, (6) what to do if ill, (7) timing of exposure to others after the disease, (8) at-risk populations, (9) what to do if you have had an exposure, and (10) additional resources.

The quality of PEMs was evaluated using the Patient Education Material Assessment Tool for Print (PEMAT-P).²⁰ PEMAT-P is a validated tool used to assess the quality of PEM by evaluating the understandability and actionability of a document.²⁰ Understandability is a metric that appraises the content, word choice, style, organization, use of numbers, lay out, design, and use of visual aids. Actionability is whether an action is identifiable in the text and how easily that action can be acted on. PEMAT-P scores range from 0-100% in either domain. A score of less than 70% reflects poor performance.²⁰

Descriptive statistics were used to summarize the data. The mean readability scores of the PEMs were compared, using a one-sample t test, to the scores that indicted a sixth grade reading level. Content proportions were analyzed using binomial distribution, with 95% confidence intervals (CIs). All data were analyzed with STATA SE (Version 12, College Station, TX). This study was exempt from Institutional Review Board (IRB) approval per Northwestern University's IRB guidelines.

Results

Of the 145 US medical schools identified, 100% of their associated medical centers had PEMs on COVID-19. Four of the Spanish-language PEMs were excluded, leaving 141 (97%) PEMs were analyzed for readability, content, and quality.

The mean readability of PEMs was above the recommended sixth grade reading level by all indices (Table 1 ). When each individual PEM was evaluated, only one PEM (0.7%) had a Flesh-Kincaid Grade Level readability at or below the sixth grade reading level. Two PEMs (1.4%) had a Simple Measure of Gobbledygook readability at or below the sixth grade reading level, while three (2%) met this threshold using the Gunning FOG index.

Table 1.

Readability indices for COVID-19 patient education materials (n = 141)

Readability indices	Mean score ± standard deviation	Compared to sixth grade reading level (P value)
FKGL*	10.5 ± 1.5	<.001
SMOG†	9.6 ± 1.2	<.001
Gunning FOG‡	12.6 ± 1.7	<.001

^⁎Flesh-Kincaid Grade Level.

^†Simple Measure of Gobbledygook.

^‡Gunning Frequency of Gobbledygook.

The PEMs were evaluated for the inclusion of 67 items across 10 domains (Table 2 ). Fever was the most frequently documented symptom in the PEMS (98% of PEMs). Ninety-seven percent of PEMs instructed patients to contact their health care provider if they were ill. Sixty-two percent of PEMs discussed seeking immediate medical care, yet very few referred to the specific signs that should prompt such care: trouble breathing 43%, persistent chest pain 30%, inability to stay awake 28%, and blue lips or face 24%. Several websites discussed at-risk populations, with 63% discussing the elderly (age >65 years) for being at risk for COVID-19, and only 15% of PEMs referencing obesity as a risk factor. While many PEMs addressed testing, only 26% addressed the implications of a positive test and only 21% addressed the implications of a negative test. Even fewer (16%) discussed the asymptomatic spread of disease. While the use of face coverings was suggested in 75% of PEMs, just 38% described the proper use of face masks. Criteria for return after illness was infrequently discussed with fewer than 15% of PEMs discussing possible criteria for return to work based on symptoms or days since a positive test. The CDC was the most commonly cited additional resource (94% of PEMs).

Table 2.

Content evaluation of Internet-based patient education materials on COVID-19 (n = 141)

Content	Proportion of websites (95% CI*)
General information
Definition of COVID-19	86% (81%-92%)
Timeframe of illness	55% (47%-64%)
Exposure self-isolate 14 days	37% (29%-45%)
Signs and symptoms
Fever	98% (95%-100%)
Cough	95% (92%-99%)
Shortness of breath	94% (90%-98%)
Sore throat	62% (53%-70%)
Myalgias	58% (49%-66%)
Anosmia (loss of smell)	54% (46%-63%)
Ageusia (loss of taste)	53% (44%-61%)
Headache	46% (37%-54%)
Chills	44% (35%-53%)
Diarrhea	35% (27%-44%)
Fatigue	30% (22%-37%)
Nausea/vomiting	29% (21%-37%)
Rhinorrhea	29% (21%-36%)
Nasal congestion	25% (17%-32%)
Testing
Viral test	79% (72%-86%)
Where can a test be obtained	58% (49%-66%)
How to schedule a test	55% (46%-63%)
Antibody test	41% (32%-49%)
What does a positive test mean	26% (19%-34%)
What does a negative test mean	21% (14%-28%)
Disease spread
Close contact (<6 feet)	71% (64%-79%)
Respiratory spread	68% (60%-76%)
Asymptomatic spread	16% (10%-22%)
How to protect yourself
Hand hygiene	95% (91%-99%)
Maintain social distancing	89% (83%-94%)
Disinfect high tough surfaces	79% (72%-86%)
Avoid touching face	77% (70%-84%)
Face coverings	75% (68%-83%)
How to wash hands	68% (60%-76%)
Define social distancing	66% (58%-74%)
Description on proper use of masks	38% (29%-46%)
If you are sick
Contact your health care provider	97% (94%-100%)
Stay home except to get medical care	72% (64%-80%)
Track your symptoms	55% (47%-64%)
Separate yourself from other people	50% (42%-59%)
Define quarantine	29% (21%-37%)
Define isolation	20% (13%-26%)
Emergency warning signs to seek care immediately	62% (53%-70%)
Emergency: trouble breathing	43% (34%-51%)
Emergency: persistent chest pain	30% (22%-38%)
Emergency: inability to stay awake	28% (20%-36%)
Emergency: bluish lips or face	24% (17%-31%)
Criteria for return after illness
Three days without fever	15% (9%-21%)
Ten days since symptoms first appeared and improving	11% (6%-17%)
Ten days since positive test if asymptomatic	8% (3%-12%)
At-risk populations
Elderly (>65 years old)	63% (55%-71%)
Serious heart conditions	52% (43%-60%)
Chronic lung disease	47% (38%-55%)
Diabetes	46% (37%-54%)
Immunocompromised	35% (26%-43%)
Cancer patients	29% (21%-37%)
Chronic immune weakening medications	23% (15%-30%)
Immune deficiencies	22% (15%-29%)
Pregnancy	16% (9%-22%)
Transplant patients	16% (10%-22%)
Obesity	15% (9%-21%)
Chronic kidney disease	15% (9%-21%)
Asthma	14% (8%-19%)
Long term care facility patients	11% (5%-16%)
Poorly controlled HIV†/AIDs‡	9% (4%-14%)
Liver disease	8% (4%-13%)
Dialysis	8% (3%-12%)
Additional resources
CDC§	94% (91%-99%)
WHO⁎⁎	37% (29%-45%)
Other	84% (78%-90%)

^⁎CI = confidence interval.

^†HIV = human immunodeficiency virus.

^‡AIDS = acquired immunodeficiency syndrome.

^§CDC = Center for Disease Control and Prevention.

^⁎⁎WHO = World Health Organization.

The PEMAT-P scores for PEMs reflected good understandability with a median score of 83% (IQR 75%-87%). Sixteen (11.6%) PEMs had below 70% score suggesting poor understandability. Actionability of the PEMs was poor with a median score of 41% (IQR 40%-60%). Only 16 (11.6%) PEMs had an actionability above 70% suggesting good actionability.

Discussion

In this study, we have demonstrated that although all medical centers associated with US medical schools had web-based PEMs about COVID-19, the readability was significantly higher than the National Institute of Health and US Department of Health and Human Services recommended sixth grade reading level and actionability of these PEMs was low.

As literacy of the general public is limited, comprehension of the general public should also be considered.²¹ The importance of comprehension of PEMs on COVID-19 should ideally allow the public not only to understand material but also to act upon it. While the majority of the PEMs had good understandability, the actionability was considered poor in most cases. Despite high reading levels of PEMs, the layout and presentation of the material, which is not assessed by the readability indices, allows for improved understanding of the PEMs. Actionability is particularly important for COVID-19 PEMs as a community faced with a global pandemic should be able to take steps to protect itself and how to proceed if illness occurs. Improvement of actionability could use decision trees emphasizing test results and symptoms to determine quarantine length, testing, and treatment as well as when to seek emergency medical care.

Our content rubric was based on the CDC's coronavirus resources.¹⁹ As the nation's health protection agency, the CDC is the leading federal agency for America's public health initiatives, and the vast majority of PEMs listed the CDC as an additional resource. Our data revealed that almost all PEMs addressed the three most common COVID-19 symptoms: fever, cough, and shortness of breath.²² However, PEMs frequently did not detail other symptoms associated with COVID-19 such as the associated gastrointestinal symptoms, which account for 18% of presenting symptoms.²³ Given the contagious nature of this disease,²⁴ it is imperative that the infected public not only recognize symptoms of the disease, but also employ safety measures to decrease risk of exposure to others.²⁵ While the majority of PEMs addressed social distancing, hand hygiene, and use of face coverings, few described the proper use of masks, which has been associated with flattening of the curve in many areas in the United States.²⁶

Almost all PEMs emphasized the importance of contacting a health care provider if a patient is ill with COVID-19 like symptoms, however few addressed emergency warning signs to seek immediate care. Further, only half of PEMs emphasized that ill individuals should isolate from other people. Testing is key to identifying and limiting spread of the novel coronavirus.²⁷ Description of testing types may prove useful in public decision making as polymerase chain reaction and immunoglobulin M testing is reflective of current infection, while immunoglobulin G tests for past infection. While many PEMs discussed testing for the virus, less than a third discussed what the results of a positive or a negative test mean. Testing for the virus has evolved over time, with some tests having a false negative rate as high as 29%.²⁸ The possibility and implications of a false negative is important to include in PEMs as a negative result may lead to a false sense of security.²⁹ As the pandemic has progressed, criteria for safe return after illness has continued to change.³⁰ Yet, it was rare for PEMs to address when patients could return after symptoms abate or after an asymptomatic positive test result.³¹

Interestingly, it was rare for PEMs to describe the high-risk populations for COVID-19. Early data released by the World Health Organization showed how those with co-existing disease are at greater risk, particularly those with obesity, hypertension, chronic lung disease, diabetes mellitus and cardiovascular disease.³² Only a quarter of those at-risk populations were addressed within the PEMs; furthermore, risk factors of obesity and residence in long-term care facilities were mentioned least. This is of concern as 68% of Americans are considered obese.^{33 , 34} Furthermore, as seen in other viral diseases such as influenza, the elderly are disproportionately affected.³⁴ Future PEMs need to ensure they address those that are most at risk.

Our study must be evaluated in the context of its limitations. PEMs were gathered at one point in time in an evolving globing pandemic and information provided may be outdated by the time patients are searching for it and using it. As guidelines from the CDC continue to change, our rubric was created prior to identifying PEMs in an attempt to minimize discrepancies in the rubric content and PEMs. The PEMs could only be collected from websites functioning properly; when a website was malfunctioning, the data collected were based on what could be readily obtained by the general public. While we chose to evaluate academic medical center online PEMs, patients may prefer to use non-academic websites when searching online health information. We chose to focus on academic medical centers, as their content may be more accurate than the information found elsewhere on the Internet. Additionally, there are multiple scoring instruments that can be used to assess readability scores; three validated scores were used for this paper to provide a varied evaluation, but these choices were not exhaustive.^{15 , 16}

Conclusions

The public uses the Internet for health care concerns, and there is a plethora of COVID-19 materials available. Our study found that the readability, content, and quality of Web-based PEMs addressing COVID-19 is lacking. The coronavirus global outbreak has been called a once-in-a-century pandemic,³⁵ necessitating adequate and effective communication from the medical community to the general public. The best way to do so is to create materials that are written at the recommended reading level, and are understandable, and are actionable.