Quality and readability of drug fact sheets for FDA-approved and emergency use authorised drug products for COVID-19 treatment: an observational study

Abstract

Objectives

To determine the quality of drug manufacturers’ fact sheets for patients for COVID-19 therapeutics for baricitinib, convalescent plasma, anakinra, molnupiravir, nirmatrelvir/ritonavir, remdesivir, tocilizumab and vilobelimab, and fact sheet readability.

Design

Cross-sectional document analysis.

Setting

Fact sheets on COVID-19 drugs approved by the US Food and Drug Administration from 2020 to 2023.

Primary and secondary outcome measures

Quality assessments with the 16-item DISCERN tool scored 16–80 points and 36-item Ensuring Quality of Information for Patients (EQIP) tool scored 0–36, where lower scores indicate low-quality information. We assessed readability with Flesch-Kincaid Reading Ease (ranges from 0 to 100 where higher scores correspond to reading ease). Higher grades indicated hard-to-read information: Flesch-Kincaid grade level (ranges from grades 0 to 18 (college graduate)), Gunning-Fog score (ranges from grades 0 to 20 (college graduate)), Coleman-Liau index (ranges from grade 4 to college graduate), automated readability index (ranging from grades 5 to 22 (college graduate)), Dale-Chall Readability (ranges from grade 4 to college graduate) and simple measure of gobbledygook (ranges from grade 3 to college graduate). Secondary outcomes were word, syllable and sentence counts. We reported percentages and the median (IQR).

Results

We found 18 fact sheets that described 11 (63.5%) anti-virals (remdesivir (n=4), molnupiravir (n=4) and nirmatrelvir/ritonavir (n=3)) and 7 (37.5%) immune modulators (tocilizumab (n=2), baricitinib (n=2), convalescent plasma (n=1), anakinra (n=1) and vilobelimab (n=1)). DISCERN (median (IQR)) reliability was 4 (IQR 3–4) and 5 (1–5), while DISCERN treatment information was 3 (1–5) and 5 (1–5) for anti-virals and immune modulators, respectively. EQIP (median (IQR)) content was 12 (11–13) and 11 (11–13), identification of information was 4 (3–4) and 3 (3–3) and structure was 9 (8–9) and 9 (9–9) for anti-virals and immune modulators, respectively. Overall, fact sheets had median readability grade levels that ranged from 6.2 to 12.4. Anti-viral and immune modulator fact sheets had median readability grade levels from 6.1 to 12.5. Median (IQR) word, >4 syllable words and sentence counts were 1646.5 (1318.3–1934.8), 25.0 (21.3–29.8) and 118.0 (92.0–152.5) overall; 1758.00 (1200.0–2181.0), 23.0 (15.0–27.0) and 134.0 (82.0–185.0) for anti-virals; and 1461.0 (1341.0–1776.0), 29.0 (23.0–46.0) and 107.0 (105.0–122.0) for immune modulators, respectively.

Conclusions

Although of fair quality, the fact sheet reading level was high, and the transparency of sources used was low. Regulatory officials should enforce readable resources from drug manufacturers to guide patients’ decision-making surrounding COVID-19 therapeutics.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• Multiple quality and readability tools were used for reliable assessments.

• Subjectivity of the quality and transparency ratings of COVID-19 drug information was lessened with reliability assessments between two raters.

• Patients’ comprehensibility of the fact sheet information was not determined in this study.

• Numeracy skills and ease of comprehension with the use of graphical presentations were not determined.

• We only assessed the readability and quality of fact sheets in English.

Introduction

The WHO declared the end of the COVID-19 pandemic on 5 May 2023. Throughout the pandemic, there were 670 million cases and over 6.8 million deaths.^{1 2} Moreover, marginalised communities and underserved populations that were disproportionally burdened by hospitalisation and death during the pandemic most likely remain vulnerable to COVID-19.^{3 4} Hospitalisations from the beginning of the pandemic totalled over 265 million worldwide and over 44 million in the USA due to COVID-19.⁵ There is an increased likelihood of death and complications from serious and critical infection, which varies according to sex, race, immunocompromising conditions and comorbidities.^{3 6 7}

Vaccination against SARS-CoV-2 that causes COVID-19 infection seeks to prevent severe disease, but uneven vaccination coverage,^{8 9} absence of immune response to available vaccines and the emergence of COVID-19 variants highlight limitations of vaccination.¹⁰ In the absence of effective vaccination, therapeutics which are intended to reduce hospitalisation or risk of death in individuals with COVID-19 who could become seriously ill are still needed. In 2020, Food and Drug Administration (FDA) permitted emergency use authorisation (EUA) of drug products in the absence of other treatments during the COVID-19 pandemic.¹¹ EUA therapeutics consist of anti-virals molnupiravir, nirmatrelvir/ritonavir and remdesivir and immune modulators baricitinib, COVID-19 convalescent plasma and anakinra. The FDA has approved baricitinib, nirmatrelvir/ritonavir, remdesivir and tocilizumab for certain child populations and hospitalised adults who receive systemic corticosteroids and require supplemental oxygen, non-invasive or invasive mechanical ventilation or extracorporeal membrane osmosis.¹¹ The current findings and associated adverse events from clinical trials about EUA COVID-19 drugs are briefly described in table 1. These findings show that there is variable evidence on the effectiveness and safety profile of COVID-19 drugs.

Table 1. Description of FDA emergency use authorised (EUA) COVID-19 therapeutics.

Drug	Effect on SARS-Co-V-2	Key findings from clinical trials	Known adverse events
Molnupiravir62 63	Causes deleterious errors in the SARS-CoV-2 genome terminating replication and infectivity	Reduced hospitalisation and death (HR 0.69, 95% CI 0.48 to 1.01) compared with placebo for mild-to-moderate COVID-19	Diarrhoea and nausea
Nirmatrelvir/ritonavir64 65	Inhibits the 3-chymotrypsin-like cysteine protease of SARS-CoV-2	Decreased hospitalisation by 59.0% (risk ratio (RR) 0.41, 95% CI 0.29 to 0.59) and death by 76.0% (RR 0.24, 95% CI 0.15 to 0.37)	Diarrhoea and muscle pain as adverse events
Anakinra66	Inhibits pro-inflammatory responses of interleukin (IL)-1α and IL-1β	A 55.0% reduction in 28-day mortality (HR 0.45, p=0.045) compared with standard care	Infections, ventilator-associated pneumonia, septic shock, multiple organ dysfunction and pulmonary embolism
Convalescent plasma67	Contains SARS-CoV-2 antibodies for treatment of COVID-19 in immunocompromised individuals	Does not reduce mortality in individuals with moderate or severe disease compared with standard care (RR 0.98, 95% CI 0.92 to 1.05)	Transfusion-related acute lung injury and fever
Remdesivir68 69	Inhibits the replication of pathogenic RNA viruses such as SARS-Co-V-2	May not reduce risk of mortality concerning moderate to severe COVID-19 (RR 0.93, 95% CI 0.81 to 1.06)	Nausea, headache and cough
Baricitinib70 71	Inhibitory effects Janus kinase 1 and 2 selective inhibitors associated with SARS-CoV-2 propagation and inflammatory effects	Patients treated with baricitinib compared with placebo experienced a 22.0% reduction (OR 0.78, 95% CI 0.56 to 1.03) in mortality	Fungal co-infections and acute kidney injury
Tocilizumab71,73	Inhibits interleukin-6 (IL-6) receptor activation since increased levels of IL-6 predict progression to respiratory failure or mortality	Mortality decreased by 26.0% (OR 0.74, 95% CI 0.58 to 0.94) in patients treated with tocilizumab compared with standard treatment	Fungal co-infections and acute kidney injury
Vilobelimab74 75	Anti-C5a monoclonal antibody	Reduced 28-day all-cause mortality in patients with severe COVID-19 in phase 2 (HR 0.65 (95% CI 0.10 to 4.14)) and 3 (HR 0.67 (95% CI 0.48 to 0.96)) trials	Acute kidney injury and septic shock

CI, confidence interval; COVID-19, coronavirus disease; HR, hazard ratio; RNA, ribonucleic acid; RR, risk ratio; SARS-Co-V-2, severe acute respiratory syndrome coronavirus 2.

More evidence on the safety and effectiveness of EUA therapeutics is needed, especially for populations that have limited English proficiency and are under-represented in clinical research.^{12 13} Materials for patients can help facilitate health-related decisions; however, the quality and readability of the information are essential.¹⁴,¹⁶ In the context of the COVID-19 pandemic, information about medication prescriptions should be readable and transparent to enable decision-making by individuals who have low English proficiency. Quality health-related information should be available to reading-level and linguistically diverse populations globally, but this report focuses on the US population for which FDA EUA drugs are approved. In the USA, 67 million people speak at least one other language than English, and among the 33% of Americans that read at or below the sixth-grade level, some individuals belonging to racial and ethnic groups self-report low English proficiency.¹⁷,²⁰ Namely, English proficiency varies by racial and ethnic group with Native Hawaiians and Pacific Islanders self-reporting at 12.2%, Latinx Americans at 29% and Asian Americans at 32%.²¹,²⁴ Additionally, Creole-speaking Haitian individuals, who comprise the largest Black demographic group in the USA, self-report 45% proficiency.²⁵ Health-related information should be written at or below the sixth-grade reading level with shorter sentence lengths to facilitate understanding of the information by readers who may have different levels of reading ability.^{26 27} Health-related materials that are not comprehendible to patients may lead to diminished health outcomes due to misunderstanding pthrescription reasons and poor access to health services.^{28 29} Drug information, especially during the COVID-19 pandemic and for future pandemics, should be tailored for people with low English proficiency at or below the sixth-grade reading level to foster inclusivity surrounding information about essential medicines.

Fact sheets are documents produced by pharmaceutical companies about EUA or FDA-approved drugs that provide key information for patients about the purpose, potential risks, adverse events and additional sources of information.¹¹ Especially in an era of misinformation about COVID-19 treatments, fact sheets by pharmaceutical companies should provide relevant and clear information to facilitate patients’ decision-making.³⁰,³³ A recent study on fact sheet readability for COVID-19 vaccines found that the information was written above the sixth-grade reading level, potentially posing challenges to comprehension.³² Fulmer and colleagues observed that the majority of package inserts that provided printed information in English in over-the-counter rapid antigen tests for COVID-19 were clearly formatted and organised, but the readability was at the eighth to ninth grade reading level.³⁴ Few studies have described the content of EUA fact sheets for drug products, and none to date have assessed information provided in fact sheets for COVID-19 therapeutics.^{32 35 36} There was still a federally imposed EUA for COVID-19 drugs as of December 2024, highlighting the extended need for medicine to treat COVID-19 after the official end of the pandemic.³⁷ Thus, we aimed to assess the quality, clarity and readability of the information in FDA EUA fact sheets for COVID-19 therapeutics produced by drug manufacturers of (1) molnupiravir, (2) nirmatrelvir/ritonavir, (3) anakinra, (4) COVID-19 convalescent plasma, (5) remdesivir, (6) baricitinib, (7) tocilizumab and (8) vilobelimab. Specifically, we determined the quality of the fact sheets concerning the transparency and relevance of treatment information, clarity of the information regarding the layout and presentation of the text, and readability of the fact sheets intended for patients and caregivers.

Methods

We followed Strengthening the Reporting of Observational Studies in Epidemiology reporting guidelines (see online supplemental appendix 1).³⁸

Identification and selection of fact sheets

One investigator (SMP) searched the FDA EUA website for the therapeutics in November 2021 to create search strings for searches in Google (see online supplemental appendix 2). Google Chrome for Windows on 17 January 2022 (V.97.0.4692.71) and 21 November 2022 (V.107.0.5304) was used by SMP to identify fact sheets for patients and caregivers describing FDA EUAs and FDA-approved therapeutics for COVID-19. SMP additionally searched the FDA EUA website on 2 March 2022 (Chrome V.97.0.4692.71) and 5 February 2023 (Chrome V.109.0.5414.120) and five additional fact sheets were obtained (see online Supplemental Appendix 3 available at https://osf.io/sr7w2/?view_only=68ae930a71bd4a4390c3ffba088691ef³⁹). Between searches, the cache, cookies, and download and browsing history were cleared. SMP downloaded and saved portable document file copies of fact sheets.

Before the selection of eligible fact sheets, SMP and JKC calibrated the extraction sheet independently. Two investigators determined eligibility of fact sheets for inclusion from the title and text. Eligible fact sheets (1) were published, updated or revised from January 2020 to 5 February 2023; (2) described information about COVID-19 EUA or FDA-approved therapeutics for patients; (3) were at least 100 words in length⁴⁰,⁴² and (4) were in English. We excluded (1) healthcare provider fact sheets, (2) journal articles, (3) newspaper articles and (4) drug labels. Disagreements between investigators were resolved through consensus discussions.

“Fact sheet” or “factsheet” and “patients” or “recipients” or “parents or caregivers” not “health care” or “healthcare providers” comprised the search. We also found fact sheets on the EUA site.¹¹ One investigator (SMP) categorised the anti-virals (nirmatrelvir/ritonavir, remdesivir and molnupiravir) or immune modulators (tocilizumab, baricitinib, COVID-19 convalescent plasma, anakinra and vilobelimab) from descriptions in FDA-issued letters of authorisation. Multiple versions of fact sheets for a single drug were analysed together due to similar DISCERN and Ensuring Quality of Information for Patients (EQIP) scores (see online Supplemental Appendixes 4 and 5 available at https://osf.io/sr7w2/?view_only=68ae930a71bd4a4390c3ffba088691ef).

Quality and clarity ratings

Two investigators (SMP, JKZ) independently rated a 10.0% (n=2) random sample of fact sheets chosen with a random number generator for quality using the DISCERN tool⁴³ and clarity using the EQIP tool⁴⁴ described below, after a training and calibration period to use the tools and discuss unclear items. The agreement for the DISCERN and EQIP ratings, which are subjective ratings, was calculated by dividing the number of concordant ratings between SMP and JKZ by the overall number of ratings multiplied by 100 to yield percent agreement. We had excellent agreement for both the DISCERN (90.0%) and EQIP extractions (95.8%). Disagreements about the sources of information for the DISCERN and document producers for the EQIP were resolved after discussions to reach a consensus without the involvement of a third author. The remaining fact sheets were rated by SMP independently due to the high agreement between the raters.

The 16-item DISCERN tool was used to rate the quality and relevance of information on COVID-19 therapeutics intended for patients using Likert scale responses with 1 (low or not available), 2–4 (moderate) and 5 (high) to rate these items. DISCERN uses four criteria to assess the authorship, attribution, currency of information and ownership of a publication (producer of the information and conflict of interest). We separated domains of DISCERN as follows: reliability includes items 1–8, treatment choices are items 9–15 and the overall quality is the total DISCERN score item 16.⁴⁵

We assessed the clarity of patient information about COVID-19 therapeutics using the 36-item EQIP tool, which consists of 18 items addressing qualitative content about the benefits vs harms of a drug, 6 items about the transparency and currency of drug information and 12 items about the format, layout and tone of the printed text. These items comprise three domains: content (items 1–18), identification (items 19–24) and structure of the information (items 25–36). We responded with yes (scored as 1) or no (scored as 0) to indicate the presence or absence of information and denoted N/A for a not applicable EQIP question. The minimum number of points overall is 0, and the maximum is 36. High-scoring fact sheets for clarity had median EQIP scores greater than 26, which corresponds to the 75th percentile of the overall or domain-specific scores.^{44 46}

Readability assessments

Fact sheets were uploaded to readable.io (SMP) and readability was automatically analysed with multiple standardised tests for a comprehensive assessment of readability and text characteristics.⁴⁷ Readability tests were the Flesch-Kincaid Reading Ease scored 0–100, where lower scores indicate difficult to understand text and higher scores indicate easier reading. Lower grade levels correspond to easier readability with the Flesch-Kincaid grade level (ranges from grade 0 to 18 (college graduate)) and Gunning-Fog score (ranges from grade 0 to 20 (college graduate)). The Coleman-Liau index and automated readability index, ranging from 5 to 22 (college graduate); New Dale-Chall Readability (NDCR) (ranges from grade 4 to college graduate); and simple measure of gobbledygook (ranges from grade 3 to college graduate) scores correspond to years of education needed to understand written text. Furthermore, we determined word count, syllables-per-word, words with >2 syllables, words-per-sentence and sentence count from readable.io. Similarities in readability and text characteristics between fact sheet versions allowed grouping of different versions (see online Supplemental Appendixes 6 and 7 available at https://osf.io/sr7w2/?view_only=68ae930a71bd4a4390c3ffba088691ef⁴⁰).

Statistical analysis

Scores from DISCERN, EQIP and readability score data were treated as non-parametric. We summarised overall and separate domain scores from the DISCERN and EQIP, as well as readability scores with medians (Md) and IQR. We used IBM SPSS Statistics for Windows, V.25.0 (IBM Corp., Armonk, New York, USA) for analyses.

Patient and public involvement

Patients and the public were not involved in the design or conduct of this study.

Results

Characteristics of the fact sheets

We retrieved 18 eligible fact sheets issued from March 2023 to November 2023. During this period, the FDA revoked EUA status for casirivimab/imdevimab (REGEN-CoV), bebtelovimab, tixagevimab/cilgavimab (Evusheld), sotrovimab and bamlanivimab/estesevimab, and we excluded these therapeutics (see online Supplemental Appendix 3 available at https://osf.io/sr7w2/?view_only=68ae930a71bd4a4390c3ffba088691ef³⁹). As of 5 February 2023 and 2 December 2023, we included 18 fact sheets, of which 11/18 (61.1%) described information about anti-viral and 7/18 (38.9%) were immune modulator therapeutics that were EUAs. Most fact sheets described information about remdesivir produced by Gilead Sciences (n=4), followed by molnupiravir manufactured by Merck & Co. (n=3), paxlovid by Pfizer (n=4), baricitinib by Eli Lilly and Company (n=2), tocilizumab by Genentech (n=2), anakinra manufactured by Swedish Orphan Biovitrum AB (n=1) and vilobelimab produced by InflaRx GmbH (n=1). There was no manufacturer listed for convalescent plasma (n=1).

DISCERN scores

Regardless of the therapeutic group of fact sheets, the median DISCERN total score was moderate (3 (IQR 3–4)). No fact sheets provided sources of information described in a reference list (see online Supplemental Appendix 3 available at https://osf.io/sr7w2/?view_only=68ae930a71bd4a4390c3ffba088691ef⁴⁰). 3 out of 16 (19%) fact sheets (1 for remdesivir and 2 for anakinra) provided high-quality information about how the therapeutic works. Fact sheets provided links to various federal websites for more information, but no links were provided about the basic mechanism of action.⁴⁸

The median DISCERN total scores for anti-virals and immune modulators were moderate (table 2). According to the domains of DISCERN, reliability of information for anti-virals was moderate, while the median reliability score for immune modulators was high (table 2).

Table 2. The DISCERN scores for the quality and transparency of the information in fact sheets intended for patients and caregivers on COVID-19 therapeutics.

DISCERN* score, median (IQR)	Total (n=18)	Anti-viral† (n=11)	Immune modulator‡ (n=7)
Overall	3 (3–4)	4 (3–4)	4 (3–4)
Domain
Reliability	5 (1–5)	4 (1–5)	5 (1–5)
Treatment	3 (1–5)	3 (1–5)	5 (1–5)

^*The quality of the information was classified according to the median score as ‘high’ (5), ‘moderate’ (2–4) or ‘absent’ (1).⁴³

^†Nirmatrelvir/ritonavir, remdesivir and molnupiravir.

^‡Tocilizumab, baricitinib, COVID-19 convalescent plasma, anakinra and vilobelimab.

EQIP scores

The overall median (IQR) EQIP score was 24.0 (23.0–26.00). The fact sheet for vilobelimab was the only one to describe information about alternative treatments. No fact sheet provided references to published evidence or mentioned involving patients in writing the fact sheets (see online Supplemental Appendix 5). Similar to DISCERN ratings about treatment risks, all fact sheets, regardless of therapeutic group, mentioned treatment risks rated with the EQIP. Both anti-viral and immunomodulator therapeutics were not highly rated overall or by domain according to the EQIP (table 3). The two latest versions of nirmatrelvir/ritonavir fact sheets were the only ones to provide depictions of drug administration.⁴⁸

Table 3. EQIP scores reflecting the clarity of the fact sheets describing COVID-19 therapeutic information for patients and caregivers.

EQIP* score, median (IQR)	Total (n=18)	Anti-viral† (n=11)	Immune modulator‡ (n=7)
Overall	24 (23–26)	25 (23–26)	23 (22–25)
Domain
Content (maximum 18 points)	12 (11–13)	12 (11–13)	11 (11–13)
Identification of information (maximum 6 points)	3 (3–4)	4 (3–4)	3 (3–3)
Structure of the information (maximum 12 points)	9 (9–9)	9 (8–9)	9 (9–9)

^*Dichotomous ratings of 0 (absent) or 1 (present). We considered high-scoring fact sheets to have median EQIP scores greater than 26, 13, 4 and 9 which respectively correspond to the 75th percentile of the overall, EQIP content, identification of information and structure of information scores in this study.

^†Nirmatrelvir/ritonavir, remdesivir and molnupiravir.

^‡Tocilizumab, baricitinib, COVID-19 convalescent plasma, anakinra and vilobelimab.

Readability scores

The reading grade level was above the recommended sixth grade for fact sheets overall and according to therapeutic (table 4). The median sentence length was similarly within the acceptable 12–17 sentence length for all fact sheets, but the maximum number of words-per-sentence for convalescent plasma was 19 (see online Supplemental Appendix 6 available at https://osf.io/sr7w2/?view_only=68ae930a71bd4a4390c3ffba088691ef³⁹).

Table 4. Readability scores and other characteristics of patient, parent or caregiver fact sheets for COVID-19 therapeutics with Food and Drug Administration Emergency Use Authorization according to therapeutic.

Readability score, median (IQR)	Total (n=18)	Anti-viral* (n=11)	Immune modulator† (n=7)
Flesch-Kincaid reading ease index	50.8 (48.8–52.3)	51.5 (48.1–52.7)	49.4 (49.0–51.9)
Flesch-Kincaid grade level	9.6 (8.9–10.2)	9.4 (8.9–10.2)	9.8 (9.3–10.1)
Gunning-Fog score	11.2 (10.4–11.7)	11.0 (10.0–11.9)	11.3 (10.5–11.6)
Coleman-Liau index	11.4 (10.6–12.0)	11.3 (10.7–12.0)	11.4 (10.5–12.0)
Simple measure of gobbledygook	12.4 (12.0–12.8)	12.4 (11.9–12.7)	12.5 (12.1–12.9)
Automated readability index	9.2 (8.0–9.7)	9.2 (7.7–9.7)	9.5 (8.1–9.8)
Dale-Chall Readability Score	6.2 (6.0–6.4)	6.1 (5.9–6.3)	6.4 (6.2–6.7)
Word count	1646.5 (1318.3–1934.8)	1758.00 (1200.0–2181.0)	1461.0 (1341.0–1776.0)
Syllables per word	1.7 (1.7–1.7)	1.7 (1.7–1.7)	1.7 (1.7–1.7)
Words with more than four syllables	25.0 (21.3–29.8)	23.0 (15.0–27.0)	29.0 (23.0–46.0)
Words per sentence	13.7 (12.44–16.1)	13.2 (12.1–14.5)	13.8 (12.5–17.1)
Sentence count	118.0 (92.0–152.5)	134.0 (82.0–185.0)	107.0 (105.0–122.0)

^*Remdesivir, molnupiravir and nirmatrelvir/ritonavir.

^†Tocilizumab, baricitinib, convalescent plasma, anakinra and vilobelimab.

Discussion

Our analysis of the quality, clarity and readability presents novel findings pertinent to patients’ understanding of information about COVID-19 therapeutics described in fact sheets intended for patients. The quality of fact sheets according to the DISCERN, regardless of therapeutic group and by therapeutic group, was of moderate quality, owing to mostly absent references to sources used to compile the information and descriptions about what would happen if no therapeutic is used. EQIP scores were similarly moderate overall and by therapeutic group due to absent information about sources of information, particular treatment alternatives or costs, and patient involvement in writing the fact sheets. Regardless of therapeutic type or group, readability was above the sixth-grade level, which may undermine most patients’ understanding.

Health-related information from the internet can influence individuals’ interaction with healthcare providers and use of healthcare resources for those who have access.^{14 49}

High-scoring fact sheets included descriptions of potential harms associated with treatment and descriptions of how the treatment works (DISCERN). EQIP high-scoring fact sheets provided figures to show the appearance of the therapeutic and regimen as well as the administration sequence. Few fact sheets provided basic information of how the therapeutics work, which could affect patients’ decision-making towards choosing a COVID-19 therapeutic. Previous studies have shown that wording used to describe treatments may play a role in patients’ choices.³² Participants were less likely to comply with EUA recommendations when provided information that included words that conveyed uncertainty or unknown drug effects in a previous study.⁵⁰

As fact sheets in this study were of moderate quality, fact sheets should provide more transparent information about sources used or basic drug mechanisms. Manufacturers have a responsibility to provide a transparent record of sources used to describe potential mechanisms, adverse events and benefits of COVID-19 therapeutics. In the context of engaging citizens in research by helping them to interpret findings from research, citations to evidence would provide an opportunity to expose the public to information about established links between treatments and outcomes. Providing enough information surrounding COVID-19 therapeutics is needed due to inequitable distribution of COVID-19 vaccines, emerging SARS-CoV-2 variants and populations who are otherwise ineligible or opt out of vaccination.

The median reading grade level found from the majority of readability tests in this study required patients to have a ninth-grade reading level, which is above the reading level of 25.0% of the US population who have low literacy skills.⁵¹ In addition, lengthy sentences combined with low readability may deem information incomprehensible for most people and for those with limited English proficiency. Although fact sheets are also available in Spanish, 31.0% of Asian Americans self-reported a lack of English fluency.²²,²⁴ Accordingly, a health equity stimulus by the Biden Administration funded several organisations to provide patients access to languages other than English.⁵²

Complex information about the purpose and risks of COVID-19 therapeutics could be conveyed using readability programmes and shorter sentences. Furthermore, complex information could be provided visually, which has been shown to increase patients’ comprehension and tendency to search for further information.^{53 54} Notably, only three fact sheets in this study illustrated dosing and administration information. Fact sheets can provide comprehensible information, but as it may be unavoidable to describe information about drugs without medical terminology, medical education could play a role in providing training for professionals to be aware of reading level barriers that their patients may face in appraising health information from fact sheets.^{32 55} Medical education that included basic training to raise healthcare providers’ ability to appraise health information to inform health-related decision-making improved their ability to communicate understandable information to patients.⁵⁶,⁶⁰ Particularly, medical education can be effective in training providers to be aware of the need to facilitate patients’ ability to read and understand health-related materials, especially individuals with limited English proficiency.^{57 61} Professionals’ awareness of these disparities is essential since Latinx and Asian Americans, who comprise the largest ethnic groups in the USA with self-reported limited English proficiency, experienced high COVID-19 hospitalisation and mortality.³²²,²⁴

There are no studies that have assessed the content of COVID-19 fact sheets for quality and readability. A strength of this study was the use of multiple readability assessments. There are some limitations to describe. We did not assess the relationship between reading ease or graphical presentations and patients’ comprehension of fact sheets in terms of health numeracy. We only assessed readability and quality of fact sheets in English. We did not focus on the visual appearance of the fact sheets in this study. The visual appeal of fact sheets in terms of the amount of text and their formatting could play a role in patients’ reading and understanding of the text. Additionally, even with comprehensive searches, it is possible that we did not retrieve some of the issuances of the fact sheets. Furthermore, the majority of the fact sheets were rated by one investigator, although we established excellent agreement after a training period.

FDA officials and drug manufacturers for COVID-19 therapeutics and for future EUAs should ensure readability and the presence of supporting evidence with the help of patient advocates and medical educators. Readability programmes supplemented with comprehension tests could be conducted prior to fact sheet release to the public. Accordingly, pharmaceutical company representatives could tailor the readability, transparency and clarity of drug information to be relevant for patients. Prospective studies could further elucidate the extent of how the use of certain wording could confound patients’ use of fact sheet information towards health-related decisions.³² Furthermore, an assessment of the quality and readability of EUA fact sheets available in other languages is needed.

Conclusion

Although of fair quality, the reading level of fact sheets intended for patients, parents or caregivers for COVID-19 therapeutics was high, reflecting a need for FDA officials to enforce readable resources from manufacturers. EUA fact sheets for COVID-19 therapeutics could be a source of publicly available information providing transparent, reliable, relevant and clear sources for patients.

Data availability statement

Data are available in a public, open access repository. All data can be found on the Open Science Framework (https://osf.io/zwuvf/?view_only=bb71291d80284d9d97b3a568ac042e6f).