Readability of online and offline written health information: a protocol of a systematic review of systematic reviews

Abstract

Introduction

Healthcare organisations should create and disseminate readable information to citizens and patients to reduce the level of health literacy required for individuals. Several systematic reviews have reported overviews of the readability of written health information for each subject area. This systematic review of systematic reviews will synthesise evidence from existing systematic reviews to provide an overview of studies that assessed readability in diverse areas and identify gaps to be filled in future research and practice to improve the readability of health information.

Methods and analysis

The following databases will be searched: PubMed, MEDLINE, CINAHL, PsycINFO, Web of Science Core Collection, Scopus, Cochrane Database of Systematic Reviews, and Google Scholar. The reference lists of the eligible studies will be searched to identify additional studies. The eligible literature will be systematic reviews of published empirical studies that quantitatively assessed the readability of health information for patients and/or citizens. Two independent reviewers will screen the titles, abstracts and full texts of all identified literature. A lead researcher will extract the data, and a second reviewer will check the results. Discrepancies will be resolved through discussion, and a third reviewer will be involved if necessary. Data, such as the characteristics of the systematic review, characteristics of the reviewed literature in the systematic review, subject area, material characteristics, definition(s) of readability, readability assessment methods, main results and quality assessment, will be extracted. The findings will be synthesised in a numerical summary and a descriptive and narrative review.

Ethics and dissemination

This review does not involve any ethical concerns. The results and implications of this systematic review will be disseminated at conferences and in peer-reviewed journals.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• This will be the first systematic review of systematic reviews to focus on the readability of written health information.

• We will use the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocol guidelines to ensure a systematic approach for searching, screening and reporting.

• We will use A MeaSurement Tool to Assess systematic Reviews V.2 instrument to assess the risk of bias of the included systematic reviews, and the findings will be interpreted considering the assessment.

• This systematic review of systematic reviews may miss important reviews published in languages other than English.

Introduction

Globally, health literacy has been recognised as the key to improving the health of individuals and communities.¹ Health literacy is also key to health equity, which is the achievement of the highest level of health for all people.² The definition of health literacy was updated with the release of the US Government’s Healthy People 2030 in August 2020.³ The update explicitly addresses organisational health literacy as well as personal health literacy.³ Healthy People 2030 defines personal health literacy as ‘the degree to which individuals can find, understand, and use information and services to inform health-related decisions and actions for themselves and others’.³ Additionally, Healthy People 2030 defines organisational health literacy as ‘the degree to which organisations equitably enable individuals to find, understand, and use information and services to inform health-related decisions and actions for themselves and others’.³ The new definitions acknowledge that organisations within the healthcare system, such as public health institutions, hospitals and pharmacies, are responsible for addressing health literacy.^{3 4} For individuals with limited personal health literacy, healthcare services provided by organisations are often difficult to access, and these individuals experience difficulties in understanding and using the information provided.⁵ To reduce the level of health literacy required for individuals, efforts by healthcare organisations are expected to make it easier for people to access healthcare services and understand and use health information; this is what is recommended by organisational health literacy.1,3

Communication practices, including providing health information, are dimensions of organisational health literacy.^{6 7} In health literacy research and practice, the ease of understanding and use of health information have often been discussed in terms of readability.^{8 9} Readability is defined as ‘the determination by systemic formulae of the reading comprehension level that a person must have to understand written materials’.¹⁰ The readability of health information written in English has been assessed using tools such as the Fry Readability Graph, the Gunning Fog Index, the Flesch-Kincaid Reading Ease test, the Flesch-Kincaid Grade Level test and the Simple Measure of Gobbledygook Grade Level (SMOG).^{11 12} Readability assessment tools using systematic formulas for other languages, such as Spanish,¹³ German¹⁴ and French, are also available.¹⁵ These systematic formulas calculate the readability of text based on factors such as word difficulty level, average sentence length and complexity of sentence structure. The text can be made easier to read easily by manipulating these factors. Ease of reading is a key requirement for healthcare organisations to provide health information that can be easily understood and used by patients and citizens.^{11 12}

However, written health information is often difficult for most intended audiences to read.¹⁶ A systematic review of readability assessment studies reported that most publicly available mental health information uses vocabulary and sentence structures that are too complex for the general public.¹⁷ Another systematic review of readability assessment studies reported that most vaccine information supplied by healthcare providers exceeded the recommended readability level.¹⁸ Another systematic review reported that most online patient information regarding common sports injuries is above the readability level recommended by the American Medical Association and National Institutes of Health.¹⁹ Other systematic reviews in the areas of ophthalmology²⁰ and paediatrics²¹ have also reported that most patient education materials are difficult to read.

Several systematic reviews have reported the readability of online or offline health information in each area. However, there is no cross-area overview of readability studies. Due to the lack of cross-area overviews, it is not known in which clinical areas, target populations (eg, adult patients, paediatric patients and general public) and media types (eg, web pages, printed materials) readability has been comprehensively investigated and what levels of readability have been identified. A review of systematic reviews of readability studies will show in which areas readability studies have already been overviewed (and need to be overviewed) and in which areas readability should be improved. Thus, synthesising evidence regarding readability from existing systematic reviews will contribute to improving organisational health literacy in terms of health communication practices, including the provision of online and offline health information. This study aims to provide an overview of systematic reviews of studies that evaluated readability in diverse areas to identify the gaps in future research and practice. The following research questions are posed:

RQ1: Which areas (eg, clinical fields, targeted population and type of media) are systematic reviews of readability assessment studies?

RQ2: What degree of readability is reported in systematic reviews in each area?

RQ3: What gaps in research and practice should be addressed in the future (eg, Which areas should systematic reviews be conducted in? Which areas should the readability be improved)?

Methods and analysis

Study design and registration

The study protocol was designed in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols (PRISMA-P) guidelines.^{22 23} The proposed review will also be conducted in accordance with the recommended methodologies for conducting and reporting systematic review of systematic reviews.24,26 Figure 1 shows the study schema. This protocol will be registered with the International Prospective Register of Systematic Reviews once it is accepted for publication in a peer-reviewed journal. We plan to begin the literature search from 1 September 2024 and to complete the analysis by the end of 2024.

Figure 1. Study schema. AMSTAR 2, A MeaSurement Tool to Assess systematic Reviews V.2.

Literature search

The following databases will be searched: PubMed, MEDLINE, CINAHL, PsycINFO, the Web of Science Core Collection, Scopus and Cochrane Database of Systematic Reviews. Google Scholar will also be searched for the triangulations. We will search abstracts and titles using the combination of keywords with reference to previous systematic reviews^{17 21 27}: AB (“readability” OR “reading level” OR “reading grade level” OR “reading ease” OR “ease of reading” OR “comprehensibility” OR “understandability” OR “ease of comprehension” OR “ease of understanding” OR “health literacy” OR “usability”) AND TI (“systematic review” OR “meta-analysis”). All search results were imported into the Rayyan QCRI software to ensure a systematic literature selection process.²⁸ We will include all publications from the time the database search is initiated to the time of the final search. The reference lists of the identified eligible studies will be searched to supplement the database searches and identify any additional potentially eligible studies.

Inclusion criteria

The proposed systematic review targets readability studies that assess the readability of written health information; these studies do not include individual participants or interventions. Therefore, the inclusion criteria of this protocol were not presented in the participants, intervention, control and outcomes format recommended by guidelines such as the A MeaSurement Tool to Assess systematic Reviews (AMSTAR) V.2 instrument.²⁹ Studies will be included in this proposed systematic review if they fulfil the following criteria:

1. Study design: systematic reviews of published literature of empirical studies that quantitatively assessed the readability of written health information for patients and/or citizens (eg, patient education materials). Where papers include both readability and other measurements (eg, understandability), we will include readability data if they are presented separately.

2. Type of media: any online or offline text-based media that was evaluated for readability will be eligible (eg, web pages, printed materials).

3. Type of measurement: any tool that uses a systematic formula or other method to assess readability will be eligible.

4. Type of literature: articles published in academic journals will be included. Grey literature (eg, conference proceedings, theses and dissertations) will also be eligible if sufficient information is provided to confirm its eligibility.

5. Publication year: we will not filter by publication year. Studies published at any time will be eligible for inclusion.

Exclusion criteria

1. Systematic reviews of studies that do not quantitatively assess readability of written health information.

2. Studies other than systematic reviews (eg, empirical studies that assessed the readability of health information).

3. Non-systematic reviews (eg, narrative review).

4. Reviews not written in English.

5. Reviews not published in full text.

Screening of studies

The Rayyan QCRI software²⁸ will be used for screening studies. Duplicates will be removed automatically using this software. First, titles and abstracts will be screened to identify eligible studies using the selection criteria. This first screening will be independently conducted by both the first (TO) and second authors (EF). Disagreements will be discussed until a consensus is reached. When a consensus cannot be reached, the third author (HO) will be involved in resolving the disagreement.

Second, the full text of the remaining literature will be screened independently by both the first (TO) and second authors (EF). Any disagreements will be resolved through discussion, and the third author (HO) will be involved if necessary. The screening process will be displayed in a PRISMA flow diagram.

Data extraction

A customised data extraction form will be created to extract all relevant data from each study. The data extraction form will be piloted with a 10% random sample of eligible literature to ensure reliability in extracting relevant data. The first author (TO) will conduct data extraction, and the second author (EF) will check the extracted data against the full texts of the eligible literature to ensure reliability. Disagreements will be resolved through discussion, and the third author (HO) will be involved if necessary. The extracted data will include:

1. Characteristics of the systematic review: journal, authors, year of publication, country, databases used, search range (years) and number of reviewed literature.

2. Characteristics of the reviewed literature in the systematic review: number of reviewed studies, publication range (years) of reviewed studies and countries of reviewed studies.

3. Study area: subject areas of non-communicable diseases (eg, cancer, mental health) or communicable diseases (eg, HIV, HPV).

4. Material characteristics: types of media (eg, printed materials, web pages), content of materials (eg, patient education), target population of materials (eg, adult patients) and total number and range of materials assessed for readability.

5. Definition(s) of readability.

6. Readability assessment methods: names of scales (eg, Flesch Reading Ease Scale, SMOG Readability Formula) or other methods that assessed readability.

7. Other measurements: names of measures that assessed other than readability when they were used.

8. Main results: readability scores, reading grade level and key findings.

9. Main discussion and conclusions.

10. Quality assessment: the tools that were used to assess risk of bias and the results of risk of bias.

Data synthesis

We will describe the characteristics of the included literature using a numerical summary in a concise table (eg, number of studies and reading grade level in each area). To answer our research questions, we will summarise and synthesise the findings of the included literature using an evidence gap map in a concise table. The evidence gap map will show, for example, the presence and absence of systematic reviews and the level of readability per area. The evidence gap map will provide an overview of which gaps and issues future research and practice should address. table 1 shows a sample of an evidence gap map. We will discuss the key findings and implications for future research and practice as we answer each of the research questions in a descriptive narrative review. Namely, we will organise what evidence exists in what areas and propose what gaps need to be filled in the research and practice of the readability of written health information.

Table 1. A sample of an evidence gap map.

Clinical areas		Targeted audience/readability level
Communicable diseases	COVID-19	Public/easy (x), public/difficult (x), patients/difficult (x)
	Ebola	None
	Hepatitis B/C	Public/difficult (x), patients/difficult (x)
	HIV/AIDS	Public/easy (x), patients/difficult (x)
	………	………
Non-communicable diseases	Cardiovascular diseases	Public/easy (x), public/difficult (x), patients/difficult (x)
	Cancers	Public/easy (x), public/difficult (x), patients/easy (x), caregivers/difficult (x)
	Chronic respiratory diseases	None
	Diabetes	Public/difficult (x), public/difficult (x), patients/difficult (x)
	………	………

‘x’ indicates a number of reference will be inserted.

Assessing risk of bias

The first author (TO) will appraise the risk of bias of each included literature using AMSTAR 2 instrument,²⁹ while the second reviewer (EF) will check all appraisals. A third reviewer (HO) will be consulted in case of disagreement. The AMSTAR 2 instrument is a 16-item questionnaire used to assess the methodological quality of systematic reviews, including the rationale for the review methods, a comprehensive search strategy and measurement of the risk of bias. The AMSTAR 2 criteria classify systematic reviews into four categories indicating confidence in the findings: high, moderate, low and critically low. The results of the assessment using the AMSTAR 2 instrument will be presented in a concise table as a quality assessment of the included systematic reviews. The findings of the systematic reviews included in our umbrella review will be interpreted in the results.

Limitations

Our study will have several limitations. First, although we will use a comprehensive search strategy and screen literature by two independent reviewers, we cannot entirely rule out the possibility of relevant studies being missed. Second, our study will include literature written in English, which may exclude relevant studies published in languages other than English. Third, our study will include systematic reviews of quantitative analyses of readability but qualitative studies will be excluded. Qualitative studies, such as interviewing participants about the ease of reading and understanding health information, are also important for improving health information. Therefore, future studies will be expected to review qualitative studies on the readability of health information. Finally, as our study includes systematic reviews, we will be unable to identify the current state of readability studies in areas where no systematic review has been conducted. However, our study will be an important first step towards a cross-area overview of readability studies of health information.

Patient and public involvement

No patients will be involved.

Ethics and dissemination

This work does not warrant any ethical or safety review. We intend to present the results of this review at a relevant conference and publish them in a peer-reviewed journal.