Research Tools and Protocols: Readability of Pediatric and Adult Patient-Reported Outcome Measures in Eosinophilic Esophagitis

Abstract

Patient-reported outcome measures (PROs) in eosinophilic esophagitis (EoE) have been utilized as research tools to assess outcomes in clinical trials. To our knowledge, adult and pediatric EoE PROs have not previously been analyzed from a health literacy perspective. We aimed to evaluate the readability of the most utilized EoE PROs for adult and pediatric populations and assess whether these PROs met national health literacy recommendations of readability at or below the sixth-grade level. We conducted a readability analysis of thirteen EoE PROs using 4 readability measures: Flesch-Kincaid Grade Level, Gunning Fog, Simple Measure of Gobbledygook, and FORCAST. Across these 4 individual metrics, the mean readability levels (years of education required) for PROs were 4.1, 5.5, 7.1, and 9.6 respectively. The 4 pediatric EoE PROs (PedsQL EoE module parent report for teens, PedsQL EoE module teen report, PEESS children and teen report, and PEESS parent report) included in this study had mean readability levels of 6.4, 6.2, 5.9 and 6.0, respectively. The 9 included adult EoE PROs (EoE-QoL-A, BEDQ, DSQ, EoE-IQ, EEsAI, EoE-SQ, PiEAQ, PROMIS Scale v1.0 – Gastrointestinal Disrupted Swallowing, and SDI) had mean readability levels ranging from 5.3 to 8.7 with a standard deviation of 1.2. The average readability for all included EoE PROs was 6.6. In conclusion, current EoE PROs as research tools are slightly above recommended readability levels. Future EoE PRO development could be strengthened by using shorter sentences, writing for the target audience, and utilizing input from age-appropriate patients.

Introduction

The prevalence of eosinophilic esophagitis (EoE) is rising in the United States, leading to an associated healthcare cost of approximately 1.3 billion dollars.¹ Diagnosis of EoE is both clinical and histologic, necessitating symptoms of esophageal dysfunction and mucosal eosinophilia in the absence of other causes.²

Patient-reported outcome measures (PROs) are key research tools to evaluate patient experiences such as treatment outcomes or quality of life, and are especially used to determine efficacy of clinical interventions.³ Nevertheless, health literacy assessment of PROs is necessary given data demonstrating that 36% of adults in the United States have basic or below basic health literacy scores.⁴ The American Medical Association recommends creating healthcare materials at or below the 6^th grade reading level to develop concordance with patient comprehension and readability of published healthcare texts.⁵

While many EoE PROs were developed and validated using best practices, prior work on assessing EoE PROs from a health literacy standpoint is limited. Our evaluation of research tools and protocols seeks to analyze the readability of the most utilized EoE PROs for adult and pediatric populations to provide insights on whether these PROs meet current national health literacy recommendations.

Methods

PRO Identification and Selection

This study was a bibliometric analysis and did not require Institutional Review Board approval. We conducted a literature search by first examining a Cochrane systematic review focused on the medical treatment of EoE which included 41 trials, 40 of which reported a PRO as a primary outcome.⁶ Of these, 14 used validated PROs, 8 were not clearly validated, and 18 included unvalidated tools. Additionally, 10 studies included QOL evaluation. We then expanded our literature search using the bibliographies of these trials and specific searches of PubMed. Ultimately, 13 PROs met the following inclusion criteria: written in the English language, intended for completion by an adult or pediatric patient population, validated in a patient population with EoE, or non-validated but studied in EoE population for dysphagia symptoms (SDI and PiEAQ).

The following PROs were selected for readability analysis within the scope of this review: Pediatric Eosinophilic Esophagitis Quality of Life Module (PedsQL EoE module) Parent Report for Teens (ages 13-18),⁷ Pediatric Eosinophilic Esophagitis Quality of Life Module (PedsQL EoE module) Teen Report (Ages 13-18,)⁷ Pediatric Eosinophilic Esophagitis (EoE) Symptom Score (PEESS) Version 2.0 Children and Teen Report (Ages 8 - 18),⁸ Pediatric Eosinophilic Esophagitis (EoE) Symptom Score (PEESS) Version 2.0 Parent Report for children and teens (ages 2-18),⁸ Adult Eosinophilic Esophagitis Quality of Life Questionnaire (EoE-QoL-A),⁹ Brief Esophageal Dysphagia Questionnaire (BEDQ),¹⁰ Dysphagia Symptom Questionnaire (DSQ),¹¹ Eosinophilic Esophagitis Impact Questionnaire (EoE-IQ),¹² Eosinophilic Esophagitis Symptom Activity Index (EEsAI),¹³ Eosinophilic Esophagitis (EoE) Symptom Questionnaire (EoE-SQ),¹² Pisa Eosinophilic Esophagitis Adaptation Questionnaire (PiEAQ),¹⁴ PROMIS Scale v1.0 – Gastrointestinal Disrupted Swallowing,¹⁵ and the Straumann Dysphagia Instrument (SDI).¹⁶ Key details of these PROs are summarized within the Supplementary Materials and Supplemental Tables 1 and 2; discussion of their development, validation, and use is beyond the scope of this paper.

Readability Analysis

Each PRO was extracted from its original form as found in publicly available or proprietary online repositories, published manuscripts, or at websites which facilitate the distribution of clinical outcome assessments, and reviewed for fidelity to the original PRO document prior to analysis on Readable.com, a website which utilizes various algorithms to score text for readability levels.¹⁷ Only text relevant to the questionnaire was included (i.e. Question stems and text, answer choices, scoring systems); we did not include formatting. The following formulas were utilized for readability analysis: Flesch-Kincaid Grade Level (FKGL), Gunning Fog, Simple Measure of Gobbledygook (SMOG), and FORCAST, descriptions of which can be found within the supplementary materials. Numeric output of the measures corresponds to the grade or years of education a reader would need for the analyzed passage’s reading level. To account for variability between readability formulas, an overall reading level was generated per PRO by adding each readability score per formula and averaging it as a mean. Further information on readability formulas and interpretations can be found in the supplementary materials.

Results

The mean readability levels of commonly used EoE PROs as calculated by averaging the four reading formula scores (FKGL, Gunning Fog, SMOG, and FORCAST) are found within the supplementary materials. Across individual metrics for FKGL grade level, Gunning Fog, SMOG, and FORCAST, the mean readability levels (years of education required) for PROs were 4.1, 5.5, 7.1, and 9.6 respectively (Figure 1A). The average readability for all included adult and pediatric EoE PROs was 6.6 (Figure 1B).

Figure 1A.

Overall mean readability levels (bar = readability level corresponding to a grade level) of EoE patient-reported outcomes by the different readability formulas. Each bar is the mean readability level as predicted by the specified formula, which corresponds to a grade level.

Figure 1B. Mean readability levels (bar = mean readability level corresponding to a grade level) per pediatric and adult EoE PRO. PR = parent report. TR = teen report. CT = children and teen report.

The 4 pediatric EoE PROs (PedsQL EoE module parent report for teens, PedsQL EoE module teen report, PEESS children and teen report, and PEESS parent report) included in this study had mean readability levels of 6.4, 6.2, 5.9 and 6.0, respectively (Supplemental Table 3).

The 9 included adult EoE PROs (EoE-QoL-A, BEDQ, DSQ, EoE-IQ, EEsAI, EoE-SQ, PiEAQ, PROMIS Scale v1.0 – Gastrointestinal Disrupted Swallowing, and SDI) had mean readability levels ranging from 5.3 to 8.7 with a standard deviation of 1.2. When measured by Gunning Fog, only 2 adult PROs (SDI 8.6 and EEsAI 7.7) were greater than the sixth-grade readability level compared to all pediatric EoE PROs which were at or below the sixth-grade readability level (Supplemental Table 3).

Discussion

While the diagnosis of EoE requires both clinical and histologic features, PROs are research tools needed to fully capture a patient’s illness experience and may span the domains of symptoms, social and emotional functioning, and behavioral adaptations. Nearly 40% of U.S. adults are estimated to have basic (approximately 5^th-8^th grade reading level) or below basic (4^th grade reading level or less) literacy scores.⁴ As such, it is crucial to assess whether EoE PROs meet national health literacy recommendations and thereby accurately capture patient disease experiences and outcomes toward guiding the delivery of patient-centered care.⁴ To assess this, we analyzed published pediatric and adult EoE PROs and found them to have a mean readability level and corresponding grade level of 6.6, suggesting most patients should be able to understand these. However, more than one-third of analyzed adult EoE PROs surpassed mean target readability levels. Providers and clinical trialists of patients with EoE should exercise caution when administering PROs with advanced readability scores.

To our knowledge, the health literacy rating of EoE PROs has not been assessed in prior studies. Overall, all four pediatric EoE PROs met target readability levels and more than one-third of adult EoE PROs surpassed target readability recommendations. The PEESS Children and Teen Report scored a mean readability level of 5.9 which is equivalent to the 6^th grade reading level and may be too difficult for the designated age group (children and teens ages 8-11 are typically in the second to fifth grade). In all, our results demonstrate that pediatric and adult EoE PROs were slightly above the target readability level set by the American Medical Association recommendation.⁵ The SDI, EoE-QoL-A, EEsAI had the highest mean readability and grade level equivalents of 8.7, 8.4, and 7.8 respectively, which was mostly driven by more words per sentence and fewer single syllable words. For example, the SDI on average utilized 10.2 words per sentence compared to 9.7 and 4.2 for the EEsAI and EoE-Qol-A respectively, resulting in the highest Gunning Fog score. Similarly, the EoE-Qol-A shared the highest FORCAST score suggesting its text contained the least amount of single syllable words. However, neither the adult nor the pediatric EoE PROs met recommendations when measured by FORCAST.

When measured by FKGL, all PROs but one met national recommendations, but when measured by SMOG, only 2 adult PROs and one pediatric PRO met the standard. The SDI has the highest SMOG score of all analyzed PROs. The SMOG score calculation relies partially on polysyllabic word count, and the SDI has a greater number of syllables per word on average compared to PROs which met the SMOG readability target.

It is crucial to conduct validity testing of future EoE PROs in more representative populations to capture patient disease experience in a generalizable way. Further discussion on such health literacy disparities and general recommendations to improve the readability of future EoE PROs is found within the supplementary materials.

Strengths of this study include conducting an objective readability analysis of adult and pediatric EoE PROs which span a variety of domains including symptoms, social and emotional functioning, and quality of life, which to our knowledge, has not been previously examined. All EoE PROs included in our study had been previously validated in EoE patients, except the SDI and PiEAQ. The questionnaires analyzed were written in English and we could not account for differences in readability for non-native English speakers. Similarly, the readability analysis did not take into consideration the original formatting or different text elements in the instruments. For PROs such as the DSQ that were initially developed to analyze clinical trial outcomes, participant bias should be considered as some participants who are more willing to participate in trials may have a higher baseline health care literacy or healthcare knowledge. Additionally, since we did not have access to primary patient data from the studies, we could not correlate readability with trial outcomes or patient characteristics like race/ethnicity or income which are associated with health literacy.

In conclusion, our readability analysis of adult and pediatric EoE PROs demonstrated a mean readability level slightly above national recommendations, with many instruments meeting the adult literacy recommendations. Given known health literacy disparities across populations, ensuring the development of future EoE PROs that meet readability criteria is crucial to capturing the disease experience of a range of patients and providing translation of clinical trials outcomes in terms of patient experience. This includes conducting validity testing of future EoE PROs in patients of many different racial and ethnic backgrounds to improve patient-centered care to guide EoE treatment for all patients. Recommendations for improving the readability of future EoE PROs include using shorter sentences, writing for the target audience, and continuing to receive input from age-appropriate patients in the development and review stages of PRO creation.

Abbreviations:

EoE

eosinophilic esophagitis

PROs

patient-reported outcome measures

FKGL

Flesch-Kincaid Grade Level

SMOG

Simple Measure of Gobbledygook

PedsQL EoE module

Pediatric Eosinophilic Esophagitis Quality of Life Module

PEESS

Pediatric Eosinophilic Esophagitis (EoE) Symptom Score Version 2.0

EoE-QoL-A

Adult Eosinophilic Esophagitis Quality of Life Questionnaire

BEDQ

Brief Esophageal Dysphagia Questionnaire

DSQ

Dysphagia Symptom Questionnaire

EoE-IQ

Eosinophilic Esophagitis Impact Questionnaire

EEsAI

Eosinophilic Esophagitis Symptom Activity Index

EoE-SQ

Eosinophilic Esophagitis (EoE) Symptom Questionnaire

PiEAQ

Pisa Eosinophilic Esophagitis Adaptation Questionnaire

SDI

Straumann Dysphagia Instrument