Readability of Epic's Obstetric Discharge Education

Abstract

Introduction

In obstetrics, patients with insufficient health literacy have a higher likelihood of adverse perinatal outcomes. The American Medical Association (AMA) recommends all patient‐facing health information be written at or below a sixth‐grade reading level. The reading level of standardized obstetric patient education materials embedded into Epic via “Patient Pass/Exit Care” package published by Elsevier is unknown. Since the Spanish versions of the materials have not been back‐translated, the readability of those materials also requires further assessment. In this study, our aims were to (1) characterize overall readability of Epic's obstetric patient education documents, (2) assess differences in readability between English and Spanish materials, and (3) understand differences in readability by subject material, stratified by language.

Methods

Using the “Clinical References” tab in Epic, obstetric patient education materials were extracted in both English and Spanish. Materials were evaluated using metrics for assessing readability that have been validated in English and/or Spanish: Flesch–Kincaid Grade Level (FKGL), Fernandez‐Heurta (FH), and Simple Measure of Gobbledygook (SMOG). Education was coded by content category. Paired t‐tests were conducted to evaluate differences in readability scores (FKGL vs. FH) and SMOG scores across language pairs. Mean differences (95% confidence intervals [CIs]) and one‐way ANOVA with Bonferroni‐adjusted post hoc tests assessed language readability gaps across clinical categories.

Results

Of 235 total texts analyzed, grade level for English was 6.2 (SD 1.0) using the FKGL metric versus 4.2 (SD 0.5) for Spanish content that used the validated FH. Overall, the English documents were 2.0 grade levels higher than the corresponding Spanish documents (95% CI, 1.8–2.1). SMOG mean grade level for English materials was 9.7 (SD 0.9) versus 6.2 (SD 0.6) for Spanish materials (p < 0.01). Across all perinatal topics, English texts scored significantly higher than Spanish texts on both SMOG and language‐specific readability metrics. A total of 41.2% of Spanish materials were at or below a 6th grade level using SMOG, but none of the English materials were.

Conclusion

Existing obstetric patient education materials embedded in the Epic electronic medical record (EMR) do not meet AMA standards for readability and comprehension. Differences between English and Spanish materials highlight a potential for disparities in patient understanding of their plans of care.

1. INTRODUCTION

In obstetrics, patients with poor health literacy have a higher likelihood of adverse maternal and neonatal outcomes: unplanned cesarean delivery, severe perineal tears, low birth weight, and low Apgar score [1]. Effective patient education materials provide an opportunity to bridge this gap. The American Medical Association (AMA) recommends all patient‐facing health information be written at or below a sixth‐grade reading level for accessibility. However, existing obstetric education materials fall short. An analysis of patient education materials by the American College of Obstetricians and Gynecologists (ACOG) found that all 134 ACOG English pamphlets assessed exceeded the recommended reading level [2]. Another study showed that across the eight major obstetric and gynecologic societies, patient education materials were all above the AMA‐recommended reading level [3].

“Epic”—the leading EMR globally—is used by 89% of acute care hospitals in the United States [4] and has discharge education published by Elsevier (Elsevier Inc) embedded into provider workflow via the “Clinical References” tab. These educational materials can be selected to both print and electronically share with the patient via the “After Visit” summary.

Despite the ubiquity of Epic, it is not known whether Epic's obstetric patient education meets AMA readability recommendations and whether differences in readability and comprehension exist by language. Furthermore, it is not known whether the Spanish versions of these materials available in Epic have been back‐translated, making it more crucial that their readability and comprehension be assessed alongside the English versions. In this study, our aims were to (1) characterize overall readability of these obstetric patient education documents, (2) assess differences in readability between English and Spanish materials, and (3) understand differences in readability by subject material, stratified by language.

2. MATERIALS AND METHODS

2.1. Study design

In this study, the patient education materials assessed were extracted in retrospective review in December 2024 using the Epic electronic medical record (EMR). These documents are published by Elsevier (Elsevier Inc) as part of “Patient Pass/Exit Care” packages and are made available to providers via Epic. They are standardized and were not edited by the members of the authors’ institution. To integrate discharge instructions into Epic, providers can navigate to “Clinical References/Exit Care” under the “Clinical References” tab to directly input these standardized discharge instructions into the patient discharge materials. These documents are provided in multiple languages. In this study, only English and Spanish materials were assessed.

The study was determined to be exempt from Institutional Review Board (IRB) review, as it did not involve access to protected health information (PHI).

2.2. Original data extraction

All “Patient Pass/Exit Care” data were extracted from Epic across specialties in December 2024. This analysis of obstetric content was a planned sub‐analysis from a broader retrospective review of Epic discharge materials in December 2024 by one of the authors [5]. Eligible documents (obstetric discharge information, written in English or Spanish) were extracted. Text from these files was migrated into individual plain‐text Microsoft Word files (2024, Version 16.68).

During data cleaning, non‐educational elements like copyright notices, timestamps, URLs, citations, and tables were purged. The documents underwent individual reformatting to improve readability assessment, removing bullet points, abbreviations, and paragraph breaks that may lead to underestimation of readability. Notably the content was not edited by this institution.

2.3. Further data categorization

Two researchers (K.U. and M.T.) reviewed all documents categorized by the package as “OB/GYN” content and then manually identified those documents that were “Obstetric,” removing gynecologic content from the analysis. A Spanish‐speaking researcher on the team (J.R.) read through all documents to ensure concordance between English and Spanish education documents’ titles and basic content. Those that were grossly incongruent between languages (e.g., a different topic entirely) were removed from the analysis.

The documents were then further manually sub‐categorized by content (by two researchers—K.U. and M.T.—who reached consensus) into the following categories:

1. Pregnancy, Prenatal Care & Maternal Health

2. Labor, Delivery & Pregnancy Loss Management

3. Postpartum Recovery, Infant Feeding & Newborn Care

Please see the Supporting Information for titles of each document and their respective categorizations.

2.4. Readability measurements

English documents were scored for readability using Flesch–Kincaid Grade Level (FKGL) (Equation 1) [6], which is validated in that language. Spanish documents were scored using the Fernandez‐Huerta (FH) Readability Index (Equation 2) [7]. Both the FKGL and FH are based on the same Flesch Reading Ease Formula and are considered comparable when applied to their respective languages.

While readability formulas provide quantitative estimates of text complexity, each metric relies on distinct factors to evaluate reading level and comprehension. Both the FKGL for English materials and the FH Index for Spanish materials focus primarily on sentence length as the central determinant of readability. However, because both formulas rely so heavily on sentence structure, neither metric guarantees actual patient comprehension of the text.

Therefore, we also applied the Simple Measure of Gobbledygook (SMOG) metric, which reflects an estimated grade level at which a reader is expected to fully comprehend the material to both English and Spanish materials [8]. SMOG is validated in both languages and supported in the literature [9].

Open‐source Python libraries (Python Software Foundation) were used to perform readability calculations. Each text was analyzed using the following readability metrics: FKGL for English texts and FH for Spanish texts, and SMOG for both languages.

The FKGL formula is as follows:

$$FKGL=0.39ASL+11.80\times ASW-15.59$$ (1)

where ASL is the average sentence length (words per sentence) and ASW is the average syllables per word.

The FH formula is as follows:

$$\text{Fern}\acute{\mathrm{a}}\text{ndez}-\text{Huerta}=206.84-60\frac{S}{W}-1.02\frac{W}{P}$$ (2)

where S is the total number of syllables, W is the total number of words, and P is the total number of sentences.

The SMOG formula is as follows:

$$SMOG=1.04\times \sqrt{30\frac{P}{N}}+3.13$$ (3)

where P is the number of polysyllabic words (3+ syllables) in the sample and N  is the number of sentences in the sample.

2.5. Statistical analysis

Because the English and Spanish documents represent paired versions of identical content, paired t‐tests were conducted to evaluate differences in readability scores (FKGL vs. FH) and SMOG scores across language pairs. To assess the magnitude of the readability gap between languages, mean differences with 95% confidence intervals (CIs) were calculated alongside p values. To compare readability scores and language gaps across clinical care categories, a one‐way analysis of variance (ANOVA) was performed. Post hoc pairwise comparisons between strata were evaluated using Bonferroni‐adjusted p values to account for multiple comparisons.

Additionally, for each category and each readability metric, the percentage of documents that were at or below a sixth‐grade reading level was calculated.

3. RESULTS

A total of 238 obstetric patient education materials were reviewed. Of these, three texts were excluded due to content discordance between the English and Spanish versions, leaving 235 documents for analysis.

Grade level for English was 6.2 (SD 1.0) using the FKGL metric versus 4.2 (SD 0.5) for Spanish content that used the validated FH. Overall, the English documents were 2.0 grade levels higher than the corresponding Spanish documents (95% CI, 1.8–2.1); see Table 1.

TABLE 1.

Readability grade level of Epic obstetric patient education materials by language and content topic.

Readability metric and content topic	English mean (SD)	Spanish mean (SD)	p value	Mean paired difference (95% confidence interval)
Language‐specific metric a
Overall	6.2 (1.0)	4.2 (0.5)	<0.01	2.0 (1.8, 2.1)
Pregnancy, Prenatal Care & Maternal Health	6.3 (1.0)	4.3 (0.4)	<0.01	2.0 (1.8, 2.2)
Labor, Delivery & Pregnancy Loss Management	6.3 (1.1)	4.2 (0.4)	<0.01	2.1 (1.9, 2.4)
Postpartum Recovery, Infant Feeding & Newborn Care	5.9 (0.9)	4.2 (0.5)	<0.01	1.7 (1.5, 1.9)
SMOG comprehension
Overall	9.7 (0.9)	6.2 (0.6)	<0.01	3.5 (3.4, 3.6)
Pregnancy, Prenatal Care & Maternal Health	9.8 (0.9)	6.2 (0.5)	<0.01	3.6 (3.4, 3.7)
Labor, Delivery & Pregnancy Loss Management	9.9 (0.9)	6.1 (0.6)	<0.01	3.8 (3.6, 4.0)
Postpartum Recovery, Infant Feeding & Newborn Care	9.4 (0.8)	6.2 (0.6)	<0.01	3.2 (3.1, 3.4)

Abbreviation: SMOG, Simple Measure of Gobbledygook.

^a Evaluated using Flesch–Kincaid Grade Level for English materials and Fernández‐Huerta for Spanish materials.

SMOG, which assesses comprehension, had a mean grade level for English materials of 9.7 (SD 0.9) versus 6.2 (SD 0.6) for Spanish materials (p < 0.01). Across all perinatal content categories, there were significant differences between English and Spanish materials in both language‐specific readability metrics and SMOG comprehension; for all categories, English materials were at a significantly higher reading level (see Table 1).

A total of 41.8% of English documents were at or below sixth‐grade reading level using the FKGL, whereas 100% of Spanish documents, when assessed by Fernandez‐Huerta, were at or below the sixth‐grade reading level. Of note, using SMOG comprehension, 41.2% of Spanish materials were at or below a sixth‐grade level, but none of the English materials were.

In comparing the gap between English and Spanish SMOG comprehension by perinatal content category, there was no significant difference in the difference between English and Spanish readability when Pregnancy, Prenatal Care & Maternal Health education is compared to Labor & Delivery education. However, when compared to Pregnancy, Prenatal Care & Maternal Health education and Labor & Delivery education, Postpartum Recovery, Infant Care education has a smaller language gap between English and Spanish, corresponding to a 0.34 grade reduction and 0.59 grade reduction, respectively, between the languages (see Table 2).

TABLE 2.

Comparison of English–Spanish SMOG comprehension gap across subject.

Group comparison a	Mean difference in language gap (grade levels)	p value b
Pregnancy versus Labor & Delivery	+0.25	0.059
Pregnancy versus Postpartum	−0.34	0.005
Labor & Delivery versus Postpartum	−0.59	<0.001

Abbreviation: SMOG, Simple Measure of Gobbledygook.

^a Pregnancy = Pregnancy, Prenatal Care, & Mental Health; Labor & Delivery = Labor, Delivery, & Pregnancy Loss; Postpartum = Postpartum Recovery, Infant Feeding, & Newborn Care.

^b Bonferroni‐adjusted.

4. DISCUSSION

In this analysis of readability of obstetric patient education materials within Epic, significant differences were found between English and Spanish materials across content categories in both language‐specific metrics of readability and SMOG comprehension scores. Evaluating English materials using only the traditional FKGL can create a misleading impression of target readability levels, as the mean readability across documents hovered around a sixth‐grade standard. However, none of the English education materials meet AMA recommendations for grade‐level readability using the SMOG. Spanish documents were at a significantly lower grade level across readability metrics. While a language gap exists across topics, the gap narrows in postpartum materials because the English content in that category was written at a lower reading level. This aligns it more closely with the Spanish content, which maintains a uniform grade level across all subjects.

Though more of the Spanish materials met AMA readability standards in comparison to the English versions, this could reflect a trade‐off in depth of content of completeness of the education. If Spanish materials are more readable, but oversimplify key messages, this could paradoxically contribute to disparities in patient understanding. Crucial next steps include examining whether differences in standardized discharge instruction readability and comprehension scores by language contribute to disparities in patient understanding of perinatal education and clinical outcomes.

Strengths of this study include using both traditional language‐specific metrics for readability (FKGL and FH) and SMOG comprehension. Another strength of this study is that the source used was a standardized data set. These findings are clinically applicable due to the widespread use of Epic and specifically for those institutions using “Patient Pass/Exit Care” by Elsevier.

Limitations include that it is unclear the manner in which these documents were originally translated, and whether they have been back‐translated. Additionally, despite the ubiquity of Epic, it is unknown how frequently clinicians distribute standard Epic discharge forms instead of custom, practice‐specific documents. Another limitation is that beyond document readability, there are other individual factors that may impact a patient's comprehension of their visit beyond the discharge documentation such as time constraints.

The ACOG emphasizes that health literacy is a critical driver of patient outcomes, noting that nearly half of all Americans struggle to understand complex medical information, putting them at greater risk for hospitalization and poorer overall health [10]. To combat these barriers, ACOG recommends creating written patient materials that are simple, actionable, and easy to read. To avoid overwhelming patients, providers should limit content to no more than four key messages that use active voice and direct, behavioral instructions rather than abstract medical principles. Additionally, materials should rely on everyday language rather than complex jargon, utilize at least a 12‐point font with plenty of white space for better readability, and incorporate culturally relevant visual aids to emphasize essential points [10].

As prior studies similarly show obstetric patient education materials exceeding readability recommendations [2, 3], next steps should also include identification, adaptation, or creation of multilingual obstetric discharge instructions that meet ACOG and AMA recommendations.

5. CONCLUSION

Existing obstetric patient education materials including those embedded in the Epic EMR do not meet AMA standards for readability and comprehension. Key differences between English and Spanish materials highlight a potential for disparities in patient understanding of their plans of care. Further research is needed to understand how these differences impact patient outcomes.

CONFLICT OF INTEREST STATEMENT

Dr. Lewkowitz was a consultant for Sonio AI in 2024. This is unrelated to this project. All authors declare no conflicts of interest.

FUNDING INFORMATION

The authors received no specific funding for this work.

Supporting information

Supporting information