Readability assessment of patient educational materials for pediatric spinal conditions from top academic orthopedic institutions

Abstract

Background:

The Internet has become a popular source of health information for patients and their families. Healthcare experts recommend that the readability of online education materials be at or below a sixth grade reading level. This translates to a standardized Flesch Reading Ease Score between 81 and 90, which is equivalent to conversational English. However, previous studies have demonstrated that the readability of online education materials of various orthopedic topics is too advanced for the average patient. To date, the readability of online education materials for pediatric spinal conditions has not been analyzed. The objective of this study was to assess the readability of online educational materials of top pediatric orthopedic hospital websites for pediatric spinal conditions.

Methods:

Online patient education materials from the top 25 pediatric orthopedic institutions, as ranked by the U.S. News and World Report hospitals for pediatric orthopedics, were assessed utilizing multiple readability assessment metrics including Flesch–Kincaid, Flesch Reading Ease, Gunning Fog Index, and others. Correlations between academic institutional ranking, geographic location, and the use of concomitant multimedia modalities with Flesch–Kincaid scores were evaluated using a Spearman regression.

Results:

Only 32% (8 of 25) of top pediatric orthopedic hospitals provided online health information at or below a sixth grade reading level. The mean Flesch–Kincaid score was 9.3 ± 2.5, Flesch Reading Ease 48.3 ± 16.2, Gunning Fog Score 10.7 ± 3.0, Coleman–Liau Index 12.1 ± 2.8, Simple Measure of the Gobbledygook Index 11.7 ± 2.1, Automated Readability Index 9.0 ± 2.7, FORCAST 11.3 ± 1.2, and Dale–Chall Readability Index 6.7 ± 1.4. There was no significant correlation between institutional ranking, geographic location, or use of video material with Flesch–Kincaid scores (p = 0.1042, p = 0.7776, p = 0.3275, respectively).

Conclusion:

Online educational material for pediatric spinal conditions from top pediatric orthopedic institutional websites is associated with excessively complex language which may limit comprehension for the majority of the US population.

Type of study/Level of evidence:

Economic and Decision Analysis/level III.

Introduction

The Internet has become a widely used resource for patients seeking orthopedic information. Over 8 million Americans use the Internet each day to learn about their health conditions and treatment options.^1–6 Parents, in particular, rate online health information as one of their most valuable tools, with an estimated 98% having used the Internet to search for information regarding the health of their child. ⁷ However, the use of medical terminology and complex writing styles may limit comprehension of online educational materials among patients and parents of different educational backgrounds. As past studies have demonstrated, there is a dramatic increase in consumption of online health information by the general public. ⁸ Recent recommendations from the National Institutes of Health (NIH) and National Academy of Medicine (NAM) recommend that patient educational materials should be written at the sixth grade reading level or below.^1,2,9–13

Readability is a numerical value that is determined by systematic formulas, reflecting the grade level of reading necessary to understand the information.^14,15 The Flesch–Kincaid (FK) reading level is a popular readability score that was created by the US military and has been validated in previous studies.^16,17 Other reading scores that emphasize different metrics compared to FK include the Coleman–Liau Index, New Dale–Chall Readability Formula, FORCAST Readability Formula, Gunning Fog Index, Simple Measure of the Gobbledygook (SMOG) Index, and Automated Readability Index. The formulas for these scores are found in Table 1. Investigators have studied the readability of patient education materials across various orthopedic subspecialties including adult reconstruction, ¹⁴ foot and ankle, ¹⁵ shoulder and elbow,^12,13 spine, ³ hand, ⁴ arthroscopy, ¹⁸ sports medicine,^11,19 pediatric orthopedics, and the American academy of orthopedics website itself. ¹ However, to our knowledge, no study has evaluated the readability of online patient education materials regarding pediatric spine conditions and procedures.

Table 1.

Formulas for readability metrics.

Readability assessment	Formula
Flesch–Kincaid	$(0.39\times mean\#ofsyllablesperword)+(11.8\times mean\#ofwordspersentence)$
Flesch Reading Ease	$206.835-(1.015\times mean\#ofwordspersentence)-(84.6\times mean\#ofsyllablesperword)$
Gunning Fog Index	$0.4\times \left(\frac{mean\#ofwords}{mean\#ofsentences}+100\times \left(\frac{mean\#ofwordswith\ge 3syllables}{mean\#ofwords}\right)\right)$
Coleman–Liau Index	$\left(0.0588\times \frac{mean\#ofletters}{word}-\left(0.296\times \frac{mean\#ofsentences}{100words}\right)\right)$
Simple Measure of the Gobbledygook (SMOG) Index	$1.043\times \sqrt{(\#ofwordswith\ge 3syllables\times \left(\frac{30}{\#ofsentences}\right)}+3.1291$
Automated Readability Index	$4.71\left(\frac{letters}{words}\right)+0.5\left(\frac{words}{sentences}\right)-21.43$
FORCAST	$20-\left(\frac{\#ofsinglesyllablewordsin150wordsample}{10}\right)$
New Dale and Chall Index	$0.0496\times \left(\frac{mean\#ofwords}{mean\#ofsentences}\right)+0.1579\times \left(\frac{unfamiliarwords}{mean\#ofwords}\right)+3.6365$

The purpose of this study was to assess the readability of patient education materials related to pediatric spine conditions available from leading pediatric orthopedic centers. We hypothesized that, on average, pediatric spine–related patient education materials from the top children’s hospitals for orthopedic surgery would be written at greater than the sixth grade reading level.

Materials and methods

In December 2021, we searched for spine-related patient education materials from the leading pediatric orthopedic institutions based on the U.S. News and World Report rankings for pediatric orthopedic surgery. ²⁰ We searched each institutions website for patient information and assessed all webpages pertinent to spine. In centers that had a specific spine section of patient education, all articles were included. On those websites that did not have a specific spine section, all articles were screened for their relevance to the spine and spine pathology by one of the senior authors (C.M.). The patient education resources were then converted into text-only format to exclude figures, disclaimers, acknowledgements, citations, references, and hyperlinks. Reformatted patient education files were then analyzed using ReadablePro 20201 (Readable, Added Bytes Ltd.; Horsham, UK). In addition, the authors screened webpages for figures, illustrations, and videos. Although these are not included in calculated readability scores, it was hypothesized that the inclusion of these multimedia may influence word selection in the webpage and affects its readability.

Statistical analysis

Using this software, the following readability scores were calculated: FK, Flesch Reading Ease, Gunning Fog Index, Coleman–Liau Index, SMOG Index, Automated Readability Index, FORCAST, and the New Dale–Chall Readability. Equations used to calculate these scores are listed in Table 1. All of the aforementioned scores, with the exception of the Flesch Reading Ease, provide a score which correlates with the grade reading level associated with the article (e.g. score of 7 equates to seventh grade reading level). A linear regression analysis was employed to generate variance inflation factors, with values ≥10 indicating collinearity between various readability scores. ²¹

Continuous variables were presented as mean values and standard deviations. Correlations between institutional ranking and FK scores were assessed using a Spearman regression. Additional factors including geographic location (urban vs rural), private versus public institution, and use of concomitant multimedia modalities (pictures or videos present on institutions website vs no media) that may impact institutional readability scores (as determined by FK) were analyzed with independent t-test and Mann–Whitney tests for parametric and non-parametric continuous variables, respectively. All tests were two-sided. Analyses were performed with RStudio 2021.09.1 (RStudio, Boston, MA, USA).

Results

In total, all 25 of the top 25 pediatric orthopedic institutions listed on the U.S. News and World Report’s website contained online resources for pediatric spinal conditions. We included 155 web pages in our final analysis. Readability scores were calculated for all web pages that were found in our search. A wide spectrum of FK scores was observed, ranging from 3.4 to 16.02. Notably, only 8 of the top 25 (32%) institutions for pediatric orthopedics as listed by U.S. News and World Report contained patient information at or below a sixth grade reading level. Overall, the mean composite scores were FK score was 9.3 ± 2.5, Flesch Reading Ease 48.3 ± 16.2, Gunning Fog Score 10.7 ± 3.0, Coleman–Liau Index 12.1 ± 2.8, SMOG Index 11.7 ± 2.1, Automated Readability Index 9.0 ± 2.7, FORCAST 11.3 ± 1.2, and Dale–Chall Readability Index 6.7 ± 1.4. Average readability scores for each institution can be found in Table 2, sorted by U.S. News and World Report ranking.

Table 2.

Readability scores for online patient resources regarding pediatric spinal pathology.

Hospital rank	Flesch–Kincaid Grade Level	Flesch Reading Ease	Gunning Fog Score	Coleman–Liau Index	SMOG Index	Automated Readability Index	FORCAST Grade Level	Dale–Chall Readability Score
1	10.8	46.2	13.3	13.6	13.1	10.7	11.1	6.8
2	8.5	51.5	9.8	11.2	10.9	7.9	11.4	6.6
3	14.4	24.9	17.1	15.9	16.1	14.5	12.0	7.7
4	9.1	50.0	11.4	13.5	11.7	9.7	11.5	6.9
5	8.0	56.0	9.1	10.8	10.5	7.5	10.9	6.3
6	8.1	53.1	8.6	11.5	10.8	7.7	11.2	6.5
7	7.5	59.4	9.2	9.8	10.4	6.8	10.8	5.8
8	9.5	42.3	8.5	10.6	10.8	8.7	12.0	7.1
9	7.0	62.5	9.1	10.0	10.3	6.6	10.4	5.4
10	9.2	50.2	11.2	12.3	11.8	9.1	11.0	6.7
11	9.0	48.1	10.4	11.9	11.6	8.4	11.4	7.0
12	7.9	50.1	7.8	10.7	9.9	6.8	11.9	7.5
13	6.9	56.2	5.7	9.7	9.1	6.1	11.5	6.0
14	8.4	55.6	10.4	11.1	11.3	8.0	10.6	5.9
15	8.9	51.9	10.5	11.6	11.4	8.5	11.0	6.2
16	11.6	35.0	12.7	13.9	13.5	10.8	11.8	7.2
17	13.3	15.5	9.8	17.5	13.3	12.4	13.8	9.2
18	7.3	63.2	8.7	10.2	10.3	7.3	10.3	5.4
19	11.0	43.0	12.7	13.3	14.1	12.5	11.7	7.0
20	10.5	42.0	13.1	13.8	12.9	10.7	11.6	7.1
21	8.9	52.5	11.5	11.2	11.9	8.2	10.8	6.1
22	12.6	25.7	12.4	15.4	14.1	11.5	13.1	9.4
23	15.6	10.6	14.3	18.5	15.9	15.7	12.9	8.4
24	13.4	30.4	15.8	15.0	15.0	13.4	12.3	8.2
25	8.7	44.5	8.9	11.8	9.6	8.5	11.5	6.8

When assessing multicollinearity, it was determined that all demonstrated a high rate of collinearity with FK scores (variance inflation factor for each score: Flesch Reading Ease = 23.6, Gunning Fog Index = 10.5, Coleman–Liau Index = 4.2, SMOG Index = 17.5, Automated Readability Index = 16.1, FORCAST = 18.8, and Dale–Chall Readability Score = 12.2). As a result, FK scores were used to analyze the relationship between readability and other institutional factors including ranking, geographic location, and presence of figures or videos.

There was no correlation between this ranking and FK Scores (ρ = 0.131, p = 0.1042). There was no significant relationship found for institutional online resource readability based on FK scores and geographic location (ρ = −0.02, p = 0.78) or use of videos (ρ = −0.08, p = 0.3275). There was a statistically significant relationship between FK scores and the presence of illustrations or figures (ρ = −0.2, p = 0.01417).

Forty-four webpages of the 155 (28.4%) included in the study contained pictures or illustrations and 7 of 155 included videos (4.5%). Fourteen of the top 25 U.S. News and World Report ranked institutions featured pictures or illustrations while only 5 included videos in their patient education materials for pediatric spinal conditions. Furthermore, 14 of the 25 institutions had dedicated spine centers associated that were not just programs or divisions of their neurosurgical or orthopedic departments. The average FK scores of the institutions with dedicated spine centers are 10, while the average FK score of the institutions that are not associated with centers is 9.65. These values are not statically significant (p = 0.73).

Discussion

The Internet has become a vital part of daily life, and its role in disseminating health care information to patients is expanding at a rapid pace. ¹⁴ With this expansion, it is important to ensure patient education is not only accurate but also at an appropriate reading level. The US adult population is composed of 5% who are illiterate in English, 14% who have below-basic literacy skills, 29% with basic literacy skills, and the remaining 52% rated at intermediate or proficient literacy based on the US Department of Education’s literacy scale.^22,23 A similar distribution was found concerning health literacy, with only 22% of patients demonstrating basic health literacy. ²⁴ Health literacy has been an ongoing topic of discussion in the orthopedic community, with previous analyses showing that many materials including those from the American Academy of Orthopaedic Surgeons (AAOS) itself were written at reading levels too complex for most patients to understand. ¹ The goal of this study was to assess the readability of parent-facing educational materials on pediatric spinal conditions from top pediatric orthopedic institutions.

The number of institutions featuring educational materials regarding pediatric spinal conditions at a suitable reading level was 8 out of 25 (32%). Eight different readability scores were employed to provide a complete and unbiased assessment of the individual webpages. Each of the formulated composite scores and metrics are influenced by different aspects of the written text. For instance, FK grades are more significantly influenced by the number of words and syllables, while the Automated Readability Index is more influenced by the ratio of letters to words. By employing a multitude of readability assessments, the authors were able to complete a more balanced evaluation of readability. By assessing for multicollinearity, we were able to demonstrate that all the scores assessed are highly correlated with FK scores. This demonstrates that the assessment of readability is not excessively influenced by a single outlying metric, and thus more representative of the true readability score.

In addition, the average FK reading level of the web pages included in this study was 9.3. Of the 155 web pages included, only 16 (10.3%) were at or below a sixth grade reading level. We found a high degree of collinearity in all readability scores, indicating there would be little difference when assessing different scores compared to the FK scores that were used. The U.S. News and World Report ranking of these institutions did not have any association with the readability of patient educational materials based on our statistical analysis (ρ = 0.131, p = 0.1042). These findings are demonstrated in Figure 1. Geographic location and use of concomitant videos were also not associated with a statistically significant change in readability levels. Interestingly, there was a statistically significant association between the presence of illustrations or figures and readability scores (ρ = −0.2, p = 0.01417), indicating that the presence of figures is associated with a lower readability score (more complex language). In addition, a lack in the inclusion of multimedia in patient educational materials was noted with only 44/155 webpages (28%) including figures or illustrations and 7/155 (5%) including videos. This represents only 14 of 25 and 5 of 25 institutions that included figures and videos, respectively.

Figure 1.

Flesch–Kincaid grade level readability scores for online patient resources for pediatric spinal pathology relative to average US reading level.

Our findings were in concordance with previous studies that assessed the readability of orthopedic patient educational materials.^{2,10,12,14,18} These studies concluded that the majority of orthopedic patient educational materials are at or beyond a sixth grade reading level and have the propensity to confuse both patients and their families. In addition, the only study able to establish an association between institution rank and readability was the study by Parsa et al. ¹⁹ which assessed the readability of hip preservation–related educational materials. This study concluded a weakly negative association between institution rank and readability scores. Institutional-related factors were not found to be statistically significant.

The lack of institutions including multimedia may also be a detriment to patient understanding and comprehension. Videos have been shown in the past to be high quality in content and enhance patient understanding. ¹⁹ In the setting of a large differential between readability scores on these web pages and reading level, a video or picture represents a simple way to supplement understanding of pediatric spinal conditions and may be helpful to increase the readability of orthopedic information.

Some other ways to improve readability of patient resources include choosing words with a single definition, using familiar words, avoiding unnecessary abbreviations or acronyms, and shorter words with decreased complexity.^11,25 The use of medical jargon and the description of specific anatomical or procedural details can decrease readability. Medical jargon is often described as shorthand language used to ease communication among medical professionals. In addition, medical jargon can also be described as the language or words that are unfamiliar to people who are not healthcare or healthcare-associated workers. ²⁶ The authors screened all the webpages for words or phrases that fit into this context of medical jargon. An overall consensus was reached among the authors for what constitutes medical jargon in each of the webpages. We found the articles with the lowest readability scores overused medical jargon and long complex sentences. Some terms, that were often encountered during the authors’ assessment of readability scores, that may be used in place of medical jargon are shown in Table 3.

Table 3.

Identification of commonly used difficult terms related to pediatric spinal pathology.

	Term	Alternative
1	Ability	Skill
2	Additional	Added, extra
3	adjacent	Next to
4	Aggressive	Forward, strong, attacking
5	alteration	change
6	Anesthetic	Pain reducing
7	Anterior	front
8	Appear	Seem, come
9	Articular	Joint surface
10	Artificial	manmade
11	Avascular	Lack of blood supply
12	Benefit	Help
13	Coalition	Joining, union
14	compress	squeeze
15	Congenital	Inborn
16	Contain	Have, hold
17	Continue	Keep, keep on
18	Create	Make
19	debilitating	weakening
20	Debridement	Joint cleaning
21	deformity	abnormality
22	Dense	Thick
23	Determine	Decide, figure
24	Develop	Make, grow
25	Difficult	Hard
26	Difficulty	Trouble
27	Ensure	Make sure
28	Evaluate	Check, rate
29	examination	check
30	Examine/examination	Check
31	External	Outer
32	external	outer
33	fracture	break
34	Frequently	Often
35	Function	Act, role
36	Identify	Name, find
37	In many cases	Mostly, most of these, often
38	In some cases	At times, sometimes
39	incorporating	joining
40	Initial	First
41	Internal	Inner, inside
42	internal	inner
43	Known as	Called, named
44	Locate	Find
45	Location	Place
46	Maintain	Keep, support
47	Monitor	Check, watch
48	Multiple	Many
49	Necessary	Needed
50	opportunities	chances
51	Option	Choice, way
52	Osteonecrosis	Dead bone
53	Participate	Take part
54	Perform	Do
55	Portion	Part
56	Position	Place
57	Primary	Main, first
58	Procedure	Rule, way, method, treatment, operation
59	Program	Plan
60	Rapid	Quick
61	Recommend	Suggest
62	Reduce	Cut
63	rehabilitate	restore
64	Remain	Stay
65	Require	Need
66	Result in	Lead to
67	Similar	Like
68	spine	back
69	subsequently	after
70	surgery	operation
71	Typically	Often
72	Usually	Often

Limitations

There are several factors that limited this study. While the readability of patient educational materials does not entirely indicate the quality of patient educational materials, it may increase patient’s ability to comprehend medical information. In addition, while there was a high degree of concordance between the readability metrics that were used, they were not entirely in agreement. There is no clear winner in terms of the best readability metric to use, and as a result, we used FK scores as that is what other papers have used in the past to assess the readability of patient educational materials.

Conclusion

This study demonstrates the readability of educational materials meant for pediatric patients or parents provided by the nation’s top pediatric orthopedic institutions often exceeds the reading level recommended by the NIH and NAM. It is concerning that so few leading institutions have resources at a sixth grade reading level or below. We recommend avoiding the use of medical jargon and removing detailed explanations regarding procedures from these web pages to improve patient understanding. This will ensure a higher degree of comprehension and help to tailor patient expectations and ultimately improve outcomes. ²⁷