Most patient education materials on shoulder conditions from the American Academy of Orthopaedic Surgeons exceed recommended readability levels

Abstract

Background

A growing number of patients use the internet to learn about their conditions and management options, but there may exist a disconnect between the readability of online education materials and a patient’s health literacy. This issue is of particular relevance for shoulder conditions, where even with traumatic injuries (eg, clavicle fracture, shoulder dislocation), treatment is discretionary, directed primarily at quality of life, and therefore highly preference-sensitive.

The purpose of this study was to utilize multiple readability algorithms to calculate the readability of the American Academy of Orthopaedic Surgeons (AAOS) patient education materials pertaining to diseases and conditions of the shoulder.

Methods

Online patient education articles from the AAOS pertaining to diseases and conditions of the shoulder were reviewed. The articles were modified for analysis using Readability Pro and readability scores were computed using the following 9 algorithms: Flesch-Kincaid Grade Level, Flesch Reading Ease, Gunning Fog Index, Coleman-Liau Index, Simple Measure of the Gobbledygook Index (SMOG), Automated Readability Index, FORCAST, and New Dale and Chall Index. A list of suggested word changes to improve the readability of included articles was compiled from Readable Pro. The average number of illustrations (images and/or videos) included per article was documented.

Results

Twenty-eight articles were included for analysis. For each of the algorithms studied, the average scores were as follows: Flesch Kincaid Grade Level was 8.8 ± .8 [range, 7.2-10.2]; recommended score: ≤ 8.0, Flesch Reading Ease 54.3 ± 5.3 [range, 45.3-64.1]; recommended score: ≥ 60, Gunning Fog 10.8 ± 1.2 [range, 8.3-13.1]; recommended score: ≤ 8.0, Coleman-Liau 11.2 ± .9 [range, 9.2-12.9]; recommended score: ≤8.0, SMOG index 11.4 ± .8 [range, 9.2-12.9]; recommended score: ≤ 8.0 , Automated Readability Index 8.4 ± .8 [range, 6.9-10.0]; recommended score: ≤ 8.0, FORCAST 11.2 ± .4 [range, 10.2-12.0]; recommended score: ≤ 9.0, and New Dale and Chall Index 5.8 ± .5 [range, 4.9-7.2 recommended score: ≤ 6.0-6.9]. The average number of illustrations per article was 4.5 ± 3.1 [range, 1-14].

Conclusion

The readability of most patient education materials from the AAOS pertaining to diseases and conditions of the shoulder is higher than recommended across a variety of algorithms. Efforts to revise the readability of online education materials are important to facilitate shared decision-making, particularly in practice settings where most decisions are preference-sensitive.

The internet has become an increasingly popular source of health information. According to recent data, more than 80% of orthopedic patients utilize the internet to learn about their conditions and treatment options prior to physician consultation.^4,5,10,11 Despite improved access to written education materials, there may exist a disconnect between the readability of these resources and a patient’s health literacy. This is especially problematic as low health literacy has been linked to poor patient outcomes, increased healthcare costs, and overutilization of emergency rooms.^3,6,21,23,28

The ability to understand written text is a critical component of health literacy. A patient’s reading skills are measured in terms of grade level, where “functional illiteracy” is equated to zero- to fifth-grade reading skills and “marginal literacy” is equated to sixth- to eighth-grade reading skills.¹² According to a recent survey study, nearly 50% of the adult US population is functionally or marginally illiterate.^13,14 Moreover, substantial limitations in health literacy have been shown to exist among the elderly, unemployed, and those of lower socioeconomic status.²⁵ In 2010, the Plain Language Initiative was implemented in an attempt to equalize care. This initiative set forth guidelines to ensure agencies use clear communication in order to help their users find, understand, and apply the information to meet their health needs.¹⁸ The guidelines are based upon writing for a specific audience, organization of information, word choice, brevity, and conversational voice.

Numerous agencies including the National Institutes of Health, Centers for Disease Control, and American Medical Association recommend that health materials be written at or below the fourth- to eighth-grade reading level.^8,15,27,29 Despite this, multiple studies have shown that the readability level of online orthopedic education materials remains above these levels.^2,7,20,22,26

The purpose of this study was to utilize multiple readability algorithms to calculate the readability of the American Academy of Orthopaedic Surgeons (AAOS) patient education materials pertaining to diseases and conditions of the shoulder. We hypothesize that the majority of these articles are written above the eighth grade reading level.

Methods

Online patient education articles from the AAOS pertaining to diseases and conditions of the shoulder were reviewed in June 2022.⁹ The article topics included for analysis were:

• Arthritis of the Shoulder

• Biceps Tendinitis

• Biceps Tendon Tear at the Shoulder

• Brachial Plexus Injuries

• Burners and Stingers

• Chronic Shoulder Instability

• Clavicle Fracture (Broken Collarbone)

• Common Shoulder Injuries

• Complex Regional Pain Syndrome (Reflex Sympathetic Dystrophy)

• Dislocated Shoulder

• Erb's Palsy (Brachial Plexus Birth Palsy)

• Frozen Shoulder

• Joint Replacement Infection

• Nerve Injuries

• Rotator Cuff Tears

• Rotator Cuff Tears: Frequently Asked Questions

• Scapula (Shoulder Blade) Fractures

• Scapula (Shoulder Blade) Disorders

• Shoulder Impingement/Rotator Cuff Tendinitis

• Shoulder Injuries in Throwing Athlete

• Shoulder Joint Tear (Glenoid Labrum Tear)

• Shoulder Pain and Common Shoulder Problems

• Shoulder Separation

• Shoulder Trauma (Fractures and Dislocations)

• SLAP Tears

• Sternoclavicular (SC) Joint Disorders

• Thoracic Outlet Syndrome

• Upper Extremity Limb Length Discrepancy

The articles were modified to remove any images, figures, citations, references, copyright notices, disclaimers, or hyperlinks and converted to plain text in Microsoft Word as previously described.^1,22,24 Analysis of the reformatted articles was performed using Readable Pro and the following readability scores were obtained (Table I):

• Flesch Reading Ease

• Flesch Kincaid Grade Level

• Gunning Fog Score

• Coleman Liau Index

• SMOG Index

• Automated Readability Index

• Dale-Chall Readability Score

• FORCAST Grade Level

• Fry Grade Level

Table I.

Individual readability scores for each of the 28 AAOS articles pertaining to disease and conditions of the shoulder.

Article Title	Flesch Reading Ease	Flesch Kincaid Grade Level	Gunning Fog Score	Coleman Liau Index	SMOG Index	Automated Readability Index	Spache Readability Score	Dale-Chall Readability Score	FORCAST Grade Level	Illustration/figures
Arthritis of the Shoulder	49.7	9.4	11.9	12.1	12.2	9.1	5.4	6.1	11.3	5.0
Biceps Tendinitis	49.8	8.9	9.8	11.9	10.7	8.3	5.2	6.2	12.0	7.0
Biceps Tendon Tear at the Shoulder	61.1	7.4	8.3	10.2	9.8	7.0	5.0	5.6	11.2	3.0
Brachial Plexus Injuries	50.0	9.9	12.6	11.9	12.6	9.8	5.8	6.5	11.4	10.0
Burners and Stingers	61.0	7.8	9.5	10.5	10.2	7.7	5.1	5.5	11.2	3.0
Chronic Shoulder Instability	49.2	9.1	10.7	12.3	11.3	8.7	5.1	5.7	11.6	4.0
Clavicle Fracture (Broken Collarbone)	63.0	7.2	9.2	10.0	10.2	6.9	4.8	4.9	10.6	6.0
Common Shoulder Injuries	55.1	8.2	10.0	11.8	10.8	8.2	5.0	5.3	11.4	1.0
Complex Regional Pain Syndrome (Reflex Sympathetic Dystrophy)	48.6	9.1	11.4	12.6	11.6	8.8	5.9	7.2	11.8	2.0
Dislocated Shoulder	56.4	8.0	9.0	11.0	10.0	7.5	4.8	5.2	11.1	1.0
Erb's Palsy (Brachial Plexus Birth Palsy)	62.6	7.8	10.1	9.4	10.7	7.3	5.1	5.5	10.5	4.0
Frozen Shoulder	53.3	8.4	9.8	12.0	10.6	8.4	5.2	5.8	11.8	7.0
Joint Replacement Infection	45.3	10.2	13.1	12.9	12.9	10.0	5.4	6.7	11.3	4.0
Nerve Injuries	64.1	8.0	10.2	9.2	10.7	7.6	5.0	4.9	10.2	2.0
Rotator Cuff Tears	57.5	8.2	10.7	10.5	11.3	7.7	5.1	5.4	10.9	14.0
Rotator Cuff Tears: Freuqently Asked Questions	53.8	9.5	12.4	10.6	12.3	8.7	5.7	5.5	10.7	2.0
Scapula (Shoulder Blade) Fractures	53.8	8.8	11.0	11.3	11.5	8.3	5.4	5.8	11.4	1.0
Scapula (Shoulder Blade) Disorders	51.1	9.1	12.0	11.8	12.2	8.7	5.4	6.3	11.3	5.0
Shoulder Impingement/Rotator Cuff Tendinitis	58.4	7.9	10.2	10.6	11.0	7.4	5.1	5.4	10.8	4.0
Shoulder Injuries in Throwing Athlete	60.4	9.2	11.0	11.3	11.6	9.6	5.6	6.0	11.1	11.0
Shoulder Joint Tear (Glenoid Labrum Tear)	49.3	9.7	11.8	11.4	11.9	8.8	5.6	6.0	11.5	2.0
Shoulder Pain and Common Shoulder Problems	53.5	8.8	10.5	11.7	11.4	8.6	5.3	5.7	11.3	1.0
Shoulder Separation	48.2	10.1	12.5	11.5	12.5	9.2	5.4	6.3	11.1	3.0
Shoulder Trauma (Fractures and Dislocations)	45.4	9.6	10.8	12.7	11.7	9.0	5.5	6.5	11.8	4.0
SLAP Tears	55.0	8.7	10.7	11.2	11.3	8.3	5.3	5.8	11.3	7.0
Sternoclavicular (SC) Joint Disorders	52.8	9.3	11.8	11.3	12.1	8.9	5.4	6.1	10.8	5.0
Thoracic Outlet Syndrome	56.9	8.5	10.5	11.9	11.4	9.0	5.1	5.3	10.9	3.0
Upper Extremity Limb Length Discrepancy	55.7	8.8	11.4	10.7	11.9	8.2	5.3	5.4	10.5	5.0
Mean (SD)	54.3 (5.3)	8.8 (0.8)	10.8 (1.2)	11.3 (0.9)	11.4 (0.8)	8.4 (0.8)	5.3 (0.3)	5.8 (0.5)	11.2 (0.4)	4.5 (3.2)

AAOS, American Academy of Orthopaedic Surgeons; SLAP, superior labrum anterior posterior; SD, standard deviation.

Each of these algorithms have been used extensively for analyzing the readability of patient education materials^2,20,26 and are based on the sample text’s syllables, words, and sentences to varying degrees (Table II). While there is no gold standard readability scoring system, each of these formulas has been shown to strongly correlate and the use of multiple scores is recommended to increase the validity.¹ A list of suggested word changes to improve the readability of included articles was compiled from Readable Pro. The average number of illustrations (images and/or videos) included per article was documented.

Table II.

Formulas used to calculate the readability scores of included articles.

Flesch-Kincaid Grade Level	(0.39 × mean # of syllables per word) + (11.8 × mean # of words per sentence)
Flesch Reading Ease	206.835 − (1.015 × mean # of words per sentence) − (84.6 × mean # of syllables per word)
Gunning Fog Index	$0.4\left(\frac{mean\#ofwords}{mean\#ofsentences}\times +100\times \frac{mean\#ofwords\ge 3syllables}{mean\#ofwords}\right)$
Coleman-Liau Index	$\left(0.0588\times \frac{mean\#ofletters}{word}\right)-\left(0.296\times \frac{mean\#ofsentences}{100words}\right)$
Simple Measure of the Gobbledygook Index	$1.043\times \sqrt{\left(\#ofwordswith\ge 3syllables\right)\times \left(\frac{30}{\#ofsentences}\right)}+3.1291$
Automated Readability Index	$4.71\left(\frac{letters}{words}\right)+0.5\left(\frac{words}{sentences}\right)-21.43$
FORCAST	$20-\left(\frac{\#ofsinglesyllablewordsin150wordsample}{10}\right)$
New Dale and Chall Index	$0.0496\times \left(\frac{mean\#ofwords}{mean\#ofsentences}\right)+0.1579\times \left(\frac{unfamiliarwords}{mean\#ofwords}\right)+3.6365$

Results

Twenty-eight articles were included for analysis. For each of the algorithms studied, the average scores were as follows: Flesch Kincaid Grade Level was 8.8 ± .8 [range, 7.2-10.2]; recommended score: ≤ 8.0, Flesch Reading Ease 54.3 ± 5.3 [range, 45.3-64.1]; recommended score: ≥ 60, Gunning Fog 10.8 ± 1.2 [range, 8.3-13.1]; recommended score: ≤ 8.0, Coleman-Liau 11.2 ± .9 [range, 9.2-12.9]; recommended score: ≤8.0, SMOG index 11.4 ± .8 [range, 9.2-12.9]; recommended score: ≤ 8.0 , Automated Readability Index 8.4 ± .8 [range, 6.9-10.0]; recommended score: ≤ 8.0, FORCAST 11.2 ± .4 [range, 10.2-12.0]; recommended score: ≤ 9.0, and New Dale and Chall Index 5.8 ± .5 [range, 4.9-7.2 recommended score: ≤ 6.0-6.9]. The average number of illustrations per article was 4.5 ± 3.1 [range, 1-14]. Overall, the average FK score was 8.8 ± 0.8 [range, 7.2-10.2], with only 21% [6 of 28] of articles at or below the eighth grade reading level (Fig. 1). A comprehensive list of suggested word changes to improve the readability of these articles is provided in Table III.

Figure 1.

Flesch Kincaid Grade Level for each of the 28 AAOS articles pertaining to disease and conditions of the shoulder. AAOS, American Academy of Orthopaedic Surgeons; SLAP, superior labrum anterior posterior.

Table III.

Suggested word changes to improve the readability of included articles.

Term	Alternative
Abnormalities	Defects
Additionally	Also
Antibiotics	Medications
Arthroplasty	Joint replacement
Arthroscopically	With a small camera
Associated	Related
Capsolabral	Joint
Chlorhexidine	Wash
Colonization	Growth
Complications	Problems
Comprehensive	Complete
Considerations	Tips
Contaminated	Polluted
Corresponding	Related
Corticosteroid	Steroid
Degeneration	Breakdown
Differentiate	Separate
Disadvantages	Downsides
Discoloration	Color changes
Discrepancy	Difference
Dramatically	Greatly
Effectiveness	Power
Electrodiagnostic studies	Nerve tests
Evaluate	Checked
Examination	Check
Familiarity	Experience with
Glenohumeral	Shoulder
Hemiarthroplasty	Partial joint replacement
Immediately	Right away
Immobilization	Casting, splinting
Immobilizer	Cast, sling, splint
Individual	Person, single
Instability	Imbalance
Laboratory	Lab
Miniaturized	Small
Modification	Change
Nonfunctioning	Nonworking
Occasionally	Sometimes
Overexertion	Over working
Particularly	Especially
Progressively	Gradually
Psychological	Mental
Pulmonologist	Lung specialist
Recommendations	Suggestions
Reconstructing	Rebuilding
Regeneration	Regrowth
Rehabilitation	Rehab
Satisfactory	Suitable
Sensitivity	Feeling
Significantly	Seriously
Spontaneously	On its own
Sterilization	Cleaning
Temporarily	Briefly
Underestimate	Misjudge
Visualization	Imaging

Discussion

Numerous agencies including the National Institutes of Health, Centers for Disease Control, and American Medical Association recommend that patient education materials be written at or below the fourth- to eighth-grade reading level.^8,15,27,29 In the present analysis, we found that the current AAOS patient education materials pertaining to disease and conditions of the shoulder are not in accordance with these standards. Overall, the average Flesch Kincaid Grade Level [FK] score was 8.8 ± 0.8 [range, 7.2-10.2], with only 21% [6 of 28] of articles at or below the eighth grade reading level. FK score has been used to reflect the overall readability in multiple studies.^1,2,20,24 Even so, the validity of reading scores is accomplished with the inclusion of multiple algorithms.¹ In our study, the readability of each article was higher than recommended by Readability Pro across each algorithm.¹⁹

Roberts et al previously assessed the change in readability scores of AAOS patient education materials across all subspecialties.²⁰ In 2008, the mean FK grade level was 10.4, which significantly reduced to 9.3 in 2014.²⁰ Although it is difficult to determine if this change represents a practical difference, these data suggest that some improvement in the readability of AAOS patient education resources has occurred.

Few studies have assessed the readability of patient education materials for shoulder conditions. In 2018, the readability of 6 patient education brochures provided by the American Shoulder and Elbow Surgeons was assessed. Topics included arthritis and total shoulder replacement, arthroscopy of the shoulder and elbow, rehabilitation of the shoulder, rotator cuff tendonitis and tears, tennis elbow, and the unstable shoulder. It was found that the brochures were written well above the eighth grade level, ranging in difficulty from a grade level of 13.4 to 15.3.²² More recently, the readability of online patient education materials for shoulder arthroplasty provided by the top 25 orthopedic institutions was assessed. Overall, the mean FK score was 9.5 and only 16% of institutions included online material at or below the eighth grade level.²⁴ The findings from these studies suggest that the readability of patient education materials differs by the source and topic of information. Therefore, while our study implies that the shoulder articles from the AAOS website may be more inclusive to patients [lower average FK score] when compared to the information provided by American Shoulder and Elbow Surgeons brochures and top academic centers, these comparisons should not be made given the differences in topic number and distribution. However, it is important to recognize that the majority of patient educational materials for shoulder conditions provided by these outlets are likely not suitable for the majority of readers in the United States.^8,16 It is therefore prudent to understand the components of the readability scores and means for improvement.

There may be several ways to improve the readability of written patient education materials. Previous studies have suggested that shorter words, using more concise sentence structure, using fewer words per paragraph, and providing more visual material may aid in lowering the readability score.²⁸ The article with the highest FK score (10.2) in our study was related to joint replacement infection. Several issues pertaining to word density and writing style were associated with this article. Specifically, 32% of the sentences contained more than 30 syllables and 52% of sentences contained more than 20 syllables. This represents a significant area for improvement given the fact that highly readable content is often associated with roughly 6% of sentences containing less than 30 syllables and roughly 12% of sentences with less than 20 syllables. Additionally, 47 words used in this article were classified as “hard words.” While the poor readability of this article may be due to the intrinsic complexity of periprosthetic infection, readability may be improved by substituting exhaustive explanations related to anatomic references, procedural steps, and implant design materials with brief descriptions.²⁴ It is important to note that this article did include a total of 3 images. The use of visual supplementary material in the form of pictures and videos has been shown to improve readability and has been cited as a missed opportunity to increase health literacy.^17,24 However, despite an average number of 4.5 illustrations per article, the overall readability score was still higher than recommended for included articles in our study. This may suggest that the complexity of shoulder topics is high and that significant improvements in readability are unable to be achieved with the incorporation of illustrations alone. As such, emphasis should be placed on improving sentence structure and writing style in addition to the incorporation of visual supplementary material.

Our study is not without limitations. The formulas used to generate readability scores are determined based upon syllable and character counts in each word, sentence, and paragraph. Therefore, readability scores may be misleading in instances where short but unfamiliar medical terms are used or in cases where short sentences are presented with complex ideas. For example, although the word “arthroplasty” has the same number of syllables as “joint replacement,” the latter may be easier to understand for the general public. Yet, based on syllable count, both words would contribute equally to the readability score. Additionally, each of the algorithms used is unable to evaluate the effect of supplementary visual aids or reader comprehension on the readability score. Therefore, despite a relatively high average number of illustrations used per article in our study, the readability scores may be inflated. Finally, the reading level of the AAOS target population may be different than that of the general patient population. Therefore, although our findings suggest that the readability of these resources is higher than the national recommendations, they may be relevant to this specific audience of readers.

Conclusion

The readability of most patient education materials from the AAOS pertaining to diseases and conditions of the shoulder is higher than recommended across a variety of algorithms. Efforts to revise the readability of online education materials are important to facilitate shared decision-making, particularly in practice settings where most decisions are preference-sensitive.

Disclaimers:

Funding: No funding was disclosed by the authors.

Conflicts of interest: The authors, their immediate families, and any research foundation with which they are affiliated have not received any financial payments or other benefits from any commercial entity related to the subject of this article.