Health literacy in rotator cuff repair: a quantitative assessment of the understandability of online patient education material

Abstract

Background

The American Medical Association and National Institutes of Health recommend online health information be written at a 6th grade or lower reading level for clear understanding. While syntax reading grade level has previously been utilized, those analyses do not determine whether readers are processing key information (understandability) or identifying available actions to take (actionability). The Patient Education Materials Assessment Tool (PEMAT-P) is a method to measure the understandability and actionability of online patient education materials. The purpose of this study was to evaluate online resources regarding rotator cuff repair utilizing measures of readability, understandability, and actionability.

Methods

The search term “rotator cuff surgery” was used in two independent online searches to obtain the top 50 search results. The readability of included resources was quantified using valid objective algorithms: Flesch-Kincaid Grade-Level, Simple Measure of Gobbledygook grade, Coleman-Liau Index, and Gunning Fog Index. The PEMAT-P form was used to assess actionability and understandability.

Results

A total of 49 unique websites were identified to meet our inclusion criteria and were included in our analysis. The mean Flesch-Kincaid Grade Level graded materials at a 10.6 (approximately a 10th grade reading level), with only two websites offering materials at a 6th grade reading level or below. The remaining readability studies graded the mean reading level at high school or greater, with the Gunning Fog Index scoring at a collegiate reading level. Mean understandability and actionability scores were 64.6% and 29.5%, respectively, falling below the 70% PEMAT score threshold for both scales. Fourteen (28.6%) websites were above the threshold for understandability, while no website (0%) scored above the 70% threshold for actionability. When comparing source categories, commercial health publishers provided websites that scored higher in understandability (P < .05), while private practice materials scored higher in actionability (P < .05). Resources published by academic institutions or organizations scored lower in both understandability and actionability than private practice and commercial health publishers (P < .05). No readability, understandability, or actionability score was significantly associated with search result rank.

Conclusion

Overall, online patient education materials related to rotator cuff surgery scored poorly with respect to readability, understandability, and actionability. Only two (4.1%) of the patient education websites scored at the American Medical Association and National Institutes of Health recommended reading level. Fourteen (28.6%) scored above the 70% PEMAT score for understandability; however, no website met the threshold for actionability.

Rotator cuff pathology (RCP) encompasses a large spectrum of injuries and degenerative disease. Rotator cuff injury is the most common cause of shoulder disability⁸ and the most common type of tendon injury in adults.²¹ Overall, approximately 30% of adults over the age of 60 have a rotator cuff tear and 62% of adults over the age of 80 have a rotator cuff tear.⁹ Overall, it is estimated that approximately half a million people undergo rotator cuff surgery each year in the United States, a number that is expected to continue to increase.^24,35

The treatment options for RCP are as diverse as the patient population who present with it, from young active patients with acute tear to older patients with chronic degenerative tears. Nonsurgical treatment includes physical therapy focused on scapular muscle strengthening,^17,22 corticosteroid injection, activity modification, rest, and platelet-rich plasma injection.⁹ There is also an extensive list of operative modalities including arthroscopic débridement, decompression, repair, soft tissue reconstruction, tendon transfers, and reverse total shoulder arthroplasty.⁹ Given the diversity of types of RCP and treatment modalities, it becomes problematic for patients to understand their injury and treatment options.

Public information regarding the use of internet information in 2011 showed that 80% of internet users looked online for information regarding specific medical diseases or treatments,¹² a percentage that has likely increased significantly since 2011. Atlas et al when researching the sources of information used by patients prior to elective surgery found online information to be used by more than one-half of the patients prior to deciding to undergo elective surgery.⁴ However, there is a question as to how well patients understand the information found online.

The American Medical Association (AMA) and the National Institutes of Health (NIH) recommend that health information be written at or below a sixth grade reading level.^15,20 Assessing the readability of online health materials is possible through validated readability algorithms, which consider word difficulty, quantity of sentences, and sentence length to determine corresponding grade levels.^6,16,19,20 The current literature has examples of studies that have used these algorithms to suggest that online health information often exceeds the AMA and NIH’s recommendations.^6,14,20,28 However, the readability indices used in previous studies are limited in their ability to recognize comprehension. As a result, the Patient Education Materials Assessment Tool (PEMAT) was created to better assess the understandability and actionability of online materials.²⁹ PEMAT uses 2 different rubrics, 1 for understandability and 1 for actionability, to provide a score to determine if the patient education material (PEM) can be understood by the reader and if it can be used to determine what actions the reader can take as a result of the information provided.²⁹

The focus of this study is to utilize the PEMAT and other readability scores to assess the current state of online resources regarding rotator cuff repair (RCR). We hypothesize that there will be a lack of resources available to patients that provide appropriate levels of readability, understandability, and actionability.

Methods and materials

Educational materials

Current analytics suggest that Google searches comprise approximately 90% of online searches; thus, the Google search engine was utilized for this study.³¹ On January 3, 2020, the average 12-month popularity of the following key words were compared using Google Trends (Google Trends, 2022): “rotator cuff repair,” “rotator cuff procedure,” “rotator cuff arthroscopy,” and “rotator cuff surgery.” (Fig. 1) “Rotator cuff surgery” had the highest search volume score and was chosen as the search term for material identification.

Figure 1.

Trends based on Google search terms formulated by trends.google.com. Value ranked 0-100 based on Google algorithm. A value of 100 indicates peak popularity of the term. Study terms demonstrated that “rotator cuff surgery” was 3.9 times more likely to be searched by the general public compared to “rotator cuff repair”.

Two internal Google searches were independently performed on January 3, 2020, by two authors. Click-through-rate analyses of internet trends show that approximately 70% of “clicks” are for the first 10 search results.¹ In our study, to be thorough, we included the first 50 results by each reviewer. The two reviewers compared their findings and removed duplicates to create a final list of websites. Inclusion criteria consisted of websites that were directed at educating patients regarding RCR. Websites were excluded from the study if they were news articles, personal anecdotes, primarily audiovisual-based materials, peer-reviewed journal articles, advertisements for specific products without patient education, articles not written for patient audience, or articles not related to RCR.

Analysis of materials

Qualitative analysis

Two members of the research team independently reviewed the PEM; conflicts between the two reviewers were resolved after discussion with a separate author acting as arbitrator when needed. The qualitative portion of the review analyzed whether the PEM was discussion of operative management (specific surgical options), nonoperative management, general background information, injury prevention, workup (diagnosis or preoperative management), advertisement of a physician or group for a certain treatment, and type of publisher (academic institution, private practice, public health information).

Readability

Flesch Reading Ease, Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook grade, Coleman-Liau Index, and Gunning Fog Index were used to evaluate the readability of PEMs. These 5 tools have often been utilized in previous studies to analyze readability.^5,6,13,14 An open source readability software (https://www.webfx.com/tools/read-able/check.php) was used to assist in scoring the PEMs for readability. Text found within each webpage that was not related to patient education was excluded (copyright, references, links).

Understandability and actionability

The PEMAT scoring system has been validated in existing literature as a tool to grade the understandability and actionability of print and audiovisual PEMs.^6,13,14 An understandability and actionability score is determined for each material reviewed with the PEMAT scoring system,² which uses two separate rubrics for understandability and actionability, respectively. Each rubric is an itemized survey for which the reviewer will answer if the PEM did or did not fulfill each item.² The understandability rubric assesses the content, word choice, organization, layout, and the possible use of visual aids to determine its score. The actionability rubric assesses the PEMs depiction of specific actions a patient can take, if the PEM does an adequate job of describing how to perform the actions described. Each rubric is then scored by taking the amount of items it did address divided by the total amount of items it could have addressed. This number is then multiplied by 100 to obtain a percentage. The scale for this scoring system is 0%-100%. A score of 70% or higher is considered adequately understandable and actionable.²⁹ Each PEM was reviewed using the PEMAT-P form by two separate reviewers.² Cohen’s kappa was used to determine inter-rater reliability. The magnitude of the kappa statistic was interpreted by a criteria set by Landis et al¹⁸ as these criteria were used by PEMAT developers and later evaluators to measure the reliability of PEMAT scoring.^29,33

Additional statistical analysis

Another variable considered during evaluation of the PEM was Google search engine ranking, which was determined by the average position at which the website appeared during the search by the 2 independently conducted queries. Correlation between search ranking, readability, understandability, and actionability was determined with Spearman’s rho. Statistical significance for this study was defined as P < .05.

Results

Two independent searches yielded 58 unique websites, of which 49 (49/58, 84.5%) met the inclusion criteria. The 9 sites not included in final analysis were excluded due to being videos (7 websites) or being research articles (2 websites). Of the websites analyzed, 31% (15/49) were private practice, 31% (15/49) were commercial health information, and 39% (19/49) were academic practice.

Qualitative analysis

While 100% of the patient-centered education material contained content regarding surgical intervention, 59% (29/49) of the websites contained information regarding nonoperative management. The majority (88%, 43/49) of the websites analyzed included background information of shoulder anatomy, pathology, or risk factors. Only 12% (6/49) of the websites addressed preventative measures. Advertisements for physicians or groups providing surgical intervention for RCR injury was very common in the websites from academic institutions (95%, 18/19) and private practice (87%, 13/15), but much less common (13%, 2/15) in the health information publisher websites.

Readability

The mean FKGL was 10.6, which is approximately a 10th grade (high school sophomore) reading level. Only 4% (2/49) of the websites offered materials at a 6th grade reading level or below. Readability was also found to be at a high school level or greater for the Gunning Fog Index: 13.7, Simple Measure of Gobbledygook grade: 10.3, automated readability: 10.6, and Coleman Liau: 10.92.

Understandability and actionability

The mean understandability and actionability scores were 64.6% and 29.5%, respectively, which means both fall below the ideal PEMAT score threshold of above 70%.²⁹ Although 28.6% (14/49) met the threshold for understandability, none of the websites (0/49) met the threshold for actionability (Fig. 2). When comparing source categories, commercial health publishers provided websites that scored higher in understandability (P < .05), while private practice materials scored higher in actionability (P < .05). Resources published by academic institutions or organizations scored lower in both understandability and actionability than private practice and commercial health publishers (P < .05). No readability, understandability, or actionability score was significantly associated with search result rank.

Figure 2.

PEMAT scores by Google search rank. There was no association between search rank and score for understandability or actionability. PEMAT, Patient Education Materials Assessment Tool.

Discussion

Shoulder pain is the 3rd most common musculoskeletal complaint of patients presenting to primary care physicians.¹⁰ The most common cause of shoulder pain is RCP.²¹ Accurate, understandable, and actionable information should be available to the large population of patients who have RCP.

This study shows that the existing online patient education materials pertaining to RCP and its treatment performs poorly on assessments for readability, understandability, and actionability. Although there is a paucity of literature regarding understandability and actionability of online resources on the topic of shoulder surgery, our findings on the readability of patient education regarding shoulder surgery echo the findings of previous studies.^3,7,27,30 For example, Akinleye et al looked at the top ten websites providing education for rotator cuff tear and found a mean FKGL of 10.6,³ similar to the FKGL of 9.47 found in our study. Given that the NIH and AMA recommend a FKGL level of 6, the current state of online medical education is inadequate.

The main principles of ethics include beneficence, nonmaleficence, autonomy, and justice.³² Autonomy, 1 of these 4 key principles of ethics, was affirmed in a court decision by Justice Cardozo in 1914 with the epigrammatic dictum, “Every human being of adult years and sound mind has a right to determine what shall be done with his [or her] own body; and a surgeon who performs an operation without his [or her] patient’s consent, commits an assault, for which he is liable in damages.”²³ Autonomy is a debated topic as it is difficult to assess if a patient fully understands what the surgery may entail, in order to be capable of providing informed consent. Ultimately, it is the responsibility of the surgeon to provide accurate, understandable information to facilitate the patient into utilizing his or her autonomy. In the case of patients deciding to undergo elective surgery for RCR it becomes imperative that they understand the possible treatment options and the outcomes related to both. Surgeons will often use online references in educating their patient about the details of the surgery. Surgeons have been found to overestimate the reading level and comprehension of their patients when explaining the outcomes of these surgeries.³⁰ When medical jargon or overcomplicated explanations prevent a patient from understanding a surgery or the outcomes associated, the patient’s autonomy can be potentially compromised.

Resources published by academic institutions and organizations scored lower in both understandability and actionability than private practice and commercial health publishers (Table I). This finding is consistent with prior studies which have found that content produced by the American Academy of Orthopaedic Surgeons (AAOS) and American Shoulder and Elbow Surgeons have demonstrated inadequate readability. In 2008, only 2% of patient directed material produced by the AAOS was at a 6th grade reading level or below.²⁶ A similar study done in 2015 found that 3% of the patient directed material produced by the AAOS was at the 6th grade reading level or below.¹¹ When using the FGKL, Eltorai et al showed that the AAOS PEMs had an average education level of 9.2.¹¹ A study by Schumaier et al in 2018 that reviewed American Shoulder and Elbow Surgeons PEMS on the topic of RCP, specifically a brochure produced entitled “Rotator Cuff Tendonitis and Tears,” found the brochure to have a grade level of 14.6, which represents the reading level of a sophomore in college.²⁷

Table I.

Comparison of academic vs. private practice vs. commercial health publishers for understandability, actionability, and various readability scores.

	Academic	Private	Health publications
Number of websites	19	16	14
Understandability mean	0.6079	0.6593	0.6833
Actionability mean	0.2406	0.3375	0.3214
FKGE mean	48.5789	52.1250	53.7714
FKGL mean	11.0632	10.4062	10.0714
GFI mean	14.1526	13.4750	13.0714
SMOG mean	10.7526	10.1188	9.8214
CLI mean	11.4737	10.5625	10.5714
ARI mean	11.1947	10.3125	9.9071

FKGL, Flesch-Kincaid Grade Level; GFI, Gunning Fog Index; SMOG, Simple Measure of Gobbledygook grade; CLI, Coleman-Liau Index; FKGE, Flesch-Kincaid Grade Ease; ARI, automated readability index.

Inadequate health literacy adversely affects patient outcomes and the doctor-patient relationship.^15,34 Current orthopedic literature demonstrates health literacy as an obstacle to patient understanding of orthopedic procedures.^5,25 Our study affirms that the current online patient education materials pertaining to RCP and its treatment performs poorly on assessments for readability, understandability, and actionability.

Limitations

Multiple limitations exist for this study. This study did not look at the accuracy of the articles, only readability, actionability, and understandability. This study’s results are limited by the given time point when the internet search was done. While the results could potentially change by repeating the study on different dates, the data were quite consistent that information regarding rotator cuff surgery on the internet is hard to comprehend. The analysis is specific to the search term used in this study, “rotator cuff surgery” and a different search term, like “rotator cuff repair,” may result in different findings. Although an attempt was made to prevent user bias by deleting search history, cookies, and cache, other information, eg, location data, may affect the search results. Our study is also limited as it is an analysis of readability, understandability, and actionability and did not account for medical accuracy of the patient education material. Readability studies remain objective based on word content; however, the PEMAT-P analysis is subjective in nature.

Conclusion

Overall, online patient education materials related to rotator cuff surgery scored poorly with respect to readability, understandability, and actionability. Only 4.1% (2/49) of patient education websites scored at the AMA and NIH recommended 6th grade or lower reading level. No (0/49) website met the PEMAT threshold of >70% for actionability, while only 28.6% (14/49) met the PEMAT threshold for understandability. Online resources distributed by academic institutions and organizations scored lower in both PEMAT measures, understandability and actionability, compared to commercial health publishers and private practices. Poor readability, understandability, or actionability of these resources regarding rotator cuff surgery may lead to inadequate informed decision-making. Surgeons, as part of academic health care institutions, private practices, and medical societies, should assist in the improvement of the current state of online education regarding this topic.

Disclaimers

Funding: No funding was disclosed by the authors.

Conflicts of interest: Dr. Wesley P. Phipatanakul is a consultant for Arthrex Consultant.

The other authors, their immediate families, and any research foundation with which they are affiliated have not received any financial payments or other benefits from any commercial entity related to the subject of this article.