Abstract
Objectives:
While the internet provides accessible medical information, often times it does not cater to the average patient’s ability to understand medical text at a 6th and 8th grade reading level, per American Medical Association (AMA)/National Institute of Health (NIH) recommendations. This study looks to analyze current online materials relating to posterior cruciate ligament (PCL) surgery and their readability, understandability, and actionability.
Methods:
The top 100 Google searchs for “PCL surgery” were compiled. Research papers, procedural protocols, advertisements, and videos were excluded from the data collection. The readability was examined using 7 algorithms: the Flesch Reading Ease Score, Gunning Fog, Flesch-Kincaid Grade Level, Coleman-Liau Index, SMOG index, Automated Readability Index and the Linsear Write Formula. Two evaluators assessed Understandability and Actionability of the results with the Patient Educational Materials Assessment Tool (PEMAT). Outcome measures included Reading Grade Level, Reader’s age minimum and maximum, Understandability, and Actionability.
Results:
Of the 100 results, 16 were excluded based on the exclusion criteria. There was a statistically significant difference between the readability of the results from all algorithms and the current recommendation by AMA and NIH. Subgroup analysis demonstrated that there was no difference in readability as it pertained to which page they appeared on Google search. There was also no difference in readability between individual websites versus organizational websites (hospital and non-hospital educational websites). Three articles were at the 8th grade recommended reading level, and all three were from healthcare institutes.
Conclusion:
There is a discrepancy in readability between the recommendation of AMA/NIH and online educational materials regarding PCL surgeries, regardless of where they appear on Google and across different forums. The understandability and actionability were equally poor. Future research can focus on the readability and validity of video and social media as they are becoming increasingly popular sources of medical information.
Key Words: Knee, Patient education materials, PCL surgery, Readability, Understandability
Introduction
Posterior Cruciate Ligament (PCL) injuries account for 5-20% of all knee-ligamentous injuries, often occurring from motor vehicle accidents and dashboard injuries. They account for 0.65%-3% of all sport-related injuries in Europe and the United States.1-5 Oftentimes, surgery for a PCL tear is only indicated when there are combined ligamentous injuries, high grade injuries, or chronic instability.6 Isolated PCL surgery is not typically performed at the time of injury and is generally delayed a few days or weeks. As PCL surgery is relatively uncommon compared to other orthopedic surgeries, there is less literature available geared towards patient education.
The internet allows patient to take a more active role in their healthcare decision making.7-10 Patients now turn to online resources as their initial source of healthcare information to better understand their injury pathology, related procedures, prognosis, and recovery timeline.11-13 The American Medical Association (AMA) and National Institute of Health (NIH) recommend that patient-centered educational materials should be written at a 6th and 8th grade reading level, respectively, for effective communication.3,14 This recommendation is in place to improve overall accessibility to healthcare data with consideration of social determinants of health and education.4,15 Healthcare literacy regularly presents as a communication barrier between physicians and patients. Spanning across all races, ethnicities, and social classes, there are direct correlations between low health literacy, education, and income levels.3,14,16–18 Many demographics who suffer from lower health literacy are also on the receiving end of worse health outcomes .17,19–22 While socioeconomic influences cannot be excluded from consideration, patients with lower health literacy are 1.5 to 3 times more likely to lack knowledge of their health status, measures of morbidity, and use of healthcare resources, overall leading to worse treatment compliance and healthcare decision making.19 Specifically in the field of orthopedic medicine, complex musculoskeletal terminology and understanding of anatomy can hinder patients’ ability to make critical medical decisions and adhere to complex instructions, impacting patient safety and autonomy.23 Therefore, it is of tremendous importance for healthcare providers in the field of orthopedics to appropriately educate and provide “understandable” resources for the benefit of their patient population.
Because PCL injuries are relatively rare amongst sport-related injuries and injuries that require surgeries, patients are likely to rely more so on these online educational materials than other avenues of information, such as the experiences of family and friends.11 The aim of this research is to assess online patient educational materials related to PCL surgery currently on Google with respect to their readability, understandability and actionability in relation to the AMA/NIH’s set reading-recommendations. Readability is defined as “ease of understanding or comprehension due to style of writing” which emphasizes sentence structure, syntax, and wording.24 Understandability accounts for the readability, information presentation, and organization of data. Actionability looks for actionable terms, straightforward methods, and visual aids to help patients act.25 Based on existing data on readability and understandability of orthopedic injuries and surgeries, it is hypothesized that PCL surgery online information will not meet the recommended guidelines for patient-facing literature.26-29
Materials and Methods
Identifying Online Educating Materials
Online educational materials for “PCL surgery” were identified using Google.com. At the time of data collection from September to October of 2022, Google was the most popular search engine, with 84-92% of all internet users.30 To prevent individualization of Google search results, the researchers deleted cookies, utilized incognito mode, and disabled location services.
To determine the number of online materials to include within the study, analysis of Google Click through Rates (CTR) and the average numbers of online educational materials performed in similar projects were taken into consideration. On average, the first five results of a Google search received more than 65% of all clicks.31-33 While Google automatically generates ten search results per page, the tenth item only received 3% of CTR.31-33 Moreover, the combined results from Pages two and three only receive 5.59%.31-33 In contrast, prior projects on Patient Educational Materials Assessment Tools (PEMAT) analyzed as little as 13 to as many as 134 total search results.26–29,34 By weighing the CTR of the average internet user versus having enough power for analysis, the first 100 online educational materials were selected to maintain the practicality and consistency of this project.
The search term “PCL surgery” was implemented. Since the choice to undergo PCL surgery after an injury is often elective and depends on patients’ subjective perceptions of functional instability, the term “PCL surgery” was deliberately chosen to properly capture educational materials utilized by patients in making informed decisions regarding surgical intervention following a diagnosis of PCL injury. The searched results that did not aim to provide educational materials, including advertisements and surgery protocols were excluded. While peer-reviewed literature is recognized for high-quality data, research papers were omitted due to their distinct target consumer audience characterized by inadequate readability for the general public.35 Video content was also excluded due to the limitation for text analysis at this time.
Qualitative Content Analysis
The results for patient educational materials underwent qualitative analysis with the following categories: discussion of general background (modality of injuries, anatomy, workup, and prognosis), discussion of operative managements, discussion of non-operative managements, discussion of injury prevention, discussion of rehabilitation and discussion of complication and risks of operative managements. The results were further categorized as resources from individual medical providers versus hospital and non-hospital related general education websites.
Readability
Seven separate algorithms were adopted to analyze the readability of websites. The algorithms included: the Flesch Reading Ease score, Gunning Fog, Flesch-Kincaid Grade Level, Coleman-Liau Index, SMOG (Simple Measure of Gobbledygook) index, Automated Readability Index, and Linsear Write Formula [Appendix Table A1]. These seven algorithms were identified from prior studies as a measure of readability and are the current recommended assessment tools from the Agency for Healthcare Research and Quality.25–28,34 The Flesch Reading Ease score assigns each article a value by analyzing the average length of the sentences and average number of syllables, with higher scores being more readable [Table 1]. The remaining algorithms directly analyzed the text and determined a United States grade levels, according to their complexity, termed Reading Grade Level (RGL). Text that was pertinent to patient education was copy-pasted into these algorithms, while irrelevant text including contact information and references were excluded from analysis.
Table 1.
The Flesch Reading Ease Score and its interpretation
| Score | Notes |
|---|---|
| 90-100 | Very easy to read, easily understood by an average 11-year-old student |
| 80-90 | Easy to read |
| 70-80 | Fairly easy to read |
| 60-70 | Easily understood by 13- to 15-year-old students |
| 50-60 | Fairly difficult to read |
| 30-50 | Difficult to read, best understood by college graduates |
| 0-30 | Very difficult to read, best understood by university graduates |
Understandability and Actionability
Two independent evaluators (Reviewer 1 and Reviewer 2) utilized the Patient Education Materials Assessment Tool for printable materials (PEMAT-P) questionnaire to assess the understandability of patient materials. The PEMAT-P was created by the Agency for Healthcare Research and Quality as the currently recommended standardized measure to distinguish understandability based on nineteen variables, including content, word choice and style, use of numbers, organization, layout and design, and use of visual aids. Actionability was assessed with seven questions, accounting for text directly addressing users, breaking down explicit steps, use of diagrams and visual aids to help users act. Scores range from 0-100% with a score above 70% representing adequate understandability and actionability.25 during data analysis of understandability and actionability, two of the website results were no longer searchable on the Internet and thus excluded.
Data Analysis
Website texts were entered into seven algorithms and results were included in our analysis. The average and standard deviation from each algorithm were calculated and compared to the AMA and NIH guidelines. The algorithms also provided an approximate age range corresponding to each reading grade level.
As stated, the first page of Google search usually accounts for more than 90% of CTR. Subgroup analysis of results from the first page of Google was compared to results overall. Another assessment was based on publication origin, divided by individual providers, hospital affiliated, and non-hospital affiliated educational websites.
Understandability and Actionability were independently analyzed by two reviewers. Scores greater than 70% are interpreted to have adequate understandability and actionability. The interrater reliability was calculated using the Cohen Kappa algorithm.
Results
Search results and Qualitative Content Analysis
Out of the 100 generated research results, 16 were excluded because they were research articles, procedure protocols for healthcare personnel, advertisements, videos, or articles that were either not meant as educational materials or not related to PCL surgery. The remaining 84 websites underwent readability analysis [Appendix Table A2]. 40 results were individual medical providers websites while 44 results were hospital and non-hospital related educational websites. During data collection for qualitative content tabulation, understandability and actionability, two websites were no longer searchable on the Internet and excluded. Of the 82 results remaining, the results of the content analysis were shown [Table 2].
Table 2.
Content analysis of 82 results with parameters: general backgrounds, operative managements, non-operative managements, injury prevention, rehabilitation, and complications and risk of operative managements
| Content | Number of PCL patient educational materials with the content |
|---|---|
| General Backgrounds (modality of injuries, anatomy, workup, and prognosis) | 72 |
| Operative Managements | 76 |
| Non-operative Managements | 54 |
| Injury Prevention | 3 |
| Rehabilitation | 41 |
| Complications and Risk of Operative Managements | 32 |
Readability
Overall, the combined results for the Flesch Reading Ease Scores were characterized as fairly-difficult to difficult to read, appropriate for college-level students [Table 3]. The readability calculated by the remaining algorithms, including the Gunning Fog Index, the Flesch-Kincaid Grade Level, the Coleman-Liau Index, the SMOG Index, the Automated Readability Index, and the Linsear Write Formula were demonstrated [Table 4]. The reading consensus, which averaged the RGL of the latter six algorithms, showed a mean RGL of 11.0th grade (SD 1.2nd grade). The corresponding age was 15- to 17-year-old.
Table 3.
The Flesch Reading Ease Scores for the 84 searched results. The score ranged from 34.5 to 66.5 meaning that the text is best understood by college-level students
| Range | Characterization | |
|---|---|---|
| The Flesch Reading Ease Scores | 34.5 – 66.5 | Fairly-difficult to difficult to read |
Table 4.
The results for the reading grade level calculated by the algorithms
| Mean Reading Grade Level (SD) | Median | |
|---|---|---|
| The Gunning Fog Index | 13.9th grade (SD 1.3) | 13.8th grade |
| The Flesch-Kincaid Grade Level | 10.7th grade (SD 1.3) | 10.8th grade |
| The Coleman-Liau Index | 11.0th grade (SD 1.3) | 11.0th grade |
| The Smog Index | 10.6th grade (SD 1.0) | 10.7th grade |
| The Automated Readability Index | 10.4th grade (SD 1.5) | 10.2nd grade |
| The Linsear Write Formula | 12.0th grade (2.1) | 11.8th grade |
The readability results in comparison to the current recommendation by AMA and NIH were demonstrated [Figure 1]. There was a statistically significant differences between the readability of the 84 results from all algorithms and the current recommendations by AMA and NIH. Among the 84 articles, none of the articles had RGL of 6th grade or below (AMA recommendation). Only 3.5% of the articles (n=three articles) had RGL at an 8th grade level (NIH recommendation). Of those, two articles were on the first page of Google search (top 10 results) while the third was on the third search page.
Figure 1.
Reading Grade Level for PCL Online Education Materials. The bar graph showed the average reading grade level and standard deviation calculated for the 84 results. The orange line is the AMA recommended reading level while the grey line represents the NIH recommended reading level
Subgroup Analysis of Readability
First 10 Google Search versus the Remaining Results:
There were no statistically significant differences in readability between first 10 Google searches compared to the remaining websites in both the Flesch Reading Ease Score and the overall reading consensus. The mean difference for the Flesch Reading Ease Score was 1.66 (CI -4.27 to 7.59, P value = 0.78). The mean difference for the overall reading consensus was - 0.21 grade (CI: -1.6 to 1.1, P value = 0.87).
Individual Providers’ Websites versus Hospital and non-hospital affiliated education websites:
Of all 84 websites, 40 were individual providers websites while 44 were hospital and non-hospital affiliated educational websites [Table 5 and 6]. There is a statistically significant difference between the hospital and non-hospital affiliated education websites and the recommendation by AMA and NIH [Figure 2]. The same difference was observed in the individual providers’ websites as well.
Table 5.
The Flesch Reading Ease Scores for personal websites and hospital and non-hospital affiliated education websites. The mean for both forums were as reported. The text was best understood by college-level students
| The Flesch Reading Ease Score Mean | Characterization | |
|---|---|---|
| Personal Websites (40 results) | 49.0 (SD 4.5) | Fairly difficult to difficult |
| Hospital and non-hospital affiliated education websites (44 results) | 49.1 (SD 8.0) | Fairly difficult to difficult |
Table 6.
The results for the reading grade level calculated by the algorithms for personal websites and hospital and non-hospital affiliated education websites
| The Gunning Fog | The Flesch-Kincaid | The Coleman-Liau | The SMOG | The automated readability | The Linsear Write Formula | |
|---|---|---|---|---|---|---|
| Personal Website (40 results) | 14.0th (SD 1.0) |
10.7th (SD 0.8) |
11.0th (SD 1.1) |
10.6th (SD 0.7) |
10.3rd (SD 1.1) |
12.0th (SD 1.6) |
| Hospital and non-hospital affiliated education websites (44 results) | 14.0th (SD 1.6) |
10.8th (SD 1.5) |
11.0th (SD 1.5) |
10.6th (SD 1.3) |
10.5th (SD 1.7) |
12.0th (SD 2.5) |
Figure 2.
Reading Grade Level for PCL Surgery: Online Educational Materials from Hospital and non-hospital affiliated education websites. The orange line indicated the AMA recommendation of 6th grade, and the grey line indicated the NIH recommendation of 8th grade. Similar finding was apparent in individual providers’ websites
There is no statistically significant difference in readability between two forums in both the Flesch Reading Ease Score and the overall reading consensus. The mean difference for the Flesch Reading Ease Score was 0.57 (CI -1.77 to 2.90, P value = 0.86). The mean difference for the overall reading consensus was -0.3 grade (CI -0.8 to 1.4, P value =0.81), where individual providers’ websites had younger RGL; however, the difference was not significant.
Overall, the three articles that were at the 8th grade reading level (recommended by NIH) were all from hospital institutes’ websites.
Understandability and Actionability
Between Reviewer 1 and Reviewer 2, the Cohen’s kappa value was approximately 0.315, with a confidence interval between 0.225 and 0.405 (p < 0.001). The interrater strength of agreement was classified as fair.
Understandability rating from Reviewer 1 demonstrated a mean score of 61.04 (SD=14.54). 21 of 82 study results (24%) showed an understandability rating greater than 70%, representing adequate understandability. 3 of which were in the top 10 Google search. The same reviewer's assessment of actionability showed an average rating score of 41.11 (SD of 23.37). 13 of the 82 study results (16%) showed actionability scores greater than 70%. two of which were among the top 10 Google search.
Understandability rating from Reviewer 2 demonstrated a mean score of 63.05 (SD of 14.56). 25 of the 82 study results (30%) showed understandability rating greater than 70%, representing adequate understandability. 6 of which were in the top 10 Google search. Actionability rating from Reviewer 2 demonstrated a mean actionability score of 34.32 (SD of 21.98), with a range from 0 to 85.7. Only eight of the 82 study results (10%) showed actionability greater than 70%. 2 of which were among the top 10 Google search.
The criteria from PEMAT-P and the numbers of studies fulfilling the criteria for understandability and actionability were displayed [Table 7 and 8]. Percentages of studies fulfilling the criteria of 82 results were calculated and exhibited below.
Table 7.
82 Study Results Fulfilling the Understandability Criteria Outlined by PEMAT-P rated by Reviewer 1 and 2
| Understandability Criteria | Reviewer 1 # of studies qualified | Percentage of studies fulfilling such criteria | Reviewer 2 # of studies qualified | Percentage of studies fulfilling such criteria |
|---|---|---|---|---|
| The material makes its purpose completely evident. | 79 | 96.34% | 78 | 95.12% |
| The material does not include information or content that distracts from its purpose. | 71 | 86.59% | 72 | 87.80% |
| The material uses common, everyday language. | 74 | 90.24% | 81 | 98.78% |
| Medical terms are used only to familiarize the audience with the terms, when used terms are defined. | 74 | 90.24% | 80 | 97.56% |
| The material uses the active voice. | 65 | 79.27% | 63 | 76.83% |
| Numbers appearing in the material are clear and easy to understand. | 72 | 87.80% | 81 | 98.78% |
| The materials do not expect the user to perform calculations. | 80 | 97.56% | 81 | 98.78% |
| The materials break information in short sections. | 77 | 93.90% | 76 | 92.68% |
| The materials have informative headers. | 76 | 92.68% | 76 | 92.68% |
| The materials present information in a logical sequence. | 76 | 92.68% | 75 | 91.46% |
| The materials provide a summary. | 8 | 9.76% | 8 | 9.76% |
| The material uses visual cues. | 18 | 21.95% | 21 | 25.61% |
| The materials use visual aids. | 25 | 30.49% | 31 | 37.80% |
| Visual aids act to reinforce rather than distract. | 20 | 24.39% | 26 | 31.71% |
| Visual aids have clear titles and captions. | 19 | 23.17% | 14 | 17.07% |
| Use of illustration and photos that are clear and uncluttered. | 17 | 20.73% | 16 | 19.51% |
| Simple tables with short and clear row and column headings. | 0 | 0.00% | 0 | 0.00% |
Table 8.
. 82 Study Results Fulfilling the Actionability Criteria Outlined by PEMAT-P, rated by Reviewer 1 and 2
| Reviewer 1 # of studies qualified | Percentage of studies fulfilling such criteria | Reviewer 2 # of studies qualified | Percentage of studies fulfilling such criteria | |
|---|---|---|---|---|
| Clearly identifies one action user can use. | 63 | 76.83% | 61 | 74.39% |
| Address the user directly. | 63 | 76.83% | 64 | 78.05% |
| Breaks down action into manageable explicit steps. | 55 | 67.07% | 41 | 50.00% |
| The materials provide a tangible tool (menu, checklists) to help users take actions. | 10 | 12.20% | 12 | 14.63% |
| Provides simple instruction and teaches how. | 40 | 48.78% | 13 | 15.85% |
| How to use charts, graphs, tables and diagrams to take actions. | 2 | 2.44% | 2 | 2.44% |
| Visual aids are used so it's easier to take action. | 3 | 3.66% | 4 | 4.88% |
Discussion
The readability of online patient educational materials related to PCL surgery did not meet the current recommendations set forth by AMA and NIH, which confirms our initial hypothesis. Around 24-30% of the materials scored adequate understandability (scoring 70% or above), while only 10-16% scored adequate actionability (scoring 70% or above). This indicated an overall poor readability, understandability and actionability of the educational materials.
In addition, only two of the ten articles from Google’s first page related to PCL surgery were appropriately at an 8th grade reading level. The RGL assessment tools (Gunning Fog, Flesch-Kincaid Grade Level, Coleman-Liau index, SMOG index, Automated Readability index, and Linsear Write formula) all demonstrated a statistically significant difference of reading grade levels relative to the recommended 6th grade and 8th grade reading levels set by the AMA and NIH. In fact, none of the articles had a RGL at a 6th grade level and only three articles (4%) had RGL below the 8th grade.
For subgroup analysis, it was suspected that perhaps materials found in the first page of the results would be more readable, increasing the click through rates. However, the analysis did not show a statistically significant difference in RGL as it pertained to where they appeared on Google search. Furthermore, there were nearly equal representations of medical information provided by personal websites (40 articles) in comparison with health institutes and non-hospital affiliated educational websites (44 articles) for the term “PCL surgery”. Both sources of information were significantly above the recommended RGL from the AMA and NIH. Subgroup analysis demonstrated that there was no statistically significant difference between the two forums. However, the three articles that met the NIH recommendation of 8th grade reading level were all from healthcare institutes. Nonetheless, these results indicate there is tremendous work to be done by both parties to provide information that is more aligned with the reading level of the general patient population.
The understandability of these websites largely considered how the overall information was presented to readers. Majority (90%) of the materials where successful in presenting information in a logical and organized manner with universal use of headings, using common language, and following a logical sequence [Table 7]. Where most websites lost points was from a lack of visual aids, illustrations, and tables which can simplify material for visual learners. In terms of actionability, over 70% of the websites identified actionable items and took an active voice towards readers. They also failed to use charts, graphs, checklists, and visual aids to help users act. For visual learners and especially patients with low-literacy level, having visual aids and illustration have shown to improve medication adherence and comprehension.36 Centers for Disease Control and Prevention also emphasized the use of visuals for effective communication, by simplifying complex information and reinforcing written health messages.
Other orthopedic procedures such as Achilles tendon or arthroscopic procedures are more common.3 Studies relating to those topics have found similar discrepancies of poor readability, understandability and actionability in their online resources 26–29. Similar findings are found across other medical fields. A 20-year analysis that examined readability of materials from High-Impact Medical Journals, including Journal of the American Medical Association (JAMA) Network, American Journal of Respiratory and Critical care Medicine (AJRCC), Circulation, Annals of the Rheumatic Diseases (ARD) and Annals of Internal Medicine demonstrated similar findings of subpar readability for the public.37 These findings also suggested that there is a large disparity between the current reading level recommendation and the online patient educational materials.
The accessibility of the Internet allows the public access to vast amounts of medical information, but healthcare professionals and educators must understand the nature of the online resources that the patients are using. The AMA and NIH recommended 6th and 8th grade reading level to ensure that this information does not represent another form of social inequality.3,4,14–18 The high level of literacy (11th-12th grade) required to comprehend PCL surgery is alarming, as it is estimated that 80 to 90 million adults in the US have limited health literacy and would not have the ability to understand what these websites trying to convey.18,19 Socioeconomic status plays a large role in health literacy across the general patient population. The downstream effect of differing socioeconomic status and poor health literacy correlate with disadvantaged people experiencing worse health outcomes, increased rates of chronic medical problems, hospitalization, and mortality.17,19–22 The AMA and NIH recommendation of all patient-educational material to be at a 6th and 8th grade reading level is an attempt to address this innate social inequality.3,4,14–18 As it stands, inequality within healthcare remains an incredibly complex issue. Improving the readability, understandability, and actionability of readily accessible online patient educational materials is a simple first step that individuals and healthcare organizations can take to begin addressing it.
Limitations
This study was limited by the design and resources that were available at the time of data collection. The term “PCL surgery” was the focus of data collection, but some patients’ searches might delve into more specific terms for rehabilitation, complications, prognosis, or treatment. Variation in keyword searches on Google have potential to largely influence the information output, and alternative phrasing should be explored. Additionally, all 100 internet searches on Google occurred within one time frame in 2022. Google is subject to changing their top 100 search results based on user activity, internet sponsorships, and user location or preferences. While researchers blinded for their specific location, deleted search history, and used private browsing, search results that were identified as the top 100 in this study will not exactly mimic what comes up for users in different regions.
It is also notable that the interrater reliability between the two independent reviewers, Reviewer 1 and Reviewer 2, was only rated as fair based on the Cohen’s kappa value of 0.315. This means many of the interpretations for the understandability and actionability for data can be subjective. This could be improved in future studies by having more than two independent reviewers analyze patient educational materials. Also, the study did not recruit patients to participate in rating of these patient educational materials. The study concluded that improvement could be done within the analyzed material; however, it is uncertain whether patients would find them truly beneficial.
The study did not analyze audiovisual resources as a medium for patient educational materials. This includes websites such as YouTube or Vimeo, but also short-form video media platforms such as Instagram or TikTok. Media outlets such as Instagram and YouTube attract around 2 billion active users per month, and patients are using these platforms to attain medical advice more than ever.38 not having data on the accuracy, understandability, and actionability of their content is a major limitation to understanding the real current climate of data distribution.
Furthermore, as technology and AI advance, new platforms such as Chat GPT are becoming a new medium for data collection. Users turn to Chat GPT because it is advertised as an all-knowing tool that scavenges all the top results from the internet to provide a condensed recommendation. With Chat GPT’s technology pulling data from Google searches, it can be assumed it’s readability would parallel the results from this study, but this was not formally assessed. Future studies can increase the range of materials analyzed to gather a more accurate understanding of the readability of online patient educational material.
Conclusion
This study demonstrated a significant discrepancy between the readability of currently available online educational materials for PCL surgery and the recommendation set by both the AMA and NIH, as well as poor understandability and actionability. Subgroup analysis showed no differences among materials’ positions in Google search and materials across different forums. As the world becomes increasingly digitized, patients can independently gain access to medical information like never before, but with this, healthcare providers and
educators must ask themselves whether this information is being appropriately written for its intended audiences.
Acknowledgment
Not applicable
Conflict of interest:
None
Funding:
None
Appendix
Table A1.
Algorithms Used for Interpretation of Calculation of Readability a
| Calculation | Interpretation | |
|---|---|---|
| Flesch Reading Ease | RE = 206.835 - (1.015 * ASL) - (84.6 * ASW) | 90.1-100.0 = 5th-grade material; 70.1-80.0 = 7th-grade material; 50.1-60.0 =10th- to 12th-grade material; 0.0-30.0 = college graduate material |
| Gunning Fog Index | 0.4*(total words/total sentences) + 100 *(words with (3+ syllables)/total words) | Estimated grade level of materials |
| Flesch-Kincaid Grade Level | 0.39*(total words/total sentences) + 11.8 * (total syllables/total words) - 15.59 | Estimated grade level of materials |
| Coleman-Liau Index | 0.0588*(letters per 100 words) - 0.3*(sentences per 100 words) - 15.8 | Estimated grade level of materials |
| The SMOG Index | 1.0430* sqrt(number of polysyllabic words*(30/total sentences) + 3.1291 | Estimated grade level of materials |
| Automated Readability Index | 4.71*(total characters/total words) + 0.5*(total words/total sentences) - 21.43 | Estimated grade level of materials |
| Linsear Write Formula | (1*(total of 1 syllabe words) + 3* (total of 3+ syllable words)/total sentences) = r; if r > 20 -> Reading grade level = r/2; if r < = 20 -> Reading grade level = (r -2)/2 | Estimated grade level of materials |
a All utilized online open-source readability calculator (https://readabilityformulas.com/free-readability-formula-tests.php)
ASL: average sentences length; ASW: average number of syllables per word.
Table A2.
Websites
| https://www.bonsecours.com/health-care-services/orthopedics-sports-medicine/knee/treatments/pcl-surgery#:~:text=PCL%20surgery%20is%20the%20surgical,ACL)%20to%20stabilize%20the%20knee. |
| https://www.mercy.com/health-care-services/orthopedics-sports-medicine-spine/specialties/knee-leg/treatments/pcl-surgery |
| https://www.orthovirginia.com/posterior-cruciate-ligament-pcl-reconstruction |
| https://www.sportsmd.com/sports-injuries/knee-injuries/pcl-tear/ |
| https://drrobertlaprademd.com/pcl-reconstruction/ |
| https://www.lmh.org/get-care/orthokansas/knee-leg/pcl-tear-reconstruction/ |
| https://www.sports-health.com/sports-injuries/knee-injuries/posterior-cruciate-ligament-pcl-tear-treatment-options |
| https://my.clevelandclinic.org/health/diseases/21793-pcl-posterior-cruciate-ligament-tears |
| https://www.herrerasportsmedicine.com/pcl-reconstruction-miami-institute.html |
| https://www.stgeorgesurgical.com/procedure/posterior-cruciate-ligament-pcl-repair/ |
| https://orthoinfo.aaos.org/en/diseases--conditions/posterior-cruciate-ligament-injuries/ |
| https://www.iskinstitute.com/kc/knee/pcl_tear/t3.html |
| https://rileywilliamsmd.com/posterior-cruciate-ligament-pcl-reconstruction-repair-manhattan-new-york-city-ny/ |
| https://www.physio.co.uk/what-we-treat/surgery/knee/posterior-cruciate-ligament-pcl.php |
| https://www.uofmhealth.org/conditions-treatments/cmc/knee/pcl |
| https://www.cosm.net.au/orthopaedic-surgery/posterior-cruciate-ligament-pcl-reconstruction-2/ |
| https://www.marshfieldclinic.org/specialties/orthopedics/knee-surgery/pcl |
| https://www.brianforsythemd.com/pcl-reconstruction%E2%80%93allograft.html |
| https://www.bjc.org/Ortho/Knee/Knee-Pain |
| https://www.physio-network.com/blog/pcl-injury-and-reconstruction/?utm_source=google-ads&utm_medium=cpc&utm_campaign=&utm_agid=&utm_term=&creative=&device=c&placement=&gclid=Cj0KCQjwuuKXBhCRARIsAC-gM0iq46b4MSaa0WT2l-nqsBrZmf4vMf8_mrhSerm_DtaWGxc6bf8eMCIaAqsdEALw_wcB |
| https://www.orthopedicsurgeonnyc.com/pcl-reconstruction-surgery |
| https://www.proortho.com/knee/pcl-surgery/ |
| https://www.webmd.com/fitness-exercise/posterior-cruciate-ligament-injury |
| https://www.seanolearyknee.co.uk/pdfs/pcl-reconstruction.pdf |
| *https://www.winchesterhospital.org/health-library/article?id=916896 |
| https://manhattansportsdoc.com/pcl-reconstruction-posterior-cruciate-ligament-orthopedic-knee-surgeon-manhattan-new-york-city-ny/ |
| https://www.orthopaedicsurgery.uci.edu/pcl-reconstruction-orthopaedic-irvine-newportbeach-california.html |
| https://www.sportssurgerychicago.com/knee/pcl-reconstruction-surgery-westchester-oakbrook-hinsdale-il/ |
| https://www.orthotrauma.com.au/procedures/knee/pcl-reconstruction/# |
| https://www.cincysportssurgeon.com/pcl-tears-sports-medicine-orthopedic-surgeon.html |
| https://www.nuffieldhealth.com/treatments/pcl-reconstruction |
| https://www.howardluksmd.com/pcl-injury-and-surgery-reconstruction/ |
| https://noyeskneeinstitute.com/specialties-and-programs/pcl-reconstruction/ |
| https://www.stlosm.com/posterior-cruciate-ligament-reconstruction-orthopedics-sports-medicine-specialists-creve-coeur-missouri.html |
| https://sportdoctorlondon.com/pcl-tear/ |
| https://www.mountsinai.org/health-library/selfcare-instructions/posterior-cruciate-ligament-pcl-injury-aftercare |
| https://fixknee.com/Non-Operative-Treatment-PCL-Tears |
| https://matthewprovenchermd.com/pcl-reconstruction-surgery-vail-aspen-denver-co/ |
| https://www.timothybertmd.com/pcl-reconstruction-sports-medicine-scottsdale-phoenix-mesa.html |
| https://treasurevalleyhospital.com/PCL-Surgery |
| https://robertboykinmd.com/pcl-reconstruction-asheville-north-carolina/ |
| https://www.uhortho.com/pcl-reconstruction-orthopaedic-treatment-chardon-oh.html |
| https://www.toshmd.com/pcl-reconstruction-orthopedic-surgeon-murray-utah-salt-lake-city.html |
| https://www.alfredmansourmd.com/posterior-cruciate-ligament-reconstruction-alfred-a-mansour-md-hip-preservation-sports-medicine-surgeon.html |
| https://www.stoneclinic.com/pcl-reconstruction-rehab-protocol |
| https://www.jordan-younginstitute.com/pcl-injuries-orthopedic-surgeon-virginia.html |
| https://www.florthocare.com/services/knee-procedures/pcl-reconstruction/ |
| https://www.aaroncoatsmd.com/pcl-reconstruction-orthopedic-surgeon-kokomo-indianapolis.html |
| https://www.kneeandshouldersurgery.com/knee-disorders/posterior-cruciate-ligament-injuries/ |
| https://www.raleighsportsmed.com/pcl-injuries-dr-barker-orthopaedic-surgeon-cary-garner-nc.html |
| https://www.travisburnsmd.com/pcl-reconstruction-orthopaedic-surgeon-san-antonio-tx.html |
| https://lasportsorthomd.com/pcl-reconstruction-surgery-van-nuys-thousand-oaks-los-angeles-ca/ |
| https://www.drtoddmooreortho.com/pcl-reconstruction-orthopedic-surgeon-fort-worth-ennis-midlothian-tx.html |
| https://www.ortho-sa.com/pcl-reconstruction-orthopedic-surgeons-san-antonio-tx.html |
| https://www.arlingtonortho.com/conditions/knee/knee-torn-posterior-cruciate-ligament/ |
| https://www.bethesdaccsc.com/pcl-injuries-surgery-center-maryland.html |
| https://www.joionline.net/trending/content/what-recovery-time-pcl-tear |
| https://www.katyorthopaedics.com/pcl-reconstruction-dr-volkan-surgeon-fort-bend-cypress-tx.html |
| https://www.bonniegregorymd.com/pcl-reconstruction-orthopaedic-surgeon-houston-sugar-land-fort-bend-tx.html |
| https://www.waremd.com/pcl-reconstruction-orthopedic-surgeon-hartford-ct.html |
| https://yorkshirekneeclinic.com/knee-conditions/complex-knee-ligament-injuries/ |
| https://www.thecambridgeknee.co.uk/pcl-reconstruction/ |
| https://www.michellewolcottmd.com/pcl-reconstruction-sports-medicine-specialist-denver-lonetree.html |
| https://www.angelinaveramd.com/pcl-reconstruction-orthopedic-sports-medicine-specialist-las-vegas-summerlin-henderson.html |
| https://www.milwaukeeorthopaedics.com/pcl-reconstruction-milwaukee-orthopaedic-group-limited.html |
| https://www.posmc.com/specialties/knee-surgery/posterior-cruciate-ligament-pcl-reconstruction/ |
| https://toportho.com/patient-information/knee-information/knee-injury-diagnosis/grade-iii-pcl-tear/ |
| https://srosm.com/orthopedic-specialties/knee/knee-treatments/pcl-reconstruction/ |
| https://rothmanortho.com/specialties/treatments/posterior-cruciate-ligament-pcl-reconstruction |
| https://www.loudounsportsmedicine.com/pcl-reconstruction-shoulder-knee-sports-medicine-specialist-va.html |
| https://www.orthonorcal.com/pcl-injuries-orthopedic-specialists-los-gatos-morgan-hill.html |
| https://www.sportssurgerynewyork.com/posterior-cruciate-ligament-pcl-reconstruction-sports-medicine-specialist-ny.html |
| https://www.toddparrymd.com/pcl-reconstruction-orthopedic-surgeon-utah-mesquite-nevada.html |
| https://www.gomberamd.com/pcl-tears.html |
| https://www.vanthielmd.com/conditions-treatments/knee-surgery/pcl-tear-surgery/ |
| *https://www.humpalphysicaltherapy.com/Injuries-Conditions/Knee/FAQs/Is-it-possible-to-rehab-a-PCL-injury-and-still-remain-active/a~6593/article.html |
| https://www.davidhartiganmd.com/pcl-reconstruction-orthopedic-sports-medicine-specialist-minneapolis-mn.html |
| https://www.shouldersandknees.com/posterior-cruciate-ligament-pcl-reconstruction/ |
| https://kneeandshoulderclinic.com.au/knees/surgical-conditions/posterior-cruciate-ligament-tear/ |
| https://www.kelechiokorohamd.com/pcl-injuries-orthopaedic-surgeon-sports-medicine-minneapolis-st-paul-rochester-mn.html |
| https://www.ryanfadermd.com/pcl-reconstruction-orthopedic-surgeon-minneapolis.html |
| https://drsunilreddy.com.au/arthroscopic-knee-surgery-and-ligament-reconstruction/pcl-reconstruction |
| https://www.billrobertsonmd.com/pcl-reconstruction-william-j-robertson.html |
| https://www.jorgechahlamd.com/knee/pcl-tear-treatment-options-chicago-il/ |
*: results that were removed from Google search for Qualitative Content Tabulation, Understandability and Actionability
References
- 1.Logan CA, Beaulieu-Jones BR, Sanchez G, et al. Posterior Cruciate Ligament Injuries of the Knee at the National Football League Combine: An Imaging and Epidemiology Study. Arthroscopy. 2018;34(3):681–686. doi: 10.1016/j.arthro.2017.08.304. [DOI] [PubMed] [Google Scholar]
- 2.Majewski M, Susanne H, Klaus S. Epidemiology of athletic knee injuries: A 10-year study. Knee. 2006;13(3):184–188. doi: 10.1016/j.knee.2006.01.005. [DOI] [PubMed] [Google Scholar]
- 3.Owesen C, Sandven-Thrane S, Lind M, Forssblad M, Granan LP, Årøen A. Epidemiology of surgically treated posterior cruciate ligament injuries in Scandinavia. Knee Surg Sports Traumatol Arthrosc. 2017;25(8):2384–2391. doi: 10.1007/s00167-015-3786-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Seurer AC, Vogt HB. Low health literacy: a barrier to effective patient care. S D Med. 2013;66(2):51, 53–57. [PubMed] [Google Scholar]
- 5.Swenson DM, Collins CL, Best TM, Flanigan DC, Fields SK, Comstock RD. Epidemiology of knee injuries among U S high school athletes, 2005/2006-2010/2011. Med Sci Sports Exerc. 2013;45(3):462–469. doi: 10.1249/MSS.0b013e318277acca. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Montgomery SR, Johnson JS, McAllister DR, Petrigliano FA. Surgical management of PCL injuries: indications, techniques, and outcomes. Curr Rev Musculoskelet Med. 2013;6(2):115–123. doi: 10.1007/s12178-013-9162-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Broom A. Virtually He@lthy: The Impact of Internet Use on Disease Experience and the Doctor-Patient Relationship. Qual Health Res. 2005;15(3):325–345. doi: 10.1177/1049732304272916. [DOI] [PubMed] [Google Scholar]
- 8.Fox NJ, Ward KJ, O’Rourke AJ. The ‘expert patient’: empowerment or medical dominance? The case of weight loss, pharmaceutical drugs and the Internet. Soc Sci Med. 2005;60(6):1299–1309. doi: 10.1016/j.socscimed.2004.07.005. [DOI] [PubMed] [Google Scholar]
- 9.Marton C, Wei Choo C. A review of theoretical models of health information seeking on the web. Journal of Documentation. 2012;68(3):330–352. [Google Scholar]
- 10.Oh HJ, Lee B. The Effect of Computer-Mediated Social Support in Online Communities on Patient Empowerment and Doctor–Patient Communication. Health Communication. 2012;27(1):30–41. doi: 10.1080/10410236.2011.567449. [DOI] [PubMed] [Google Scholar]
- 11.Diaz JA, Griffith RA, Ng JJ, Reinert SE, Friedmann PD, Moulton AW. Patients’ use of the Internet for medical information. J Gen Intern Med. 2002;17(3):180–185. doi: 10.1046/j.1525-1497.2002.10603.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Gualtieri LN. The doctor as the second opinion and the internet as the first InCHI'09 Extended Abstracts on Human Factors in. Computing Systems. 2009:2489–2498. [Google Scholar]
- 13.Kivits J. Informed Patients and the Internet: A Mediated Context for Consultations with Health Professionals. J Health Psychol. 2006;11(2):269–282. doi: 10.1177/1359105306061186. [DOI] [PubMed] [Google Scholar]
- 14.Weiss BD. “Health Literacy: A Manual for Clinicians,” American Medical Association. Chicago: American Medical Foundation; 2003. [Google Scholar]
- 15.Wynia MK, Osborn CY. Health Literacy and Communication Quality in Health Care Organizations. J Health Commun. 2010;15(Suppl 2):102–115. doi: 10.1080/10810730.2010.499981. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011;155(2):97–107. doi: 10.7326/0003-4819-155-2-201107190-00005. [DOI] [PubMed] [Google Scholar]
- 17.Nielsen-Bohlman L, Panzer AM, Kindig DA. Health Literacy: A Prescription to End Confusion. 1st ed. Institute of Medicine (US) Committee on Health Literacy: National Academies Press (US); 2004. [PubMed] [Google Scholar]
- 18.Kim H, Xie B. Health literacy in the eHealth era: A systematic review of the literature. Patient Educ Couns. 2017;100(6):1073–1082. doi: 10.1016/j.pec.2017.01.015. [DOI] [PubMed] [Google Scholar]
- 19.Dewalt DA, Berkman ND, Sheridan S, Lohr KN, Pignone MP. Literacy and health outcomes: a systematic review of the literature. J Gen Intern Med. 2004;19(12):1228–1239. doi: 10.1111/j.1525-1497.2004.40153.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Kilfoyle KA, Vitko M, O’Conor R, Bailey SC. Health Literacy and Women’s Reproductive Health: A Systematic Review. J Womens Health (Larchmt) 2016;25(12):1237–1255. doi: 10.1089/jwh.2016.5810. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Miller TA. Health literacy and adherence to medical treatment in chronic and acute illness: A meta-analysis. Patient Educ Couns. 2016;99(7):1079–1086. doi: 10.1016/j.pec.2016.01.020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Morrison AK, Glick A, Yin HS. Health Literacy: Implications for Child Health. Pediatr Rev. 2019;40(6):263–277. doi: 10.1542/pir.2018-0027. [DOI] [PubMed] [Google Scholar]
- 23.Rosenbaum AJ, Pauze D, Pauze D, et al. Health Literacy in Patients Seeking Orthopaedic Care: Results of the Literacy in Musculoskeletal Problems (LIMP) Project. Iowa Orthop J. 2015;35:187–192. [PMC free article] [PubMed] [Google Scholar]
- 24.Dubay WH. The principles of readability. Costa Mesa, CA: Impact Information; 2004. [Google Scholar]
- 25.Assess, Select, and Create Easy-to-Understand Materials: Tool #11. Health Literacy Universal Precautions Toolkit. [August 30, 2023]. Available at: https://www.ahrq.gov/health-literacy/improve/precautions/tool11.html.
- 26.Doinn TÓ, Broderick JM, Abdelhalim MM, Quinlan JF. Readability of Patient Educational Materials in Hip and Knee Arthroplasty: Has a Decade Made a Difference? J Arthroplasty. 2020;35(11):3076–3083. doi: 10.1016/j.arth.2020.05.076. [DOI] [PubMed] [Google Scholar]
- 27.Gao B, Shamrock AG, Gulbrandsen T, Femino JE, Netto CD, Duchman KR. Achilles Tendon Repair: Can Patients Read, Understand, and Act on Online Educational Resources? Foot Ankle Orthop. 2020;5(4):2473011420S00224. [Google Scholar]
- 28.Gao B, Shamrock AG, Gulbrandsen TR, et al. Can Patients Read, Understand, and Act on Online Resources for Anterior Cruciate Ligament Surgery? Orthop J Sports Med. 2022;10(7):23259671221089977. doi: 10.1177/23259671221089977. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Mehta MP, Swindell HW, Westermann RW, Rosneck JT, Lynch TS. Assessing the Readability of Online Information About Hip Arthroscopy. Arthroscopy. 2018;34(7):2142–2149. doi: 10.1016/j.arthro.2018.02.039. [DOI] [PubMed] [Google Scholar]
- 30.Bianchi T. Global search engine desktop market share 2023. Statista; Available at: https://www.statista.com/statistics/216573/worldwide-market-share-of-search-engines / [Google Scholar]
- 31.Iden K. How Far Down the Search Results Page Will Most People Go? [Accessed June 3, 2023]. https://www.theleverageway.com/blog/how-far-down-the-search-engine-results-page-will-most-people-go /
- 32.Meyer C. Results in Google Get Almost 70% of All Clicks. Advance Metrics. [Accessed June 3, 2023]. Available at: https://www.advance-metrics.com/en/blog/the-top-5-results-in-google-get-almost-70-of-all-clicks /
- 33.Robbins Ron. What is the Average Google Click Through Rate by Position - Clicta Digital. clicta digital. [Accessed June 3, 2023]. Available at: https://clictadigital.com/what-is-the-average-google-click-through-rate-by-position.
- 34.Williams AM, Muir KW, Rosdahl JA. Readability of patient education materials in ophthalmology: a single-institution study and systematic review. BMC Ophthalmology. 2016;16(1):133 . doi: 10.1186/s12886-016-0315-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Josey M, Gaid D, Bishop LD, Blackwood M, Najafizada M, Donnan JR. The Quality, Readability, and Accuracy of the Information on Google About Cannabis and Driving: Quantitative Content Analysis. JMIR Infodemiology. 2023;3:e43001. [Google Scholar]
- 36.Mbanda N, Dada S, Bastable K, Ingalill GB, Ralf W S. A scoping review of the use of visual aids in health education materials for persons with low-literacy levels. Patient Educ Couns. 2021;104(5):998–1017. doi: 10.1016/j.pec.2020.11.034. [DOI] [PubMed] [Google Scholar]
- 37.Rooney MK, Santiago G, Perni S, et al. Readability of Patient Education Materials From High-Impact Medical Journals: A 20-Year Analysis. J Patient Exp. 2021;8:2374373521998847. doi: 10.1177/2374373521998847. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Dixon S. Biggest social media platforms 2023. Statista. [Accessed June 3, 2023]. Available at: https://www.statista.com/statistics/272014/global-social-networks-ranked-by-number-of-users /