Total Knee Arthroplasty: A Quantitative Assessment of Online Patient Education Resources

Abstract

Background

Patients often turn to the online resources to learn about orthopedic procedures. As the rate of joint arthroplasty is projected to increase, the corresponding interest in relevant online education material will increase as well. The American Medical Association (AMA) and National Institutes of Health (NIH) recommend that publicly available online health information be written at the 6th grade or lower reading level to be fully understood by the average adult in the United States. Additionally, educational resources should be written such that readers can process key information (understandability) or identify available actions to take (actionability). The purpose of this study was to quantify the readability, understandability, and actionability of online patient educational materials regarding total knee arthroplasty (TKA).

Methods

The most common Google™ search term utilized by the American public was determined to be “knee replacement”. Subsequently two independent online searches (Google.com) were performed. From the top 50 search results, websites were included if directed at educating patients regarding TKA. Non-text websites (audiovisual), articles (news/research/industry), and unrelated resources were excluded. Readability was quantified using the following valid objective algorithms: Flesch-Kincaid Grade-Level (FKGL), Simple Measure of Gobbledygook (SMOG) grade, Coleman-Liau Index (CLI), and Gunning-Fog Index (GFI). PEMAT was utilized to assess understandability and actionability (0-100%; score ≥70% indicates acceptable scoring). The relationship between search rank with FKGL and PEMAT scores was quantified.

Results

A total of 34 (68%) unique websites met inclusion criteria. The mean FKGL, SMOG, CLI, and GFI was 11.8±1.6, 11.1±1.2, 11.9±1.4, and 14.7±1.6, respectively. None of the websites scored within the acceptable NIH/AMA recommended reading levels. Mean understandability and actionability scores were 54.9±12.1 and 30.3±22.0. Only 5.9% (n=2) and 9.2% (n=1) of websites met the ≥70% threshold for understandability and actionability. Only 29.4% (n=10) sources used common language and only 26.9% (n=9) properly defined complicated medical terms. Based on website type, the mean understandability scores for academic institution, private practice, and health information publisher websites were 57.2±8.8%, 52.6±11.1%, and 54.3±15.3% (p=0.67). Readability (rho: -0.07; p=0.69), understandability (rho: -0.02; p=0.93), and actionability (rho: -0.22; p=0.23) scores were not associated with Google™ search rank.

Conclusion

TKA materials scored poorly with respect to readability, understandability, and actionability. None of the resources scored within the recommended AMA/NIH reading levels. Only 5.9% scored adequately on understandability measures. Substantial efforts are needed to improve online resources to optimize patient comprehension and facilitate informed decision-making.

Level of Evidence: III

Introduction

Osteoarthritis (OA) of the knee is a significant health problem, with up to 14 million people in the United States (US) reporting symptoms.¹ OA is a leading cause of disability, as joint pain can produce sequelae, including depressed mood, poor sleep, and loss of independence.^1,2 The volume of primary total knee arthroplasties in the US is projected to have an 85% increase by the year 2030, surmounting to approximately 1.26 million cases per year.³

When considering an elective surgical procedure, patients often turn to the internet for additional information. In 2019, 90% of US adults utilized the internet, with 72% of adults accessing the internet for health information.⁴ This includes the TKA specific population, consisting of older patients who are at increased risk for lower health literacy.⁵ However, despite online patient education materials being recognized as an important component of health literacy,⁶ studies continue to demonstrate variable quality and validity of online information. Furthermore, inconsistent, and poorly written material may negatively impact understanding, shared decision-making, and overall outcomes.^5,7-11

The American Medical Association (AMA) and National Institutes of Health (NIH) recommend that publicly available online health information be written at the 6th grade or lower reading level to be fully understood by the average adult in the United States. Currently the quality of TKA patient education materials is poorly understood. Past literature has investigated the readability of TKA online patient education materials.^12-15 However, readability assessments are subjected to linguistics, syllables, and syntax, which limits their measuring ability to effectively assess a resource’s capacity to convey data such that readers can process and understand the information. This limitation has been previously recognized and the Patient Educational Materials Assessment Tool (PEMAT) was developed to assess the ability of readers to process and explain key messages (understandability) and identify what they can do based on the information presented (actionability).⁶ The aim of this study was to utilize PEMAT and validated readability algorithms to quantify the readability, understandability, and actionability of online TKA patient education resources.⁶ We hypothesize that the existing TKA resources will score favorably on the readability, understandability, and actionability metrics.

Methods

Educational Material Identification

The most common TKA search term was determined by utilizing Google™ trends (trends.google.com).¹⁶ This trend analysis tool collects the individual Google™ search inputs in the United States, and normalizes the one-year search data. Subsequently, terms can be compared resulting in geographic and quantity-based values ranked 0-100 based on a well designed Google™ algorithm. A value of 100 indicates peak popularity of the term. The following terms were compared from 2/2019-2/2020: “total knee arthroplasty“, “total knee surgery“, “total knee replacement“, “knee arthroplasty“, and “knee replacement“. Google™ reported “knee replacement” was 28.1 times more likely searched by the general public compared to “total knee replacement”, the second most common TKA input. (Figure 1)

Figure 1.

Trends based on google search terms formulated by trends.google.com. Value ranked 0-100 based on Google™ algorithm. A value of 100 indicates peak popularity of the term. Study terms demonstrated that “knee replacement” was 28.1 times more likely to be searched by the general public compared to “total knee arthroplasty”.

The Google™ search engine was the search engine of choice because at the time of this study Google™ searches comprised a majority (88-92%) of the internet search market share.^17,18 Two searches of the term “knee replacement” were independently performed by two reviewers on 2/17/20 (TRG) and 2/18/20 (MKS). Prior studies have reported that approximately 70% or more “clicks” originate from the first 10-50 search results,^19-21 with, previous PEMAT studies targeting the first 10 to 50 websites.^22-26 Therefore, the searches were entered to imitate real user experience. From the independent searches, each reviewer subsequently recorded the first 50 websites in order of search result.

The two sets of search results were consolidated, and duplicates were removed. Each website was meticulously assessed with strict inclusion and exclusion criteria applied. Inclusion criteria consisted of websites that contained primary content educational information focused on total knee arthroplasty. Exclusion criteria included news articles, primarily audio-visual resources, personal experiences/blogs, references specifically written for health care professionals, peer-reviewed journal articles, advertisements of product or service without patient education, articles unrelated to TKA and articles not directed at patients as the primary consumer. Primarily audiovisual resources were excluded because these could not undergo readability analysis.

Content Analysis

Content analysis was performed on every included resource via two qualitative reviews consisting of the following categories: i) discussion of operative management ii) discussion of non-operative management iii) advertisement of a physician or group who provided the described management iv) discussion of general background information of the disease (anatomy, pathology, prognosis, risk factors) v) discussion of work-up or activities related to diagnosis and/or preoperative management vi) discussion of postoperative management vii) discussion of complications and/or risks of operative management. For this study, an advertisement was defined as any website that included information directing the reader towards a specific institution or group for management of TKA.

Subgroup analysis based on type of practice that created the resource was performed. This included the following groups: academic institution, private practice, or health publisher.

Statistical Analysis

Readability

Objective quantification of the patient education resources was performed by employing the following validated algorithms: Flesh-Kincaid Grade Ease (FKGE), Flesch-Kincaid Grade-Level (FKGL), Simple Measure of Gobbledygook (SMOG) grade, Coleman-Liau Index (CLI), Gunning-Fog Index (GFI), and Automated Readability Index (ARI). These algorithms were accessed using an open-source readability software (https://webfx.com/tools/read-able/check.php). A larger numerical FKGE score signifies the text material is grammatically simpler to read. FKGL, SMOG grade, CLI, GFI, and ARI readability scores represent grade level per the United States education system. Selected algorithms have been utilized in previous studies to assess the readability of other surgical online materials.^{10,22,23,27-29} Text unrelated to patient education, including copyright, references, and links independent of the main text were excluded from the readability analysis.

Understandability and Actionability

Understandability and actionability of each website were analyzed by applying the validated Patient Education Material Assessment Tool (PEMAT), a reliable instrument from the Agency for Healthcare Research and Quality (AHRQ).^6,30,31 The PEMAT tool specifies independent understandability and actionability scores for each educational material on a scale from 0-100%. A larger percentage represents a higher level of understandability or actionability for the reader. The PEMAT developers have established a threshold of 70% as the minimum score required for a resource to have adequate actionability and understandability.⁶ Therefore, a resource having a PEMAT score less than 70% is considered poorly understandable or poorly actionable.⁶ Two reviewers (MKS, TRG), individually conducted separate understandability and actionability analysis on the included websites using the PEMAT-P form.^6,30-32 As previously applied by the PEMAT developers,^6,31 interrater reliability was calculated using Cohen’s Kappa.

Search Rank Analysis

Google search rank was averaged from two independently conducted searches of “knee replacement”. Spearman’s rho was used to assess the correlation between website search rank and its readability, understandability, and actionability. Statistical significance was defined as p<0.05.

Results

Following the two independent searches and the removal of duplicate websites, a total of 52 unique online materials were identified, while 34 (65.4%) websites ultimately met the inclusion criteria. Three (5.8%) websites were excluded as primary literature, five (9.6%) were primarily audiovisual materials, and seven (13.5%) were excluded as news articles, two (3.8%) websites were excluded as primarily advertisements, and one (1.9%) was excluded as other/miscellaneous.

Of the 34 included online educational resources, 15 (44.1%) included background information (anatomy, pathology, prognostic factors), 15 (44.1%) discussed nonoperative management and 31 (91.1%) discussed operative management. Less than half the websites (n=15, 44.1%) discussed the preoperative workup of TKA, while 29 (85.3%) discussed the postoperative course. Risks and complication of operative management were discussed in 21 (61.8%) of the websites. Overall, 17 (50.0%) included an advertisement for a physician or group who provided the described management. Based on website type: 13 (38.2%) were from an academic institution, 12 (35.3%) from private practice, and 9 (26.5%) from a health publisher.

Readability

The mean FKGE was 45.9±8.3. Readability grade-levels per score ranged from 10.7 to 14.2. The mean FKGL, SMOG, CLI, GFI, and ARI representing grade level were 11.8±1.6, 11.1±1.2, 11.9±1.4, 14.7±1.6 and 12.1±1.7, respectively (Table 1). None (n=0) of the websites scored within the AMA/NIH recommended levels (≤ 6th grade reading level). (Table 1)

Table 1.

Flesch-Kincaid Grade Ease

Score	School Level	Interpretation	Number of Websites (n, %)
100-90:	5th Grade	Easy to read and understand	0 (0%)
90-80:	6th Grade	Easy for conversational English consumers	0 (0%)
80-70:	7th Grade	Fairly easy to read	0 (0%)
70-60:	8th/9th Grade	Understood by most 13–15-year-olds	1 (2.9%)
60-50:	10th/12th Grade	Fairly difficult to read	8 (23.5%)
50-30:	College	Difficult to read	21 (61.8%)
30-0:	College Graduate	Very difficult to read (University graduate level)	4 (11.8%)

Understandability and Actionability

Mean understandability and actionability scores were 54.9±12.1 and 30.3±22.0. Only 2 (5.9%) websites met the threshold for adequate (≥70%) understandability. Only 1 (2.9%) met the threshold for actionability. (Figure 2) Interrater reliability demonstrated substantial agreement (kappa = 0.80+/-0.002).

Figure 2.

Knee Replacement Patient Educational Material PEMAT Scores

There were 10 (29.4%) sources that used common language and 12 (35.3%) defined complicated medical terms. The most frequently missed understandability criteria was the lack of a comprehensive summary with 30 (88.2%) missing this valuable component. The second most frequently missed criteria was a lack of clear titles/ captions (n=26, 76.5%). While 94.1% (n=32) scored well regarding layout and design, only 73.5% (n=25) of websites used visual aids. Additionally, under word choice and style, only 10 (29.4%) websites used common, everyday language and only 9 (26.9%) appropriately used and defined medical words.

Based on website type, the mean understandability scores for academic institution, private practice, and health information publisher websites were 57.2±8.8%, 52.6±11.1%, and 54.3±15.3% (p=0.67), respectively. (Figure 3)

Figure 3.

Mean Understandability per Website Category

Search Rank

Google search rank was not associated with the online material’s readability (rho: -0.07; p=0.69), understandability (rho: -0.02; p=0.93), and actionability (rho: -0.22; p=0.23) scores.

Discussion

This study investigated the readability, understandability, and actionability of online resources regarding the diagnosis and treatment of total knee arthroplasty. While readability assessments on TKA have been performed previously, this is the first study to utilize the validated PEMAT algorithm to commonly accessed online TKA patient education materials.^12-15 Our results demonstrated that these resources were not sufficiently formulated for the American public based on readability, understandability, and actionability measures. Additionally, this study demonstrated no difference in measures between academic institution, private practice, or health information publisher websites. Overall, the findings of this study corroborates with other studies, confirming an existing concern on the lack of online patient materials that are both readily accessible to patients.^6,33-41 As there was no association between Google rank and any of the measures utilized in this study, patients must be made aware that “top hits” are not necessarily equivalent with highest quality or utility.

Though there are concerns about the quality and reliability of health information found online,^{6,32,33,35,37,40-47} patient education material websites have been increasingly recognized as a crucial part of health literacy in the internet era.^6,46,47 Health literacy is the capacity to “obtain, process and understand basic health information and services needed to make appropriate health decisions.”⁴⁸ While personal health literacy is a common public health concern,⁵ it has been shown to have detrimental impacts on cost as well as patient outcomes including adherence, length of stay, and complication rates.^6,7,12,49,50 Therefore, it is crucial to investigate and emphasize the critical importance of health literacy in surgical specialties, especially those involving elective procedures.

Previous studies have demonstrated that health literacy impacts patient expectations following orthopedic procedures.^49-51 Hadden et al. investigated the effect of health literacy by evaluating 200 patients who underwent a total joint arthroplasty procedure (THA/TKA). They reported that patients with higher health literacy reported higher expectations for walking following the arthroplasty procedure, whereas patients with lower health literacy reported lower expectations.⁵⁰ Mendenez and colleagues reviewed audio-recordings of 84 upper extremity clinic visits. They found that patients with limited health literacy were less likely to engage in question-asking behavior during the visits with the surgeon.⁵²

Considering best practices to optimize organizational health literacy is crucial in the orthopaedic field. This is especially true in patients undergoing TKA, as this patient population is consistently older, and therefore at a higher risk of not fully understanding basic health information and services.^46,50,53 Furthermore, limited health literacy results in an increased risk for the inability to make appropriate health decisions both preoperatively and postoperatively. As increasing numbers of older individuals access the internet for health information, organizations will need to strive to provide accurate, accessible, and high-quality online patient education materials that are easy to read and understand by the public.⁵

This current study demonstrated that the included websites scored inappropriately above the NIH and AMA’s recommended reading level (6th grade reading level or lower).^12-15 Utilizing common readability index tools, none of the included websites were written below a 6th grade reading level, with all being at the 10th grade or higher. Previous studies analyzing the readability of TKA materials have reported similar results with readability scores too advanced for the general public.^13,15,54 A study by Schnaekel et al. found that the mean FKGL of nine arthroplasty materials were written at an 11th grade level.¹³ Similarly, Bahadori et al. investigated the readability of arthroplasty-related smartphone apps.¹⁵ The mean FKGL level was 9.7, with only 3 apps (20%) meeting the NIH/AHA readability threshold. Finally, a study investigating the readability of online patent education materials from major implant manufacturers found that the mean grade level was 11.3, with only 2.2% (n=13) of articles at the recommended 6th grade or lower reading level.¹²

While readability instruments measure the complexity of the vocabulary and syntax, it does not directly measure the understandability and actionability. Importantly, the ability to process key information (understandability) and identify key actions (actionability) are both crucial to the functional application of health literacy. Therefore, the PEMAT was established to compensate for this limitation of previous instruments utilized to measure patient education materials. Utilizing this reliable and valid instrument with a threshold of 70% for a text to be considered adequately understandable and actionable by readers.⁵ Only 5.9% (n=2) of the included TKA materials met the threshold for understandability while only 2.9% (n=1) met the threshold for actionability. These scores are similar to those reported in other medical and surgical subspecialities.^{10,22,23,27-29}

There has been a lack of literature comparing academic, private practice, and health publisher patient educational resources. Rozental and colleagues published a study that compared academic and private practice patient educational resources. They reported that although private practice websites were more complete and informative compared to academic websites, both lacked in many informative and patient education aspects.⁵⁵ However, they did not investigate if the resources were suitable to the general public or if they were designed in order for patients to understand the material and take action when necessary. This current study demonstrated no difference in PEMAT understandability scores.

The TKA patient resources included in this study have several themes when evaluating the understandability criteria that was missed. These included missing summaries, lack of visual aids, and unclear titles. Frequently missed actionability criteria included failing to address the patient directly, failing to breakdown instructions into explicit steps, and failing to provide a tangible action tool, such as a checklist. To address these deficits, website authors should consider incorporating PEMAT guidelines (accessible at https://www.ahrq.gov/ncepcr/tools/self-mgmt/pemat2.html) to ensure development of patient-appropriate resources.

Limitations

There are several limitations that should be considered. Due to constant algorithm variabilities, Google™ search results consistently change; therefore, the top 50 websites could be different at various times and geographic search locations. To minimize variability, the authors cleared all cookies and cache prior to the search. Another limitation includes the choice of search engine, search term, and country of origin, as these factors can influence the provided search results. However, the authors utilized the most common search engine with the most common term searched by the public. Limitations with readability measures should be addressed as well. The readability scores can be skewed by certain healthcare vocabulary. Words with increased characters, including “arthroplasty”, can inherently increase the grade level of the content. Therefore, this aspect may inflate all the grading scores used in this study. However, readability is known to have its limitation in all healthcare and medical content.⁵⁶ Another limitation is the subjectivity of the PEMAT grading and implicit bias could not be fully eliminated. To limit this bias and subjectivity, two authors independently performed the grading, which demonstrated substantial interrater reliability, consistent with prior studies utilizing PEMAT.³¹

Conclusion

Overall, total knee arthroplasty online patient educational materials scored poorly with respect to readability, understandability, and actionability. None of the online resources scored at the AMA and NIH recommended reading level. Additionally, only 5.9% (n=2) and 2.9% (n=1) of websites met the threshold for adequate (≥70%) understandability and actionability. Optimization of the most accessible TKA websites is necessary and special attention on simplifying and/or defining key medical terms. Providers, health care institutions, and medical societies should lead this effort, as ensuring accessible, high-quality materials is a crucial component of patient care and overall outcomes. Additionally, while studies have demonstrated an association between mass media and health behavior,⁵⁷ further research is required to interpret the impact of online patient education materials on arthroplasty related health behaviors and treatment decision-making.