Factors Associated with the Quality of Online Information about Scapholunate Interosseous Ligament Insufficiency

Abstract

Background The Internet is increasingly used by patients to seek health information about their medical conditions. The online information is of variable quality, often difficult to read, and sometimes inaccurate or misleading. This study assessed factors associated with the quality, readability, and dominant tones of online information about scapholunate interosseous ligament (SLIL) insufficiency.

Materials and Methods Using the three most used search engines, we entered the terms “wrist sprain,” “scapholunate ligament injury,” and “SL dissociation” and assessed the quality of the 45 Web sites identified using the DISCERN tool, readability by the Flesch Reading Ease Score, the Flesch–Kincaid Grade Level, the Gunning Fog Index, and the Simple Measure Of Gobbledygook, and dominant tones using the IBM Watson Tone Analyzer and the Linguistic Inquiry and Word Count.

Results Online information about SLIL injuries had a mean DISCERN score of 39 ± 8.2. A dominant Web site tone of “sadness” correlated with lower DISCERN scores. A dominant tentative tone in text was associated with easier to comprehend texts.

Conclusion The online information regarding SLIL insufficiency is of generally low quality, limited readability, and the underlying tones may be misleading. Professional societies might consider efforts to provide appealing, readable, information about SLIL insufficiency and other less common diagnoses on the Internet.

Introduction

Rupture of the scapholunate (SL) interosseous ligament (SLIL) is one of the most common causes of traumatic disruptions of carpal kinematics. 1 The altered kinematics make the wrist vulnerable to a specific pattern of osteoarthritis known as SL advanced collapse (SLAC). 2 3 4 5 Several operations for SLIL injury are offered to reduce painful clicking and to delay arthritis by attempting to improve carpal alignment, for example, ligament repair or reconstruction, capsulodesis, or tenodesis. 6 7 8 9 Operative treatment has inconsistent results for improving carpal malalignment, stiffens the wrist, introduces operative risks such as infection and errant implant, does not clearly improve symptoms better than sham surgery or alternative management strategies, and may not alter the natural history of SLAC. 10 11 The many areas of debate in the diagnosis and optimal treatment of SL ligament instability create complicated treatment decisions for patients and difficult discussions for the clinicians that care for them.

Accurate and trustworthy health information can help patients make decisions based on their values and not on misconceptions. The Internet is one of the most common places people seek health information. 12 Online information has variable quality of content, can be difficult to read, and are often inaccurate or misleading. 13 Prior studies that analyzed carpal tunnel syndrome 14 and distal radius fracture 15 on Web sites report low value and poor quality information, and exceed the recommended sixth grade reading level. 16 The online available information for less common problems such as SLIL insufficiency might be even more problematic.

It can be helpful for clinicians to know what patients read online so that they can understand the influences on patient explanatory model and be prepared to address misconceptions. Little is known about the quality of online information regarding SLIL insufficiency. This study tested the hypothesis that there are no factors independently associated with SLIL insufficiency in online information content quality, measured using the DISCERN instrument. 17 Additionally, we assessed factors associated with readability of text, measured using the Simple Measure Of Gobbledygook (SMOG), and present dominant tones in text, measured using the IBM Watson Tone Analyzer.

Materials and Methods

Data Source

This study is exempt from Institutional Review Board approval. On December 18, 2017, we entered the terms “wrist sprain,” “scapholunate ligament injury,” and “SL dissociation” into three of the most used search engines (Google, Yahoo!, and Bing). 18 We cleared search history and cookies and searched with the browser in privacy mode to assure prior searches did not influence our results. Only 7% of online health information seekers leave the first result page. 19 To obtain enough Web sites, we included all first 40 hits of each search engine, resulting in 360 potential eligible Web sites. After excluding duplicates ( n = 141) and irrelevant Web sites ( n = 174, e.g., unrelated to the disease, research journals, Web-shops, videos, etc.), 45 Web sites remained for further analyses.

Measurements

We recorded whether Web sites were for profit (commercial or from private physician clinics) or nonprofit (affiliated with a university or academic center, government-funded, or from nonprofit organizations), treatment options that were discussed (operative, nonoperative, both, or none), and if there was a bias in favor of a specific treatment. Additionally, we assessed if Web sites were certified with a Health On the Net (HON) code provided by the HON Foundation. 20 When Web sites meet their 8 principles (authority, complementarity, confidentiality, attribution, justifiability, transparency, financial disclosure, and advertising), a HON code seal on the Web page is accredited. 20 If this seal was not found on the original Web site, the HON code online database was consulted. 21

The quality of online information content was evaluated using the DISCERN instrument. 17 The DISCERN instrument is a validated questionnaire designed to assess the quality of written health information for patients ( Supplementary Table S1 , available in the online version). It consists of 16 questions, 8 on reliability of the publication, 7 on quality of treatment information, and 1 on overall rating. Each question is rated on a 5-point Likert scale ranging from 1 “no” to 5 “yes.” Total scores range from 0 to 80 with higher scores indicating greater overall Web site quality.

Readability was assessed using four validated and often used tools: the Flesch Reading Ease Score (FRES), 22 the Flesch–Kincaid Grade Level (FKGL), 23 the Gunning Fog Index (GFI), 24 and the SMOG 25 ( Supplementary Table S2 , available in the online version). We used a free online calculator to determine all readability ratings. 26 The FRES score ranges from 0 to 100, with higher scores indicating that a text is easier to comprehend. The FKGL reflects the required U.S. grade level to read and understand text. Scores range from 3 to 12 and lower scores indicate that the text is easier to read. The GFI gauges how many years of formal education is required to understand text with scores ranging from 0 to 15. The SMOG reflects how many years of education (ranging from 4 to 18) a person needs to understand a text. This readability tool is considered to be preferable in the biomedical field. 27 28 For this reason, we only used the SMOG tool for bivariate and multivariate analysis. Since readability tools rely on word and sentence count, we formatted all Web site texts prior to analyses to avoid any under- or overestimation of readability levels. We removed headings, sentence fragments, references, lists with bullets, and periods that did not mark the end of a sentence. Web site texts were copied and pasted in Microsoft Word (Microsoft Corp., Redmond, Washington, United States).

We used the IBM Watson Tone Analyzer 29 and the Linguistic Inquiry and Word Count (LIWC) 30 to assess dominant tones in text. The IBM Watson Tone Analyzer measures dominant tones in written texts (emotion, language style, and social tendencies) with scores ranging from 0 to 1.0. Tones with a score of < 0.5 are considered to be “unlikely present” in the text. Scores > 0.75 are considered to be “very likely present” in text. We copied text from Web sites into the tone analyzer to measure present tones of the Web site. The LIWC is a lexicon-based validated tool that gauges language used in text. 30 It reflects the percentage of words in text that can be categorized into one of the following categories: basic linguistic like articles, big words (words with more than 6 letters), and on psychological level like self-references, social words, positive emotion, negative emotion, and overall cognitive words.

Reliability

One reviewer rated all Web sites on DISCERN. A recent study conducted by the same research team showed almost perfect intraobserver reliability of the DISCERN tool, with intraclass correlation coefficients of 0.95 (confidence interval [CI] = 0.92–0.96; p ≤ 0.001). 31 In this same study, the interobserver reliability for DISCERN, measured by intraclass correlation, was 0.95 (CI = 0.91–0.98; p ≤ 0.001).

The IBM Watson Tone Analyzer was developed to compare tones in customer service conversations. IBM trained a machine-learning model based on a data set that was built on 96,000 customer conversations and the tone was rated by five trained annotators. 32 IBM states to achieve high accuracy when they compare their machine model to a benchmark data set, but they do not give actual numbers about reliability. 32 The LIWC showed high internal reliability scores for social processes (Crohnbach's α = 0.97), affective processes (Crohnbach's α = 0.97), and cognitive processes (Crohnbach's α = 0.97) when written essays were rated by LIWC as well as by independent judges. 33

Statistical Analysis

Continuous variables are reported as mean and standard deviation and categorical variables as number and percentage. To compare continuous and dichotomous variables, we used Student's t -test, for two continuous variables we used Pearson's correlation, and analysis of variance test was used to compare continuous and ordinal variables. We indented to create three multivariable regression models to identify factors associated with (1) online information content quality measured using DISCERN, (2) readability measured using SMOG, and (3) very likely present dominant tones in text measured using the IBM Watson Tone Analyzer.

A priori power analysis indicated that a sample size of 65 Web sites would provide 80% statistical power, with α = 0.05, for a regression with 7 independent variables our complete model would account for 20% of the overall variability in quality, and a single variable would account for 10% or more of the variability in quality.

We could only include 45 Web sites and because of this lack of statistical power we did not perform multivariable analysis as this could result in unreliable findings.

Web Site Characteristics

Of the 45 Web sites, 6 (13%) were HON code certified, 20 (44%) Web sites were nonprofit, 37 (82%) discussed both treatment options (operative and nonoperative), and 43 (96%) had no clear preference for treatment ( Table 1 ). The mean DISCERN score was 39 ± 8.2. Of all Web sites, 19 had a DISCERN score of ≥ 40 and 6 Web sites > 50. Mean readability score for FRES was 53 ± 12 (indicating “fairly difficult to read”), for FKGL it was 10 ± 2.5, for GFI it was 14 ± 2.7, and for SMOG it was 10 ± 1.9. Four Web sites were written below 7th grade level and required 7 years of education according to all readability tools. No Web site scored below 6th grade on all four readability tools. Tentative was the only very likely present dominant tone in text (0.82 ± 0.10). Mean LIWC score for negative emotions was 2.3 ± 1.1 and for positive emotions it was 1.2 ± 0.48.

Table 1. Web site characteristics.

Variables	Values
Abbreviations: FKGL, Flesch–Kincaid Grade Level; FRES, Flesch Reading Ease Score; GFI, Gunning Fog Index; HON code, Health On the Net code; LIWC, Linguistic Inquiry and Word Count; SMOG, Simple Measure of Gobbledygook. Note: Continuous variables as mean (± standard deviation); discrete variables as number (percentage).
Web sites	45
HON code, n (%)	6 (13)
Nonprofit, n (%)	20 (44)
Treatment options discussed, n (%)
None	2 (4.4)
Only nonoperative treatment	1 (2.2)
Only operative treatment	5 (11)
Both treatments	37 (82)
Clear preference for treatment, n (%)
None	43 (96)
Nonoperative treatment	1 (2)
Operative treatment	1 (2)
DISCERN	39 ± 8.2
Readability scores
FRES	53 ± 12
FKGL	10 ± 2.5
GFI	14 ± 2.7
SMOG	10 ± 1.9
IBM dominant tones
Fear	0.51 ± 0.012
Joy	0.53 ± 0.36
Sadness	0.61 ± 0.043
Analytical	0.67 ± 0.12
Tentative	0.82 ± 0.10
LIWC
Self-references	0.13 ± 0.22
Social words	3.1 ± 2.9
Positive emotion	1.2 ± 0.48
Negative emotion	2.3 ±1.1
Overall cognitive words	4.5 ± 1.6
Articles (a, an, the)	11 ± 2.4
Big words	26 ± 4.3

Results

DISCERN

In bivariate analyses, a dominant Web site tone of sadness was the solely significant factor associated with DISCERN scores ( r =–0.40, p = 0.0072) ( Table 2 ). Web sites containing a HON code scored slightly greater on DISCERN (42 ± 5.5, p = 0.27) than Web sites without a HON code (38 ± 8.5, p = 0.27), but this was not significant. Readability scores on GFI ( r =–0.25, p = 0.10) and SMOG ( r =–0.25, p = 0.10) were negatively but not significantly correlated with DISCERN. No multivariable analysis was performed due to our small sample size and a lack of more than one significant factor.

Table 2. Bivariate analyses.

Variables	DISCERN	p –Value	SMOG	p –Value	Tentative tone	p –Value
Abbreviations: FKGL, Flesch–Kincaid Grade Level; FRES, Flesch Reading Ease Score; GFI, Gunning Fog Index; HON code, Health On the Net code; LIWC, Linguistic Inquiry and Word Count; SMOG, Simple Measure of Gobbledygook. Note: Pearson's correlation indicated by r; boldfaced values indicate statistically significant difference.
DISCERN	–	–	–0.25	0.10	–0.099	0.53
HON code
Yes	42 ± 5.5	0.27	8.9 ± 2.0	0.072	0.88 ± 0.043	0.17
No	38 ± 8.5		10 ± 1.8		0.81 ± 0.11
Nonprofit
Yes	41 ± 9.3	0.13	10 ± 2.1	0.090	0.85 ± 0.098	0.10
No	37 ± 6.9		11 ± 1.5		0.80 ± 0.099
Treatment options discussed
None	27 ± 4.9	0.11	12 ± 0.07	0.065	0.79 ± 0.0	0.0016
Only nonoperative treatment	33 ± 0.0		13 ± 0.0		0.74 ± 0.0
Only operative treatment	44 ± 13		11 ± 2.1		0.66 ± 0.097
Both treatments	39 ± 7.2		9.9 ± 1.8		0.85 ± 0.086
Clear preference for treatment
None	39 ± 8.3	0.52	10 ± 1.9	0.34	0.82 ± 0.10	0.70
Nonoperative treatment	33 ± 0.0		13 ± 0.0		0.74 ± 0.0
Operative treatment	31 ± 0.0		10 ± 0.0		0.85 ± 0.0
Readability scores (r)
FRES	0.15	0.31	–0.95	< 0.001	0.47	0.0015
FKGL	–0.19	0.21	0.98	< 0.001	–0.42	0.0051
GFI	–0.25	0.10	0.99	< 0.001	–0.42	0.0060
SMOG	–0.25	0.10	–	–	–0.44	0.0034
IBM Watson Tone Analyzer (r)
Fear	0.064	0.96	1.0	0.015	–0.61	0.58
Joy	0.0086	0.99	0.23	0.61	–0.39	0.51
Sadness	–0.40	0.0072	0.042	0.79	–0.031	0.85
Analytical	–0.058	0.75	0.16	0.36	–0.0045	0.98
Tentative	–0.099	0.53	–0.44	0.0034	–	–
LIWC
Self–references	–0.0095	0.95	–0.16	0.30	–0.14	0.38
Social words	0.20	0.18	0.61	< 0.001	0.28	0.074
Positive emotion	0.19	0.22	–0.034	0.83	0.14	0.39
Negative emotion	0.031	0.84	–0.25	0.094	0.25	0.11
Overall cognitive words	0.057	0.71	–0.33	0.028	0.23	0.15
Articles (a, an, the)	–0.18	0.23	0.40	0.0072	–0.29	0.06
Big words	–0.080	0.60	0.77	< 0.001	–0.36	0.019

Readability

SMOG scores were negatively correlated with FRES ( r = –0.95, p ≤ 0.001) and positively correlated with FKGL ( r = 0.98, p ≤ 0.001) and GFI ( r = 0.99, p ≤ 0.001) scores. A dominant tone of “fear” ( r = 1.0, p = 0.015) and “tentative” ( r = –0.44, p = 0.0034), the use of social words ( r = 0.61, p ≤ 0.001), cognitive words ( r = –0.33, p = 0.028), articles used ( r = 0.40, p = 0.0072), and big words ( r = 0.77, p ≤ 0.001) were factors associated with the SMOG.

Dominant Tones

Dominant tentative tone in text was associated with easier readability levels (FRES [ r = 0.47, p = 0.0015], FKGL [ r = –0.042, p = 0.0051], GFI [ r = –0.42, p = 0.0051], SMOG ( r = –0.44, p = 0.0034], and use of big words ( r = –0.36, p = 0.019)].

Discussion

Patients increasingly search the Internet to obtain health information, but Web site texts can be misleading, of low quality, and sometimes inaccurate. 12 34 This study assessed factors associated with the quality of online information content regarding SLIL measured using the DISCERN instrument. Additionally, we assessed factors associated with readability levels, measured using SMOG, and present dominant tones in texts measured using the IBM Watson Tone Analyzer.

Our study has several limitations. First, besides a broad search with use of different search terms only 45 Web sites met our inclusion criteria. We could not include enough Web sites that addressed SLIL insufficiency to reach adequate power according to a priori power analysis. Among the excluded Web sites, 45% were peer-reviewed publications and 29% were written for health care professionals, for example, summaries for radiologist, and presentation sheets. Second, we evaluated Web sites written in English that were available on the day of our search, December 18, 2017. The findings of this study may not be reproducible within the next few years. Third, our search contained three terms but patients might use other terms, find other Web sites, and read other information than included in our study. Fourth, although DISCERN is a validated tool with specific rating criteria, it partly relays on subjective input. Fifth, readability tools are based on word and sentence count only. They do not account for Web site structure, layout, or illustrations—items that can enhance understanding.

The finding that a dominant tone of sadness was the only factor associated with DISCERN scores might be a result of the small sample size, and relative dearth of information regarding SLIL insufficiency on the Internet. If the existing information reinforces sadness rather than emphasizing adaptiveness, this material may do harm.

The finding that the SMOG tool was associated with the FRES, FKGL, and GFI shows the very strong correlation among all readability tools, although they use different formulas and scales. A dominant tone of fear correlated with difficult to read texts, and a dominant tentative tone with easier texts. It shows that there is variation among Web sites in word choice and concepts which can affect illness beliefs, symptom intensity, and magnitude of limitations. Additionally, we found that most Web sites were above the recommended reading levels. None of the Web sites addressing SL insufficiency met standards for readability with fewer than 6 years of education on all readability tools as advised by the National Institution of Health and the American Medical Association. This increases the risk of misunderstandings and misconceptions if information is too difficult to read or comprehend.

We found that a tentative tone was more likely present in Web sites that were easier to read. It shows that if Web sites use easier words they present their information in another way than more difficult Web sites. Word choice and emotional texts could influence a patients’ attitude and expectations toward an illness or treatments. However, little is known about the influence of sentiments in online information. Some recent studies in other medical fields than musculoskeletal illnesses assessed sentiments in online text. 35 36 37 A study about online cancer screening information found that colonoscopy information had a greater prevalence of negative sentiments than mammography. 36 Another study about Zika virus images on social media found that many images contained negative sentiments and induced fear. 37 Understanding dominant and present sentiments in information can give a useful insight in patients’ attitudes and expectations toward disease and treatment.

Less common diseases and illnesses may be at a greater risk for limited and lower quality information on the Internet. Professional societies might consider putting effort to provide appealing, readable, information about less common problems such as SLIL insufficiency on the Internet.

Note

This work was performed at the Dell Medical School–University of Texas at Austin.

Funding Statement

Funding None.