Down the Rabbit Hole: Evaluation of Internet Information Quality in Parathyroid and Thyroid Surgery

Abstract

Introduction:

Patients often discuss information obtained from Internet sources during clinic visits, which can be of variable quality and depth. We sought to review and assess information on the Internet regarding common operations within endocrine surgery.

Methods:

Using Google’s search engine, the top 100 websites from the search phrase “parathyroid surgery,” and the top 100 websites from the phrase “thyroid surgery” were identified. Each website was evaluated for accessibility, accuracy, and completeness of information about gland hormone function, associated disease processes, and surgery itself. Results were stratified based on the website type, and bivariate analysis was performed to determine accuracy by category. Presence of author credentials, last webpage update, and presence of advertisements were also assessed. Inter-rater reliability was calculated for each variable.

Results:

For parathyroid surgery, at least two-thirds of all websites included information about surgery, hormone function, and disease processes. For thyroid surgery, 71% of websites included procedure information, but only 52% included information about hormone function and 56% about disease processes. Less than 30% of all websites included advertisements and less than 25% listed author credentials or provided references. Academic or research-affiliated sources were most likely to have zero inaccuracies, but 44% of all websites had at least one potential inaccuracy. Inter-rater reliability achieved at least moderate agreement (>0.41) for 56% of variables.

Conclusions:

There is a wide array of information available to patients online, and accuracy varies based on multiple factors including the type of website. Endocrine surgeons and related practitioners must be cognizant of this fact when discussing treatment plans with patients.

Introduction

Beginning in the late 20th century, the Internet has become a source of health information for people across the globe. For almost any healthcare-related question a person may have, there are hundreds of articles online to provide a possible answer. Many patients also turn to the Internet to self-refer for surgical problems.^1,2 Within the past decade, most studies have estimated that 54%-97% of patients search the Internet for medical information about their diagnosis or treatment plan.^3–8 With the continued growth of Internet use and technology, the percentage of adults seeking medical information online has likely grown.

In many cases, it can be difficult for a patient to determine whether the information they find online is accurate.⁹ Several previous studies have evaluated the accuracy of information regarding specialty-specific topics in health care and have shown that misinformation is ubiquitous on the Internet.^10–12 Inaccurate information, when widely available, could potentially cause harm to patients who choose to make medical decisions based on information found on a website. While a website that offers medical information without providing author credentials or assurance that the data within is evidence-based may raise red flags for the medical community, these dangers may be less readily apparent to those outside of health care. Furthermore, in some cases, Internet misinformation could sow seeds of mistrust if the physician’s recommendations aren’t congruent with what is found on a website, and a patient chooses to believe the online source over their doctor.

Despite rising concerns about misinformation on the Internet, little is known about the prevalence of this problem related to thyroid and parathyroid disorders. Internet users searching for information about these common operations in endocrine surgery have likely been diagnosed with thyroid or parathyroid disease and may be searching for treatment options for their diagnoses, including the possibility of surgery. In this study, we sought to evaluate the accuracy of Internet information that would easily be viewed by a patient seeking to learn about thyroid or parathyroid surgery.

Methods

To evaluate the quality of Internet information, we used Google’s search engine to identify the top 100 websites which resulted from the search phrase “parathyroid surgery” and the top 100 websites from the phrase “thyroid surgery.” The web search was conducted in June 2021, and the link to each website was copied into a database for later retrieval in the event of search result changes. The Google location function was turned off, ensuring that results were generalizable beyond the physical location the search was performed. Websites were excluded if they required a login or subscription to view the information. When evaluating each website, reviewers did not deviate from the initial page which resulted from the search engine or follow any hyperlinks to additional tabs or outside websites to ensure repeatability.

Three separate evaluators visited each website and assessed whether it was accessible, along with a range of variables based on the conceptual framework from prior studies of Internet information reliability.¹³ Reviewers included three individuals at different stages of medical training, working in coordination with an American Association of Endocrine Surgeons (AAES) endocrine surgeon member. Websites were assigned one point for each framework component present on the page, including categorizations about both structure and content (Table 1). Websites did not receive credit for components which were absent. For each procedure, the degree of misinformation on each website was stratified categorically based on the page having zero, one, two, or three or more potential inaccuracies, and points were assigned accordingly. The scoring system was standardized for all websites, independent of the website source (i.e., academic, research-based, private clinic, and so on.) or the type of practice (endocrine surgery, general surgery, ear, nose and throat, and so on.) providing information. Since accuracy can be subjective, each evaluator determined the degree of misinformation by comparing it to information publicly available on the AAES patient information website,¹⁴ with review of any questions or discrepancies by an AAES endocrine surgeon member.

Table 1 –

Scoring system for evaluation of Internet misinformation.

Is the information available and complete?	Points
Is the website accessible?	1	0
Is information available about disease processes?	1	0
Is information available about hormone function?	1	0
Is information available about surgical treatment?	1	0
Are author credentials present?	1	0
Are references available?	1	0
Was the website updated in the past year?	1	0
Are advertisements present?	1	0
Is the information accurate?
0 inaccuracies	0
1 inaccuracy	1
2 or more	2
3 or more	3

For information about hormonal function of the thyroid and parathyroid glands, websites received a point if they even briefly mentioned the role of hormones such as thyroxine, thyroid-stimulating hormone, or parathyroid hormone . For disease processes, websites which mentioned thyroid cancers or functional thyroid disorders (such as Graves’ disease), and parathyroid hyperplasia, adenomas, or cancer received a point for that category. For the surgery itself, websites received a point if they included any basic procedural information such as placement of incision(s), open versus minimally invasive techniques, and which glands or portions of glands should be removed. Author credentials were included if they were listed as a health care professional (MD/DO, advanced practitioner, nurse, and so on.). A point was assigned for references if they cited textbooks, peer-reviewed journal articles, or links to widely accepted sources such as UpToDate, Medscape, the AAES, or the American Thyroid Association, among others. If the date of the last webpage update was not identifiable, the website did not receive a point for having been updated within the past year.

Following data collection, the degree of agreement between the three raters was evaluated by calculating inter-rater reliability (IRR) using the kappa coefficient for each variable. Additionally, a consensus score between the three evaluators was synthesized for each variable, in order to perform comparative analysis. Websites were also stratified into five categories based on type. Academic websites were defined as those affiliated with a school of medicine. Association or research-based websites included research bodies such as the National Institutes of Health or nonprofit associations raising awareness for disease such as the American Thyroid Association. Other categories included hospitals or health care systems, private clinics, and all other informational websites. Bivariate analysis was performed across categories using chisquare tests on the Stata 17 platform (College Station, TX). This project was deemed exempt from review by the institutional review board.

Results

A total of 100 websites were evaluated for both thyroid surgery and parathyroid surgery, with 100% of them being accessible. Parathyroid surgery websites were significantly more likely than thyroid websites to include information about hormone function and disease processes (77% versus 52% and 73% versus 56%, respectively, P < 0.05 for both). Slightly fewer parathyroid websites included procedural information (67%) compared to thyroid websites (71%). Only 25% of parathyroid websites listed author credentials (being a health care provider) and only 19% provided credible references, versus 23% and 25% for thyroid. Of note, 17% of parathyroid and 28% of thyroid websites displayed advertisements. Only 37% of parathyroid and 60% of thyroid websites had been obviously updated within the past year (Table 2). Of the top 100 websites for each type of procedure, 26 websites overlapped between parathyroid and thyroid search results and were included in both data sets. IRR ranged from 0.14 to 0.86 and achieved at least moderate agreement (>0.41) for 56% of variables, including the presence of advertisements, author credentials, references, and information about the procedure and hormone function. When evaluating website accuracy, IRR was fair for parathyroid data (0.24), and slight for thyroid data (0.14). All the IRRs were statistically significant.

Table 2 –

Presence of information within endocrine surgery.

Variable	Parathyroid (%)	Thyroid (%)	P-value
Content
Procedure information	67	71	0.54
Hormone information	77	52	0.002
Disease information	73	56	0.01
Accuracy	57	55	0.77
Structure
Author credentials	25	23	0.74
References	19	25	0.31
Advertisements	17	28	0.06
Updated <1 y	37	60	0.001

Bold values indicate statistical significance (P < 0.05).

Comparing website categories, information about parathyroid surgery was most likely to come from academic-affiliated sources (30%), while only 6% of websites were association or research-based. Twenty-two percent of websites were affiliated with hospitals, 22% with private clinics, and 20% were related to miscellaneous or informational sources such as health and wellness pages or patient blogs. For thyroid websites, the most common type of website was informational (41%), a minority were academic (19%), only 4% were association/research-based, and 36% were associated with a hospital or clinic.

Examining rates of overall accuracy of Internet information, 43% of parathyroid websites and 45% of thyroid websites had at least one potential inaccuracy. When stratifying based on the degree of misinformation present, 57% of parathyroid websites had no obvious inaccuracies, 21% had one, 11% had two, and 11% had three or more. For thyroid websites, 55% of websites had no obvious inaccuracies, 17% had one, 21% had two, and 7% had three or more (Table 3).

Table 3 –

Number of errors in endocrine surgery websites by type.

Variable	0 (%)	1 (%)	2 (%)	3 (%)	P value
Parathyroid					<0.001
Total	57	21	11	11
Academic	26 (87)	4 (13)	0 (0)	0 (0)
Research	6 (100)	0 (0)	0 (0)	0 (0)
Hospital	12 (55)	4 (18)	4 (18)	2 (9)
Clinic	8 (37)	10 (45)	3 (14)	1 (5)
Other	5 (25)	3 (15)	4 (20)	8 (40)
Thyroid					0.114
Total	55	17	21	7
Academic	15 (79)	2 (11)	1 (5)	1 (5)
Research	3 (75)	0 (0)	1 (25)	0 (0)
Hospital	8 (47)	3 (18)	6 (35)	0 (0)
Clinic	12 (63)	5 (26)	2 (11)	0 (0)
Other	17 (41)	7 (17)	11 (27)	6 (15)

Italicized values indicate statistical significance (P < 0.05).

Examining misinformation based on website type, the majority of academic and association/research-based websites had zero inaccuracies, including 87% of parathyroid and 79% of thyroid websites for academic sources, and 100% and 75% for research-based sources. Miscellaneous/informational websites were most likely to have three or more potential inaccuracies for parathyroid data (40%), but 41% of thyroid websites from informational sources had no inaccuracies, and only 15% had three or more. The difference in rate of misinformation based on website type was statistically significant for parathyroid websites (P < 0.001), but not for thyroid websites (P = 0.114) (Table 3).

Discussion

Medicine today is dependent on the Internet for many aspects of patient care and communication, including electronic medical records, patient portals, and patient and provider educational resources. Many studies have attempted to quantify patient use of the Internet for medical purposes, with a wide range of results. A 2019 review of Internet use among patients with thyroid cancer reported that 87% of patients used the Internet for medical information, with 94% of respondents primarily using Google’s search engine.¹⁵ Although there have been reviews of patient Internet use in the setting of thyroid pathology and specifically thyroid cancer,^5,15 there has been only qualitative review of information regarding parathyroid surgery.¹⁶ Patients requiring parathyroid surgery are often older and may not use the Internet as often or as easily, which could explain this discrepancy.

This study has identified significant variability in the information available on the Internet regarding thyroid and parathyroid surgery. Although many of the top websites were affiliated with academic institutions or association/research-based sources, only one quarter of all websites provided credentials for authors with health care experience. Additionally, overlap among 26% of websites between parathyroid and thyroid search results and inclusion in both data sets may account for similar rates of misinformation for at least a quarter of the data.

Overall rates of misinformation were similar when comparing thyroid and parathyroid surgery websites, with almost half of all web pages containing at least one potential inaccuracy. For both types of surgery, information was often either vague or very specific, especially if the page was associated with a clinic which provided practice-specific information or preoperative and postoperative expectations. Since preoperative and postoperative care likely differs between independent providers, one physician’s recommendations may not universally apply to every patient’s experience. Very few pages discussed specialized options such as minimally invasive radio-guided parathyroidectomy, or robotic/laparoscopic options for surgery, likely because these techniques may not be available in all surgical settings, and our search only evaluated the top 100 hits.¹⁷

Completeness of information was similar among both parathyroid and thyroid surgery websites, with a majority of all websites including procedural information, hormone function, and disease pathophysiology. There were slightly higher rates of parathyroid information, which may be due to the nature of the disease, which requires surgery for each of these glands. Parathyroidectomy is typically performed based on well-defined indications for surgery, the most common being symptomatic primary hyperparathyroidism or asymptomatic disease meeting specific criteria.^18,19 Indications for thyroid surgery are more subjective, with guidelines available for multiple types of cancer,²⁰ but variability in the workup and treatment of goiters or conditions such as Graves’ disease. Additionally, there may be more information to digest about the thyroid, and we only evaluated results from the specific search phrase “thyroid surgery”. There may have been lower rates of information about thyroid hormone function and disease process because this information could have been available on separate, linked pages we did not examine. Many thyroid surgery websites focused on nodules or cancer, and less frequently mentioned surgical treatment of Grave’s disease.

Multiple websites mentioned subtotal thyroidectomy, which is no longer recommended.^21,22 Misinformation about the recovery process was also common, though some of this variation can be attributed to individual providers’ practice style. Some websites mentioned drain placement (which may be provider-specific and is not always necessary in routine cases²³), mandatory hospital admission (though many thyroidectomies are performed on an outpatient basis²⁴), and one website included a picture of an incision closed with staples. Some websites claimed that thyroid surgery may not be necessary at all, and thyroid disease can be controlled with medication. This may be true in some circumstances but may give patients biased ideas or incomplete information about their treatment options. Interestingly, many websites included disclaimers on the bottom of the page in small font which stated that information should not be used to make medical decisions, or similar words to that effect. These were frequently not easily identifiable.

Examining information based on website type allows providers to better understand the resources available for patients on the Internet. Academic and association/research-based sources were most likely to have zero inaccuracies, which can be reassuring to providers and patients receiving care within academic institutions. Around half of the hospital or clinic-affiliated websites had at least one potential inaccuracy, which may represent more variability in private practice or nonacademic settings. Interestingly, 41% of informational thyroid websites had no inaccuracies, but this category was still most likely to have the highest rate of misinformation for both parathyroid and thyroid data overall. This is important for providers to consider when attempting to guide patients to reliable sources for information. Academic or research-based websites are highly likely to have accurate information, but alternative options might include physician-approved handouts, or providing patients with personalized written information during clinic visits.

Limitations to this study include the sample size of 200 websites, which represents a small proportion of all of those which are available on the Internet. This search was conducted using only one search engine (Google) and provided results from one fixed point in time in June 2021. This study was conducted as a feasibility pilot study to determine whether this type of information gathering was possible, and search terms were strictly defined and limited to ensure that they were repeatable. There is a wide variety of additional search terms patients may use, which could lead to a different set of search results. One strength of this approach is that the Google search algorithm prioritizes highly-trafficked websites or those for which publishers have paid to promote them, making each included website much more likely to be viewed by members of the public. We only evaluated information on the initial website from our search results and did not follow hyperlinks to open additional tabs or webpages to ensure results were repeatable. This may not accurately represent how patients search the literature, however, as they may click these links and read additional information which was not evaluated in this study. Additionally, this search did not examine information available from social media sources which are commonly used today. In terms of the scoring system, three researchers used one document to standardize what should be deemed accurate, but different levels of medical knowledge and training may be the underlying differences in ratings and correspondent IRR. The IRR was calculated in attempt to minimize the subjective nature of independent review, and a standardized scoring system was used to simply state the presence or absence of information when examining multiple variables and when attempting to stratify the degree of misinformation. The methodology used in this study is similar to a few other studies published in this arena.^10–13,15

Conclusions

In conclusion, an abundance of medical information about thyroid and parathyroid surgery is available on the Internet from a variety of sources, and patients are presented with many options which may provide biased, incomplete, or incorrect information. Overall, approximately half of the websites evaluated in this study provided correct information with no obvious inaccuracies, but the remainder of websites had at least one potential inaccuracy. This has the potential to lead to clinician distrust or patient disagreement with medical recommendations, which can be difficult if patients come to the clinic with predetermined ideas about their diagnoses. As a result, providers may spend valuable time during clinic visits correcting such misinformation instead of providing patients with correct information about their specific diagnosis and treatment plan. Given the degree of misinformation present online, providers should attempt to guide patients to where they can find reliable information about their condition and be willing to answer questions to clarify any misconceptions to promote provider trust and shared decision-making.